+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/fakes --output-run DEEP/fakes/science#partitionFakes/20240325T150327Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 75.1251 seconds; current memory usage: 0.178 Gibyte, delta: 0.039 Gibyte, peak delta: 0.051 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0027 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0043 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/pipeline_orig.qgraph DEEP/fakes/science#partitionFakes/20240325T150327Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 0.2612 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 2.4505 seconds; current memory usage: 0.213 Gibyte, delta: 0.035 Gibyte, peak delta: 0.023 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.214 Gibyte (main), 0.202 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0331 seconds; current memory usage: 0.214 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.214 Gibyte (main), 0.202 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_fakes_science#partitionFakes_20240325T150327Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0218 seconds; current memory usage: 0.214 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.214 Gibyte (main), 0.202 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 3 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.6478 seconds; current memory usage: 0.218 Gibyte, delta: 0.004 Gibyte, peak delta: 0.004 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.218 Gibyte (main), 0.202 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: 6cf5440c-ecac-4089-a7a4-f618aa4c5290
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 23347291821824)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a 169.254.95.120,n3133,127.0.0.1,198.48.92.26,10.64.129.133,10.64.65.133 -p 0 -c 1.0 -m None --poll 10 --task_port=54812 --result_port=54779 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x153bfde10a90>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-25 15:05:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3133,127.0.0.1,198.48.92.26,10.64.129.133,10.64.65.133 -p 0 -c 1.0 -m None --poll 10 --task_port=54812 --result_port=54779 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 169.254.95.120,n3133,127.0.0.1,198.48.92.26,10.64.129.133,10.64.65.133 -p 0 -c 1.0 -m None --poll 10 --task_port=54812 --result_port=54779 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-25 15:05:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 169.254.95.120,n3133,127.0.0.1,198.48.92.26,10.64.129.133,10.64.65.133 -p 0 -c 1.0 -m None --poll 10 --task_port=54812 --result_port=54779 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 169.254.95.120,n3133,127.0.0.1,198.48.92.26,10.64.129.133,10.64.65.133 -p 0 -c 1.0 -m None --poll 10 --task_port=54812 --result_port=54779 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-25 15:05:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-25 15:05:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-25 15:05:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3133,127.0.0.1,198.48.92.26,10.64.129.133,10.64.65.133 -p 0 -c 1.0 -m None --poll 10 --task_port=54812 --result_port=54779 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 169.254.95.120,n3133,127.0.0.1,198.48.92.26,10.64.129.133,10.64.65.133 -p 0 -c 1.0 -m None --poll 10 --task_port=54812 --result_port=54779 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.providers.local.local DEBUG: Launching in remote mode
2024-03-25 15:05:08 proc_lsst.multi:170 [INFO]  [multi] job_id 43004
proc_lsst.multi INFO: [multi] job_id 43004
2024-03-25 15:05:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-25 15:05:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-25 15:05:08 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 43004
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 43004
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 43004
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App partitionFakes, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x153bf7f104d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 8e2ac21b32f2843f57bf82eed4684145
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153bf7ed34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/logs/partitionFakes/c1a5d6e4-7726-4f7f-8b3d-95df328ffe8d_partitionFakes_.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/logs/partitionFakes/c1a5d6e4-7726-4f7f-8b3d-95df328ffe8d_partitionFakes_.stderr
2024-03-25 15:05:09 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 0 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:05:39 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:06:09 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:06:39 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:07:09 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:07:39 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:08:09 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:08:39 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:09:09 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:09:39 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:10:09 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:10:39 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
2024-03-25 15:11:09 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23347307020496
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e2ac21b32f2843f57bf82eed4684145 with result from task 0
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1
parsl.dataflow.dflow INFO: Tasks in state States.failed: 0
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/logs/partitionFakes/c1a5d6e4-7726-4f7f-8b3d-95df328ffe8d_partitionFakes_.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/submit/DEEP/fakes/science#partitionFakes/20240325T150327Z/logs/partitionFakes/c1a5d6e4-7726-4f7f-8b3d-95df328ffe8d_partitionFakes_.stderr
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=1, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 1 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: adf3ef86acab
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: adf3ef86acab
2024-03-25 15:11:17 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
2024-03-25 15:11:17 proc_lsst.multi:201 [INFO]  cancelling 43004 on provider local
proc_lsst.multi INFO: cancelling 43004 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 43004
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-25 15:11:17 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-25 15:11:17 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['43004']
proc_lsst.multi INFO: new jobs since last cancel ['43004']
2024-03-25 15:11:17 proc_lsst.multi:146 [INFO]  found job 43004 in provider local
proc_lsst.multi INFO: found job 43004 in provider local
2024-03-25 15:11:17 proc_lsst.multi:201 [INFO]  cancelling 43004 on provider local
proc_lsst.multi INFO: cancelling 43004 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 43004
parsl.providers.local.local WARNING: Failed to kill PID: 43004 and child processes on local
2024-03-25 15:11:18 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 376.9274 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_fakes_science#partitionFakes_20240325T150327Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 376.9361 seconds; current memory usage: 0.214 Gibyte, delta: -0.003 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 455.2847 seconds; current memory usage: 0.214 Gibyte, delta: 0.076 Gibyte, peak delta: 0.079 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.218 Gibyte (main), 0.218 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_fakes_science#partitionFakes_20240325T150327Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	7m38.114s
user	0m14.336s
sys	0m5.640s