Runtime Environment Variables

The following table describes runtime flags that are set in the environment to change the behavior as well as enable or disable some features.

Flag

Default

Description

Consumer

PT_HPU_LAZY_MODE

1

Controls Execution mode:

  • 0x1 - Lazy mode

  • 0x2 - Eager mode

Habana PyTorch Bridge Modules

PT_HPU_LOG_MOD_MASK

All Modules

A Bitmask specifying Habana PyTorch Bridge module to enable logging.

  • 0x1 - Device logs

  • 0x2 - PT kernel/ops logs

  • 0x4 - Bridge logs

  • 0x8 - SynapseAI helper logs

  • 0x10 - Distributed module logs

  • 0x20 - Lazy mode logs

  • 0x80 - CPU fallback logs

Habana PyTorch Bridge Modules

PT_HPU_LOG_TYPE_MASK

1

A Bitmask specifying Habana PyTorch Bridge logging level from SynapseAI and perf_lib.

  • WARNING = 0x1

  • TRACE = 0x2

  • DEBUG = 0x4

Habana PyTorch Bridge

ENABLE_CONSOLE

False

If set to true, enables printing SynapseAI logs to the console.

SynapseAI

LOG_LEVEL_ALL

5

Logging level from SynapseAI and perf_lib.

  • 6 is no logs

  • 0 is verbose

By default, logs are placed either in the console (if ENABLE_CONSOLE=true) or under ~/.habana_logs/.

SynapseAI

GRAPH_VISUALIZATION

False

Creates of graph visualization files. The output dump graphs are in ./.graph_dumps folder

SynapseAI

PT_RECIPE_CACHE_PATH

Unset

Path (directory), where compiled graph recipes are stored to accelerate a scale up scenario. Only one process compiles the recipe, and other processes read it from disk.

If unset (default), compiled graph recipes are not stored on disk (recipe disk caching disabled).

Note: If a recipe cache is shared among a few processes (scale up), it must be stored on a local physical disk. Avoid using remote drives (such as NFS) where file locks are not supported, as it it may lead to instability and unpredictable behavior.

Habana PyTorch Bridge

PT_CACHE_FOLDER_DELETE

True

If set to True, the directory provided in PT_RECIPE_CACHE_PATH will be cleared when the workload starts. If set to False, cached recipes from previous executions of the workload will be used if graphs remain the same.

Habana PyTorch Bridge

PT_HPU_MAX_COMPOUND_OP _SIZE

INT64_MAX

Limits internal graph size to specified number of ops Reduces the lazy mode memory overhead. This will be improved in future releases.

Note: This may affect performance.

Habana PyTorch Bridge

HABANA_PGM_LRU_MAX

30000

If cache evictions cause performance degradation, increasing the cache size will increase performance. The default value is 30000. Note: Boost in performance may cause an increase in host memory consumption.

Habana PyTorch Bridge

PT_HPU_ENABLE_MEMCOPY _CONCAT_CTL_EDGE_FIX

False

Enables fix for memcopy and concat operations. Memcopy will be treated as in-place and concat will correctly add control edges. Enabling this flag could negatively impact performance.

Habana PyTorch Bridge

PT_HPU_LAZY_ACC_PAR_MODE

1

This flag turns on host time optimization of lazy ops accumulation. It offloads ops accumulation to a separate thread, thus reducing computation time of the main thread.

Habana PyTorch Bridge