Configuration

The configuration of the SynapseAI Profiling Subsystem is specified through a simple JSON file that can be edited using the hl-prof-config CLI tool.

The default path for this file is ~/.habana/prof_config.json. This is the file that will be loaded when you set HABANA_PROFILE=1 environment variable, and will be modified when using the hl-prof-config CLI tool.

You can specify a different file path using HABANA_PROF_CONFIG=<path_to_configuration.json> environment variable.

Sample Configurations

The following are common configuration options. Other options can be found in hl-prof-config CLI tool.

Capturing SynapseAI API Calls

To capture SynapseAI API calls using the profiling subsystem, you only need to execute the following command before running your model:

$ hl-prof-config -e off --hw-trace off

Here, -e off restores the configuration to its default, while --hw-trace off turns off hardware traces that have been captured by default from two synLaunch calls.

Capturing Hardware Trace

To capture hardware trace automatically, it is necessary to define when to start and stop tracing. It is recommended to capture the entire execution. This configuration captures hardware trace from the moment the script acquires the Gaudi device until it releases it. -b 256 sets the trace buffer size on the device to 256 MB.

$ hl-prof-config -e off --phase=device-acq -b 256

Note

Python scripts may exit the application in an abrupt manner, which can hinder the configuration from capturing the hardware trace.

Selecting the Captured Steps

To avoid capturing excessive amounts of data that are challenging to analyze, capture the hardware trace from a specific point in the script’s execution. The Profiler allows you to choose when to start and stop capturing by counting the number of synLaunch calls, as shown below:

$ hl-prof-config -e off --phase=multi-enq -g 50-100

This configuration captures hardware trace starting from the 50th synLaunch call and stops capturing at the 100th.

Resetting to Default Configuration

In case a configuration file is not found, the SynapseAI Profiler will use the default configuration. This configuration logs Synapse API calls on the host and captures hardware traces starting from the first two synLaunch API calls (which execute a graph on Gaudi).

To reset the configuration to its default settings, run:

$ hl-prof-config -e off