Configuration

No configuration is required when using the profiling default settings. To configure the settings, a profiling configuration tool is included in the SynapseAI installation.

Pre-configured Instrumentation

The profiling configuration tool enables adjusting the software and hardware settings of the profiling subsystem using either the CLI based or GUI based hl-prof-config. The configurable settings include changing the session name, output directory, output formats and basic hardware settings of the instrumentation. After using hl-prof-config GUI or CLI tool, a new configuration file, prof_config.json, is created.

prof_config.json is stored in a hidden folder, '.habana', located in your home directory. All subsequent profiling sessions will use settings present in the:file:prof_config.json file. The settings can be reset to default using the profiling configuration tool, or by deleting the prof_config.json file located in the '.habana' directory.

CLI/GUI Configuration Tools

To configure the profiler settings, run the compatible application:

Note

GUI configuration tool is not supported on Docker environment. Please create your configuration file using the GUI tool outside the docker, and copy it into the docker. Alternatively, you can use the CLI tool inside the docker to create a configuration file.

CLI Configuration Tool - hl-prof-config <args>

hl-prof-config <args> is a command line interface for CLI. Command line parameters must be used when calling the CLI in order to configure the SynapseAI Profiler operation:

hl-prof-config -h
Usage: /home/[user]/builds/latest/hl-prof-config [-v|--version] [-h|--help] [--legacy]

If no options are given, The configuration GUI tool will be opened.

General options:
   -h [ --help ]                    Produce help message
   -v [ --version ]                 Print version information
   -s [ --session-name ] arg        Name for output file(s). Add '#' to concatenate a runtime generated timestamp of the format YYYYMMDD_hh-mm-ss
   --add-pid arg                    Add PID to output files when applicable
   --graph-data-cl arg              Clean Graph Data files produced during the execution
   -o [ --output-dir ] arg          Directory to store the trace output file(s)
   -r [ --per-recipe ] arg          Create sub-directory for each recipe, num invocations (givven by -n) is per-recipe
   -a [ --add-pci ] arg             Add device PCI-BUS id to the output file name
   --simulationMode arg             Enable parsing in simulation mode (coral)
   -e [ --edit-existing ] arg       Configuration file will be updated (default is overwrite)
   --list-templates                 Display list of supported templates use HABANA_PROFILE=<template_id>
   --gaudi                          Target architecture is Habana Gaudi
   --gaudi2                         Target architecture is Habana Gaudi2
   -c [ --chip ] arg                Target architecture (gaudi/gaudi2)

Output options:
   --invoc arg                      Output file per trace invocation (hltv/csv/json - e.g 'json,csv')
   --merged arg                     Output merged file with all HW + Host traces (hltv/csv/json - e.g 'json,csv')
   --host arg                       Enable/Disable host API profiling (default on)
   --host-api arg                   Set list of host APIs to enable, separated by comma (scal/synapse/hccl/hlthunk - e.g 'scal,hlthunk'), implies --host on
   --hw-trace arg                   Enable/Disable HW Trace (default on)
   --recipe-dump-format arg         Dump recipe command buffers format (text, html, json)
   --trace-analyzer arg             Add trace analyzer data to json
   --trace-analyzer-csv arg         Create trace analyzer data in CSV
   --data-flow arg                  Add data flow arrows indication
   --memory-reuse arg               Add memory reuse arrows indication
   --no-merge arg                   Do not merge device profiles with host profile in same json output (default merged)
   --fuser arg                      Add fuser metadata to the json output (default off)
   --enq-flow arg                   Add arrows between enqueue/synLaunch API call to the start of HW activity (supported in Gaudi2 only)

Instrumentation mode options:
   -p [ --phase ] arg               Profile a specific API (enq/multi-enq/mem/dev ice-acq)
   -i [ --instrumentation ]         Enable user profiler activation via API (disables all phases)
   -g [ --invocations-range ] arg   Range of enqueue invocations to profile. Example: 1-3,4,10-12 (default 1-2)
   -b [ --buffer-size ] arg         Size in MB of trace buffer to allocate

To enable profiler for hardware specific events, run the following accordingly:

 hl-prof-config -h -gaudi

Device specific options for gaudi:

Device options:

NIC:
   --nic arg                        [ on|off|default ] Set profile mode of all NIC units
   --nic0_0 arg                     [ on|off|default ] Set profile mode of NIC0 0
   --nic0_1 arg                     [ on|off|default ] Set profile mode of NIC0 1
   --nic1_0 arg                     [ on|off|default ] Set profile mode of NIC1 0
   --nic1_1 arg                     [ on|off|default ] Set profile mode of NIC1 1
   --nic2_0 arg                     [ on|off|default ] Set profile mode of NIC2 0
   --nic2_1 arg                     [ on|off|default ] Set profile mode of NIC2 1
   --nic3_0 arg                     [ on|off|default ] Set profile mode of NIC3 0
   --nic3_1 arg                     [ on|off|default ] Set profile mode of NIC3 1
   --nic4_0 arg                     [ on|off|default ] Set profile mode of NIC4 0
   --nic4_1 arg                     [ on|off|default ] Set profile mode of NIC4 1

DMA IF:
   --dma_if arg                     [ on|off|default ] Set profile mode of all DMA IF units
   --dma_if_w_s arg                 [ on|off|default ] Set profile mode of DMA IF W S
   --dma_if_e_s arg                 [ on|off|default ] Set profile mode of DMA IF E S
   --dma_if_w_n arg                 [ on|off|default ] Set profile mode of DMA IF W N
   --dma_if_e_n arg                 [ on|off|default ] Set profile mode of DMA IF E N

DMA CH:
   --dma_ch arg                     [ on|off|default ] Set profile mode of all DMA CH units
   --dma_ch0 arg                    [ on|off|default ] Set profile mode of DMA CH0
   --dma_ch1 arg                    [ on|off|default ] Set profile mode of DMA CH1
   --dma_ch2 arg                    [ on|off|default ] Set profile mode of DMA CH2
   --dma_ch3 arg                    [ on|off|default ] Set profile mode of DMA CH3
   --dma_ch4 arg                    [ on|off|default ] Set profile mode of DMA CH4
   --dma_ch5 arg                    [ on|off|default ] Set profile mode of DMA CH5
   --dma_ch6 arg                    [ on|off|default ] Set profile mode of DMA CH6
   --dma_ch7 arg                    [ on|off|default ] Set profile mode of DMA CH7

MME:
   --mme arg                        [ on|off|default ] Set profile mode of all MME units
   --mme0_acc arg                   [ on|off|default ] Set profile mode of MME0 ACC
   --mme0_sbab arg                  [ on|off|default ] Set profile mode of MME0 SBAB
   --mme0_ctrl arg                  [ on|off|default ] Set profile mode of MME0 CTRL
   --mme1_acc arg                   [ on|off|default ] Set profile mode of MME1 ACC
   --mme1_sbab arg                  [ on|off|default ] Set profile mode of MME1 SBAB
   --mme1_ctrl arg                  [ on|off|default ] Set profile mode of MME1 CTRL
   --mme2_acc arg                   [ on|off|default ] Set profile mode of MME2 ACC
   --mme2_sbab arg                  [ on|off|default ] Set profile mode of MME2 SBAB
   --mme2_ctrl arg                  [ on|off|default ] Set profile mode of MME2 CTRL
   --mme3_acc arg                   [ on|off|default ] Set profile mode of MME3 ACC
   --mme3_sbab arg                  [ on|off|default ] Set profile mode of MME3 SBAB
   --mme3_ctrl arg                  [ on|off|default ] Set profile mode of MME3 CTRL

TPC:
   --tpc arg                        [ on|off|default ] Set profile mode of all TPC units
   --tpc0 arg                       [ on|off|default ] Set profile mode of TPC0
   --tpc1 arg                       [ on|off|default ] Set profile mode of TPC1
   --tpc2 arg                       [ on|off|default ] Set profile mode of TPC2
   --tpc3 arg                       [ on|off|default ] Set profile mode of TPC3
   --tpc4 arg                       [ on|off|default ] Set profile mode of TPC4
   --tpc5 arg                       [ on|off|default ] Set profile mode of TPC5
   --tpc6 arg                       [ on|off|default ] Set profile mode of TPC6
   --tpc7 arg                       [ on|off|default ] Set profile mode of TPC7

GENERAL UNITS:
   --general_units arg              [ on|off|default ] Set profile mode of all GENERAL UNITS units
   --cpu arg                        [ on|off|default ] Set profile mode of CPU
   --mmu arg                        [ on|off|default ] Set profile mode of MMU
   --pcie arg                       [ on|off|default ] Set profile mode of PCIE
   --psoc arg                       [ on|off|default ] Set profile mode of PSOC

GUI Configuration Tool - hl-prof-config

This section describes the hl-prof-config graphical tool as a method to configure Habana profiler.

Terms

  • Template: A pre-defined configuration that can be loaded using HABANA_PROFILE=<>, or can be used as a base configuration with the GUI tool.

  • Plugin: A library implementing a particular task during execution of an API call.

  • Field: A configurable parameter in the plugin configuration. Its default value is stored in the plugin’s library.

  • Group: A collection of fields, and/or other group(s) in the plugin configuration.

  • Scheme: Each plugin has its own scheme. A json structured data representing the hierarchy of groups and fields of the configuration of that plugin. The json object can be illustrated as a tree: the leaves are the fields; and the non-leave nodes are groups. Each node has an ID. For each field, the scheme contains its properties; which GUI widget to use (Checkbox/Drop down menu/ Text field, etc) and its default value.

  • Profiler Configuration File: A JSON file containing an array of plugins. Per plugin, the following properties should exist:

    • name - The name of the plugin as it will be displayed.

    • lib - Which library is used for this plugin.

    • enable - True/false; controls whether the plugin is enabled or not (optional, default: false).

    • values - A json object containing the differences from the default configuration. Any configuration value which does not exist in the configuration file will be as the default value written in the plugin’s scheme.

Note

Although entries in values should be different from their default values, entries might be equal to their defaults, and will not be treated as a misformed configuration file.

  • Default Configuration File: ~/.habana/prof_config.json. When a SynapseAI program starts, this file (by default) is parsed and its content determines which plugins are enabled with which configuration. In order to make a SynapseAI application use a different configuration file set the environment variable HABANA_PROF_CONFIG=<path-to-config file> before running the app.

hl-prof-config

hl-prof-config is a graphical application allowing users to view, edit, and save profiler configuration files in a visual way.

Prerequisites

shim_ctl is necessary on the system since hl-prof-config uses it in the background. It is assumed that shim_ctl exists in PATH, however, if not, the environment variable SHIM_CONFIG_PATH=<path to shim_config> hl-prof-config can be set.

Usage

To use hl-prof-config application, see the figure and options below:

../../_images/welcome.png

Figure 11 Welcome Screen

  1. File Menu

  2. Help menu

  3. Main window Screen

  4. Plugins Selection Screen

  5. Plugins Configuration Screen

  6. General Settings Screen

  7. Reset Configuration Button

File Menu

The file menu allows loading a configuration file, and saving the current configuration to a file. Under the file menu, the following options are availabe:

  • Open Configuration File: Browse dialog will open to choose an existing configuration file and open it.

  • Open Default Configuration File: The default configuration file of the profiler is located at : ~/.habana/prof_config.json. Clicking this option will open it without showing a browse dialog.

  • Save: Save the current configuration to the loaded file.

  • Save As: Browse dialog will be opened, allowing you to choose which file should store the current configuration.

  • Save to Default Configuration File: Save current configuration to : ~/.habana/prof_config.json without showing a browse dialog.

Note

Only after opening a file, saving options and other screens will be available.

Help Menu

To Navigate back to the opening screen, click the “Welcome” button.

../../_images/welcome_screen2.png
Main Window Screen

Allows you to open a predefined template or a recently opened configuration file.

If a pre-define template configuration setting is opened, it can be saved to a default config or by using save as.

../../_images/MainWindowScreen.png
Plugins Selection Screen

This section describes the available plugins. Upon opening a configuration file, the ‘Select Plugins’ screen is shown, and the user can select the specific plugins to enable and set their execution order (see Fig. 13).

../../_images/select_plugins.png

Figure 13 Plugins Selection Screen

  1. Loaded file indicator.

  2. Available Plugins - Displays a list of available plugins in the system. The checkbox determines whether a specific plugin is enabled or not. Clicking the ‘spanner’ icon will navigate you to the plugin’s configuration view. Clicking on a plugin name will show you information in the ‘Plugin Information’ section (see below).

  3. Plugin Information - Displays additional details about the selected plugin.

  4. Restore Defaults - Clicking this button will set all the plugin’s parameters back to their default values.

  5. Plugins Sequence - View and adjust the order of the plugins’ execution in the profiler. You can move a plugin by dragging it or by selecting it and using the arrows (to select multiple plugins press the ‘ctrl’ key).

Note

If Host Profiler plugin is enabled, it is recommended to position it last.

Plugins Configuration Screen

This section explains the plugin parameters. Upon selecting plugins, the user can adjust the selected plugin parameters (see Fig. 14).

../../_images/plugins_config.png

Figure 14 Plugins Configuration Screen

  1. Search - The search is case-insensitive, and is done on:

    • Group names

    • Field names

    • Drop-down menu options

  2. Plugin tabs - Each plugin will appear on a separate tab. Click on the ‘Plugin tab’ to view its configuration in #3 and #4.

  3. Tree view of the plugin configuration - Click on a group to expand it, and view its content in #4.

  4. View group and its parameters and subgroups - This view allows you to modify parameters.

  5. Show Advanced parameters switch - If some groups/fields were marked in the scheme as ‘advanced’, they will be hidden by default. Turn on this switch if you would like to unhide them.

  6. Save Button - Click to save the current configuration.

  7. Undo and Redo Buttons - Click to undo or redo changes in the configuration.

  8. Multiple Selection - To select multiple profile units or engines, hold down the Control (Ctrl) key while clicking on more than one profile unit. Alternatively, the user can select a range of profile units by holding the Shift key. The displayed fields are the common denominator between the selected profile units. Upon multiple selection, the user can change a field value across multiple profile units. In addition, the user can select all events by clicking on ‘Select All’ checkbox, which will select all the events, even the ones that are not displayed. On the other hand, the user can click on ‘Select All Displayed’, which will select only the fields being displayed. The multiple selection features an indeterminate state as well for values that are not the same across the selected profile units.

Multiple Siblings Selection Screen

This section describes the available options for enabling and disabling all the siblings simultaneously. You can right click the Profile Unit to open a context menu window. See the figure and options below:

  1. Disable, Disable All, Reset to default and Undo changes.

  2. Enable, Enable All, Reset to default and Undo changes.

../../_images/plugins_config2.png

Keyboard Shortcuts

To facilitate easier use of hl-prof-config, a few key combinations were added enabling the execution of well-known commands as described below:

Save File

Ctrl + S

Save as File

Ctrl + Shift + S

Undo

Ctrl + Z

Redo

Ctrl + Shift + Z

Navigate to search box

Ctrl + F

General Settings Screen

This section contains global profiler parameters, such as output directory and profiling session name:

../../_images/general_settings.png

Figure 16 General Settings Screen

Reset Configuration Button

Clicking this button will restore all values to default configuration:

../../_images/reset_settings.png

Figure 17 Reset Settings Screen

Source Code Instrumentation

For application level activation of the profiler in user code, select API controlled option in the main window. This can be set separately for device instrumentation and host instrumentation.

Note

Please use a pre-defined template configuration profiling from api as a base configuration for Source code instrumentation. Make sure that options under General settings-> Phase Options remain unchecked.

Such configuration disables all automatic profiling of API calls. After saving this configuration, calls to the synProfilerStart() and synProfilerStop() APIs should be placed in the relevant places in the code. Each time a pair of start and stop calls are encountered, the profiling subsystem generates a new trace buffer. The trace buffer may then be retrieved from memory using the synProfilerGetTrace() API. The trace buffer will be available for retrieval until a new one is generated. In this context, the other phases and max invocations are irrelevant. All other settings apply both to manual and automatic profiling. Runtime usage is also the same for both, as detailed in the following section.

The trace buffer may be retrieved in memory as a C/C++ struct, and parsed in the application, or written to the file system in JSON format. If using the latter, the file will be saved according to the output settings. Further details are available in the SynapseAI API documentation.

The below is an example of using an API controlled instrumentation:

// start profiling (enables device trace modules)

status = synProfilerStart(synTraceDevice, deviceId);

// do something here e.g., enqueue

// ...

// wait on some handle for completion

// stop profiling (disables the device trace modules)

status = synProfilerStop(synTraceDevice, deviceId);

// output to file system by passing buffer=nullptr & size=nullptr

status = synProfilerGetTrace(synTraceDevice, deviceId, synTraceFormatTEF, nullptr, nullptr);