Configuration
On this Page
Configuration¶
No configuration is required when using the profiling default settings. To configure the settings, a profiling configuration tool is included in the SynapseAI installation.
Pre-configured Instrumentation¶
The profiling configuration tool enables adjusting the software and hardware settings of the profiling
subsystem using either the CLI based or GUI based hl-prof-config. The configurable settings
include changing the session name, output directory, output
formats and basic hardware settings of the instrumentation.
After using hl-prof-config GUI or CLI tool, a new configuration file, prof_config.json, is created.
prof_config.json is stored in a hidden folder, '.habana',
located in your home directory. All subsequent profiling sessions
will use settings present in the:file:prof_config.json file. The settings can be reset to default using the profiling configuration tool, or by
deleting the prof_config.json file located in the '.habana'
directory.
CLI/GUI Configuration Tools¶
To configure the profiler settings, run the compatible application:
CLI Configuration Tool -
hl-prof-config <args>GUI Configuration Tool -
hl-prof-config
Note
GUI configuration tool is not supported on Docker environment. Please create your configuration file using the GUI tool outside the docker, and copy it into the docker. Alternatively, you can use the CLI tool inside the docker to create a configuration file.
CLI Configuration Tool - hl-prof-config <args>¶
hl-prof-config <args> is a command line interface for CLI. Command line parameters must be used when calling the CLI in order to configure the SynapseAI Profiler
operation:
hl-prof-config -h
Usage: /home/[user]/builds/latest/hl-prof-config [-v|--version] [-h|--help] [--legacy]
If no options are given, The configuration GUI tool will be opened.
General options:
-h [ --help ] Produce help message
-v [ --version ] Print version information
-s [ --session-name ] arg Name for output file(s). Add '#' to concatenate a runtime generated timestamp of the format YYYYMMDD_hh-mm-ss
--add-pid arg Add PID to output files when applicable
--graph-data-cl arg Clean Graph Data files produced during the execution
-o [ --output-dir ] arg Directory to store the trace output file(s)
-r [ --per-recipe ] arg Create sub-directory for each recipe, num invocations (givven by -n) is per-recipe
-a [ --add-pci ] arg Add device PCI-BUS id to the output file name
--simulationMode arg Enable parsing in simulation mode (coral)
-e [ --edit-existing ] arg Configuration file will be updated (default is overwrite)
--list-templates Display list of supported templates use HABANA_PROFILE=<template_id>
--gaudi Target architecture is Habana Gaudi
--gaudi2 Target architecture is Habana Gaudi2
-c [ --chip ] arg Target architecture (gaudi/gaudi2)
Output options:
--invoc arg Output file per trace invocation (hltv/csv/json - e.g 'json,csv')
--merged arg Output merged file with all HW + Host traces (hltv/csv/json - e.g 'json,csv')
--host arg Enable/Disable host API profiling (default on)
--host-api arg Set list of host APIs to enable, separated by comma (scal/synapse/hccl/hlthunk - e.g 'scal,hlthunk'), implies --host on
--hw-trace arg Enable/Disable HW Trace (default on)
--recipe-dump-format arg Dump recipe command buffers format (text, html, json)
--trace-analyzer arg Add trace analyzer data to json
--trace-analyzer-csv arg Create trace analyzer data in CSV
--data-flow arg Add data flow arrows indication
--memory-reuse arg Add memory reuse arrows indication
--no-merge arg Do not merge device profiles with host profile in same json output (default merged)
--fuser arg Add fuser metadata to the json output (default off)
--enq-flow arg Add arrows between enqueue/synLaunch API call to the start of HW activity (supported in Gaudi2 only)
Instrumentation mode options:
-p [ --phase ] arg Profile a specific API (enq/multi-enq/mem/dev ice-acq)
-i [ --instrumentation ] Enable user profiler activation via API (disables all phases)
-g [ --invocations-range ] arg Range of enqueue invocations to profile. Example: 1-3,4,10-12 (default 1-2)
-b [ --buffer-size ] arg Size in MB of trace buffer to allocate
To enable profiler for hardware specific events, run the following accordingly:
hl-prof-config -h -gaudi
Device specific options for gaudi:
Device options:
NIC:
--nic arg [ on|off|default ] Set profile mode of all NIC units
--nic0_0 arg [ on|off|default ] Set profile mode of NIC0 0
--nic0_1 arg [ on|off|default ] Set profile mode of NIC0 1
--nic1_0 arg [ on|off|default ] Set profile mode of NIC1 0
--nic1_1 arg [ on|off|default ] Set profile mode of NIC1 1
--nic2_0 arg [ on|off|default ] Set profile mode of NIC2 0
--nic2_1 arg [ on|off|default ] Set profile mode of NIC2 1
--nic3_0 arg [ on|off|default ] Set profile mode of NIC3 0
--nic3_1 arg [ on|off|default ] Set profile mode of NIC3 1
--nic4_0 arg [ on|off|default ] Set profile mode of NIC4 0
--nic4_1 arg [ on|off|default ] Set profile mode of NIC4 1
DMA IF:
--dma_if arg [ on|off|default ] Set profile mode of all DMA IF units
--dma_if_w_s arg [ on|off|default ] Set profile mode of DMA IF W S
--dma_if_e_s arg [ on|off|default ] Set profile mode of DMA IF E S
--dma_if_w_n arg [ on|off|default ] Set profile mode of DMA IF W N
--dma_if_e_n arg [ on|off|default ] Set profile mode of DMA IF E N
DMA CH:
--dma_ch arg [ on|off|default ] Set profile mode of all DMA CH units
--dma_ch0 arg [ on|off|default ] Set profile mode of DMA CH0
--dma_ch1 arg [ on|off|default ] Set profile mode of DMA CH1
--dma_ch2 arg [ on|off|default ] Set profile mode of DMA CH2
--dma_ch3 arg [ on|off|default ] Set profile mode of DMA CH3
--dma_ch4 arg [ on|off|default ] Set profile mode of DMA CH4
--dma_ch5 arg [ on|off|default ] Set profile mode of DMA CH5
--dma_ch6 arg [ on|off|default ] Set profile mode of DMA CH6
--dma_ch7 arg [ on|off|default ] Set profile mode of DMA CH7
MME:
--mme arg [ on|off|default ] Set profile mode of all MME units
--mme0_acc arg [ on|off|default ] Set profile mode of MME0 ACC
--mme0_sbab arg [ on|off|default ] Set profile mode of MME0 SBAB
--mme0_ctrl arg [ on|off|default ] Set profile mode of MME0 CTRL
--mme1_acc arg [ on|off|default ] Set profile mode of MME1 ACC
--mme1_sbab arg [ on|off|default ] Set profile mode of MME1 SBAB
--mme1_ctrl arg [ on|off|default ] Set profile mode of MME1 CTRL
--mme2_acc arg [ on|off|default ] Set profile mode of MME2 ACC
--mme2_sbab arg [ on|off|default ] Set profile mode of MME2 SBAB
--mme2_ctrl arg [ on|off|default ] Set profile mode of MME2 CTRL
--mme3_acc arg [ on|off|default ] Set profile mode of MME3 ACC
--mme3_sbab arg [ on|off|default ] Set profile mode of MME3 SBAB
--mme3_ctrl arg [ on|off|default ] Set profile mode of MME3 CTRL
TPC:
--tpc arg [ on|off|default ] Set profile mode of all TPC units
--tpc0 arg [ on|off|default ] Set profile mode of TPC0
--tpc1 arg [ on|off|default ] Set profile mode of TPC1
--tpc2 arg [ on|off|default ] Set profile mode of TPC2
--tpc3 arg [ on|off|default ] Set profile mode of TPC3
--tpc4 arg [ on|off|default ] Set profile mode of TPC4
--tpc5 arg [ on|off|default ] Set profile mode of TPC5
--tpc6 arg [ on|off|default ] Set profile mode of TPC6
--tpc7 arg [ on|off|default ] Set profile mode of TPC7
GENERAL UNITS:
--general_units arg [ on|off|default ] Set profile mode of all GENERAL UNITS units
--cpu arg [ on|off|default ] Set profile mode of CPU
--mmu arg [ on|off|default ] Set profile mode of MMU
--pcie arg [ on|off|default ] Set profile mode of PCIE
--psoc arg [ on|off|default ] Set profile mode of PSOC
GUI Configuration Tool - hl-prof-config¶
This section describes the hl-prof-config graphical tool as a method to configure Habana profiler.
Terms¶
Template: A pre-defined configuration that can be loaded using
HABANA_PROFILE=<>, or can be used as a base configuration with the GUI tool.Plugin: A library implementing a particular task during execution of an API call.
Field: A configurable parameter in the plugin configuration. Its default value is stored in the plugin’s library.
Group: A collection of fields, and/or other group(s) in the plugin configuration.
Scheme: Each plugin has its own scheme. A json structured data representing the hierarchy of groups and fields of the configuration of that plugin. The json object can be illustrated as a tree: the leaves are the fields; and the non-leave nodes are groups. Each node has an ID. For each field, the scheme contains its properties; which GUI widget to use (Checkbox/Drop down menu/ Text field, etc) and its default value.
Profiler Configuration File: A JSON file containing an array of plugins. Per plugin, the following properties should exist:
name- The name of the plugin as it will be displayed.lib- Which library is used for this plugin.enable- True/false; controls whether the plugin is enabled or not (optional, default: false).values- A json object containing the differences from the default configuration. Any configuration value which does not exist in the configuration file will be as the default value written in the plugin’s scheme.
Note
Although entries in values should be different from their default values, entries might be equal to their defaults, and will not be treated as a misformed configuration file.
Default Configuration File:
~/.habana/prof_config.json. When a SynapseAI program starts, this file (by default) is parsed and its content determines which plugins are enabled with which configuration. In order to make a SynapseAI application use a different configuration file set the environment variableHABANA_PROF_CONFIG=<path-to-config file>before running the app.
hl-prof-config¶
hl-prof-config is a graphical application allowing users to view, edit, and save profiler configuration files in a visual way.
Prerequisites¶
shim_ctl is necessary on the system since hl-prof-config uses it in the background. It is assumed that shim_ctl exists in PATH,
however, if not, the environment variable SHIM_CONFIG_PATH=<path to shim_config> hl-prof-config can be set.
Usage¶
To use hl-prof-config application, see the figure and options below:
Figure 11 Welcome Screen¶
Main Window Screen¶
Allows you to open a predefined template or a recently opened configuration file.
If a pre-define template configuration setting is opened, it can be saved to a default config or by using save as.
Plugins Selection Screen¶
This section describes the available plugins. Upon opening a configuration file, the ‘Select Plugins’ screen is shown, and the user can select the specific plugins to enable and set their execution order (see Fig. 13).
Figure 13 Plugins Selection Screen¶
Loaded file indicator.
Available Plugins - Displays a list of available plugins in the system. The checkbox determines whether a specific plugin is enabled or not. Clicking the ‘spanner’ icon will navigate you to the plugin’s configuration view. Clicking on a plugin name will show you information in the ‘Plugin Information’ section (see below).
Plugin Information - Displays additional details about the selected plugin.
Restore Defaults - Clicking this button will set all the plugin’s parameters back to their default values.
Plugins Sequence - View and adjust the order of the plugins’ execution in the profiler. You can move a plugin by dragging it or by selecting it and using the arrows (to select multiple plugins press the ‘ctrl’ key).
Note
If Host Profiler plugin is enabled, it is recommended to position it last.
Plugins Configuration Screen¶
This section explains the plugin parameters. Upon selecting plugins, the user can adjust the selected plugin parameters (see Fig. 14).
Figure 14 Plugins Configuration Screen¶
Search - The search is case-insensitive, and is done on:
Group names
Field names
Drop-down menu options
Plugin tabs - Each plugin will appear on a separate tab. Click on the ‘Plugin tab’ to view its configuration in #3 and #4.
Tree view of the plugin configuration - Click on a group to expand it, and view its content in #4.
View group and its parameters and subgroups - This view allows you to modify parameters.
Show Advanced parameters switch - If some groups/fields were marked in the scheme as ‘advanced’, they will be hidden by default. Turn on this switch if you would like to unhide them.
Save Button - Click to save the current configuration.
Undo and Redo Buttons - Click to undo or redo changes in the configuration.
Multiple Selection - To select multiple profile units or engines, hold down the Control (Ctrl) key while clicking on more than one profile unit. Alternatively, the user can select a range of profile units by holding the Shift key. The displayed fields are the common denominator between the selected profile units. Upon multiple selection, the user can change a field value across multiple profile units. In addition, the user can select all events by clicking on ‘Select All’ checkbox, which will select all the events, even the ones that are not displayed. On the other hand, the user can click on ‘Select All Displayed’, which will select only the fields being displayed. The multiple selection features an indeterminate state as well for values that are not the same across the selected profile units.
Multiple Siblings Selection Screen¶
This section describes the available options for enabling and disabling all the siblings simultaneously. You can right click the Profile Unit to open a context menu window. See the figure and options below:
Disable, Disable All, Reset to default and Undo changes.
Enable, Enable All, Reset to default and Undo changes.
Keyboard Shortcuts¶
To facilitate easier use of hl-prof-config, a few key combinations were added enabling
the execution of well-known commands as described below:
Save File |
Ctrl + S |
Save as File |
Ctrl + Shift + S |
Undo |
Ctrl + Z |
Redo |
Ctrl + Shift + Z |
Navigate to search box |
Ctrl + F |
Source Code Instrumentation¶
For application level activation of the profiler in user code, select API controlled option in the main window. This can be set separately for device instrumentation and host instrumentation.
Note
Please use a pre-defined template configuration profiling from api as a base configuration for Source code instrumentation.
Make sure that options under General settings-> Phase Options remain unchecked.
Such configuration disables all automatic profiling of API calls. After saving this configuration, calls to the
synProfilerStart() and synProfilerStop() APIs should be placed in the
relevant places in the code. Each time a pair of start and stop calls
are encountered, the profiling subsystem generates a new trace
buffer. The trace buffer may then be retrieved from memory using the
synProfilerGetTrace() API. The trace buffer will be available for
retrieval until a new one is generated. In this context, the other
phases and max invocations are irrelevant. All other settings apply both
to manual and automatic profiling. Runtime usage is also the same for
both, as detailed in the following section.
The trace buffer may be retrieved in memory as a C/C++ struct, and parsed in the application, or written to the file system in JSON format. If using the latter, the file will be saved according to the output settings. Further details are available in the SynapseAI API documentation.
The below is an example of using an API controlled instrumentation:
// start profiling (enables device trace modules)
status = synProfilerStart(synTraceDevice, deviceId);
// do something here e.g., enqueue
// ...
// wait on some handle for completion
// stop profiling (disables the device trace modules)
status = synProfilerStop(synTraceDevice, deviceId);
// output to file system by passing buffer=nullptr & size=nullptr
status = synProfilerGetTrace(synTraceDevice, deviceId, synTraceFormatTEF, nullptr, nullptr);