Profiling Tips and Tricks

This section explains the most common options to create a configuration file in various modes for data collection purposes.

Configure Data Collection per Enqueue

This command generates a configuration file for profiling per enqueue mode:

hl-prof-config --gaudi --buffer-size 256 --trace-analyzer on  --trace-analyzer-csv on --phase enq -e off --invoc json --merged hltv

It generates a dump of two files: a JSON file per enqueue (each enqueue is numbered), and one large hltv file that contains all enqueues (not numbered). In this mode, trace is collected after every enqueue. The enqueues are executed in a serial mode where each enqueue is executed once the previous one ends. Since the trace buffer is limited in size, for initial trace collection, it is recommended to check the desired enqueues rather than collecting specific enqueues in a multi-enqueue mode.

Configure Data Collection in Multi-Enqueue Mode

This command generates a configuration file for profiling in multi-enqueue mode:

hl-prof-config --gaudi --buffer-size 256 --trace-analyzer on  --trace-analyzer-csv on --phase multi-enq --invocations-range 1-100 -e off --merged hltv

It generate a dump of CSV and JSON files for specified enqueues. Set enqueues 10 to 13.

Note

The above two commands generate trace analyzer data to provide further understanding of each executed node.