hl_smi_async Tool

hl_smi_async is a utility tool for monitoring and managing the Gaudi devices asynchronously. Its basic functionality is a telemetry data collection and logging the collected data. hl_smi_async allows reading the different telemetry packets and handling all the related configurations and handshakes. For more information about the telemetry packets, see the Gaudi 3 In-Band Telemetry for Hypervisor document located in the Intel Gaudi vault and Intel RDC. The hl_smi_async application is located in the hl-smi repository: hl-smi/app/hl-smi-async.

Note

  • The tool can run with or without driver loaded.

  • It is only possible to run the tool on a bare-metal machine or a Hypervisor. Running the tool on a guest VM may cause undefined behavior.

Options and Usage

The following table lists the available hl_smi_async options and their usage to help you effectively configure the tool for your specific needs.

Example:

sudo /usr/sbin/hl-smi-async -D 0000:b1:00.0 -O console -L info -I 5

Option

Description

-h, --help

Outputs the help message and exits.

-V, --version

Outputs version information and exits.

-D, --device [device]

Specifies the PCIe address of the device (e.g. 0000:b1:00.0).

-O, --output [output]

Specifies the telemetry output type. Valid values:

  • console (default) - Prints all collected telemetry to the console.

  • logfile - Prints all collected telemetry to a logfile: telem_log.txt

-L, --loglevel [level]

Specifies the log level. Valid values:

  • info (default) - Log informative messages.

  • debug - Log debug info.

  • error - Log all errors.

-I, --iterations [number]

Specifies the number of iterations. If not set (default), the tool runs in an endless loop.

Output example:

  • Driver is not up (preboot mode):

    Starting telemetry data collection
Reading all synced telemetry packets
Timestamp: 11795 ms, Packet: Temperature, Field: temperature.aip, Data: 41
Timestamp: 12212 ms, Packet: Power, Field: power.draw.54v, Data: 185
Timestamp: 11836 ms, Packet: Health, Field: health, Data: 1
Timestamp: 12212 ms, Packet: Perf, Field: uptime, Data: 12
Timestamp: 12212 ms, Packet: Perf, Field: utilization.aip, Data: 0
Timestamp: 3709 ms, Packet: System Status, Field: ib_fw_update.stat, Data: 2
Timestamp: 3709 ms, Packet: System Status, Field: ethernet_ports.state, Data: 16777215
Timestamp: 3709 ms, Packet: Security, Field: security.hash_spi_code, Data:
e5 49 5f 40 6e db 8b 92 23 41 d1 a3 c6 09 28 06 25 d8 da da 46 0a 9d 0a e5 f5 12 56 0c bc 3d 1d 30 50 c0 94 4b 77 ec 50 16 b3 c3 fd a9 05 3d a4
Timestamp: 3709 ms, Packet: Mem_Utility, Field: utilization.memory, Data: 0
Timestamp: 3709 ms, Packet: System Information, Field: fw_version.spi, Data: .0

  • Driver is up:

    Starting telemetry data collection
Reading all synced telemetry packets
Timestamp: 103144 ms, Packet: Temperature, Field: temperature.aip, Data: 1271310189
Timestamp: 102991 ms, Packet: Power, Field: power.draw.54v, Data: 169
Timestamp: 103107 ms, Packet: Health, Field: health, Data: 2
Timestamp: 103107 ms, Packet: Perf, Field: uptime, Data: 102
Timestamp: 103107 ms, Packet: Perf, Field: utilization.aip, Data: 0
Timestamp: 989 ms, Packet: System Status, Field: ib_fw_update.stat, Data: 2
Timestamp: 989 ms, Packet: System Status, Field: ethernet_ports.state, Data: 16777215
Timestamp: 990 ms, Packet: Security, Field: security.hash_spi_code, Data:
e5 49 5f 40 6e db 8b 92 23 41 d1 a3 c6 09 28 66 25 d8 da da 46 0a 9d 0a e5 f5 12 56 0c bc 3d 1d 30 50 c0 94 4b 77 ec 50 16 b3 c3 fd a9 05 3d a4
Timestamp: 990 ms, Packet: Security, Field: security.hash_boot_fit, Data:
b2 c7 87 3a 79 a7 2e cc 9c 27 70 66 b8 8f d3 11 2d c1 0a 78 cf 9f c2 a2 72 0d 25 5f be 5f 0c af 47 45 c0 7b 37 eb 97 bd 62 58 55 53 ab 71 cd 52
Timestamp: 7770 ms, Packet: Mem_utility, Field: utilization.memory, Data: 0
Timestamp: 1090 ms, Packet: System Information, Field: fw_version.spi, Data: .0
Timestamp: 1090 ms, Packet: System Information, Field: fw_version.os, Data: 1.20.0.0
Timestamp: 102991 ms, Packet: Errors, Field: ecc.errors.uncorrected.aggregate.total, Data: 0
Timestamp: 102991 ms, Packet: Errors, Field: ecc.errors.uncorrected.volatile.total, Data: 0
Timestamp: 102991 ms, Packet: Errors, Field: ecc.errors.corrected.aggregate.total, Data: 0
Timestamp: 102991 ms, Packet: Errors, Field: ecc.errors.corrected.volatile.total, Data: 0
Timestamp: 102991 ms, Packet: Errors, Field: ecc.errors.dram.aggregate.total, Data: 0
Timestamp: 102991 ms, Packet: Errors, Field: ecc.mode.current, Data: 1
Timestamp: 102991 ms, Packet: Errors, Field: ecc.mode.pending, Data: 1

Some fields are invalid when the driver is not up (preboot mode). For more information, refer to the Gaudi 3 In-Band Telemetry for Hypervisor document located in the Intel Gaudi vault and Intel RDC.

Validating FW Images Authenticity

This section explains how to validate the authenticity of firmware images using the hl-smi-async tool by comparing its output with the corresponding SHA files included in the habanalabs-hypervisor-utils package:

  • img-hash-gaudi3-boot-fit.sha384

  • img-hash-gaudi3-images-pointers.bin.be.sha384

After downloading and installing habanalabs-hypervisor-utils package which includes hl-smi-async tool, as described in Installing Hypervisor Tools Package section, the SHA files will be located in /lib/firmware/habanalabs/gaudi3.

To retrieve the hashes of the latest FW version, perform the following:

  1. Upgrade FW version to the latest SPI flash version as described in Firmware Upgrade section.

  2. Load the LKD driver on the VM.

  3. Run the hl-smi-async utility on the hypervisor:

    sudo /usr/sbin/hl-smi-async -D <pci_addr> -O console -L info -I 1

  4. Using a command line utility (such as xxd), compare the outputs of the security.hash_spi_code and security.hash_boot_fit hashes with the output of the hl-smi-async tool. security.hash_spi_code hash is the signature of the flash image, while security.hash_boot_fit hash is the signature of the FW application (mgmt app) which is loaded directly into RAM. The hash_spi_code is accessible during both the preboot and FW application run stages, while the hash_boot_fit is only available during running the FW application after the LKD driver has been loaded. Therefore, prior to running the LKD driver, only the hash_spi_code is displayed, while the hash_boot_fit is shown only after the LKD is running. See the examples below. The values displayed in the outputs vary depending on the release build number in use:

    In the file output:

    $ xxd /lib/firmware/habanalabs/gaudi3/img-hash-gaudi3-images-pointers.bin.be.sha384
00000000: 5c49 ed19 051e d6a6 0381 13d0 7487 dce4  \I..........t...
00000010: 906b bb74 ee06 3569 f67a 6906 eb8a c8a9  .k.t..5i.zi.....
00000020: abbd 0c32 0e59 1e55 48fe aa8d 8732 a6e1  ...2.Y.UH....2..

    Look for the following from hl-smi-async tool output:

    Security, Field: security.hash_spi_code, Data:
5c 49 ed 19 05 1e d6 a6 03 81 13 d0 74 87 dc e4 90 6b bb 74 ee 06 35 69 f6 7a 69 06 eb 8a c8 a9 ab bd 0c 32 0e 59 1e 55 48 fe aa 8d 87 32 a6 e1

    In the file output:

    $ xxd /lib/firmware/habanalabs/gaudi3/img-hash-gaudi3-boot-fit.sha384
00000000: 0d8d d302 3aea d1d1 8821 030f 404d bf98  ....:....!..@M..
00000010: a05c 52b1 3e6e 16c4 1678 0cc7 b195 8c42  .\R.>n...x.....B
00000020: 6675 e039 9adf 4dfb 40d8 f622 3b11 d46c  fu.9..M.@..";..l

    Look for the following from hl-smi-async tool output:

    Security, Field: security.hash_boot_fit, Data:
0d 8d d3 02 3a ea d1 d1 88 21 03 0f 40 4d bf 98 a0 5c 52 b1 3e 6e 16 c4 16 78 0c c7 b1 95 8c 42 66 75 e0 39 9a df 4d fb 40 d8 f6 22 3b 11 d4 6c