HL-325 In-Band (IB) CPLD Programming

The primary purpose of the CPLD (Complex Programmable Logic Device) interface is to facilitate communication and interaction between the BMC and the CPLD.

The CPLD in the HL325 can be flashed with custom images by:

  • Using the BMC via JTAG interface. Refer to HLB-325 BMC Programming Guide available in the Intel Gaudi vault and Intel RDC.

  • Using the OAM via OOB with PLDM type 5. Refer to Gaudi 3 OOB Management Specification available in the Intel Gaudi vault and Intel RDC.

  • Using the OAM via In-Band with hl-fw-loader.

This document provides guidelines for updating the CPLD via In-Band with hl-fw-loader.

Updating CPLD

Using BMC’s JTAG interface requires the porting of a dedicated utility such as jam-stapl.

Updating the CPLD using the OAM’s JTAG can be accomplished either from the BMC (OOB via I2C) or the host (IB via PCI). The hl-fw-loader utility is employed in the IB scenario.

CPLD Version Matrix

The CPLD version and the corresponding .itb file required for the update depend on the CPLD device type (15D_ES3 or 15D_HBN), the existing CPLD version, and the security key as described in the table below.

CPLD Device Type

Existing CPLD Version

Security Key

CPLD Key ID

Updated CPLD Version

Update File (*.itb)

15D_ES3

Any version

Any key

RnD_KAK

03_TS67EC5839

#1

15D_HBN

04_TS67B607AE

Key 0

RnD_KAK

04_TS67E45AE9

#2

04_TS67B607AE _KAK0_ISK1_revoke _ISK0_signed

Key 1

Production_KAK0

04_TS67E45AE9 _KAK0_ISK1_revoke _ISK0_signed

#3

All files required for the CPLD update are included in the release and can be found in gaudi3-system-1.21.2-76.tgz located in the vault and Intel RDC:

  1. HL325_15D_R0B_20250401a_FPGA_03_TS67EC5839.itb

  2. HL325_15D_HBN_R0C_20250326a_FPGA_04_TS67E45AE9.itb

  3. HL325_15D_HBN_R0C_20250326a_FPGA_04_TS67E45AE9_KAK0_ISK1_revoke_ISK0_signed.itb

Retrieve CPLD Type and Keys

To retrieve the CPLD FW, type, and keys, perform one of the options listed below. In case you do not have access to CPLD registers, refer to Option 2.

  1. Download gaudi3-sec-fit-fw-60.1.0.tgz from vault or Intel RDC.

  2. Untar the .tgz file.

  3. Unload the drivers. The order depends on the version of the driver currently running:

    • v1.21.1 and above:

      sudo modprobe -r habanalabs_ib && sudo modprobe -r habanalabs_en && sudo modprobe -r habanalabs_cn && sudo modprobe -r habanalabs && sudo modprobe -r habanalabs_compat

    • v1.20.X and below:

      sudo modprobe -r habanalabs && sudo modprobe -r habanalabs_cn && sudo modprobe -r habanalabs_ib && sudo modprobe -r habanalabs_en

  4. Retrieve the CPLD FW, type, and security key by using the gaudi3-cpld-verify-version.itb file:

    sudo hl-fw-loader -f gaudi3-cpld-verify-version.itb

    Output example for the CPLD type 15D_ES3:

    Habanalabs firmware flashing utility version hl-1.19.1-fw-57.2.2.0 (Jan 16 2025 - 17:32:39)

#### Start scanning PCI for devices
####      Found PCI device(s) at: b1:00.0(Gaudi3)
#### Finished scanning: 1 device(s) found
#### Read file gaudi3-cpld-verify-version.itb  Ok
#### Read file /lib/firmware/habanalabs/gaudi3/gaudi3-agent-fw_loader-fit.itb          Ok
#### File component versions
####     agent:                  hl-gaudi3-1.19.1-fw-57.2.2-sec-2
#### About to send FW for the following devices:
####      b1:00.0(Gaudi3)
#### PLEASE DO NOT POWER OFF THE SYSTEM!!!
#### Are you sure [y/N]? y
#### Start sending firmware

####      Sending to 0000:b1:00.0(Gaudi3)
.
Waiting for device to become ready...
Ready
CPLD model: LFMXO5-15D-ES3
CPLD firmware version: 00000003
CPLD firmware timestamp: 677A4813
CPLD key ID: RnD_KAK

....
.
Waiting for device to become ready...
Ready
....
Missing firmware's version information - skipping version check
Ok (17.463844 seconds)
####
#### Finished sending firmware:           Ok

    If the CPLD version in use is lower than 04_TS66F2DEBE, the output of sudo hl-fw-loader -f gaudi3-cpld-verify-version.itb might be incorrect. In this case, use the below command to retrieve the CPLD information:

    hl-smi -q |grep -e "PCB Version" -e "CPLD Version"

    Output example:

    PCB Version               : R0C
CPLD Version              : Version: 0x4, Timestamp(epoch): 0x66f2debe […]

    Use the following table to map the PCB versions to their corresponding CPLD types:

    PCB Version

    CPLD Type

    R01

    Production (15D_HBN, LFMXO5-15D-HBN-HL325)

    R0C

    QS (15D_HBN, LFMXO5-15D-HBN-HL325)

    R0B

    ES (15D_ES3, LFMXO5-15D-ES3)

  1. From your operating system, retrieve your existing CPLD FW:

    hl-smi -L |grep CPLD

  2. Unload the drivers. The order depends on the version of the driver currently running:

    • v1.21.1 and above:

      sudo modprobe -r habanalabs_ib && sudo modprobe -r habanalabs_en && sudo modprobe -r habanalabs_cn && sudo modprobe -r habanalabs && sudo modprobe -r habanalabs_compat

    • v1.20.X and below:

      sudo modprobe -r habanalabs && sudo modprobe -r habanalabs_cn && sudo modprobe -r habanalabs_ib && sudo modprobe -r habanalabs_en

  3. Retrieve your CPLD device type by reading the CPLD register 0x2b bits [2:3]. The table below outlines the possible read values with their corresponding CPLD type:

    Read Value [2:3]

    CPLD Type

    0b’01

    15D-ES3

    0b’10

    15D-HBN

  4. Retrieve your existing security key by reading the CPLD register 0x68. The table below outlines the possible read values with their corresponding keys:

    Read Value

    Key Used

    0x00

    Key 0

    0x11

    Key 1

Update CPLD Flow

  1. Install or upgrade Intel Gaudi SW Stack and driver on a Fresh OS. Refer to Driver and Software Installation section. If you are using an older version of CPLD than 04_TS66F2DEBE, you must first update the OAM CPLD to 04_TS66F2DEBE with v1.18.0 before upgrading to v|Version|. Attempting to update directly to OAM CPLD version 04_TS67E45AE9 will result in a failure. Refer to v1.18.0 of the document for the full process.

  2. Update preboot:

    sudo hl-fw-loader -y

  3. Download the gaudi3-system-1.21.2.tgz file from vault or Intel RDC.

  4. Untar the .tgz file.

  5. Update the CPLD FW.

  1. Retrieve your PCI addresses (BDFs):

    lspci -nn |grep -i Habana

  2. Open 8 terminals and update the OAM CPLD FW for each card using the corresponding BDF in each terminal:

    sudo hl-fw-loader -f <CPLD File.itb> -d <BDF>

  1. Power cycle the device.

  2. From your operating system, verify the CPLD FW update by running one of the below commands:

    hl-smi -L |grep CPLD

    sudo hl-fw-loader -f gaudi3-cpld-verify-version.itb