HL-338 In-Band (IB) CPLD Programming

The primary purpose of the CPLD (Complex Programmable Logic Device) interface is to facilitate communication and interaction between the BMC and the CPLD.

The CPLD in the HL-338 can be flashed with custom images by:

• Using the HL-338 PCIe card via OOB with PLDM type 5. Refer to Gaudi 3 OOB Management Specification in the vault.

• Using the HL-338 PCIe card via in-band with hl-fw-loader.

This document provides guidelines for updating the CPLD via In-Band with hl-fw-loader.

Updating CPLD

The hl-fw-loader utility is used for updating the CPLD via in-band.

CPLD Version Matrix

The CPLD version and the corresponding .itb file required for the update depend on the sample version and the security key as described in the table below.

CPLD Device Type	Sample Version	CPLD Key ID	Updated CPLD Version	Update File (*.itb)
15D_HBN	ES	RnD_KAK	01_TS67B64F7B	#1
15D_HBN	QS	Production_KAK0	01_TS67B64F7B	#2

All files required for the CPLD update are included in the release and can be found in gaudi3-system-1.21.4-3.tgz located in the vault and Intel RDC:

1. HL338_15D_HBN_R0B_20250219a_FPGA_01_TS67B64F7B.itb

2. HL338_15D_HBN_R0B_20250219a_FPGA_01_TS67B64F7B_KAK0_ISK1_revoke_ISK0_signed.itb

Retrieve CPLD Type and Keys

To retrieve the CPLD FW, type, and keys, perform the following.

1. Download gaudi3-sec-fit-fw-61.0.2.tgz from vault or Intel RDC.

2. Untar the .tgz file.

3. Unload the drivers. The order depends on the version of the driver currently running:

   v1.21.1 and above

   sudo modprobe -r habanalabs_ib && sudo modprobe -r habanalabs_en && sudo modprobe -r habanalabs_cn && sudo modprobe -r habanalabs && sudo modprobe -r habanalabs_compat
   

   v1.20.X and below

   sudo modprobe -r habanalabs && sudo modprobe -r habanalabs_cn && sudo modprobe -r habanalabs_ib && sudo modprobe -r habanalabs_en
   

4. Retrieve the CPLD FW, type, and security key by using the gaudi3-cpld-verify-version.itb file. Note that this .itb file has been downloaded in Step 1:

   sudo hl-fw-loader -f gaudi3-cpld-verify-version.itb
   

   Output example for the CPLD type 15D_ES3:

   Habanalabs firmware flashing utility version hl-1.19.1-fw-57.2.2.0 (Jan 16 2025 - 17:32:39)
   
   #### Start scanning PCI for devices
   ####      Found PCI device(s) at: b1:00.0(Gaudi3)
   #### Finished scanning: 1 device(s) found
   #### Read file gaudi3-cpld-verify-version.itb  Ok
   #### Read file /lib/firmware/habanalabs/gaudi3/gaudi3-agent-fw_loader-fit.itb          Ok
   #### File component versions
   ####     agent:                  hl-gaudi3-1.19.1-fw-57.2.2-sec-2
   #### About to send FW for the following devices:
   ####      b1:00.0(Gaudi3)
   #### PLEASE DO NOT POWER OFF THE SYSTEM!!!
   #### Are you sure [y/N]? y
   #### Start sending firmware
   
   ####      Sending to 0000:b1:00.0(Gaudi3)
   .
   Waiting for device to become ready...
   Ready
   CPLD model: LFMXO5-15D-ES3
   CPLD firmware version: 00000003
   CPLD firmware timestamp: 677A4813
   CPLD key ID: RnD_KAK
   
   ....
   .
   Waiting for device to become ready...
   Ready
   ....
   Missing firmware's version information - skipping version check
   Ok (17.463844 seconds)
   ####
   #### Finished sending firmware:           Ok
   

   If the output of sudo hl-fw-loader -f gaudi3-cpld-verify-version.itb indicates a failure:

   #### Finished sending firmware: Failed!
   

   Use the following command to retrieve the CPLD information:

   hl-smi -q |grep -e "PCB Version" -e "CPLD Version"
   

   Output example:

   PCB Version               : R02
   CPLD Version              : Version: 0x1, Timestamp(epoch): TS67B64F7B […]
   

   Use the following table to map the PCB versions to their corresponding CPLD types:

   PCB Version	CPLD Type
   R0B/R01	ES (15D_HBN, LFMXO5-15D-HBN-HL338)
   R02	Production (15D_HBN, LFMXO5-15D-HBN-HL338)

Update CPLD Flow

1. Install or upgrade Intel Gaudi SW Stack and driver on a Fresh OS. Refer to Driver and Software Installation section.

2. Update preboot:

   sudo hl-fw-loader -y
   

3. Download the gaudi3-system-1.21.4.tgz file from vault or Intel RDC.

4. Untar the .tgz file.

5. Update the CPLD FW.

   1. Retrieve your PCI addresses (BDFs):

   lspci -nn |grep -i Habana
   

   2. Based on the HL-338 PCIe cards available, open the terminals and update the PCIe card CPLD FW for each card using the corresponding BDF in each terminal:

   sudo hl-fw-loader -f <CPLD File.itb> -d <BDF>
   

6. Power cycle the device. This should be a full AC power cycle.

7. From your operating system, verify the CPLD FW update by running one of the below commands:

   Using hl-smi

   hl-smi -L |grep CPLD
   

   Using hl-fw-loader

   sudo hl-fw-loader -f gaudi3-cpld-verify-version.itb