Optimizing Training Platform

The following optimization methods should be applied in your on-premise environment.

Ensure MPI Command Binds Cores/Socket to Correct Number

You can use this method to ensure CPU affinity and core allocation are correctly done for your processes. In the below example, six cores are allocated for one process, which controls one Gaudi in the same socket as the six cores:

--bind-to core --map-by socket:PE=6

Note

This optimization method also applies for AWS DL1.

The CPU affinity can improve performance not only for distributed training using multiple Gaudi devices in one server, but also for training using a single Gaudi device.

This number can be calculated by using the number of CPU(s) thread(s) per core (these two numbers are from the `lscpu` command), and the number of Gaudi devices. For example the HLS-1 platform contains eight devices while other servers may have four devices. In the below example, there are 96 CPU(s) and 2 thread(s) per core, therefore, 96 / 2 / 8 = 6 cores per Gaudi.

This sample code can also be used to calculate the number of physical CPU cores and HPU count to generate the appropriate PE value, shown as MPI_PE below. This can be incorporated into any model:

export PHY_CPU_COUNT=$(lscpu --all --parse=CORE,SOCKET | grep -Ev "^#" | sort -u | wc -l)
export PHY_HPU_COUNT=$(ls /dev/accel/accel? | wc -l)
export MPI_PE=$(($PHY_CPU_COUNT/$PHY_HPU_COUNT))

The PE value in the Model-References examples may be set to a common number to ensure functionality, but the directions above should be used for optimal system performance depending on the Host CPU.

Set CPU Setting to Performance

BIOS Configuration Requirement for CPU Frequency Scaling

To enable the operating system to manage CPU performance states dynamically, the system BIOS must expose and allow control over CPU frequency scaling features. Specifically, the BIOS must support ACPI CPU frequency scaling interfaces and not restrict access to performance states (P-states) or governors. This ensures that the OS can read and modify the CPU frequency governor settings via cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor. Without proper BIOS support, this interface may be unavailable or return errors, preventing the OS from applying desired power or performance policies.

The below is an example of setting the CPU to performance for Ubuntu:

#Get setting:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

#Set setting:
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Note

The CPU settings must be updated on bare metal before starting the container.

Update CPU Settings

This section describes how to update CPU settings on Gaudi 3 using Sapphire and Granite Rapids to optimize performance.

Sapphire and EMR Rapids - 4th and 5th Gen Xeon® Processors

  1. Set Energy Perf BIAS:

    wrmsr -a 0x1b0 0x0

  2. Set the Core Frequency (HWP):

    wrmsr -a 0x774 0x2708
Granite Rapids - 6th Gen Xeon® Processor

Note

Enumerated lists that are auto-numbered start with #..

  1. Disable SubNUMA in BIOS.

  2. Disable the following states:

    1. C6:

      echo 1 | tee /sys/devices/system/cpu/cpu*/cpuidle/state3/disable

    1. C6P:

      echo 1 | tee /sys/devices/system/cpu/cpu*/cpuidle/state4/disable

    Note

    The C6P state may not be supported on all systems. To view the available states, run the following:

    cat /sys/devices/system/cpu/cpu*/cpuidle/state*/name

  3. Set Latency Optimized Mode (LOM). This can be done either by cloning Intel® PCM API or in BIOS:

    cd ~
git clone --recursive https://github.com/intel/pcm
cd pcm
git submodule update --init --recursive
mkdir build
cd build
cmake ..
cmake --build .
sudo cp ./bin/pcm-tpmi /usr/local/bin/pcm-tpmi
cd ~/pcm/scripts
sudo bash ./bhs-power-mode.sh --latency-optimized-mode

    For more details, see Building PCM Tools.

    Make sure to be on a relatively new BIOS:

    1. Click Socket Configuration.

    2. Navigate to Advanced Power Management Configuration.

    3. Locate and choose CPU - Advanced PM Tuning.

    4. Enable Latency Optimized Mode.

  4. Set Energy Perf BIAS:

    wrmsr -a 0x1b0 0x0

  5. Set the Core Frequency (HWP)

    wrmsr -a 0x774 0x2708

Ensure System Components are Updated

Refer to the Installation Guide to check for the latest Docker run commands, FW and SW packages, BMC and CPLD versions and make sure they are installed properly.

Ensure Dataset/Model Code/Output Utilize High-Performance NVMe/SSD Storage

For best performance, use NVMe for all datasets, model code, and output locations when running training.