Optimization in Training Platform

Ensure MPI Run Command Includes Core/Socket Binding/Mapping to the Right Number of Cores

--bind-to core --map-by socket:PE=7

This is to ensure CPU affinity is done correctly for your processes and core allocation is distributed as well. In this example, 7 cores are allocated for one process, which controls one Gaudi in the same socket as the 7 cores. The CPU affinity can improve performance not only for distributed training using multiple Gaudi devices in one server, but also for training using a single Gaudi device.

This number can be calculated by using the number of CPU(s). the Thread(s) per core (these 2 numbers are from the command `lscpu`), and the number of Gaudi devices (for example 8 for Habana’s HLS-1, other servers may have 4 devices). In the below example, there are 112 CPU(s) and 2 Thread(s) per core, therefore, 112 / 2 / 8 = 7 cores per Gaudi.

$ lscpu

CPU(s): 112

On-line CPU(s) list: 0-111

Thread(s) per core: 2

Set CPU Setting to Performance

The below is an example of setting the CPU to performance for Ubuntu:

Get setting: “cat
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor”

Set setting: “echo performance \| sudo tee
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor”

Use Low Latency Kernel

There is a 15-20% performance drop on generic kernel in case of BERT, compared with low latency kernel (5.4.0-72-lowlatency), for example, in Ubuntu 18.04. You can use the command line below to install low latency kernel:

sudo apt install linux-lowlatency-hwe-18.04

Ensure FW/SW Packages/BMC/CPLD are the Latest

Refer to Installation Guide to check for the latest FW/SW Packages/BMC/CPLD versions and make sure they are installed properly.

Ensure Docker Run Command Used is the Latest

Refer to Installation Guide to make sure docker run command is installed accordingly.

Ensure Dataset/Model Code/Output are Placed on High Performing Hard Drive (NVME/SSD)

For best performance, use NVME for all datasets, model code, and output locations when running the training.

Ensure Network ifs for Habana Cards Have Static IP

Set the Habana card ifs to have static IPs by running the below command:

./manage_network_ifs.sh --set-ip

For further details, refer to manage_network_ifs.sh.