Intel Gaudi RDMA PerfTest Tool

This document provides guidelines for installing and running the Intel® Gaudi® RDMA PerfTest tool habanalabs-perf-test on Gaudi accelerator. This tool is designed for low-level, high-performance testing of connectivity through ping-pong and bandwidth communication tests. It utilizes the Reliable Connection (RC) method and RDMA Write operations to deliver performance measurements.

Note

• The tool is supported only with Gaudi 3 on Ubuntu 22.04.

• The tool can only be used in a container and is not supported on a VM.

• The tool can only be used with Gaudi 3 NICs.

Prerequisites

Make sure you have the following packages installed:

• habanalabs-thunk

• habanalabs-dkms

• habanalabs-rdma-core

For more information about the packages installation, see Custom Driver and Software Installation.

Note

If you have upgraded to the 1.19.1 software version, the above packages are already included, and no additional installation is required.

Installation

1. Download the package files:

   wget https://vault.habana.ai/artifactory/debian/jammy/pool/main/h/habanalabs-perf-test/habanalabs-perf-test_1.19.1-26_amd64.deb
   

2. Install the package files:

   sudo apt install ./habanalabs-perf-test_1.19.1-26_amd64.deb
   

Options and Usage

RDMA PerfTest tool is executed using cloud_run.py Python wrapper script. It runs across an entire data center, covering all pairwise permutations for thorough evaluation.

The following port connectivity options are supported for testing on multiple nodes:

Prerequisites

The below lists the prerequisites needed to run the cloud_run.py Python wrapper script:

• Install the requirements file:

  pip install -r requirements.txt
  

• Make sure all tested nodes have SSH keys configured for seamless access. The script relies on SSH sessions established via SSH keys.

  Note

  Starting from 1.18.0 release, SSH host keys have been removed from Dockers. To add them, make sure to run /usr/bin/ssh-keygen -A inside the Docker container. If you are running on Kubernetes, make sure the SSH host keys are identical across all Docker containers. To achieve this, you can either build a new Docker image on top of Intel Gaudi Docker image by adding a new layer RUN /usr/bin/ssh-keygen -A, or externally mount the SSH host keys.

• Make sure all external NIC ports are active and accessible on each node. For more details, see Disable/Enable NICs.

• Prepare a host file listing all tested nodes with their SSH connection details (IP and port). Use the following format: SSH-IP:SSH-PORT. For example:

  kuku-kvm12-lake:22
  kuku-kvm13-lake:22
  kuku-kvm14-lake:22
  kuku-kvm15-lake:22
  

• Make sure to write the LD_LIBRARY_PATH environment variable to a file. This ensures it can be accessed in remote SSH sessions during testing:

  echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" | tee ~/.ENV_SCALEUP
  

Python Wrapper Options

Option	Description
-h, –help	Show the help message and exit
-hf, –host_file	Path to a host_file that includes a host IP list
-skt, –ssh_key_type	SSH key type
-skf, –ssh_key_file	SSH private key file path (default: /home/username/.ssh/id_rsa)
-o, –output	Save all the log files in a specific path (the flag must be set)

Example:

python3 ./cloud_run.py --host_file ./hostfile --output /tmp/output perftest

Test-specific Options

Ping-pong Test (ping-pong)

Option	Description
-h, –help	Show the help message and exit
-tp, –tcp_port	Specify the TCP port range script will use (default:1100)
-s, –size	Size of message to exchange (default: 4096)
-r, –rx_depth	Number of receives to post at a time (default: 128)
-n, –iters	Number of exchanges (default: 10)
-c, –chk	Validate received buffer

Example:

python3 ./cloud_run.py --host_file ./hostfile --output /tmp/output perftest ping_pong --size 4096 --rx_depth 128 --iters 10 --chk

Bandwidth Test (write_bw)

Option	Description
-h, –help	Show the help message and exit
-tp, –tcp_port	Specify the TCP port range script will use (default:1100)
-s, –size	Size of message to exchange (default: 1048576)
-r, –rx_depth	Number of receives to post at a time (default: 128)
-n, –iters	Number of exchanges (default: 100000)
-c, –criteria	Pass/fail criteria value for the test threshold in Gbps (no default, optional)

Example:

python3 ./cloud_run.py --host_file ./hostfile --output /tmp/output perftest write_bw --size 1048575 --rx_depth 128 --iters 100000

Expected output:

* CloudReport_<timestamp>.txt - Tested nodes summary.
* <server_host_name>_<client_host_name>
  └── scaleUpRepor_<timestamp>.txt - Specific server and client pair summary.
  └── perftest
    └── <network_ip>
      └── <server - device (ib_dev)>
        └── <device_port (ib_port)>
          └── <client - device (ib_dev)>
            └── <device_port (ib_port)>.txt - Both device application prints.

The output is saved in the output directory in a timestamp-named folder.