Power Stress and EDP Tests Plugins Design, Switches and Parameters

This section describes plugin specific switches only. Common plugin switches and parameters are described in hl_qual Common Plugin Switches and Parameters.

Note

To run power stress and EDP tests on HL-328, make sure to follow the steps described in the Limitations section.

Gaudi 3/2 Power Stress Plugin Design Consideration and Responsibilities

The power stress plugin runs a power stress test, putting the device in constant and equal level power load for an extended time (more than two hours). The following device functionalities are tested:

• Thermal stress test, cooling system functionality, temperature dissipation and thermal protection mechanisms. These are tested while running the power stress plugin in extreme load.

• PID module which is responsible for power management and keeping power usage below allowable MAX power. See Power Stress Test Expected Maximum Power Levels.

Prerequisites

For Gaudi 2 only: Before running this test, load the driver:

sudo modprobe habanalabs timeout_locked=0

Power Stress Test Expected Maximum Power Levels

The power stress plugin performs a maximum power stress on the device. The following lists the expected maximum power levels:

Gaudi 3

• HL-325 and HL-325L - Measured power level for 54V power supply: 850 [watts]

• HL-338 PCIe card - Measured power level for 54V power supply: 600 [watts]

• HL-328 PCIe card - Measured power level for 54V power supply: 645 [watts]

Gaudi 2

• HL-225H and HL-225C:

  • Extreme - Measured power level: 530-560 [watts]

  • High - Measured power level: 450-490 [watts]

• HL-225D:

  • Extreme - Measured power level: 405 [watts]

  • High - Measured power level: 200 [watts]

• HL-288 PCIe card:

  • Extreme - Measured power level: 400 [watts]

  • High - Measured power level: 190 [watts]

Power Stress Test - Pass/Fail Criteria

The pass/fail criteria consists of the following:

• Test must not trigger an over-temperature shutdown.

• MME engine calculation must be bit-exact.

First-gen Gaudi Power Stress Plugin Design Consideration and Responsibilities

The power stress plugin runs a power stress test, putting the device in constant and equal level power load for an extended time (more than two hours). The following device functionalities are tested:

• Thermal stress test, cooling system functionality, temperature dissipation and thermal protection mechanisms. These are tested while running the power stress plugin in extreme load.

• Power limiter and clock relaxation mechanisms. The power limiter is a mechanism that limits the power usage below 300 [watts]. When the power limit is met, the device clocks are lowered. To test the power limiter mechanism, the plugin must run at an extreme power level.

• Long work periods in typical power levels (extreme, high).

Power usage and reported temperature may differ according to the tested server as they depend on the ambient condition of the server (fan speed, ambient temperature and number of devices been tested).

The power stress plugin conducts multi-level power stress test. The test plugin supports the following power levels:

• Extreme - Measured power level: 345 - 355 [watts]

• High – Measured power level: 230 - 240 [watts]

Note

• The above power measurements were achieved on an HL-205 device and are peak power levels measured by the HLML API.

• Tests should run longer than 30 seconds to reach maximum power.

Power Stress Test - Pass/Fail Criteria

The power stress tests must run until completion without overheating or power supply failures.

Power Stress Test Plugin Switches and Parameters

Gaudi 3

hl_qual -gaudi3 -c <pci bus id> [-t <time in seconds>] -rmod <serial | parallel> [-dis_mon] -s [-enable_ports_check <all | int>] [-toggle]

Switches and Parameters | Description
-t		Power stress test duration in seconds. Applicable range: 60-172800. If this switch is omitted, the default value is 60 seconds.
-s		Power stress test selector.
-enable_ports_check <all | int>		Indicates whether the ports are UP or DOWN. If the ports are DOWN, the test fails: all - Checks all the external and internal ports. int - Checks the internal ports only.
-toggle		Enables ports toggle check and count number of port toggles during the test.

./hl_qual -gaudi3 -c all -rmod parallel -s -t 120

Gaudi 2

hl_qual -gaudi2 -c <pci bus id> [-t <time in seconds>] -rmod <serial | parallel> [-dis_mon] -s [-l <extreme | high>] [-enable_ports_check <all | int>] [-toggle]

Switches and Parameters	Description
-t	Power stress test duration in seconds. Applicable range: 60-172800. If this switch is omitted, the default value is 60 seconds.
-s	Power stress test selector.
-l <extreme | high>	Power level selector for 54V power supply. For the power level, measurements, refer to Power Stress Test Expected Maximum Power Levels. If the value is not specified, the default value is High.
-enable_ports_check <all | int>	Indicates whether the ports are UP or DOWN. If the ports are DOWN, the test fails: all - Checks all the external and internal ports. int - Checks the internal ports only.
-toggle	Enables ports toggle check and count number of port toggles during the test.

./hl_qual -gaudi2 -c all -rmod parallel -s -l extreme -t 120

First-gen Gaudi

hl_qual -gaudi -c <pci bus id> [-t <time in seconds>] -rmod <serial | parallel> [-dis_mon] -s [-l <extreme | high>]

Switches and Parameters	Description
-t	Power stress test duration in seconds. If this switch is omitted, the default value is 40 seconds.
-s	Power stress test selector.
-l <extreme | high>	Power level selector: extreme - 345-355 [w] measured on HL-205 high - 230-240 [w] measured on HL-205 If the value is not specified, the default value is High.

./hl_qual -gaudi -c all -rmod parallel -s -l extreme -t 120

EDP Stress Test Plugin Design Consideration and Responsibilities

The EDP stress test verifies the functionality of the power supply by generating a fast power usage transient from low power to high power and vice versa. The power cycles repeat throughout the test’s execution time.

Figure 18 EDP Stress Test Power Cycles

The test’s configurable parameters, -tw/-ts <time>, set the duration of the power cycle which consists of high-power level and low-power level (idle state) usage. The high power levels are configurable via the -l switch.

EDP Stress Test Expected Maximum Power Levels

The EDP stress plugin performs a maximum power stress on the device. The following lists the expected maximum power levels:

Gaudi 3

• HL-325 and HL-325L:

  • Extreme - Measured power level for 54V power supply: 850 [watts]

  • High – Measured power level for 54V power supply: 820-830 [watts]

• HL-338 PCIe card:

  • Extreme - Measured power level for 54V power supply: 600 [watts]

  • High – Measured power level for 54V power supply: 600 [watts]

• HL-328 PCIe card:

  • Extreme - Measured power level for 54V power supply: 645 [watts]

  • High – Measured power level for 54V power supply: 645 [watts]

Gaudi 2

• HL-225H and HL-225C:

  • Extreme - Measured power level: 530-560 [watts]

  • High - Measured power level: 450-490 [watts]

  • inc_power (extreme + Serdes) - Measured power level: 570 [watts]

• HL-225D:

  • Extreme - Measured power level: 405 [watts]

  • High - Measured power level: 200 [watts]

  • inc_power (extreme + Serdes) - Measured power level: 410 [watts]

• HL-288 PCIe card:

  • Extreme - Measured power level: 400 [watts]

  • High - Measured power level: 190 [watts]

  • inc_power (extreme + Serdes) - Measured power level: 390 [watts]

First-gen Gaudi

HL-205:

• Extreme - Measured power level: 345-355 [watts]

• High - Measured power level: 250-260 [watts]

Prerequisites

For Gaudi 2 only: Before running this test, load the driver:

sudo modprobe habanalabs timeout_locked=0

EDP Stress Test - Pass/Fail Criteria

The EDP stress test must run until completion without overheating or power supply failures.

EDP Stress Test Plugin Switches and Parameters

Gaudi 3

./hl_qual -gaudi3 -c <pci bus id> [-t <time in seconds>] -rmod <serial | parallel> [-dis_mon] -e [-sync] [-Tw <time in seconds>] [-Ts <time in seconds>]
                                  [-enable_ports_check <all | int>] [-toggle] [-l <extreme | high>]

Switches and Parameters	Description
-t	Power stress test duration in seconds. Applicable range: 60-172800. If this switch is omitted, the default value is 60 seconds.
-e	EDP test selector.
-l <extreme | high>	Power level selector for 54V power supply. For the power level, measurements, refer to EDP Stress Test Expected Maximum Power Levels. If the value is not specified, the default value is High.
-sync	Enables device sync. When using this switch, the rising edge of the power cycle is synchronized between all devices.
-Tw <time in seconds>	Sets the duration of the high power time for the EDP power cycle which is measured in seconds. Applicable range: 1-20.
-Ts <time in seconds>	Sets the duration of the idle time for the EDP power cycle which is measured in seconds. Applicable range: 1-20.
-enable_ports_check <all | int>	Indicates whether the ports are UP or DOWN. If the ports are DOWN, the test fails: all - Checks all the external and internal ports. int - Checks the internal ports only.
-toggle	Enables ports toggle check and count number of port toggles during the test.

The below command line executes the EDP stress test for 60 seconds and runs 10 power cycles. Each power cycle runs 3 seconds of high power usage and 1 second of low power usage.

./hl_qual -gaudi3 -c all -rmod parallel -t 60 -e -Tw 3 -Ts 1

Gaudi 2

./hl_qual -gaudi2 -c <pci bus id> [-t <time in seconds>] -rmod <serial | parallel> [-dis_mon] [-mon_cfg <monitor INI path>]
      -e [-sync] [-enable_ports_check <all | int>] [-toggle] [-Tw <time in seconds>] [-Ts <time in seconds>] [-l <extreme | high | inc_power>]

Switches and Parameters	Description
-t	Power stress test duration in seconds. Applicable range: 60-172800. If this switch is omitted, the default value is 60 seconds.
-e	EDP test selector.
-l <extreme | high | inc_power>	Power level selector for 54V power supply. For the power level, measurements, refer to EDP Stress Test Expected Maximum Power Levels. If the value is not specified, the default value is High.
-sync	Enables device sync. When using this switch, the rising edge of the power cycle is synchronized between all devices.
-enable_ports_check <all / int>	Indicates whether the ports are UP or DOWN. If the ports are DOWN, the test fails: all - Checks all the external and internal ports. int - Checks the internal ports only.
-Tw <time in seconds>	Sets the duration of the high power time for the EDP power cycle which is measured in seconds. Applicable range: 1-20.
-Ts <time in seconds>	Sets the duration of the idle time for the EDP power cycle which is measured in seconds. Applicable range: 1-20.
-toggle	Enables ports toggle check and count number of port toggles during the test.

The below command executes the EDP stress test for 60 seconds and runs 10 power cycles. Each power cycle runs 3 seconds of high power usage and 1 second of low power usage:

./hl_qual -gaudi2 -c all -rmod parallel -t 60 -e -l extreme -Tw 3 -Ts 1

First-gen Gaudi

hl_qual -gaudi -c <pci bus id> [-t <time in seconds>] -rmod <serial | parallel> [-dis_mon] [-mon_cfg <monitor INI path>]
      -e [-tw <time in seconds>] [-ts <time in seconds>] [-l <extreme | high>]

Switches and Parameters	Description
-t	Power stress test duration in seconds.
-e	EDP test selector.
-l <extreme | high>	Power level selector for 54V power supply. For the power level, measurements, refer to EDP Stress Test Expected Maximum Power Levels. If the value is not specified, the default value is High.
-tw <time in milliseconds>	Time duration of high power usage in the EDP test power cycle. tw should be a multiple of 200 ms. hl_qual rounds them to the nearest multiple of 200 ms.
-ts <time in milliseconds>	Time duration of low power usage (idle mode) in the EDP test power cycle. tw should be a multiple of 200 ms. hl_qual rounds them to the nearest multiple of 200 ms.

The below command executes the EDP stress test for 40 seconds and runs 10 power cycles. Each power cycle runs 2 seconds of high power usage and 2 seconds of low power usage.

./hl_qual -gaudi -c all -rmod parallel -t 40 -e -l extreme -tw 2000 -ts 2000

Limitations

Before running power stress and EDP tests on HL-328, make sure to perform the following:

1. Open the file mme_pv_gaudi3.json which is located at /opt/habanalabs/qual/gaudi3/bin.

2. Add {"Gaudi3DPmuRLConfig": ["mode4"]} between the following lines:

   {"wMaxVal": ["10"]},
   {"geometry":["4xh"]}]
   

   The lines order should be as follows:

   {"wMaxVal": ["10"]},
   {"Gaudi3DPmuRLConfig": ["mode4"]},
   {"geometry":["4xh"]}]