Release Notes
On this Page
Release Notes¶
Note
For previous versions of the Release Notes, please refer to Previous Release Notes.
New Features - v1.21.0¶
The following documentation and packages correspond to the latest software release version from Intel® Gaudi®: 1.21.0-555 We recommend using the latest release where possible to stay aligned with performance improvements and updated model coverage. Please refer to the Installation Guide for further details.
The Intel Gaudi Software Suite version v1.21.0 may not include all the latest functional and security updates. A new version, v1.22.0, is targeted to be released in July 2025 and will include additional functional and security updates. Customers should update to the latest version as it becomes available.
General¶
Added support for Gaudi 3 with the Intel® Tiber™ AI Cloud. See Intel Tiber AI Cloud Quick Start Guide.
In PerfTest tool, added support for Gaudi 2 and the new
--basic_check
switch. The switch enables testing between the corresponding ports across two systems.OFI Wrapper and libfabric are no longer installed by default. When using Host NIC with OFI in Intel Gaudi containers, make sure to follow the instructions in Using Host NIC over OFI section.
Firmware¶
For Gaudi 3:
Added support for GetSensorThresholds and SetSensorThresholds sensors.
Added support for I2C direct and IPMI protocols in out-of-band.
Added GET_UID MCTP control command.
For Gaudi 3 and Gaudi 2:
Added hlml_device_get_power_management_limit_constraints API to the Habana Labs Management Library and Habana Labs Python Management Library. See HLML API Reference and PYHLML API Reference.
For HL-225D:
Enabled FP32 workloads.
PyTorch¶
Intel Gaudi offers a wide range of models using Eager mode and
torch.compile
, which is the default mode. However, any model requiring Lazy mode will need to set thePT_HPU_LAZY_MODE=1
flag, as it is set to 0 by default. Usingtorch.compile
with Eager mode is recommended, as Eager mode alone can be slower due to its limited optimization of computation graphs. Note that Lazy mode will be deprecated in future releases.Enabled multi-threaded graph compilation (only with PT_HPU_LAZY_MODE=0) to improve performance in Eager mode and reduce time-to-train (TTT) in Compile mode. See Parallel Compilation section
Added vLLM profiling methods. See Profiling with vLLM.
Added support for Reducing vLLM FP8 Warmup Time feature to enhance performance. See reducing-vLLM-fp8-warmup-time.
Added support for the following features to the Intel Gaudi vLLM fork:
Automatic Prefix Caching
Pipeline Parallelism - see Pipeline Parallelism section.
Guided Decoding
V1 Support - initial support
Multimodality
Exponential Bucketing - initial support
Delayed Sampling - initial support
FP16 support - limited models (for further details, see Intel Gaudi vLLM fork)
Multi node
INT4 support (AWQ/GPTQ) - limited models (for further details, see Intel Gaudi vLLM fork)
Support for RedHat Openshift AI
Split QKV optimizations for BF16
Added support for new models in Intel Gaudi vLLM fork:
DeepSeek-R1
Codellama-34b-instruct-hf
Llama 3.3-70B-instruct-hf
mistralai/Mistral-Small-Instruct-2407
Enabled Context Parallelism in training with BF16 precision for MLM.
Added support for NumPy 2.x. Intel Gaudi Docker image includes NumPy 1.26.4 by default but can be upgraded to the latest NumPy 2.x version as needed.
The
mixture_of_experts
(MoE) custom op has been extended to support the following new flows. For further details, refer to Mixture of Experts Forward (MoE):Dynamic Per-Token Quantization: This flavor runs the operation in FP8 precision. Downscale GEMM scales are computed internally, while the scales for the upscale GEMM and weights are provided by the caller. It supports per-token scales, which can be either squeezed 1D or unsqueezed 2D tensors.
Blockwise Quantization: In this flavor, the operation runs in BF16 precision, with weights provided using FP8 blockwise quantization. Weight dequantization is performed internally within the operation. Only square blocks are supported.
In MediaPipe, added support for HEVC video decode. See MediaPipe.
v1.21.0 release has been validated with Hugging Face Optimum for Intel Gaudi library and model version v1.16.0. Future releases of the Optimum for Intel Gaudi library may be validated with this release. See the Support Matrix for a full list of version support.
The Hugging Face Optimum for Intel Gaudi library now supports the updated TGI-gaudi version 2.3.2. See https://github.com/huggingface/tgi-gaudi.
Validated the Intel Gaudi 1.21.0 software release with vLLM v0.7.2. See the Support Matrix for a full list of version support.
Validated the Intel Gaudi 1.21.0 software release on PyTorch Lightning version 2.5.1. See https://lightning.ai/docs/pytorch/stable/integrations/hpu/. PyTorch Lightning will be deprecated in the subsequent release.
Upgraded to Intel Neural Compressor (INC) v3.4.
Enhancements - v1.21.0¶
Firmware¶
For Gaudi 3:
Improved I2C busses stability for sensors readings and PLDM transmissions.
Fine-tuned ADC power sensor to achieve more accurate readings.
Improved firmware update transfer mechanism over PLDM/MCTP.
For HL338 PCIe card, fine-tuned idle power consumption as a base for utilization calculation.
Improved OAM health reports over PLDM.
HuggingFace¶
Integrated HPU as the native backend for Transformers and Accelerate, successfully validated Llama 70B and Mixtral PEFT across 8 cards.
Transformer Engine¶
Improved FP8 training performance by introducing a smoother Swiglu implementation.
Intel Gaudi, Kernels, tpc_guid
and HCCL¶
Stability and performance of the compiler in Eager mode for transpose operations.
Stability of the compiler in Eager mode for FP8 data type (Llama inference).
Embedded System Tools¶
hl_qual test (F2 extreme) support to prevent interrupt storms on certain cards.
hl-smi-async tool to provide valid Ethernet status even when NICs are disabled.
Card recognition when it appears on a non-zero PCIe domain, resolving unexpected behavior during maintenance.
Extended sensor tool configuration to support new variants of Gaudi 3, such as HL-328.
Bug Fixes and Resolved Issues - v1.21.0¶
Firmware¶
For Gaudi 3:
Fixed the values of the engines utilization.
Fixed PLDM package component classification values to use 10 (firmware) instead of 0 (unknown).
Fixed onboard temperature sensors readings on PCB revision R0C.
Aligned PCIe HDR sensors to the PLDM specification.
For Gaudi 2:
Fixed the reporting of ASIC serial number out-of-band when LKD is loaded.
Fixed Ethernet error injection to conform the out-of-band Ethernet opcode specification.
DeepSpeed¶
Fixed graph breaks caused by the
fetch_sub_module
routine, which improved DeepSpeed (Zero3) performance when using PyTorch compile.Using
torch.use_deterministic_algorithms
no longer breaks memory mapping whenkeep_module_on_host
flag is set while running inference on large models.Added missing components to support DeepSeek V3 AutoTP.
Transformer Engine¶
Fixed the following:
An accuracy issue in Fused Attention during training with FP8 precision on Gaudi 3.
A bug in full recompute mode for HL_PP > 1 configurations during FP8 precision training.
An accuracy bug in configurations using Sequence Parallel mode during FP8 precision training on Gaudi 2.
Intel Gaudi, Kernels, tpc_guid
and HCCL¶
Reduced compilation (warm-up) time through various backend software improvements, resulting in overall device time improvements on Gaudi 3 through optimizations.
Fixed the following
Compilation error: “Message: getDramOffset: DRAM offset is not set!” that appeared in the Schwarz model.
Oversubscription of the MME Suspension Buffer, which caused Compute or DMA timeout errors.
Long and unbalanced MME computation caused by oversubscription of the Dcore cache.
Support for the torch.block_diag operation.
Segmentation fault and endless loop compilation error: “Message: calculator stuck in endless loop” on Gaudi 2.
For
mixture_of_experts
(MoE):Removed limitation on the number of operands for the MoE operation.
Fixed support for updating tensors in scatter-add operations for partial weight gradient accumulation.
HL_QUAL¶
Removed nic_base dir_bw
test variant as the calculated bandwidth was fluctuating. Instead, the bandwidth
test variant can be used,
which provides stable and predictable results. See Serdes Base Test Testing Modes.
Known Issues and Limitations - v1.21.0¶
General¶
Enabling IOMMU passthrough is required only for Ubuntu 24.04.2/22.04.5 with Linux kernel 6.8. For more details, see Enable IOMMU Passthrough.
Running functional test in high power mode experiences a performance failure with an 8.5% degradation, whereas the functional test in extreme power mode is fully operational.
Intel Gaudi Media Loader is not supported in RHEL9.4 OS with Python 3.11.
PyTorch¶
Multiprocess worker creation using the
fork
start method is not supported with PyTorch dataloader. Instead, use thespawn
orforkserver
start method. See Torch Multiprocessing for DataLoaders.In certain models, there is performance degradation when using HPU graphs with Lazy collectives. To mitigate this, set
PT_HPU_LAZY_COLLECTIVES_HOLD_TENSORS=1
flag, though it may lead to increased memory consumption on the device side. This will be fixed in the subsequent release.When running DeepSpeed inference with
keep_module_on_host = True
flag, make sure to setDEEPSPEED_USE_HABANA_FRAMEWORKS_DETERMINISTIC_API=1
. This prevents OOM issues on the host when loading the model’s parameters.Sporadic numerical instability may occur when training with FP8 precision.
Starting from PyTorch 2.5.1,
torch.nn.functional.scaled_dot_product_attention
automatically casts inputs to FP32, which can lead to performance degradation or GC failures. To avoid this, settorch._C._set_math_sdp_allow_fp16_bf16_reduction(True)
to enabletorch.nn.functional.scaled_dot_product_attention
to operate on BF16. Without this setting, the inputs will always be cast to FP32.DeepSpeed is not compatible with PyTorch Lightning in SW releases greater than 1.17.X. However, it is supported in older releases that include lightning-habana 1.6.
To bypass a performance issue in Linux kernel version >= 5.9 (e.g. Ubuntu 22.04.5), the
intel_idle
driver must be disabled by addingintel_idle.max_cstate=0
to the kernel command line.Support for
torch.compile
is in early stage. Models may not work (due to missing OPs implementation) or performance may be affected.Support for Eager mode is in early stages. Models may not work (due to missing OPs implementation) or performance may be affected. The functionality of Eager mode as a subset of Lazy mode can be emulated by using
PT_HPU_MAX_COMPOUND_OP_SIZE
environment variable and limiting cluster sizes to 1. See Eager Mode.Model checkpointing for ResNet50 in
torch.compile
mode is broken. This will be fixed in the next release.Timing events where
enable_timing=True
may not provide accurate timing information.Handling Dynamic shapes can be initiated by setting the
PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES
flag. Flag is disabled by default but enabled selectively for several models. For best performance, users should follow the guidance on how to work with Dynamic Shapes in the Handling Dynamic Shapes document.Graphs displayed in TensorBoard have some minor limitations, eg. operator’s assigned device is displayed as “unknown device” when it is scheduled to HPU.
HPU tensor strides might not match that of CPU as tensor storage is managed differently. Reference to tensor storage (such as torch.as_strided) should take into account the input tensor strides explicitly. It is recommended to use other view functions instead of torch.as_strided. For further details, see Tensor Views and TORCH.AS_STRIDED.
Weights sharing:
Weights can be shared among two or more layers using PyTorch with Gaudi only if they are created inside the module. For more details, refer to Weight Sharing.
Weights are not shared with operators outside of the PyTorch library (i.e. PyBind11 functions).
User-defined attributes in HPU
torch.nn.Parameter
are not preserved aftertorch.nn.Parameter
is assigned with a CPU tensor.Python API
habana_frameworks.torch.hpu.current_device()
returns 0 regardless of the actual device being used.For
torch.nn.Parameter
which is not created insidetorch.nn.Module
:When two
torch.nn.Parameter
are on CPU storage and referencing the same parameter, the connection will be lost if one of them is moved to HPU.Assigning a CPU tensor to HPU
torch.nn.Parameter
is not supported.
Saving metrics to a file configured using Runtime Environment Variables is not supported for workloads spawned via
torch.multiprocessing
.Using torch.device(hpu:x) - (for example, as model.to) - where x is rank > 0 may lead to memory leaks. Instead, always use torch.device(hpu) to access the current rank.
Added the capability to serialize constant tensors, enabling recipe caching to disk for inference scenarios. However, due to a technical limitation, sharing recipes between cards on a single server is not possible. Recipes from each card are stored in separate directories, leading to increased usage of disk space.
Performing view-related operations on tensors with INT64 data type (
torch.long
) in Lazy mode can lead to incorrect results. If this data type is not required, the script should work with INT32 tensors (torch.int
). By default, PyTorch creates integer tensors withtorch.long
data type, so make sure to explicitly create INT32 tensors. This limitation does not apply to Eager +torch.compile
mode (PT_HPU_LAZY_MODE=0
).Launching ops with tensor inputs from mixed devices (
hpu
andcpu
) is not supported in Eager +torch.compile
mode (PT_HPU_LAZY_MODE=0
). All tensors need to reside onhpu
. Launching ops with tensor inputs from mixed devices is supported in Lazy mode in which internal transfers tohpu
are performed.