Scale-Out Topology

The following table summarizes the scale-out topology further described in DDP-based Scaling of Gaudi on PyTorch.

Scale Out Topology

Distributed Framework

Scaling Via

Flags

Notes

AWS DL1 / Host NIC Scaling

PyTorch DDP

Libfabric

HCCL_OVER_OFI=1

Refer to Scale-Out via Host-NIC over OFI

Gaudi based

PyTorch DDP

RDMA

None

Default mode = RDMA