AMD Validated Reference Architecture: Data Storage for GPU-Accelerated AI

Learn More

Velocity • Durability

Book a Briefing
Contact Sales
Book a Briefing
Book A Demo
Contact Sales
Contact Sales
Documentation

V5000 Installation and Service Guide

Download PDF

4. Configure InfiniBand with Subnet Manager

This chapter briefly describes how to use a Subnet Manager tool when configuring InfiniBand installations.

4.1 Subnet Manager Requirements

For InfiniBand networks, the Subnet Manager is a centralized agent responsible for defining and managing an InfiniBand fabric. The Subnet Manager discovers all InfiniBand fabric devices and configures them per the specified subnet configuration. A full overview of the functions of a Subnet Manager are beyond the scope of this document, however, NVIDIA provides their own high-level overview in NVIDIA’s Subnet Manager Overview document located at https://docs.nvidia.com/networking/display/mlnxosv3112006/subnet+manager.

The centralized nature of the Subnet Manager mostly avoids the need to individually configure the nodes of the InfiniBand fabric. This does mean that InfiniBand nodes will not appear online until they have been discovered and configured by a valid Subnet Manager.

PanFS places few requirements on the InfiniBand subnet, however, the requirements it does place are:

  • All realm nodes must be within same partition.
  • The partition used for the realm must enable support for IPoIB along with the corresponding IPoIB broadcast group.
  • The Subnet Manager must enable support for Virtualization (virt_enabled 2 for OpenSM). This is because the VPODs are attached to the fabric via SR-IOV virtual functions.

Most InfiniBand switches provide an embedded Subnet Manager. However, in many cases these embedded tools do not support the ability to configure virtual endpoints. In such circumstances, an additional secondary instance of OpenSM may be installed to a node on the InfiniBand subnet to provision the virtual endpoints. Due to OpenSM limitations, it is not possible for a VDURA Data Platform to act as the Subnet Manager.

4.2 OpenSM

OpenSM is a fully-featured Subnet Manager that can be installed on most Linux InfiniBand nodes. It is provided with the NVIDIA DOCA OFED software but is also available in the standard packages offered by many Linux distributions. The full documentation for OpenSM can be found at https://docs.nvidia.com/networking/display/mlnxofedv24102180lts/nvidia+sm. For the purposes of this installation the procedure for installing and configuring a minimal OpenSM instance to be able to run a realm is covered below.

  1. Install the latest NVIDIA DOCA/OFED suite for your Linux distribution using the instructions at https://docs.nvidia.com/doca/sdk/doca+installation+guide+for+linux/index.html.

Alternatively use your Linux distribution’s own copy of OpenSM (e.g. yum install opensm, zypper install opensm, or apt install opensm).

  1. Create a new partition file in /etc/opensm/partitions.conf:
mkdir -p /etc/opensm
touch /etc/opensm/partitions.conf
  1. Using an editor of your choice, add the following content to /etc/opensm/partitions.conf:
Default=0x7fff, ipoib, rate=3, mtu=5, scope=2, defmember=full:
ALL, ALL_SWITCHES=full;
  1. Enable and start the opensm service:
systemctl start opensmd
systemctl enable opensmd