12. Remote Direct Memory Access in VDURA Systems

This chapter explains the VDURA implementation of Remote Direct Memory Access (RDMA). This chapter includes the following topics:

• Introduction
• Prerequisites
• Enable or Disable RDMA on the Realm
• Mount the RDMA Client

12.1 Introduction

PanFS can be configured to move bulk read/write data between a client and Storage Nodes over RDMA instead of TCP. Each client-to-node session keeps its TCP connection for control traffic, but adds an RDMA connection for data. Currently, VDURA systems provide one cluster-wide on/off switch for a single RDMA device per client and two client mount options. There are no other tunable parameters.

RDMA is turned off by default and must be enabled in two places: once cluster-wide on the realm, and once per client at mount time.

12.2 Prerequisites

To use RDMA with your VDURA system, you must have an RDMA-capable device on the client, e.g., an NVIDIA/Mellanox adapter (device names such as mlx5_0 and mlx5_1). The RDMA device must sit on the network interface that carries the storage traffic. If it is on a different interface, RDMA mounts fail even though the device exists.

Check what the client sees:

rdma link

The output should resemble the following, where the device name is listed before the “/”:

link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev ens1f0

In this example, mlx5_0 is the device name.

12.3 Enable or Disable RDMA on the Realm

RDMA is disabled by default on a freshly initialized realm. Follow the steps below to enable it.

1. Enter service mode:

   [pancli] support servicemode yes

2. Issue the command to enable RDMA:

   [pancli_service] mconsole_feature RDMA on

If you wish to disable RDMA, use the off option instead:

[pancli_service] mconsole_feature RDMA off

3. Verify the current state:

   [pancli_service] mconsole_feature

This command lists every feature; look for the RDMA row.

4. Leave service mode:

   [pancli_service] support servicemode no

If you are logged in to a bash shell on a Director instead of the PanCLI shell, you can run a command in service mode directly, for example: pancli -s -c mconsole_feature RDMA on

Be aware of the following points when enabling or disabling RDMA.

• The only relevant RDMA options are on and off. For more information, see the fifth point in Known Limitations.
• Setting the state to what it is already results in a “No change to RDMA needed.” message.
• Changing the state restarts services and the file system is briefly unavailable. Wait for the realm to come back online before mounting volumes or creating new ones.

12.4 Mount the RDMA Client

Use rdma-mode to mount the PanFS file system. Optionally, you can select a specific RDMA device with rdma-dev.

12.4.1 Mount Options

RDMA options behave identically for volume mounts, realm-root mounts (panfs:// [IP]/), and global mounts (panfs://[IP]:global).

Option	Description
rdma-dev [RDMA device name, e.g. mlx5_0]	By default, this is not set; the client selects a device. When rdma-mode is set to off, rdma-dev is accepted but ignored.
rdma-mode off auto enforce	Any rdma-mode value other than the three listed below will fail the mount. off: • Off is the default setting. • If RDMA is available, TCP is used and no RDMA connection is made. • If RDMA is unavailable, or rdma-dev names a device that does not exist, TCP is used. • If the realm RDMA feature is turned off, TCP is used. auto: • If RDMA is available, RDMA is used. • If RDMA is unavailable, the mount succeeds but falls back to TCP with no error. • If the realm RDMA feature is turned off, the mount succeeds and uses TCP. enforce: • If RDMA is available, RDMA is used. • If RDMA is unavailable, the mount fails. • If the realm RDMA feature is turned off, the mount fails. For more information, see the third point in Known Limitations.

Examples:

mount -t panfs -o rdma-mode=auto panfs://<director_ip>/
<volume> /mnt/<volume>

mount -t panfs -o rdma-mode=enforce,rdma-dev=mlx5_0 panfs://
<director_ip>/<volume> /mnt/<volume>

mount -t panfs -o rdma-mode=auto panfs://<director_ip>/ /mnt/
panfs

12.4.2 Error Messages

You may see several error messages if RDMA is incorrectly configured.

• If there is an invalid mode value:

  mount.panfs: invalid rdma-mode 'test'; valid values are
  'auto', 'off', or 'enforce'

• If enforce is used with no usable RDMA:

  mount.panfs: Invalid argument: inspect options and try again.
  /var/log/messages may have more details about the mount
  failure.

See /var/log/messages or dmesg for detailed information:

pan_rdma_device_init: no RDMA-capable device found on any
network interface
RDMA: init failed, mode=enforce, aborting, 0x7 (Invalid
argument)

• If a mount inherits settings from an earlier PanFS mount, mount.panfs prints a warning beginning with:

  warning: inheriting the rdma-mode=… setting from existing
  PanFS mount; per-mount RDMA options are not supported

For more information, see the first point in Known Limitations.

12.4.3 Known Limitations

Be aware that there are currently several limitations to this implementation.

• RDMA settings are per client, not per mount. The first PanFS mount on a client fixes rdma-mode and rdma-dev for the whole client. Every later mount inherits them and silently ignores its own values. To change the setting, unmount every PanFS file system on that client first.
• Volumes with a stripe unit smaller than 64 KiB fall back to TCP for data I/O, even with rdma-mode=enforce.
• Currently, rdma-mode=enforce succeeds even when the realm RDMA feature is off, and subsequent writes hang. Ideally, it should fail the mount.
• Automatic device selection is unreliable on some hardware realms; rdma-mode=enforce fails unless rdma-dev is used.
• Currently, mconsole_feature treats anything that is not on as off. For example, trying to use mconsole_feature RDMA enable results in a “Turned RDMA off.” message instead of reporting an invalid value.
• Toggling the RDMA feature interrupts the file system. Operations issued immediately afterwards can fail. For example volume creation may fail resulting in an “rmm: the realm root volume is not online” error.