AMD Validated Reference Architecture: Data Storage for GPU-Accelerated AI

Learn More

Velocity • Durability

Book a Briefing
Contact Sales
Book a Briefing
Book A Demo
Contact Sales
Contact Sales
VDURA — RDMA
AI SOLUTIONS / RDMA

Take the CPU out of the data path, and give those cores back to the model.

GPU-to-storage transfers move by direct memory access instead of through the host CPU, so the cores and DRAM you bought for AI stay available to AI.

✓ Runs on all V5000 systems ✓ InfiniBand or Ethernet ✓ Zero dedicated CPU cores
TWO WAYS TO MOVE THE SAME BYTES
CONVENTIONAL PATH Storage node NVMe flash Host CPU + host DRAM copy GPU memory waits its turn Every byte is handled twice. Cores and DRAM are consumed to move data, not to train. RDMA PATH V5000 storage node NVMe flash · VPOD 400 GbE or NDR IB direct memory access GPU server DirectFlow client zero dedicated cores CPU · out of the data path
Same bytes, one less place to queue. Direct memory access between GPU server nodes and the VDURA Data Platform removes the CPU bottleneck on the paths that matter most in training and inference.
WHY THE HOST CPU IS A STORAGE PROBLEM

The slowest part of a 400 GbE path is often the server in the middle of it.

Fabric bandwidth has outrun what a host CPU can shepherd. When every byte is copied through host DRAM on its way to the accelerator, the CPU becomes the governor on a storage tier that is otherwise fast enough.

THE HOST TAX
Every byte handled twice
On a conventional path, data lands in host DRAM before it reaches the accelerator. That copy costs memory bandwidth, cache pressure, and interrupt handling on cores you bought to feed GPUs.
VDURA: RDMA moves data by direct memory access between GPU server nodes and the platform, eliminating the CPU bottleneck rather than scheduling around it.
THE CLIENT TAX
Peak numbers that cost you cores
Some architectures only reach their headline throughput in kernel-bypass mode, which means pinned HugePages and cores permanently reserved for storage on every node in the fleet.
VDURA: DirectFlow uses roughly 191 MB of DRAM per compute node, zero dedicated cores, and the standard Linux page cache. Cycles are borrowed during active I/O and returned immediately.
THE GATEWAY TAX
A funnel in front of the flash
Aggregate throughput caps at the protocol heads regardless of how much capacity sits behind them. Adding drives does not widen the neck.
VDURA: DirectFlow clients talk straight to storage nodes in parallel, with Director Nodes coordinating metadata out of band. RDMA is the transport on that direct path, not a shortcut around a gateway.
WHAT THE STORAGE CLIENT COSTS YOU · 500-NODE GPU CLUSTER
Fleet-wide, per the V12 white paper

Fast is not free if the client takes the hardware with it. Architectures that need kernel-bypass modes to hit peak numbers pay for them in pinned memory and permanently reserved cores.

DRAM committed to storage clients, VDURA ~93 GB
Roughly 191 MB per compute node, using the standard Linux page cache.
DRAM committed, kernel-bypass architectures 2.5 TB
Pinned HugePages reserved whether or not I/O is in flight.
CPU cores dedicated to storage, VDURA 0
Cycles are borrowed opportunistically and returned to the application.
CPU cores locked, kernel-bypass architectures 500 to 2,000
Permanently unavailable to the AI applications on the same fleet.
Every core and every gigabyte the storage client does not claim stays available to the applications. RDMA extends that principle to the wire itself.
HOW IT WORKS

A parallel client, an RDMA transport, and no gateway anywhere in the path.

RDMA is not a bolt-on accelerator sitting in front of a slow file system. It is the transport under DirectFlow, the POSIX parallel client that already talks straight to storage nodes with the control plane out of band.

STEP 1 · CLIENT
DirectFlow mounts the namespace
A POSIX-compliant kernel module on x86 and Arm, deployed fleet-wide with Ansible or Puppet. Full POSIX semantics and cache coherence, mounted like any file system.
STEP 2 · CONTROL PLANE
Directors answer out of band
VeLO on the Director Nodes resolves metadata and file layout separately from the data plane, so coordination traffic never competes with payload traffic.
STEP 3 · TRANSPORT
RDMA carries the payload
Direct memory access between GPU server nodes and the platform over InfiniBand or Ethernet, with the host CPU out of the data path.
STEP 4 · PARALLELISM
Files stripe wide
Clients read and write across storage nodes simultaneously with client-side network erasure coding. Throughput scales with node count, not with gateway count.
FABRIC AND SILICON

Your fabric, your NICs, your accelerators.

RDMA runs on the network you already chose. V5000 storage nodes support InfiniBand NDR and NDR200 as well as 400, 200, and 100 GbE Ethernet, with two high-speed network slots per node and LACP link aggregation by default.

✓ InfiniBand NDR and NDR200, or Ethernet at 400, 200, and 100 GbE
✓ Two high-speed network slots per V5000 storage node, 2 × 400 GbE by default with LACP
✓ Static LAG, single link, and failover modes are all supported configurations
✓ Validated on AMD Pensando Pollara 400G and NVIDIA ConnectX-7 adapters, with AMD EPYC Turin processors
✓ Hardware-agnostic: commodity components throughout, no proprietary enclosure required
THREE FABRICS, ONE CLUSTER
Front-end and storage WHERE VDURA LIVES
Ingest, compute-to-storage traffic, in-band management, and user access. This is the fabric the V5000 attaches to and the one RDMA runs on.
Back-end scale-out GPU COLLECTIVES
RoCEv2 or UEC-ready RDMA carrying GPU-to-GPU collective traffic. Physically separate from the storage fabric by design.
Accelerator scale-up INSIDE THE NODE
The high-bandwidth interconnect between accelerators on and between baseboards. Storage never touches it.
Front-end and back-end networks stay physically separated, so storage traffic never contends with GPU collectives.
WHERE IT PAYS OFF

Different stages, same win: fewer hands on the data.

INGEST AND PREP
One copy, not three
Land data over S3 and train against it as files in the same volume. RDMA carries the transform traffic without a detour through host memory.
Zero copies between stages
MODEL LOAD
Cold start gets shorter
Multi-terabyte model files move into accelerator memory over a direct path, so the cluster reaches its first step sooner.
Faster time to first step
TRAINING
Ranks read in parallel
Every rank pulls its shard from storage nodes simultaneously over RDMA. The data loader stops being the thing the GPUs wait on.
Sustained parallel reads
CHECKPOINT
Burst writes land on flash
Model and optimizer state flush in parallel across flash nodes. A shorter pause per checkpoint means you can afford to checkpoint more often.
Minutes, not hours
WHAT THIS MEANS FOR THE BUSINESS

A cheaper path per byte is a cheaper token.

Cores Back To The Workload
Every core the storage client does not lock is a core running your pipeline. At fleet scale that is hundreds of cores returned to revenue work.
Utilization Per Rack
A shorter path means more GPU-hours converted to training progress rather than spent waiting on data. Utilization is the P&L.
Performance Per Watt
Moving bytes without a host copy is work you no longer power or cool. In a power-capped facility that headroom goes to accelerators.
No Forklift Required
RDMA runs on all V5000 systems, over the InfiniBand or Ethernet fabric you already have. It is a software capability, not a new appliance.
Open Silicon
Validated on both AMD and NVIDIA networking with commodity servers underneath, so your NIC and server vendor choices stay yours.
Standard Protocols Stay
NFS, SMB, S3, and Kubernetes CSI are always there. RDMA and DirectFlow are the accelerated path when a workload needs everything.

See what RDMA does for your cluster.