AMD Validated Reference Architecture: Data Storage for GPU-Accelerated AI

Learn More

Velocity • Durability

Book a Briefing
Contact Sales
Book a Briefing
Book A Demo
Contact Sales
Contact Sales
AI SOLUTIONS ·
PERSISTENT CONTEXT

Persistent context: any GPU node can recall any context instantly.

Within the node, across the east/west network, or via north/south tiers. Context that outlives the pod that created it, at the cost of flash rather than the cost of recomputation.

✓ Context survives restarts ✓ Persistence with an SLA ✓ RDMA data paths
WHERE CONTEXT CAN LIVE
GPU HBM GIGABYTES · NANOSECONDS
The only place the model reads from directly. Fast, tiny, and the first thing to run out.
Host memory HUNDREDS OF GB · MICROSECONDS
Buys you one order of magnitude. Still capped per node, and gone when the pod restarts.
Node-local SSD TERABYTES · IN THE NODE
Flash you already bought, sitting in the GPU node. DirectFlow extends context onto it rather than leaving it stranded.
VDURA flash tier PETABYTES · PERSISTENT
Read over RDMA in parallel from every node. Survives eviction, pod restarts, and rescheduling.
Recompute from scratch BILLED AT GPU RATES
The fallback when there is no tier below memory. Quadratic in context length, and it produces nothing new.
Without a persistence tier, the bottom of that hierarchy is recomputation, billed at GPU rates.
THE PREFILL TAX

Every miss is a GPU doing homework it already finished.

Prefill is quadratic in context length. A long document, a long conversation, an agent with history: rebuilding that context is one of the most expensive operations your fleet performs, and it produces nothing new.

The cache does not miss because your model is wrong. It misses because GPU memory is small, sessions outlive pods, and there is nowhere fast enough to put the overflow.

HBM runs out before your context does
Context grows with length and concurrency at the same time. Long-context serving hits the memory ceiling far sooner than the compute ceiling.
Sessions outlive pods
Rescheduling, autoscaling, and node maintenance all evict context. The user comes back to a conversation your infrastructure has forgotten.
Shared prefixes get rebuilt per request
System prompts, tool definitions, and retrieved documents repeat across requests. Without persistence, each one is prefilled again.
Bursty tenants destroy locality
On a shared fleet, one busy tenant evicts everyone else’s context. Hit rate collapses exactly when traffic is highest.
WHAT THIS MEANS FOR THE BUSINESS

Context recall is a financial metric.

Every point of recall is prefill you did not pay for, latency your customer did not feel, and GPU capacity you did not have to buy. It is the rare lever that improves margin and experience at the same time.

Prefill You Do Not Pay For
A cache hit replaces a quadratic prefill with a read. That is GPU time returned directly to serving new tokens.
Time to First Token
TTFT is what your customer actually experiences and what your SLA commits to. Reading context beats rebuilding it.
Higher Effective Concurrency
Freeing HBM from cache overflow lets you hold more sessions per GPU without adding accelerators.
Predictable Tail Latency
Hit rate that does not collapse under burst is what makes a latency SLA safe to sign.
Tenant Density Becomes Safe
Per-tenant isolation on the cache tier turns density from a risk into a revenue lever.
Agents Become Affordable
Multi-step reasoning multiplies context reuse. Persistence is what keeps agentic workloads from repricing themselves.
Defer GPU Purchases
Capacity recovered from redundant prefill is capacity you did not have to buy at current accelerator prices.
Cost Per Token Served
All of the above resolve into one number, and it is the one your finance team can act on.
WITH AND WITHOUT A PERSISTENCE TIER
Same model. Same GPUs. Different bill.
The only variable is whether context had to be rebuilt from scratch when the session came back. Longer bars are better on every row.
GPU MEMORY ONLY
First-token speed, long context rebuild from zero
Sessions held per GPU capped by HBM
Hit rate under burst collapses
Capped by HBM. Overflow is dropped, and dropped context means prefill again.
WITH VDURA PERSISTENCE
First-token speed, long context read, not rebuild
Sessions held per GPU cache spills to flash
Hit rate under burst holds
Context spills to a flash tier fast enough to read back. Sessions resume rather than restart.
HOW IT WORKS

Four steps, and the GPU never leaves the fast path.

1 Extend
HBM UNDER PRESSURE
Rather than discarding cold context, DirectFlow extends its buffer into host memory and node-local SSD inside the GPU server.
No new hardware
2 Persist
WRITEBACK WITH AN SLA
KVCache Writeback holds context durably and keeps it addressable across pod restarts, rescheduling, and node maintenance.
Persistence you can hold an SLA against
3 Locate
METADATA AT SCALE
VeLO resolves which context blocks exist and where, at billions of operations per second, so lookup is never the bottleneck.
Lookup in the noise floor
4 Recall
RDMA, PARALLEL READ
The Context Cache Tiering Framework serves blocks back at LMCache speed, in parallel across storage nodes, with the CPU out of the path.
Recall instead of recompute
CONTEXT-AWARE TIERING

Three components extend context beyond GPU memory.

One inside the node, one that makes persistence a commitment rather than a best effort, and one that serves it back at the speed long-context serving and RAG require.

V12 IN THE NODE
Extended DirectFlow Buffer to Local SSD
The DirectFlow client extends its buffer onto SSDs already installed in the GPU server, turning stranded flash into usable context space with no new hardware.
Activates capacity you already own
V12 PERSISTENCE
KVCache Writeback
Context is written back and held durably rather than discarded under memory pressure, so persistence is a commitment you can hold an SLA against, not a best effort.
Persistence you can hold an SLA against
V12 SERVING BACK
Context Cache Tiering Framework
Serves context back at LMCache speed for long-context LLM serving and RAG, placing each block on the right media as access patterns shift.
Fast enough to beat recompute
THE INDUSTRY IS MOVING THIS WAY
Offloading context to NVMe is becoming standard.
NVIDIA is standardizing context offload to NVMe across the serving stack. As that happens, the questions that will separate implementations are economics, isolation, and hardware freedom.
Where implementations will differ
Everyone will support offload. Not everyone will do it at mixed-fleet economics, on open hardware, with per-tenant isolation.
WHY A PARALLEL FILE SYSTEM WINS HERE

A persistence tier is only useful if recalling context beats recomputing it.

That is a latency bar, and most storage cannot clear it. VeLO handles metadata at billions of operations per second, every DirectFlow client reads across storage nodes in parallel over RDMA with the CPU out of the path, and throughput comes from parallel node count rather than flash capacity. Add nodes, get latency headroom, linearly.

2.7 TB/s
per rack, all-flash
12 nines
durability with MLEC
1,500+
nodes, one namespace
6+ nines
availability
Extended DirectFlow Buffer
The client extends context onto SSDs already sitting in the GPU node, so flash you paid for stops being stranded capacity.
KVCache Writeback
Context held durably rather than discarded under memory pressure, so persistence is something you can hold an SLA against.
Context Cache Tiering Framework
Serves context back at LMCache speed for long-context LLM serving and RAG, with placement that follows access patterns.
VeLO Metadata Engine
Flash-optimized distributed metadata built for billions of operations per second, so lookup never becomes the bottleneck.
RDMA Data Paths
Direct GPU-to-storage transfers with the CPU out of the path, on NVIDIA ConnectX or AMD Pensando fabrics.
One Namespace
Context, model weights, RAG corpus, and session archive all live on one platform. No external data movers between them.
WHERE IT PAYS OFF FIRST

The longer the context, the bigger the return.

LONG CONTEXT
Document and codebase assistants
Hundreds of thousands of tokens of context per session. Rebuilding it on every return visit is the single largest avoidable cost.
AGENTIC
Multi-step agents
Planning, tool calls, and reflection each reuse accumulated state. Losing it mid-task means restarting the reasoning chain.
SHARED PREFIX
System prompts and RAG grounding
The same prefix repeats across thousands of requests. Persist it once and every subsequent request reads instead of prefills.
MULTI-TURN
Consumer and support chat
Sessions that pause for hours and resume. Persistence is the difference between resuming and starting over.
CONTEXT IS TENANT DATA
Context is the most sensitive thing you cache.
KV cache holds a verbatim reflection of what your customer typed and what your model retrieved for them. Persisting it means treating it with the same isolation and encryption discipline as any other tenant data.
Keys Per Tenant
AES-256 end to end with KMIP, scoped to the tenant rather than the cluster.
Namespace Isolation
Hard boundaries between customers sharing the same physical cache tier.
Per-Tenant QoS
One bursty tenant cannot evict another tenant’s context or spend their latency budget.
Policy Retention
Context and session history age out on your schedule, not whenever memory happens to fill.
FITS THE SERVING STACK YOU ALREADY RUN

No rewrite. No second software stack.

PROTOCOLS AND INTEGRATIONS
DirectFlow POSIX NFS SMB S3 Kubernetes CSI RDMA REST API Terraform Ansible KMIP
Serving pods get persistent volumes through the native CSI plug-in with no custom scripting. The cache tier is reachable through standard POSIX and S3 paths, so integration is configuration rather than engineering.
HOW IT SHOWS UP IN YOUR RACK
Software-defined on open hardware
Commodity servers with standard Ethernet or InfiniBand. You keep your choice of server, drive, and fabric vendor.
Add flash nodes for latency headroom
Throughput comes from parallel node count, not flash capacity. Scale the cache tier independently of bulk capacity.
Operational in a day, upgraded in place
Deployment is measured in hours. Expansions and updates happen online while you keep serving traffic.

Stop paying twice for the same context.

Tell us how you serve today and we will show you what a persistence tier is worth against your traffic.