AMD Validated Reference Architecture: Data Storage for GPU-Accelerated AI

Learn More

Velocity • Durability

Book a Briefing
Contact Sales
Book a Briefing
Book A Demo
Contact Sales
Contact Sales
VDURA — S3
AI SOLUTIONS / S3

An S3 object is a file in the volume. Not a copy of one.

POSIX and S3 are two doors into the same store. A file written through DirectFlow is readable through S3, and the reverse. One namespace, one set of bytes, no sync jobs and no second silo to buy, power, and reconcile.

✓ AWS-standard S3 API ✓ Scales with Directors ✓ Per-tenant credentials
TWO DOORS · ONE SET OF BYTES
TWO SILOS, ONE COPY JOB Object store the data lake copy job Training tier second copy, second bill Pay twice for capacity. Stale data between runs. ONE VOLUME, EVERY DOOR S3 ingest NFS · SMB DirectFlow Gateway services run on Directors direct parallel path · no gateway One global namespace NVMe flash scale-out HDD capacity tier ≠ protocol Data arrives once and lands once. The object an ingest pipeline writes is the file the training job reads. Scale object throughput by adding Directors, independent of how much capacity sits behind them.
WHY THE TWO-SILO STACK PERSISTS

Nobody wanted two storage systems. They wanted object economics and file performance.

The lake holds the data and the training tier holds the speed, so a copy job runs between them forever. That copy is capacity you buy twice, a window where the two sides disagree, and a pipeline step that can fail at three in the morning.

THE COPY TAX
Capacity you buy twice
A dataset in the lake and the same dataset staged on the training tier are two bills for one set of bytes, plus the power and floor space under both.
VDURA: The S3 object is the file. Data arrives once and lands once, so there is no staging copy and no second capacity pool to fund.
THE SYNC TAX
A pipeline step that can fail
Copy jobs are the part of the pipeline nobody owns. They run long, they run stale, and they fail on the night before a deadline.
VDURA: Ingest over S3 and train against the same volume through DirectFlow. There is nothing to sync, so training starts the moment the data lands.
THE GATEWAY TAX
Object throughput that will not grow
When object access is a bolted-on front end, throughput caps at the heads. Adding capacity behind them does not widen the neck.
VDURA: The S3 service runs on Directors. Add Directors to grow object throughput and requests per second, independent of storage capacity.
HOW S3 WORKS HERE

The object service runs on the Directors, against the global namespace.

One namespace, every protocol
✓ The S3 service runs on the Directors against the global namespace
✓ A file written via POSIX is readable via S3, and the reverse
✓ DirectFlow, NFS, SMB, and S3 all address the same volume
✓ Tier is not protocol: an object can live on flash or on HDD and stay reachable either way
Compatible with what you already run
✓ AWS-standard S3 API: boto3, Go, and TypeScript SDKs work unmodified
✓ Open table formats including Delta Lake and Apache Iceberg
✓ Dataset versioning through DVC, Git LFS, and lakeFS
✓ No proprietary client required for object access
Scales on its own axis
✓ Add Directors to grow object throughput and requests per second
✓ Object performance scaling is independent of storage capacity
✓ High-concurrency sequential ingest without degrading concurrent training I/O
✓ Point S3 at a higher performance tier when a workload needs it, same namespace and same mechanism
Tenant-ready
✓ Per-tenant S3 credentials and quotas, scoped per volume
✓ Usage metered hourly per volume, feeding your billing directly
✓ Objects tier to HDD under the same placement policies as file data
✓ TLS-only endpoints, SigV4-only authentication
✓ Tenant-facing S3 endpoints against the same namespace
WHAT THE OBJECT TIER DELIVERS

Meets the AI cloud object spec, and can go faster if the workload needs it.

Object performance here is a function of which tier you point S3 at and how many Directors you run. Both are decisions you make, not limits the architecture imposes.

PROVISIONED PER TiB
Throughput you choose
Object throughput and request rate are per-TiB ratings you provision, sized against what AI cloud object specs actually require.
SCALES ON ITS OWN AXIS
Add Directors
Object throughput and requests per second grow with Directors, independent of how much capacity sits behind them.
HOT-TIER LATENCY
Milliseconds
On the hot tier, so object access is viable for interactive work rather than batch only.
NEED MORE
Point it higher
Aim S3 at a higher performance tier and object inherits it. Same namespace, same mechanism, no migration.
WHERE TEAMS POINT S3

Four jobs, one namespace behind all of them.

Dataset ingest and lakes
Land raw data over S3 at high concurrency. It becomes the file every later stage reads, with no copy into a separate training silo.
Arrives once, lands once
Checkpoint archive
Age checkpoint history off flash to scale-out HDD under policy, while every checkpoint stays online and readable through any protocol.
Still online, still reachable
Model registry
Artifacts and model versions in object storage that the training fleet can also mount as files, with DVC and lakeFS working normally on top.
One artifact, two doors
Tenant-facing endpoints
Hand each tenant their own S3 endpoint with scoped credentials and quotas, metered hourly, against the pool you already run.
A second product, same fleet
boto3 Go SDK TypeScript SDK Delta Lake Apache Iceberg DVC Git LFS lakeFS
WHAT THIS MEANS FOR THE BUSINESS

One capacity pool, priced as more than one product.

Sell The Data Lake, Too
Tenants land data over S3, train against it as files, and archive it in place. One capacity pool, one namespace, priced as more than one product.
Stop Funding The Copy
Removing the staging tier removes its capacity, its power, and the operational time spent nursing copy jobs that nobody wants to own.
Object Economics Without Object Limits
Cold bulk settles to HDD under policy while hot records stay on flash, so the price per terabyte follows the access pattern rather than the protocol.
No Migration To Get There
Full S3 API compatibility means existing pipelines point at a new endpoint and keep working. The SDKs your teams already use do not change.
Scale The Part That Is Busy
Directors grow object throughput, storage nodes grow capacity. You buy the axis that is actually constrained instead of a whole new system.
Billing That Matches Usage
Hourly metering per volume means the invoice reflects what a tenant consumed, whether they reached the data as objects or as files.

See how one namespace serves both objects and files, and size it for your workload.