# What Did AWS and NVIDIA Actually Ship?

The AWS Machine Learning Blog published a post titled "Fault tolerant distributed training on Amazon EKS using NVRx" on 16 September 2026, describing an integration between the NVIDIA Resiliency Extension and PyTorch's Fully Sharded Data Parallel (FSDP) training path running on Amazon Elastic Kubernetes Service. The post covers three distinct mechanisms: asynchronous checkpointing that overlaps checkpoint I/O with ongoing training, in-process restart that recovers without tearing down the process, and ft_launcher in-job restart that restarts the job inside the existing Kubernetes pod rather than rescheduling it. According to the AWS Machine Learning Blog, the benchmarks were run on NVIDIA H100 instances at 2 to 8 nodes, showing 99%+ training efficiency and second-scale recovery from GPU faults. That is the entire published evidence base. There is no disclosed node count above 8, no cross-generation comparison, and no fault-injection methodology described in the summary material. My reading: the engineering is real and the mechanisms are the right ones. Async checkpointing is the single highest-leverage change any training team can make, because synchronous checkpointing is where most wall-clock time disappears on long runs. But "second-scale recovery" and "99%+ efficiency" are being presented as product properties when they are, at best, properties of a small-cluster configuration under unspecified fault conditions.

# Why Does Sub-10-Node Evidence Not Settle the Question?

Fault tolerance economics scale badly. At 2 to 8 nodes, the probability that any single GPU faults during a training window is low, so the measured efficiency is dominated by checkpoint overhead rather than by recovery events. At hundreds or thousands of GPUs, the mean time between failures collapses, and the system spends a much larger fraction of its life in the recovery path. The AWS Machine Learning Blog reports 99%+ training efficiency at 2 to 8 nodes. That number is plausible for that configuration. It is not evidence for the same number at scale, because the two regimes have different dominant costs. A cluster that never faults will show high efficiency regardless of how good the recovery path is; a cluster that faults constantly will expose every weakness in the restart logic, the checkpoint consistency guarantees, and the collective reformation time. This is not a criticism of the work. It is a criticism of the framing. The post is a technical walkthrough, and walkthroughs are allowed to use small benchmarks. The problem is that "99%+ efficiency" will be lifted out of context by procurement teams and repeated as a cluster-scale claim.

# How Does NVRx-on-EKS Compare to the Alternatives?

The realistic comparison is not against doing nothing — it is against the other ways teams currently buy resilience on AWS. The table below compares the approaches a platform team would actually weigh.

Approach	Recovery granularity	Checkpoint overhead	Operational burden	Evidence quality
NVRx on EKS (this post)	In-process and in-job restart, seconds	Async, overlapped with training	Managed via EKS primitives	2–8 H100 nodes, undisclosed fault injection
Vanilla FSDP + sync checkpointing	Full job reschedule	Blocking, minutes per checkpoint	Team-owned scripts and glue	Widely observed in practice
Third-party orchestration (e.g. managed training platforms)	Job-level restart, platform-dependent	Varies; often async	Vendor contract	Vendor-reported, rarely reproducible
Slurm + custom health checks on EC2	Node-level drain and requeue	Team-implemented	High; bespoke	Strong at scale in some labs, undocumented publicly
Verdict	NVRx-on-EKS wins for teams already standardized on EKS and H100; it loses to bespoke Slurm stacks at the largest scales until AWS publishes evidence above 8 nodes.

# What Does This Change for Platform Teams?

The practical effect is that fault tolerance stops being a bespoke engineering project and becomes a configuration choice on EKS. That is a genuine shift. Teams that previously needed a dedicated reliability engineer to build checkpoint coordination and restart logic can now adopt a documented pattern. NVIDIA's own repository for the resiliency extension documents the library as an open-source component, which matters because it means the integration is not locked to AWS semantics — the same primitives can be ported to other Kubernetes environments. NVIDIA reported the extension as part of its broader reliability tooling effort, and the open-source posture is consistent with that. But there is a real cost: teams adopting this inherit a dependency on both AWS's EKS integration quality and NVIDIA's library release cadence. If the library changes its restart semantics, the EKS pattern changes with it. That is a different risk profile from a bespoke stack the team fully controls.

# Who Gains and Who Loses?

Gains: AWS, because resilience becomes a differentiator it can bundle rather than a feature customers build; NVIDIA, because NVRx adoption deepens its software footprint on top of its hardware position; and mid-size labs, who get a credible resilience story without hiring a distributed-systems specialist. Loses: third-party training-orchestration vendors whose primary pitch is fault tolerance, because a managed EKS pattern compresses their differentiation; and teams on non-H100 accelerators, who have no equivalent published evidence and will be making an unvalidated bet if they assume the same numbers transfer.

# What Should Be Predicted From Here?

1. AWS will publish an expanded NVRx-on-EKS benchmark covering at least 64 nodes, with explicit fault-injection methodology, within two quarters of the September 2026 post.
2. At least one competing hyperscaler — most likely Google Cloud with its GKE training stack — will publish a directly comparable fault-tolerance benchmark for FSDP within three quarters, framing recovery time as the headline metric.
3. Third-party training-orchestration vendors will shift their messaging away from raw fault tolerance and toward multi-cloud portability, because single-cloud resilience is being commoditized by the hyperscalers.

# What Should a Reader Remember?

• The mechanisms AWS and NVIDIA shipped are the right ones; async checkpointing and in-process restart are genuinely high-leverage for long training runs.
• The evidence base is 2 to 8 H100 nodes, and the 99%+ efficiency figure should be read as a small-cluster result, not a system guarantee.
• The strategic move is AWS bundling resilience as a managed EKS pattern, which compresses the differentiation of third-party orchestration vendors.
• Teams on non-H100 hardware are making an unvalidated assumption if they expect the same recovery characteristics to transfer.
• The next decisive data point is a fault-injection benchmark at four-digit GPU counts, and no one has published one yet.