Fault tolerant distributed training on Amazon EKS using NVRx

A technical guide to using NVIDIA's Resiliency Extension with PyTorch FSDP on Amazon EKS, covering async checkpointing and fast GPU fault recovery with H100 benchmarks.

By AIWI EditorialReviewed by Chayanika DivekarAI-assisted summary · human-reviewed1 min readPublished Sep 16, 2026

Single source1 source

Reality Check

Availability: Shipping now
Hype-o-meter: Mostly substance
Who it's for: ML Engineers · Enterprises
Do this now: Review the integration steps for adding NVRx to your PyTorch FSDP training on Amazon EKS.
Claims check: Benchmark claims are attributed to H100 runs at 2-8 nodes in the post.
Corroboration: Single source · 1 source

Key Takeaways

NVRx integrates with PyTorch FSDP training on Amazon EKS
Async checkpointing overlaps checkpoint I/O with training
In-process and ft_launcher in-job restart enable second-scale recovery
H100 benchmarks at 2-8 nodes show 99%+ training efficiency

The AWS Machine Learning Blog describes how to integrate the NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training running on Amazon EKS. The approach aims to make distributed training fault tolerant by overlapping checkpoint I/O with training and recovering from GPU faults in seconds.

The post covers several mechanisms: asynchronous checkpointing, in-process restart, and in-job restart via ft_launcher. It also presents H100 benchmarks across 2 to 8 nodes, reporting more than 99% training efficiency and recovery times on the order of seconds.

Why it matters

GPU faults during large distributed training runs can waste compute and time. Techniques that overlap checkpointing with training and restart quickly help maintain high training efficiency, as indicated by the reported benchmarks.

Who should care

Engineers and teams running distributed PyTorch FSDP workloads on Amazon EKS with NVIDIA GPUs, and those focused on training resiliency and infrastructure efficiency.

Was this helpful?

Sources & Further Reading

Fault tolerant distributed training on Amazon EKS using NVRx — AWS Machine Learning Blog · Sep 16, 2026

Frequently Asked Questions

What efficiency do the benchmarks report?

The post reports more than 99% training efficiency with second-scale recovery on H100 across 2 to 8 nodes.

Which recovery mechanisms are covered?

Asynchronous checkpointing, in-process restart, and ft_launcher in-job restart.

Share GPU clusters across teams with isolation and fairness using Amazon SageMaker HyperPod — Oct 8, 2026
Manage Amazon SageMaker HyperPod Spaces directly from SageMaker Studio — Oct 6, 2026
Scaling MoE reinforcement learning on Amazon EKS with EFA and DeepEP with 40% more throughput — Sep 25, 2026
Accelerate multimodal RL training with SkyRL on Amazon SageMaker HyperPod — Sep 25, 2026

#training #gpu #cloud #mlops #benchmarks