# What did the AWS benchmark actually test, and why does it matter for production inference?

According to the AWS Machine Learning Blog (published September 8, 2026), the benchmark compared two 30B Mixture-of-Experts models — Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B — across G5 (A10G), G6 (L4), G6e (L40S), and G7 (Blackwell) GPU instances on SageMaker AI. The test measured throughput, latency, and cost-per-token under real-time conditions. This is not a synthetic microbenchmark; it's the kind of data engineers need when deciding which instance family to standardize on for serving small MoE models in production. What matters here is that MoE models like these are increasingly the sweet spot for cost-effective inference — they activate only a fraction of their parameters per token. The benchmark shows that hardware choice amplifies or negates that advantage. AWS reported that G7 instances delivered significantly higher throughput and lower latency across both models, translating to a cost-per-token advantage of up to 40% versus G5 instances.

# Why does G7 beat G5 and G6 so decisively on cost-per-token?

The core reason is architectural. G7 instances are powered by NVIDIA Blackwell GPUs, which introduce FP4 precision support and faster HBM3e memory. According to AWS's blog, the G7 instances achieved higher token generation rates per second, especially at larger batch sizes, because Blackwell's memory bandwidth reduces the bottleneck that typically limits autoregressive generation. But the gains aren't uniform. AWS said the cost-per-token improvement was more pronounced for Qwen3-Coder-30B than for Nemotron-3-Nano-30B, likely due to differences in model architecture and how well each exploits FP4 quantization. This is a crucial nuance: not every model will see the same benefit, so teams need to benchmark their own workloads rather than assume G7 is universally better.

# Who should migrate to G7 immediately, and who should stay on G6e?

If your workload is real-time — chat assistants, copilots, or any application where users wait for a response — G7 is the clear winner. The benchmark showed lower latency across the board, and for interactive applications, latency is the metric that directly impacts user experience. AWS's data suggests that the higher hourly cost of G7 instances is more than offset by the reduced time per request, making it the cost-effective choice for latency-sensitive serving. Conversely, if your workload is batch-oriented — offline summarization, data augmentation, or nightly jobs — you may still be better off on G6e. The throughput per dollar at high batch sizes might favor G6e, especially if you can tolerate longer completion times. The AWS blog did not provide a full cost breakdown for batch scenarios, so teams should run their own tests with representative batch sizes before migrating wholesale.

# What are the operational tradeoffs of moving to G7 on SageMaker AI?

Migrating from G5/G6 to G7 is not a simple lift-and-shift. First, you need to ensure your model can exploit FP4 quantization — that may require re-quantizing weights and validating accuracy. Second, SageMaker AI deployment configurations need to be updated to use the new instance types, and you'll need to test for any software compatibility issues with your inference container. Third, availability matters. G7 instances are newer and may have limited capacity in some regions. AWS said the instances are available in select AWS regions, so check your region's instance availability before committing. Finally, pricing is higher per hour, so you'll need to adjust cost monitoring and potentially implement auto-scaling to avoid idle costs.

# How does this benchmark change the cloud inference competitive landscape?

This benchmark is a shot across the bow at other cloud providers. AWS is positioning SageMaker AI as the go-to place for running small MoE models cost-effectively, and G7 gives it a hardware advantage that rivals using older GPUs (like A100 or H100) may not match. According to AWS's blog, the price-performance gains are "measurable," and that's a direct message to enterprises evaluating multi-cloud AI strategies. The loser here is any provider that hasn't yet deployed Blackwell at scale. For example, if a competitor offers only A100-based instances, they'll struggle to match G7's cost-per-token for these workloads. This could accelerate the migration of inference workloads to AWS, especially for startups where every millisecond and cent counts.

Metric	G5 (A10G)	G6e (L40S)	G7 (Blackwell)
Throughput (tokens/sec)	Baseline	~1.3x (estimated)	~1.7x (estimated)
Latency (ms/token)	Baseline	~20% lower (estimated)	~40% lower (estimated)
Cost per token	Baseline	~15% lower (estimated)	~40% lower (estimated)
FP4 support	No	No	Yes
Best for	Legacy, low-cost batch	Batch and moderate latency	Real-time, latency-sensitive
Verdict	Skip for new workloads	Still viable for batch	Winner for real-time inference

1. By March 2027, AWS will report that G7 instances account for over 50% of new SageMaker AI inference deployments for models under 40B parameters. 2. NVIDIA will release a similar benchmark for its DGX Cloud platform, showing comparable gains, within six months. 3. At least two major AI startups will publicly announce migrating their production inference from G5 to G7 by Q2 2027, citing cost savings.

• G7's cost advantage is real but not uniform — model architecture and batch size matter more than the hype suggests.
• FP4 support is the key technical enabler, but teams must validate accuracy after quantization.
• Real-time workloads should migrate now; batch workloads should wait for more data.
• This benchmark is a competitive signal for multi-cloud strategies — G7 raises the bar for inference cost efficiency.
• Don't assume G7 is universally better; run your own benchmarks with your own models and traffic patterns.