CODA Rewrites Transformer Kernels: FlashAttention Obsolete?

A new paper from researchers at MIT and Stanford proposes CODA, a system that rewrites entire transformer blocks as single GEMM-epilogue programs. If validated, this could eliminate the need for specialized kernels like FlashAttention, potentially reshaping the economics of AI inference.

CODA rewrites transformer blocks as single GEMM-epilogue programs, eliminating the need for separate attention and feed-forward kernels.
Initial benchmarks show CODA matching or exceeding FlashAttention on A100 GPUs for common transformer configurations.
The key tension: CODA's abstraction requires compiler support that may not align with NVIDIA's proprietary software stack.

How Does CODA Actually Rewrite a Transformer Block?

According to the CODA paper published on arXiv, the core insight is to treat the entire transformer block—attention, feed-forward, layer norms, and residual connections—as a single matrix multiplication followed by a customized epilogue program. The authors, led by researchers from MIT CSAIL and Stanford, demonstrate that by carefully partitioning the GEMM and fusing element-wise operations into the epilogue, they can eliminate the overhead of launching multiple kernels and transferring intermediate tensors between global memory and registers. The paper reports that for a 6-layer transformer with 512 hidden dimensions, CODA achieves 1.2x speedup over a baseline using separate FlashAttention and MLP kernels on an NVIDIA A100.

Does CODA Really Match FlashAttention Performance?

CODA Rewrites Transformer Kernels: FlashAttention Obsolete?

What Does This Mean for AI Infrastructure Engineers?

According to the paper's authors, the primary benefit is reduced engineering complexity. Instead of hand-tuning separate kernels for each new architecture variant, developers could express the entire block in a high-level DSL that compiles to the GEMM-epilogue pattern. This is particularly relevant for the growing number of transformer variants—like Mamba, RWKV, and hybrid models—that deviate from the standard attention pattern. The authors claim that CODA's abstraction can automatically generate efficient kernels for these variants, a task that currently requires weeks of expert CUDA programming.

Who Wins and Who Loses If CODA Gains Traction?

Metric	CODA (GEMM-Epilogue)	FlashAttention (Hand-Tuned CUDA)
Engineering effort per new architecture	Days (DSL-level changes)	Weeks (custom kernel development)
Peak throughput (A100, 6-layer, 512-dim)	1.2x baseline	Baseline
Compiler dependency	High (requires GEMM-epilogue support)	Low (direct CUDA)
Portability across GPU vendors	Potentially high (if DSL is vendor-neutral)	Low (NVIDIA-specific)
Verdict	Wins on flexibility and portability potential	Wins on maturity and peak optimization

What Are the Remaining Uncertainties?

The paper's benchmarks are limited to small models (up to 12 layers, 768 hidden dimensions) and a single GPU architecture (A100). According to a discussion on Hacker News, several commenters questioned whether the approach scales to large language models with hundreds of layers and billions of parameters. The authors acknowledge that memory bandwidth for the epilogue program could become a bottleneck for very deep models. Additionally, the paper does not evaluate training performance, only inference. The key open question is whether the compiler optimization required for CODA can match the hand-tuned peak performance of libraries like FlashAttention on newer architectures like H100 or B200.

My thesis: CODA is a genuinely promising abstraction that could democratize transformer kernel development, but its adoption depends on NVIDIA's willingness to support the required compiler primitives, which is far from guaranteed. In the short term, the paper provides a compelling research prototype, but production deployment will require either NVIDIA to extend its compiler stack or a third-party compiler like Triton to mature enough to handle the epilogue fusion automatically. The winners are startups building custom transformer architectures who currently struggle with kernel engineering; the losers are companies like NVIDIA if the abstraction reduces lock-in to their proprietary software stack. I predict that within 18 months, at least one major inference framework (e.g., vLLM or TensorRT-LLM) will announce experimental support for a CODA-like abstraction, but it will remain niche until the compiler tooling matures.

By Q1 2028, at least one open-source inference engine will integrate a CODA-inspired kernel generator, citing the paper as direct inspiration.
NVIDIA will not natively adopt CODA's abstraction in its proprietary compiler stack, instead offering a limited version only through CUDA 13's epilogue API.
Within 24 months, a startup will raise a seed round specifically to commercialize CODA-like compiler technology for non-NVIDIA hardware (AMD, Intel, or custom ASICs).

May 2026
CODA paper published on arXiv
MIT and Stanford researchers release the CODA paper proposing GEMM-epilogue transformer blocks.
May 2026
Hacker News discussion
The paper gains traction on Hacker News, with debate about scalability and compiler support.
Q1 2028 (predicted)
First open-source integration
Predicted integration of CODA-like abstraction in an inference framework.

CODA's key innovation is fusing an entire transformer block into a single GEMM with an epilogue, eliminating multi-kernel overhead.
The approach trades peak theoretical performance for engineering simplicity and portability, a trade-off that may not suit hyperscalers optimizing for the last 5%.
Adoption hinges on compiler ecosystem support, not just algorithmic merit—NVIDIA's incentives may block or limit the abstraction.
The paper's limited scale (small models, single GPU) leaves open questions about applicability to frontier LLMs.
This is the most practical attempt so far to unify transformer kernels, but it is not yet production-ready.