TPU Porting Reveals PyTorch's Hidden Abstractions Fail...

When developer tucan9389 attempted to port the nanochat conversational AI model from PyTorch to a Google TPU using JAX, the result was a stark lesson in hardware-software mismatch. While the model architecture (a small transformer) transferred conceptually, nearly every PyTorch idiom—from dynamic control flow to gradient handling—broke under JAX's functional constraints.

Porting nanochat from PyTorch to JAX for TPU broke due to JAX's functional constraints: no mutable state, no dynamic control flow, and no straightforward gradient tape.
High-level transformer architecture carried over, but low-level operations like attention masking and tokenization required complete rewrites.
The effort highlights a critical gap: PyTorch's ease of use does not extend to TPU hardware, creating a barrier for developers exploring Google's accelerators.

Why Did PyTorch's Dynamic Execution Fail on TPU?

According to the developer's discussion on GitHub (July 2026), the core issue was JAX's requirement for pure functions. PyTorch's nn.Module relies on mutable state and dynamic computation graphs, which are incompatible with JAX's functional programming model. For example, the nanochat model's attention mechanism used PyTorch's `masked_fill` with a dynamically computed mask; in JAX, this required converting to a static mask via `jax.lax.select` or restructuring the entire forward pass. The developer reported that "every line of PyTorch code that depended on runtime state had to be rewritten as a pure function."

What Specific Operations Broke During the Port?

The most painful breakage occurred in tokenization and padding. PyTorch's `DataLoader` with dynamic padding (using `collate_fn`) is trivial; JAX/TPU requires fixed-size tensors because TPU compilation is static. The developer had to pad all sequences to a maximum length before compilation, wasting memory on shorter sequences. Additionally, PyTorch's `nn.Dropout` with `training=True/False` broke because JAX expects dropout to be a separate function with an explicit `rng` key. According to the TensorFlow documentation on transformer models, TPU inference typically uses deterministic attention, but nanochat's original PyTorch code used stochastic dropout, requiring a complete removal of dropout layers for inference portability.

TPU Porting Reveals PyTorchs Hidden Abstractions Fail on JAX

How Does the Developer's Experience Compare to Official Guidance?

Google's official JAX documentation emphasizes that "JAX is not a drop-in replacement for PyTorch," but the nanochat porting experience reveals just how far the gap extends. The developer noted that while Google provides `flax` and `optax` libraries, they lack the maturity of PyTorch's ecosystem—for instance, there is no direct equivalent to `torch.nn.TransformerEncoder` in Flax that handles masking automatically. The developer had to manually implement attention masking using `jax.numpy.where` and `jax.lax.broadcast_in_dim`, which the developer described as "10x more verbose than the PyTorch version."

What Are the Performance Implications for TPU Inference?

After the port, the developer reported that TPU inference was faster for batch sizes above 32 (approximately 2x throughput vs. a V100 GPU) but slower for single-sample inference due to compilation overhead. The TPU's static graph compilation took 15-20 seconds for the nanochat model, making interactive use cases impractical. This aligns with Google's own benchmarks, which show TPUs excel at large-batch training but struggle with latency-sensitive inference. The developer concluded that "for a chatbot that needs to respond in real-time, TPU is not the right choice unless you can batch requests."

Comparison: PyTorch (GPU) vs. JAX (TPU) for nanochat

Dimension	PyTorch (GPU)	JAX (TPU)
Programming Model	Imperative, dynamic graph	Functional, static graph
Memory Management	Automatic, dynamic	Manual pre-allocation required
Attention Masking	Dynamic mask via masked_fill	Static mask via lax.select
Dropout Implementation	Built-in, stateful	Explicit rng key, stateless
Tokenization	Dynamic padding in DataLoader	Fixed-length padding before compilation
Inference Latency (single sample)	~50ms	~2s (including compilation)
Ecosystem Maturity	Mature (Hugging Face, etc.)	Emerging (Flax, Optax)
Verdict	Winner for flexibility and ease of use	Winner for large-batch throughput

My thesis: The nanochat porting effort is a canary in the coal mine for TPU adoption outside of Google's internal workloads. In the short term, developers will continue to choose GPUs for conversational AI because PyTorch's dynamic model maps naturally to the unpredictable nature of chat interactions. Google's TPU v5e and v5p are optimized for large-batch training of models like Gemini, not for serving small, interactive models. The long-term consequence is that TPU adoption will remain confined to hyperscale training clusters and a few specialized inference scenarios (e.g., batch processing of long documents). The loser here is Google's TPU ecosystem, which risks becoming a niche product for large-scale training only. The winner is NVIDIA, whose GPUs offer a seamless path from development to deployment. However, if Google invests in JAX-native inference serving tools (e.g., a JIT compiler that caches compiled graphs), TPUs could become viable for interactive use cases within 2-3 years.

Google will release a JAX-native inference server with graph caching by Q2 2027 to address the compilation overhead issue, but it will initially only support static-sized inputs.
NVIDIA will maintain its dominance in conversational AI inference for models under 7B parameters through 2028, as TPU latency remains prohibitive for real-time chat.
The Hugging Face ecosystem will not add first-class TPU support for inference before 2029, leaving TPU porting as a manual effort for developers.

July 2026
nanochat porting discussion published
Developer tucan9389 details the challenges of porting nanochat from PyTorch to JAX for TPU, highlighting breakages in attention, dropout, and tokenization.
August 2026
Google TPU v5e announcement
Google announces TPU v5e with improved inference performance, but still focused on large-batch workloads.
September 2026
Hugging Face JAX backend (experimental)
Hugging Face releases an experimental JAX backend for transformers, but it is limited to training, not inference.

July 2026: Developer tucan9389 publishes porting experience on GitHub, detailing breakages.
August 2026: Google announces TPU v5e with improved inference performance.
September 2026: Hugging Face releases experimental JAX backend for transformers (limited to training).

Insight 1: PyTorch's dynamic execution is not a bug—it's a feature that TPU's static graph cannot replicate without major rewrites.
Insight 2: The porting effort reveals that TPU's strength (throughput) is its weakness (latency); for interactive AI, GPUs remain superior.
Insight 3: Google's TPU strategy depends on developers adopting JAX, but the ecosystem gap vs. PyTorch is widening, not closing.
Insight 4: The nanochat case is a template for what other developers will face: porting costs often outweigh TPU performance gains for small models.
Insight 5: Until Google provides a PyTorch-to-JAX transpiler or a TPU-compatible PyTorch backend, TPU adoption for inference will remain a niche effort.