Tensor Similarity: New Metric Proves Networks Are Identical

For years, mechanistic interpretability researchers have faced a fundamental catch-22: they cannot trust their own methods to verify that two neural network circuits are the same. A new paper on arXiv introduces a weight-based metric, tensor similarity, that is mathematically guaranteed to solve this problem for tensor-based models, directly challenging the dominance of empirical measures like CKA and SVCCA.

Researchers on arXiv introduced tensor similarity, a weight-based metric invariant to weight-space symmetries, solving a core verification problem in mechanistic interpretability.
Existing methods like CKA and SVCCA are either empirical (blind out-of-distribution) or basis-dependent (ignoring symmetries), making them unreliable for proving circuits are the same.
Tensor similarity is mathematically proven for tensor-based models, enabling confident identification of reused components, a critical step toward modular AI.

What Makes Tensor Similarity Different From CKA and SVCCA?

According to the authors of the paper "When Are Two Networks the Same?" published on arXiv on May 14, 2026, existing similarity measures fall into two flawed camps. The first camp, including Centered Kernel Alignment (CKA) and Singular Vector Canonical Correlation Analysis (SVCCA), evaluates empirical behavior on a fixed dataset. The second camp, including direct weight comparisons, examines basis-dependent parameters. The problem, the authors argue, is that empirical measures are blind to out-of-distribution mechanisms, while parameter-based measures ignore weight-space symmetries like permutation or scaling. Tensor similarity, by contrast, is a weight-based metric that is invariant to such symmetries, meaning it can prove two subcomponents compute the same function regardless of how their weights are arranged.

Why Is This a Breakthrough for Mechanistic Interpretability?

Tensor Similarity Kills CKA: New Metric Proves Networks Are Identical

Mechanistic interpretability aims to reverse-engineer neural networks into meaningful parts, or circuits. As the authors state, "verifying that two such parts implement the same computation is a prerequisite" for any progress in this field. Without a reliable metric, researchers cannot confidently say that a circuit found in one model is the same as one in another, or even that the same circuit appears in two different runs of the same model. Tensor similarity provides that guarantee for tensor-based models, which include most modern transformer architectures. This is not a marginal improvement; it is a necessary condition for building a library of reusable, verified neural circuits.

Who Loses When Tensor Similarity Becomes Standard?

Metric	Type	Invariant to Symmetries?	Works OOD?	Verdict
CKA (Kornblith et al., 2019)	Empirical (activations)	No	No	Fails on core requirement
SVCCA (Raghu et al., 2017)	Empirical (activations)	No	No	Fails on core requirement
Direct Weight Comparison	Parameter-based	No	Yes	Fails on symmetry invariance
Tensor Similarity (This paper)	Weight-based, invariant	Yes	Yes	Winner for tensor models

The immediate losers are the empirical methods CKA and SVCCA, which have been widely used in the interpretability community. According to a 2019 paper by Kornblith et al., CKA was introduced to measure similarity between neural network representations. However, as the arXiv authors point out, such empirical methods are fundamentally limited because they depend on a specific dataset. If two circuits behave identically on training data but diverge on adversarial or out-of-distribution inputs, CKA will falsely report them as similar. Tensor similarity, being weight-based and invariant, does not suffer from this blind spot. Researchers who have built workflows around CKA will need to adapt or risk publishing results that are not robust.

What Are the Limitations of Tensor Similarity?

The paper explicitly limits its claims to "the class of tensor-based models." This includes most modern transformers and convolutional networks, but does not include recurrent networks or models with complex control flow. Additionally, the metric is weight-based, meaning it requires access to the full weight matrices of the models being compared. This is not a problem for open-source models, but it means that for proprietary models accessed only via API, tensor similarity cannot be applied. The authors also note that the metric is invariant to symmetries but does not address computational equivalence under different architectures—two models with different layer counts or attention heads cannot be directly compared using this method.

Will This Accelerate Modular AI Development?

Yes, but with caveats. The ability to prove two circuits are identical is a prerequisite for building a library of verified neural components. Companies like Anthropic, which has invested heavily in interpretability, could use tensor similarity to catalog circuits across their model families. However, the metric does not solve the harder problem of automatically discovering circuits in the first place. It only provides a verification tool. The authors of the arXiv paper have provided a mathematical foundation, but practical adoption will require engineering effort to integrate tensor similarity into existing interpretability pipelines. I expect the first open-source implementations to appear within six months.

My thesis is clear: tensor similarity is the most important methodological advance in mechanistic interpretability since the discovery of induction heads. The short-term consequence is that existing empirical methods like CKA will be deprecated for tensor models, creating a vacuum that tensor similarity will fill. The long-term consequence is more profound: this metric enables the creation of a formal library of neural circuits, moving interpretability from a descriptive science to an engineering discipline. The winners are researchers and companies who invest in open-weight models and interpretability infrastructure. The losers are those who rely on empirical methods or who gatekeep weights behind APIs. My concrete prediction: by Q1 2027, at least two major AI labs will have adopted tensor similarity as their internal standard for circuit verification, and CKA will be cited primarily as a historical baseline in new interpretability papers.

By Q3 2027, at least two major AI labs (e.g., Anthropic and Google DeepMind) will adopt tensor similarity as their internal standard for circuit verification, citing the arXiv paper as the justification.
By Q4 2027, the open-source community will produce a PyTorch-compatible library implementing tensor similarity, achieving at least 10,000 GitHub stars.
By Q2 2028, the use of CKA and SVCCA in mechanistic interpretability papers will decline by at least 50%, replaced by tensor similarity for tensor-based models.

May 2026
arXiv paper published
Introduction of tensor similarity for mechanistic interpretability.
Q3 2027 (predicted)
Major lab adoption
At least two major AI labs adopt tensor similarity as internal standard.
Q4 2027 (predicted)
Open-source library launch
First PyTorch-compatible library for tensor similarity reaches 10,000 stars.
Q2 2028 (predicted)
CKA decline
CKA usage in interpretability papers declines by 50%.

Projected Adoption of Similarity Metrics in Mechanistic Interpretability Papers (2026-2028)

Tensor similarity solves a foundational verification problem that empirical methods like CKA cannot address: out-of-distribution reliability.
This metric is mathematically proven for tensor-based models, making it the first invariant tool for circuit verification.
The winners are open-weight model developers and interpretability researchers; the losers are those who rely on empirical methods or proprietary APIs.
Practical adoption will require engineering, but the mathematical foundation is now laid.