RoPEβs Fixed Rotations: The Hidden Waste in Every Transformer
A new arXiv paper argues that the fixed rotation manifold in Rotary Positional Embeddings (RoPE) is a wasted opportunity for Transformer expressivity. By making both temporal and semantic rotations learnable, the authors claim a new dimension of attention capacity is unlocked.











