PEFT-Arena: The Benchmark That Exposes LLM Forgetting

The arXiv paper 'PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective' (published May 27, 2026) introduces a new benchmark that jointly evaluates downstream accuracy and general capability retention across 13 PEFT methods. The authors argue that the field's obsession with downstream metrics has hidden a dangerous trend: many PEFT methods, including LoRA and AdaLoRA, cause significant forgetting of pretrained knowledge, especially in reasoning and factual recall.

PEFT-Arena, a new benchmark from arXiv (May 2026), evaluates 13 PEFT methods on both downstream accuracy and retention of pretrained capabilities across 10 tasks.
The study finds that most PEFT methods, including LoRA and AdaLoRA, exhibit significant forgetting of reasoning and knowledge tasks, while full finetuning (FFT) retains superior stability.
This challenges the prevailing assumption that PEFT methods preserve pretrained knowledge by default, and forces the field to adopt a stability-plasticity framework for evaluation.

What Does PEFT-Arena Measure That Existing Benchmarks Miss?

According to the authors of the PEFT-Arena paper (arXiv, May 27, 2026), existing evaluations of parameter-efficient finetuning (PEFT) methods focus almost exclusively on downstream task accuracy, such as GLUE or SuperGLUE scores. This narrow lens overlooks a critical dimension: the retention of general capabilities learned during pretraining, including reasoning, factual knowledge, and language understanding. The paper states: 'We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting.'

PEFT-Arena addresses this gap by constructing a benchmark that jointly measures performance on 10 diverse tasks—ranging from math reasoning (GSM8K) to factual recall (MMLU) to code generation (HumanEval)—both before and after finetuning. The benchmark includes 13 PEFT methods, including LoRA, AdaLoRA, IA3, Prefix Tuning, Prompt Tuning, and full finetuning (FFT) as a baseline. The key metric is the 'stability score,' which quantifies how much of the pretrained capability is retained after adaptation, and the 'plasticity score,' which measures downstream task accuracy.

PEFT-Arena: The Benchmark That Exposes LLM Forgetting

Which PEFT Methods Perform Best on Stability-Plasticity?

The paper reports that full finetuning (FFT) achieves the highest stability score (0.92 out of 1.0) while also achieving strong plasticity (0.89). Among PEFT methods, IA3 emerges as the best performer with a stability score of 0.87 and plasticity of 0.84. In contrast, LoRA—the most widely adopted PEFT method—scores only 0.78 on stability and 0.81 on plasticity, indicating significant forgetting of pretrained capabilities. AdaLoRA, a variant that adaptively allocates parameters, fares worse with a stability score of 0.71.

To put these numbers in context: a stability score of 0.71 means that nearly 30% of the model's pretrained knowledge is lost or degraded after finetuning. For enterprise deployments that rely on LLMs for diverse tasks (e.g., customer support, code generation, and knowledge retrieval), this level of forgetting could be catastrophic. The paper notes that 'on MMLU, AdaLoRA forgets 12% of factual knowledge after finetuning on a single downstream task.'

Why Does LoRA—the Industry Favorite—Perform So Poorly on Stability?

LoRA's popularity stems from its efficiency: it injects low-rank matrices into the model, requiring only a fraction of the parameters to be updated. However, the PEFT-Arena results suggest that this efficiency comes at a cost. The paper hypothesizes that LoRA's low-rank updates may disproportionately affect the model's internal representations of general knowledge, especially in deeper layers where factual recall is encoded.

The authors write: 'We hypothesize that the low-rank constraint in LoRA forces the model to repurpose existing representations, leading to interference with pretrained features.' This is consistent with prior work on catastrophic forgetting in continual learning, where limited parameter updates often lead to overwriting of old knowledge. The implication is that practitioners who rely on LoRA for rapid adaptation may be unknowingly sacrificing the model's broader capabilities.

What Are the Practical Implications for Enterprises and Developers?

For enterprises deploying LLMs in production, the PEFT-Arena findings suggest a need to reevaluate the choice of finetuning method. If a model is used for a narrow, static task (e.g., sentiment analysis), LoRA's plasticity may suffice. But for models that need to handle diverse queries—such as a chatbot that must answer factual questions, generate code, and reason about math—the stability loss may be unacceptable.

According to the paper, 'for applications requiring broad generalization, such as open-domain QA or multi-task agents, full finetuning or IA3 may be preferable despite higher computational cost.' This is a direct challenge to the current trend of using LoRA for everything, and it may spur development of new PEFT methods that explicitly optimize for stability. The paper also notes that 'no single PEFT method dominates both metrics across all tasks,' suggesting that practitioners should benchmark their own use cases rather than relying on default choices.

Here is a comparison of key PEFT methods based on the paper's findings:

Method	Stability Score	Plasticity Score	Forgetting on MMLU (%)	Compute Cost
Full Finetuning (FFT)	0.92	0.89	2%	High
IA3	0.87	0.84	5%	Low
LoRA	0.78	0.81	9%	Low
AdaLoRA	0.71	0.79	12%	Medium
Prefix Tuning	0.83	0.77	7%	Low
Prompt Tuning	0.80	0.76	8%	Low
Verdict	IA3 offers the best trade-off for general-purpose use; LoRA is suitable only for narrow, static tasks.

My thesis is that PEFT-Arena is the most important LLM evaluation contribution of 2026 because it forces the field to confront a hidden cost of efficiency: the silent erosion of pretrained capabilities. The paper's methodology is sound—jointly measuring stability and plasticity across diverse tasks—and its findings are robust across model sizes (7B, 13B, 70B). However, I have two reservations. First, the benchmark uses a single finetuning dataset per task (e.g., a fixed math dataset), which may not capture the full variability of real-world adaptation. Second, the stability metric is averaged across all pretrained capabilities, but some capabilities (e.g., factual recall) may be more critical than others (e.g., stylistic fluency).

In the short term, this paper will pressure PEFT library maintainers (e.g., Hugging Face's PEFT library) to add stability metrics to their default evaluations. In the long term, it could lead to a new generation of PEFT methods that explicitly optimize for stability, perhaps through regularization or multi-task learning. The losers are the current champions of LoRA and AdaLoRA, which will need to justify their forgetting rates. The winners are IA3 and full finetuning, which now have empirical evidence of superior stability.

Predictions:

By Q4 2026, Hugging Face will add a 'stability score' to its PEFT model cards, based on the PEFT-Arena methodology, to help users evaluate forgetting risk.
By Q2 2027, at least two new PEFT methods will be published that explicitly optimize for stability, achieving stability scores above 0.90 while maintaining plasticity above 0.85.
By Q1 2027, enterprises using LLMs for multi-task agents (e.g., customer support chatbots) will shift from LoRA to IA3 or full finetuning, citing the PEFT-Arena findings in their technical blog posts.

Stability Scores of PEFT Methods (from PEFT-Arena)

Article Summary:

PEFT-Arena reveals that most PEFT methods, including LoRA, cause significant forgetting of pretrained capabilities, challenging the assumption that PEFT preserves general knowledge.
IA3 and full finetuning are the top performers on the stability-plasticity trade-off, while AdaLoRA is the worst, forgetting 12% of factual knowledge on MMLU.
Practitioners should benchmark their own use cases rather than defaulting to LoRA, especially for models that need broad generalization.
The benchmark's methodology is sound but has limitations: it uses fixed datasets and averages across capabilities, which may not reflect real-world variability.
This paper will likely shift the PEFT landscape toward methods that explicitly optimize for stability, with new methods expected within 12 months.