ScarfBench: AI Agents Flunk Enterprise Java Migration

IBM Research just dropped ScarfBench, a benchmark that measures how well AI agents can migrate enterprise Java codebases. The results are brutal: no model exceeds 30% accuracy. For companies betting on AI to modernize legacy systems, this is a cold dose of reality.

IBM Research released ScarfBench, a benchmark for AI agents performing enterprise Java framework migration.
Top models like GPT-4o and Claude 3.5 Sonnet scored below 30% accuracy on the benchmark.
The results expose a critical gap: current AI agents cannot reliably handle complex, multi-file codebase transformations.
This creates an opportunity for vendors who invest in domain-specific training and synthetic data generation.

What Makes Java Migration So Hard for AI Agents?

According to the ScarfBench paper published by IBM Research on the Hugging Face blog, the benchmark consists of 100 real-world migration tasks derived from open-source Java projects. These tasks involve migrating from older Java frameworks like Java EE to modern ones like Jakarta EE. According to IBM Research, the tasks require understanding of not just syntax but also framework-specific APIs, dependency management, and architectural patterns. The benchmark measures three dimensions: correctness of the migrated code, completeness of the migration, and adherence to framework conventions. The average score across all tested models was 24.3%. Even the best-performing model, GPT-4o, achieved only 28.7%. This is far below the 80% threshold that IBM considers acceptable for production use.

Which Models Performed Best and Worst?

IBM tested seven models: GPT-4o, GPT-4 Turbo, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3 70B, and Mixtral 8x22B. The results were uniformly poor, but there were notable differences. Claude 3.5 Sonnet scored 26.3%, slightly behind GPT-4o. Gemini 1.5 Pro achieved 22.1%. The open-source models fared worse: Llama 3 70B scored 15.4%, and Mixtral 8x22B scored 12.8%. IBM Research reported that all models struggled with dependency resolution and API migration. For example, when migrating from Java EE's `@Stateless` to Jakarta EE's `@Stateless`, models often missed required import changes or incorrectly modified business logic. The models also failed to handle multi-file dependencies, where a change in one file requires corresponding changes in several others.

Model	Accuracy (%)	Strengths	Weaknesses
GPT-4o	28.7	Syntax correctness	Dependency resolution
Claude 3.5 Sonnet	26.3	API migration	Multi-file changes
Gemini 1.5 Pro	22.1	Error handling	Framework conventions
Claude 3 Opus	19.5	Code completeness	Dependency management
GPT-4 Turbo	18.2	Import resolution	Business logic preservation
Llama 3 70B	15.4	Basic syntax	Complex migrations
Mixtral 8x22B	12.8	Simple tasks	All complex tasks
Verdict	No model is production-ready. GPT-4o leads but fails at core migration tasks. Open-source models are not viable without fine-tuning.

How Does ScarfBench Compare to Other Code Benchmarks?

Existing code benchmarks like HumanEval and SWE-bench focus on function-level coding or bug fixing. According to IBM Research, ScarfBench is different because it tests multi-file, framework-level migration tasks that require understanding of enterprise patterns. The benchmark includes tasks like migrating EJB session beans, converting JPA entity mappings, and updating web.xml configurations. IBM Research stated that ScarfBench fills a gap because "existing benchmarks do not capture the complexity of real-world enterprise migrations." The paper notes that a typical migration task in ScarfBench involves 5-15 files and requires knowledge of 3-5 different APIs. This is orders of magnitude more complex than generating a single function.

What Does This Mean for Enterprise AI Adoption?

My thesis: ScarfBench proves that the AI industry has been overselling its ability to handle enterprise software modernization. The gap between marketing claims and actual capability is wide.

Short-term consequences: Enterprises that have been testing AI agents for Java migration will hit a wall. Projects that promised 80% automation will deliver less than 30%. This will cause a slowdown in AI adoption for legacy modernization. Vendors like GitHub Copilot and Amazon CodeWhisperer will face pressure to show real results.

Long-term consequences: The winners will be companies that invest in domain-specific fine-tuning and synthetic data generation. IBM Research has already shown that fine-tuning on migration tasks improves accuracy by 15-20 percentage points. I predict that within 12 months, a fine-tuned model will exceed 50% on ScarfBench. The losers will be vendors that rely on generic frontier models without adaptation.

Who gains: IBM, because it owns the benchmark and can use it to drive consulting revenue. Also, any startup that builds a migration-specific AI tool. Who loses: Enterprises that already invested in generic AI agents, and open-source model vendors who cannot match fine-tuned performance.

What Are the Biggest Open Questions?

First, can synthetic data close the gap? IBM Research hinted that generating realistic migration examples is a challenge. Second, will frontier models improve on their own, or is fine-tuning essential? Third, how will this benchmark evolve as frameworks change? Jakarta EE itself is updated regularly, so any benchmark must adapt. According to the ScarfBench paper, the authors plan to release a leaderboard and accept community contributions. This could make ScarfBench the standard for Java migration evaluation, similar to how GLUE became standard for NLP.

Predictions

1. By June 2026, IBM will release a fine-tuned model that scores above 50% on ScarfBench, using synthetic data generated from open-source Java repositories. This will give IBM a clear lead in enterprise migration. 2. GitHub Copilot will add Java migration capabilities by Q4 2025, but will score below 35% on ScarfBench, forcing Microsoft to partner with IBM or another specialist. 3. At least two startups will raise Series A funding in 2025 specifically to build AI agents for Java framework migration, citing ScarfBench as the validation metric.

June 2025
ScarfBench released
IBM Research publishes ScarfBench on Hugging Face, showing all models score below 30%.
Q4 2025
GitHub Copilot migration features
Expected release of Java migration capabilities by GitHub.
June 2026
IBM fine-tuned model
Predicted release of IBM's fine-tuned model scoring above 50%.

Article Summary

ScarfBench is the first benchmark specifically designed for enterprise Java framework migration by AI agents.
No current model achieves even 30% accuracy, showing a massive gap between marketing and reality.
The benchmark measures multi-file, framework-level understanding, which is absent from existing code benchmarks.
Fine-tuning and synthetic data are the most promising paths to improvement, not larger generic models.
IBM is positioned to lead this market because it owns both the benchmark and the domain expertise.