# What Actually Changed in Agent Evaluation?

According to the AWS Machine Learning Blog, the September 22, 2026 post frames the problem bluntly: "Skills let you encode domain-specific procedures as reusable, portable instructions for agents, but a fluent answer doesn't prove the agent picked the right skill or followed it." That sentence is the entire thesis of the release. AWS is not shipping a new model or a new agent runtime — it is shipping a measurement discipline. The mechanics matter more than the marketing. Strands Evals provides the harness for defining test cases against an agent, and Amazon Bedrock AgentCore Evaluations provides the scoring layer that runs those cases and returns structured judgments. The combination lets a team assert two distinct things: that the agent selected skill X for input Y, and that the agent's behavior conformed to skill X's instructions. Those are separate assertions, and conflating them is exactly how teams end up shipping agents that look compliant but aren't. My read: this is AWS quietly admitting that the previous generation of agent evals — end-to-end answer scoring — was insufficient. That admission is more valuable than the tooling itself.

# Who Is Affected First, and Why Should They Care?

The immediate beneficiaries are teams already running agents on AWS with skill libraries — think customer support triage, financial document processing, and multi-step internal ops. If an agent has more than about five skills, routing ambiguity becomes the dominant failure mode, and answer-level evals will not catch it. Strands Agents documentation describes the Evals SDK as a way to run structured experiments against agents with defined test cases and scorers, which is the right primitive. But the operational reality is that every eval case costs tokens and wall-clock time. A team with 40 skills and 200 regression cases is now looking at a meaningful CI bill. The second affected group is platform teams at non-AWS shops. They now have a reference design they must either replicate or explain away. "We score the final answer" is no longer a defensible position in a design review, and that pressure will land on every agent framework vendor within two quarters.

# What Are the Real Operational Tradeoffs?

Three tradeoffs dominate. First, determinism versus coverage. Skill selection can often be asserted deterministically — did the agent call skill X? — which is cheap and reliable. Instruction following usually cannot, because it involves judging whether behavior matched a procedure. That requires an LLM-as-judge, which introduces judge variance. Teams that treat both as the same kind of test will get noisy results and lose trust in the suite. Second, eval latency versus release velocity. Running a full skill-selection suite on every commit is the correct answer and also the slow one. The pragmatic pattern is a small blocking suite of high-traffic skills on every PR, plus a broader nightly run. Third, cost versus signal. AWS's own framing implies that the failure being caught is expensive — a wrong skill routed to a customer. That justifies a higher eval budget than most teams currently allocate. The counterargument, which I find weaker, is that human spot-checks are cheaper. They are not, once you account for the fact that humans cannot run the same case 500 times without drifting.

# How Does This Compare to the Alternatives?

Approach	Measures Skill Selection	Measures Instruction Following	Operational Cost	Best Fit
Strands Evals + AgentCore Evaluations	Yes, natively	Yes, via scorers	Medium — token and CI cost	AWS-native agent teams with skill libraries
Answer-only evals (most frameworks today)	No	No	Low	Single-skill or prototype agents
Hand-rolled trace assertions	Yes, if instrumented	Partially	High engineering time	Teams with strong observability already
Human spot-check QA	Inconsistently	Inconsistently	High, non-repeatable	Regulated workflows requiring sign-off
Third-party agent eval vendors	Varies	Varies	Vendor license plus integration	Multi-cloud or framework-agnostic stacks
Verdict	AWS wins for AWS-native teams; everyone else should copy the two-assertion model even if they don't adopt the tools.

# What Should Teams Do in the Next 90 Days?

Start by inventorying skills and labeling each one as high-traffic, high-risk, or neither. High-traffic, high-risk skills get deterministic selection assertions in the blocking CI suite. Everything else goes nightly. Second, write instruction-following scorers as explicit rubrics, not open-ended prompts. A scorer that asks "did the agent follow the skill?" will produce garbage. A scorer that asks "did the agent perform steps 1, 3, and 5 in order and skip step 2?" will produce usable signal. Third, wire eval failures into the same alerting path as production incidents. The AWS post's implicit argument is that a misrouted skill is a production bug, not a QA finding. Treat it that way and the investment pays back.

# Predictions

1. AWS will ship a managed, no-code skill-selection scorer inside Amazon Bedrock AgentCore Evaluations by Q1 2027, collapsing the current custom-scorer requirement.
2. At least two of LangChain, CrewAI, or Microsoft's agent tooling will announce skill-routing eval primitives by mid-2027, explicitly citing selection accuracy as a first-class metric.
3. Enterprise agent teams with more than ten production skills will make skill-selection assertions a blocking CI gate by the end of 2027, and teams without that gate will see measurably higher misroute incident rates.

# What Should Readers Remember?

• Skill selection and instruction following are two separate assertions, and conflating them is the most common evaluation mistake in skill-equipped agents.
• Deterministic checks belong on skill selection; LLM-as-judge scorers belong on instruction following — mixing them produces noise.
• AWS's framing reframes misrouted skills as production bugs, which justifies a larger eval budget than most teams currently allocate.
• The two-assertion model is portable: teams on non-AWS stacks can adopt the discipline without adopting the tooling, and they should.
• The competitive pressure from this release will land on agent framework vendors, not on model providers — expect eval primitives to become table stakes by 2027.