How to Trust an AI Agent's 'Done'—Evaluating Output and Trajectory

Executive Summary

Even if an Agent returns the correct final answer, the work cannot be considered acceptable if it accessed data beyond its authorization, skipped required approval, or made unnecessary changes along the way.

Conversely, even if it calls the correct Tools in the correct order, it still fails if the amount or recipient in the final answer is wrong.

An Agent therefore needs to be evaluated from two perspectives.

• Output Evaluation: What it ultimately returned and created
• Trajectory Evaluation: Which Tools it used, and in what order, to get there

This article connects the Skill evaluation from Build With Google Day 1 with ADK Graph verification, then explores how to choose among EXACT, IN_ORDER, and ANY_ORDER, how to practice Evaluation-Driven Development, and where to preserve human judgment.

Where We Are in the Series

Across five installments, this series examines the question: “AI has become extraordinarily intelligent. But can we truly entrust it with work?”

1. What Is Agentic Engineering?
2. Antigravity as a Workplace
3. Agent Skills as Operating Manuals
4. Process Design with Agents CLI and ADK Graph
5. Evaluation Through Output and Trajectory (This Installment)

Connecting from the Previous Installment—A Path That Worked Once Is Not a Guarantee

In the previous installment, we examined a structure that uses an LLM to classify shipping inquiries, receives the result through a Schema, and routes it to either shipping or unrelated using Python and Graph edges.

On the VM, two representative inputs followed the expected paths, and the Unit test for the route function also passed.

However, this only proves that the observed cases worked.

• Does it classify correctly when the wording changes?
• Does it remember to call the required Tool?
• Does it avoid calling unnecessary or prohibited Tools?
• Does it follow the required Tool order?
• Is the content of the final answer also correct?
• Does it stop safely when something fails?

Each of these must be tested separately.

In other words, a working example is the beginning of understanding, not the completion of trust.

Separating Deliverables from Work Records

Suppose you tell a new employee, “Confirm approval before arranging the product.”

The response you receive is, “The order has been arranged.” Judging by the sentence alone, the answer is correct. But what if the employee actually accessed customer data beyond their authorization and arranged the product without approval?

Even when the final state is the same, the assessment of the work changes.

Output Evaluation

This examines the final result.

• Is the answer correct?
• Does it follow the specified format?
• Were the required files created?
• Do the tests pass?
• Does it avoid unsupported claims?

Trajectory Evaluation

This examines the observable execution path.

• Did it call the expected Tool?
• Did it follow the required order?
• Did it avoid calling unnecessary Tools?
• Did it traverse only permitted Graph edges?
• Did it avoid skipping steps that require human confirmation?

Here, Trajectory does not mean forcing the model to disclose its private internal reasoning. It means evaluating actions that the system can record, such as Tool calls, arguments, state transitions, and external effects.

In other words, Output is “what was delivered,” while Trajectory is “how it was delivered.” Only when both are correct do we achieve operational quality.

Why the Final Answer Alone Is Not Enough

Evaluating only the final answer leaves three blind spots.

1. A Coincidentally Correct Answer

Even when an Agent uses incorrect evidence or an irrelevant Tool, the final sentence can still happen to be correct. The same applies when it has memorized a test case.

2. Side Effects

The answer may be correct even though unnecessary actions—such as sending an email, updating data, or deleting a file—occurred along the way.

3. Irreproducibility

If we cannot trace why the Agent produced an answer, we cannot fix it when it fails. Nor can we determine whether it will achieve the same level of quality next time.

At the same time, strictly matching the Trajectory alone is insufficient. The Agent can follow the correct sequence of Tools and still summarize the wrong data or make a calculation error.

Only Output is correct       → Misses coincidences and dangerous paths
Only Trajectory is correct   → Misses errors in the final content
Evaluate both                → Allows separate inspection of the result and process

In other words, these two forms of evaluation are not substitutes; they are orthogonal dimensions of quality.

What Do EXACT, IN_ORDER, and ANY_ORDER Allow?

Google ADK’s official TrajectoryEvaluator implementation compares the expected sequence of Tool calls with the actual sequence in three ways.

Suppose the expected process consists of the following three steps.

A = Verify the customer
B = Check inventory
C = Confirm the order

expected = [A, B, C]

EXACT—Allows No Extras, Omissions, or Ordering Differences

[A, B, C]       PASS
[A, X, B, C]    FAIL
[B, A, C]       FAIL
[A, B]          FAIL

This mode requires the expected and actual sequences to match exactly. The evaluation implementation compares not only the call names but also the Tool arguments.

Suitable examples: Refunds, sending, deletion, approval, and other processes where the procedure and number of executions must be strictly fixed.

IN_ORDER—Allows Additional Calls as Long as the Order Is Preserved

[A, B, C]          PASS
[A, X, B, Y, C]    PASS
[B, A, C]          FAIL

This mode allows additional calls between the expected calls, provided that the expected calls appear in the same order.

Suitable examples: Processes where required steps must remain in sequence, while intermediate log retrieval or auxiliary read operations may vary.

ANY_ORDER—Checks for the Expected Calls Without Requiring an Order

[A, B, C]          PASS
[C, X, A, B]       PASS
[A, C]             FAIL

This mode accepts any order and permits additional calls between them, provided that all expected calls are present.

Suitable examples: Reading from multiple independent information sources when the order does not affect the result or safety.

One point requires caution. Because IN_ORDER and ANY_ORDER allow additional calls, they may not automatically reject a case in which the expected Tools are present but a dangerous Tool is also called. If necessary, use separate criteria to verify that no prohibited Tools were called.

Furthermore, restricting ANY_ORDER to read-only operations is not a rule imposed by the ADK API itself. This article recommends using it primarily for read operations whose side effects are unlikely to change when order is ignored, as a safety-oriented design practice.

In other words, do not choose the most permissive Mode. Choose the Mode that allows only the differences that are operationally safe to ignore.

Evaluation-Driven Development—Define the Acceptance Criteria First

If you wait until after implementation and merely observe that it “seems to work,” you may end up adapting your expectations to the result.

In the Build With Google presentation, the practice of preparing structured evaluation cases—covering the input, expected Tools, and expected Output—before implementation was introduced as Evaluation-Driven Development (EDD).

It resembles test-driven development, but for an Agent, the targets include not only final values but also the Tool trajectory and semantic quality.

{
  "input": "How much is standard shipping?",
  "expected_route": "shipping",
  "expected_tools": ["classify_query", "shipping_faq"],
  "forbidden_tools": ["issue_refund"],
  "expected_output": {
    "contains": ["$5.99"],
    "must_not_claim": ["I issued a refund"]
  }
}

This is a simplified example for explanatory purposes. An actual ADK evaluation set can define conversations, expected intermediate Tool use trajectories, final responses, criteria, and more. The ADK evaluation documentation explains how to run evaluations from the CLI, Web UI, and pytest.

Writing the evaluation first exposes ambiguities in the design.

• Does “related to shipping” include damage and returns?
• Is matching the amount sufficient for an FAQ answer?
• Are additional read-only Tools allowed?
• For an unrelated question, may the Agent answer with anything other than a refusal?
• If classification fails, which safe default should it follow?

In other words, an evaluation specification is both a grading rubric for the Agent and a specification in which humans explicitly define what counts as correct work.

Skill Evaluation and Graph Evaluation Share the Same Structure

In the third installment, we organized Skill failures into four categories: Trigger, Execution, Token Budget, and Regression. Mapping these onto Output and Trajectory reveals where evaluation should occur.

Trying a Skill once does not reveal Regression. Whenever the Skill library changes, existing representative inputs and out-of-scope inputs must be rerun.

Similarly, a Graph should be evaluated not only with shipping inputs but also with unrelated inputs, boundary cases, Schema violations, and Tool failures.

In other words, the evaluation target is not a single answer but the entire lifecycle, including selection, execution, and results.

Reading VM Verification by Strength of Evidence

For the Customer Support Agent on the VM, we separated the checks.

• Lint: Passed
• Unit test: Passed
• Representative shipping input: Passed
• Representative unrelated input: Passed
• Integration test: Passed under explicitly stated assumptions about the expected execution environment contract

The Integration test revealed an important observation.

When running raw pytest, the environment configuration assumed by the Lab was not loaded, so execution stopped before reaching Graph routing. When the same verification was performed with only the names of the required environment variables explicitly specified, the workflow routing test passed. The values themselves were not included in the verification record.

This does not mean that the first failure was caused by an error in the Graph logic. Conversely, the second success does not guarantee the entire production environment.

What this demonstrates is the unit of reproducibility.

Reproducible execution
  = Code
  + Dependency versions
  + Runtime
  + Environment contract
  + Test data
  + Command

Even if “the command is the same,” it is not the same verification when the assumed environment differs.

In other words, test results require more than PASS/FAIL: they must also state under which environment contract they ran and how far execution progressed.

A Small Example—Inspect the Actual Artifact, Not Just the Report

Suppose an Agent reports, “I excluded unnecessary cache files and packaged the deliverables.” Even if the report is polished, the actual contents of the archive should be checked separately.

For the post-exercise artifacts on the VM, the Manifest indicated that cache files had been excluded, while the archive itself still contained 14 cache-like members.

This is neither a problem with the official Lab itself nor a failure mode unique to AI. The same discrepancy can occur in a script written by a human.

This small example demonstrates only one point.

Report stating "excluded"  = claim
List of archive members    = artifact

Verify the claim and artifact independently

If the final result is a file, open the file. If it is an archive, list its members. If it changes an API, read back the actual state. An Agent’s self-report is part of the evidence, but it should not be the only evidence.

In other words, the shortest path to trusting a completion report is not to make the report more elaborate, but to make the actual artifact mechanically verifiable.

Where Do Humans Remain?

Automating evaluation can make humans appear unnecessary. However, machine evaluation has boundaries.

What Code Can Verify Rigorously

• Whether the result conforms to a Schema
• Whether the expected Tool was called
• Whether a prohibited Tool was not called
• Whether state transitions remained within the permitted set
• Whether Unit tests or property tests pass

What an LLM Judge Can Help Assess

• Whether different wording expresses the same meaning
• Whether the answer is supported by the evidence
• Whether the response satisfies the rubric’s standards for courtesy and completeness

What Humans Should Decide Ultimately

• Whether the objective should be pursued in the first place
• How exceptions affect the business, legal obligations, and customers
• Design trade-offs without a single correct answer
• Whether the evaluation metrics themselves distort the original objective
• Responsibility for approving irreversible operations

An LLM Judge is also probabilistic. Its evaluation can be affected by positional bias arising from answer order, gaps in the Judge model’s own knowledge, and ambiguity in the rubric. Countermeasures include calibrating by reversing the order, evaluating multiple times, and combining the Judge with deterministic checks.

In other words, the human role shifts from manually reading every output to designing the objectives, exceptions, responsibilities, and the evaluation system itself.

Input boundary
  └─ Can it classify in-scope, out-of-scope, and dangerous intent?

Context boundary
  └─ Did it retrieve the necessary materials without mixing in outdated ones?

Schema boundary
  └─ Is the output typed so that the next process can read it?

Workflow boundary
  └─ Did it use only permitted state transitions?

Tool boundary
  └─ Does it follow least privilege, with no prohibited operations?

Output boundary
  └─ Do the content, format, and evidence satisfy the requirements?

Artifact boundary
  └─ Does the report match the actual artifact?

Bringing the Five Installments Back Together

At the beginning of the series, we asked:

AI has become extraordinarily intelligent. But can we truly entrust it with work?

After five installments, the answer looks like this.

Model        The capability of an excellent new employee
  +
Antigravity  A workplace with projects, permissions, connections, and evidence
  +
Skills       Operating manuals opened when needed
  +
ADK Graph    Permitted sequences and branches
  +
Evaluation   Inspection of deliverables and execution paths
  =
Closer to an Agent system that can be entrusted with work

“Can be entrusted” is not a declaration that the system will never fail.

It means that we can define what is correct. We can restrict what the Agent is allowed to do. We can observe its progress. We can detect failures. We have boundaries at which control returns to a human. When a problem occurs, we can identify the cause and improve the system.

Designing these properties one by one is Agentic Engineering.

Key Takeaways

• Evaluate Output and Trajectory separately. Both the correct answer and the correct path to reach it are required.
• Choose evaluation strictness based on risk. Use EXACT, IN_ORDER, and ANY_ORDER to allow only differences that are safe to ignore.
• Humans do not disappear outside the evaluation system. They decide objectives, exceptions, irreversible operations, and the evaluation metrics themselves.

Toward Day 2—Observing What Comes After “We Built It”

This concludes the Day 1 retrospective. However, rather than being a conclusion, it also serves as a blueprint for reading Day 2 more deeply.

Day 1 covered the foundations for entrusting work to an Agent. Place the objective in a Project, separate the required knowledge into Skills, restrict Tools and permissions, fix the execution order with a Workflow, and evaluate Output and Trajectory. Only when these elements are in place can AI capabilities be connected to continuous development and operations.

On Day 2, rather than merely following new features, we will use the following five questions as our analytical framework.

1. What counts as completion?
2. Which context and Skill should it read, and when?
3. Which Tools and permissions should be allowed?
4. Which state transitions should be permitted?
5. How should Output and Trajectory be inspected?

What answers do Day 2’s features and implementations provide to these five questions? Conversely, which decisions remain with humans? By following the analysis that far, we can understand them not merely as feature introductions, but as designs for bringing Agents into real-world work.

Once AI has become intelligent, what we need is not an even longer instruction prompt. We need a mechanism that transforms intelligence into work that is inspectable, controllable, and subject to human accountability. On Day 2, we will continue from this foundation.

References

• Build With Google Day 1 — Lab 1
• Build With Google Day 1 — Lab 2
• Google ADK Agent evaluation
• Official ADK TrajectoryEvaluator implementation
• Evaluating Agent Skills