Translating Natural-Language Requests into Production-Ready AI Agents—The App Builders Track

The “exceptionally talented new hire” we welcomed on Day 1 is now going to build work tools for us.

The request looks simple at first.

Build a support Agent that answers customer questions, searches internal data when necessary, and can also perform calculations.

Here, an Agent is software that reads a user’s request, selects the necessary steps, and carries out the work. A Tool is a function the Agent invokes to search, calculate, update data, or perform another operation. Treating these two concepts separately is the starting point of this article.

A talented new hire can write code astonishingly quickly. But the requester does not really want to know how many lines were generated.

• Which questions will it answer, and which will it refuse?
• Which Tool will it use, and under what conditions?
• Is the value returned by a Tool real data or test data?
• How long will it remember information from a conversation, and at what scope?
• Was not only the answer correct, but also the path taken to reach it?
• If it fails, can it stop safely and allow the cause to be investigated?

In the App Builders Track at Build with Gemini Tokyo Day 2, we worked with AI to create an Agent scaffold, test it, run it interactively, inspect its processing trail, and add external capabilities. The latter half covered memory across conversations, internal-document search, safe code execution, screen presentation, and post-release operations. The individual product names are introduced from Section 1.2 onward, where their roles are explained.

This article interprets that workflow not as a single scene in which “AI wrote code,” but as a process for progressively translating ambiguous human intent into inspectable software.

A Map of This Article

Let us begin with the conclusion.

A Coding Agent, as defined here, is a development-support Agent that reads a developer’s request and proceeds not only with code creation, but also with investigation, editing, execution, and testing.

The value of a Coding Agent is not limited to producing code quickly. Its value lies in converting natural-language requests into specifications, implementations, Tool contracts, tests, execution evidence, and distributable deliverables that humans can verify.

This transformation is somewhat like a compiler. It gradually turns high-level intent into an executable form while detecting errors at each intermediate stage.

Read the following table from left to right. It shows how something requested verbally is converted into intermediate deliverables and what should be inspected at each stage. The terms in the rightmost parts of the table are restated in everyday language immediately afterward.

We Will Follow a Single Development Project Throughout

Because many feature names appear, we will keep the narrative centered on one project. Aya, a fictional product manager created for this explanation, works with a new developer to build “a support Agent that answers inquiries, looks up order status and charges, and, when necessary, drafts a proposed shipping-address change.” However, the Agent is not allowed to issue refunds or modify orders on its own.

Aya’s project is a fictional scenario used to explain the technologies as one continuous story. Test results and screens introduced with the phrase “in the exercise” are things observed in the hands-on session. The fictional project and the exercise findings are explicitly distinguished wherever they appear.

The initial request is a single sentence. From there, we define specifications, build Tools, discover failures caused by simulated data, evaluate the Agent, separate conversational state from information sources, and finally turn the design into promises that must be upheld in production. When long-term memory, internal-document search, and dynamic UI generation appear later, they do not begin unrelated stories. They are challenges involved in advancing the same support Agent from “a demo we managed to build” to “a product we can trust with the work.”

First, Restate the Terms in Everyday Language

We will add precision where necessary. For now, rather than asking “What component are we building?”, use “Which concern about the support Agent does this mechanism resolve?” as your guide.

The article proceeds in three stages. Stage 1, Chapters 1–5, covers the foundations for transforming ambiguous requests into Tools and evaluations. Stage 2, Chapters 6–10, covers productization by connecting conversations, search, execution environments, user interfaces (UIs), and authentication. Stage 3, Chapters 11–18, covers design and operations for continuing to make changes safely. Whenever a new term appears, return to the question of which concern in Aya’s project it addresses.

1. The First Job Is Not Writing Code, but Finding Ambiguity

1.1 “Build a Support Agent” Does Not Define Enough

People can fill in gaps in natural language from context. To run a request as software, however, too many matters remain undecided.

• Does the scope include only product information, or order information as well?
• Is the information source the official FAQ, an internal database, or the Web?
• May it execute refunds or cancellations?
• How will it verify that the customer is the person they claim to be?
• If a Tool fails, should it retry or hand the case to a human?
• When should the conversation history be deleted?
• What may be retained in long-term Memory?
• Which evaluation cases will determine correctness?

The better the Coding Agent is, the more likely it is to complete something that “looks plausible” even while the request remains ambiguous. A good development workflow therefore does not conceal ambiguity; it converts ambiguity into questions, plans, types, and tests.

flowchart LR
    A[Human request] --> B[Specifications and prohibited actions]
    B --> C[Agent and Tool implementation]
    C --> D[Deterministic tests]
    D --> E[Conversation and path evaluation]
    E --> F[Execution evidence]
    F --> G[Distributable deliverable]
    D -->|Failure| B
    E -->|Difference from expectations| B

The important point in this diagram is that the process does not move in only one direction until completion. Differences discovered through tests or traces must feed back into the initial specifications, not merely prompt rewording.

1.2 What Role Does Each Development Tool Play?

A series of names can make the subject suddenly seem difficult, but each element has a distinct role.

The Agents CLI Project Structure also separates Agent code, App, Tests, Eval, Manifest, dependency lockfiles, and other elements. There is a reason for this separation: it isolates the impact of changes and the location of failures.

1.3 A Skill Is Not the Capability Itself, but a Way of Working

The Agents CLI Getting Started and Skills Reference provide Coding Agents with Skills for Workflow, ADK code, Scaffold, Evaluation, Deployment, Observability, and other areas.

Thinking of a Skill as a “magic plugin” leads to misunderstanding. What a Skill primarily supplies is procedural knowledge such as the following:

1. Find the Skill relevant to the current request
2. Read only the necessary content
3. Follow the specified sequence and constraints
4. Use the required tools and templates
5. Verify the work and leave deliverables behind

The diagram also illustrates another important distinction.

• Capability: An ability that is technically available
• Policy: An ability permitted in the current situation
• Trace: An ability actually used

Being listed among the capabilities does not necessarily mean something may be used, and being permitted does not necessarily mean it was actually used. Separating these three concepts makes Tool-selection reviews easier.

Adding more Skills does not automatically make the Agent smarter. If the number of Skills with similar descriptions grows, their selection can conflict. If more information is loaded at all times, less context remains for reasoning about the request itself. Skills also require version management, applicability conditions, and regression testing.

Review so far

We decomposed Aya’s one-sentence request into scope, information sources, prohibited operations, failure handling, memory, and evaluation conditions. We also clarified the roles of the tools supporting development. We have not yet reached the stage of discussing code quality, but if ambiguity remains here, fast implementation later will only produce fast rework.

The next unresolved problem: The support Agent cannot perform order searches or calculations by itself. We need to design Tools—the gateways to the outside world—as contracts that neither the model nor the runtime environment will misunderstand.

2. A Tool Is Broader Than a “Function”—It Is a Contract Between the Agent and the Outside World

2.1 Why Design It More Carefully Than an Ordinary Function?

In an ordinary program, a developer specifies the function name and arguments in code. With an Agent, the model reads the user’s natural language and selects which Tool to invoke and what arguments to provide.

A Tool therefore has two kinds of readers.

1. The runtime environment, which interprets types, return values, and exceptions
2. The model, which determines “when to use it” from the name, description, and meaning of the arguments

A docstring or schema is not supplementary documentation; it is part of the execution contract.

A Tool should be tested at no fewer than three layers.

2.2 Reading the Contract Through a Tip-Calculation Example

Connecting directly to customer data or order changes would introduce Tool-design and authorization-design issues at the same time. In the exercise, we therefore began with tip calculation, which does not update an external system, as a small teaching example. We observed only the contract for inputs, calculations, rounding, and failures. We later extend this approach to order and shipping Tools.

In the exercise, we added a Function Tool that returns the tip and total from a bill amount and tip percentage.

The goal is not merely to perform multiplication. It is to communicate promises to both the model and the runtime environment: “use this only when both the bill amount and percentage are available,” “18 means 18%,” and “return the tip and total.”

The following example also makes the currency, rounding rules, and abnormal values explicit. Readers who are not familiar with code may skip it. The point to notice is that ambiguous business rules have been moved into inspectable conditions.

from decimal import Decimal, ROUND_HALF_UP
from typing import Literal

Currency = Literal["USD", "JPY"]

def calculate_tip(
    bill_amount: str,
    tip_percent: str,
    currency: Currency,
) -> dict[str, str]:
    """Calculate the total only when the bill amount, tip rate, and currency are all provided."""
    bill = Decimal(bill_amount)
    percent = Decimal(tip_percent)

    if bill < 0:
        raise ValueError("The bill amount must be zero or greater")
    if percent < 0 or percent > 100:
        raise ValueError("Specify the tip rate between 0 and 100")

    scale = Decimal("1") if currency == "JPY" else Decimal("0.01")
    tip = (bill * percent / Decimal("100")).quantize(
        scale,
        rounding=ROUND_HALF_UP,
    )
    total = (bill + tip).quantize(scale, rounding=ROUND_HALF_UP)

    return {
        "currency": currency,
        "tip_amount": str(tip),
        "total_amount": str(total),
        "rounding": "ROUND_HALF_UP",
    }

In this example, negative bill amounts, extreme percentages, and currency-specific decimal precision are moved into deterministic processing instead of being entrusted to the Agent’s conversational ability. Organizational rules such as “how should fractions of a yen be handled?” and “should the percentage be applied before or after tax?” must be added to the specifications and tests.

2.3 Interpret What a Successful Test Actually Proves

After the Function Tool was added, both the processing logic and the test verifying that the Agent selects the Tool succeeded.

This is important evidence, but it is not proof of correctness for every possible input. Untested negative numbers, very large values, mixed currencies, and ordering with taxes or discounts still require separate coverage.

Instead of focusing on the number of tests, examine which parts of the contract they cover.

3. The Most Interesting Failure: The Model Was Faithful, but the Answer Was Wrong

3.1 Tokyo’s Weather Was Reported as 90°F

After a Tool that calculates a fixed formula, we examine a Tool that obtains a value externally. Before moving on to order data, we investigated the weather Tool used in the exercise. This revealed an important failure: “the processing succeeded, but the information source was wrong.”

When asked in the Playground for Tokyo’s weather and current time, the Agent used both the weather Tool and the time Tool. For the time Tool, it corrected the place name and retried the call.

However, the final answer reported that Tokyo’s weather was sunny and 90°F, or about 32°C. It is tempting to conclude that “the model fabricated a fact.” When we inspected the Tool implementation, however, we found that it was not using a real weather API. It was a simulated test Tool that returned a fixed value for all but certain cities.

The model was faithful to the Tool result it had been given. The error was not in text generation, but in the information-source contract.

This type of failure is dangerous. Whether the input is a stale cached value, another customer’s data, an estimate, or fixed test data, the Agent can turn it into natural and persuasive prose. The fluency of the text “launders” the Tool output, making it appear to be a reliable fact.

3.2 Return Not Only the Value, but Also Its Provenance

The remedy is to attach provenance—the origin of the value—to the Tool result.

In addition to the temperature itself, the following JSON reports whether it is real or simulated data, when it was observed, and what was queried. The purpose is not merely to add fields, but to allow the Agent to determine whether the value may be stated as a fact.

{
  "value": {
    "temperature_f": 90,
    "condition": "sunny"
  },
  "provenance": {
    "source_kind": "simulation",
    "upstream": "local_fixture",
    "normalized_query": "Tokyo",
    "observed_at": null,
    "valid_until": null,
    "dataset_revision": "demo-v1",
    "is_complete": true
  }
}

With this result, the Agent can explicitly identify it as “a demo response” rather than “the actual weather.” Depending on the use case, a production system should include retrieval time, expiration time, upstream API, data version, missing values, pagination, quality flags, evidence URIs, and similar metadata.

3.3 Do Not Mistake Which Layer Failed

Scoring only the final answer cannot produce this classification. That is why an execution Trace is necessary.

Review so far

A Tool was not merely a function, but a contract covering when it should be invoked, what should be passed to it, and how its returned values should be interpreted. When Tokyo’s weather was wrong, the simulated Tool—not the model—was the cause. By examining value provenance and the execution path instead of looking only at the final prose, we can identify which layer needs to be corrected.

The next unresolved problem: Even if a Tool can technically be invoked, it does not follow that it may be invoked for a particular user and purpose. Write Tools can also cause duplicate processing through well-intentioned retries. Next, we separate capability, permission, and execution fact.

4. Separate “Can Use,” “May Use,” and “Actually Used”

The list of Tools registered with an Agent is a map of its capabilities. It is not the same as the scope permitted for the current user or the path actually taken.

• Capability graph: The Agents and Tools that are technically callable
• Policy graph: The scope that may be called for this user, purpose, and situation
• Execution trace: The order, arguments, and results of calls in this execution

For example, a write Tool may be registered but prohibited for inquiry-only use. Even if the Agent invokes a Tool, a Gateway—the shared entry point for communication—or IAM—the data-side access-control system—may reject it. The Agent may also switch to a different Tool after a failure.

4.1 Retries Become a Different Problem When Side Effects Are Involved

Correcting a place name and retrying a time lookup is generally safe. Retrying orders, reservations, emails, or payments in the same way, however, can execute an operation twice.

A Tool with side effects needs at least the following:

• An identifier that prevents the same request from being executed twice (idempotency key)
• A fingerprint of all arguments
• Detection of previously executed duplicates
• A commit point at which the operation is actually finalized
• A distinction between retryable failures and failures that must not be retried
• Binding between the human-approved content and the execution arguments

“An Agent retries intelligently” can provide resilience for read operations, but become a cause of incidents for write operations.

5. Turn Evaluation from an Answer Contest into Quality Assurance

Following unit testing in the exercise, we ran an evaluation through the CLI and confirmed a high score.

This result is evidence that “the specified evaluation data and evaluation dimensions succeeded.” It is not proof that the Agent “answers every question correctly.”

The Agents CLI Evaluation Guide covers general quality, instruction following, Tool use, multi-turn paths, task success, hallucinations, grounding, safety, and other dimensions.

5.1 Divide Tests into Layers

Calculations, authorization, Tool names, and citation-URI matches should be evaluated deterministically wherever possible. For aspects without a single correct answer, such as clarity of explanation or writing style, use a method in which another large language model assigns a score (LLM Judge), together with sample reviews in which humans inspect a subset.

5.2 An LLM Judge Is Also a Measuring Instrument

Scoring by an LLM is convenient, but it can vary according to the Judge model version, the order in which candidates are presented, ambiguity in the rubric, the position of information in long text, and errors in the reference answer.

Evaluation results should therefore lock and record not only the Agent-side configuration, but also the following:

• Judge model and version
• Rubric version
• Input dataset version
• Sampling conditions
• Number of executions
• Agreement rate with human evaluation
• Results from separately evaluating critical conditions

A single unauthorized operation must not be buried in a high average score.

Review so far

We have distinguished between a Tool being callable, being permitted in the current case, and actually having been invoked. We then selected measurements suited to each type of failure: exact tests for calculations, Traces for Tool paths, and rubrics for explanatory quality.

The next unresolved problem: A product cannot be built from a single correct answer. We must decide what to retain across multiple conversational turns, how to search official documents, where to run calculation code safely, and what to display in the UI.

6. Session, State, and Memory Are Not All the Same Kind of “Memory”

For a human new hire, notes from the conversation on the desk, the case-management record for work in progress, and long-term customer knowledge are separate things. The same is true for an Agent.

• Session: A unit grouping a sequence of conversations or executions
• State: Progress within the current work, such as the current step, selections, and Tool results
• Memory: Long-term information referenced across separate Sessions

Agent Architecture Components separates short-term Session/State from external Persistent Memory. Memory Bank can generate Memory from Session events and Content, then retrieve it by Scope.

6.1 Memory Is Not a Convenient Cache, but a New Data Product

Long-term Memory retains summaries, preferences, and past decisions. You therefore need to design the following:

• Whose Memory is it: the user’s, organization’s, case’s, or Agent’s?
• What may be stored?
• How are facts distinguished from inferences?
• Are the basis and generation time retained?
• When does it expire?
• How are corrections and deletions propagated?
• How is retrieved Memory audited?

If stale Memory takes precedence over current facts, the Agent will be consistently wrong. Memory quality should be measured not by the number of stored items, but by accuracy, freshness, deletability, and access boundaries.

7. Code Execution Requires Distinguishing Two Execution Planes

“An Agent executes Python” may sound like a single feature, but there are execution planes with different purposes.

7.1 Calculations Performed While the Model Produces a Response

Gemini API Code Execution is a Tool through which the model generates and executes code while constructing a response, then uses the calculation or analysis results in its answer. It is suitable for short calculations and data processing, but is subject to constraints such as execution time and File I/O.

7.2 A Sandbox Independent of the Agent

The Code Execution Sandbox is treated as an isolated resource separate from the Agent’s own execution environment (Runtime). It can support state across multiple executions and File reads and writes, but external communication, lifetime, capacity, and artifact export must be explicitly defined.

When selecting an option, do not ask only whether “Python can run.” Verify the following:

• Under which Identity it runs
• Whether the Network is closed by default
• Which Files it can read and write
• Limits on execution time, CPU, Memory, and capacity
• Deletion when the Session ends
• Package sources and electronic fingerprints (hashes) derived from their contents
• Whether stdout, generated Files, and exceptions can be retained as evidence

8. RAG Is Not a Single Quality Metric

RAG is a mechanism that searches for necessary material and then uses its contents to produce an answer. The important point is not to turn “retrieval” and “answer generation” into a single black box.

The RAG Engine in Gemini Enterprise Agent Platform also consists of multiple stages: document ingestion, splitting into retrieval-friendly sizes, numerical representation of meaning, creation of a retrieval index, retrieval, and answer generation.

When “the RAG answer is poor,” distinguish at least the following possibilities:

1. The correct document was not registered
2. The document exists, but splitting severed its meaning
3. The search query was poor
4. The correct passage did not rank highly
5. The answer ignored the retrieved passage
6. The citation did not correspond to the claim
7. The document was stale or should not have been visible to the user

The evidence retained should include more than the final answer. By preserving the search query, filter, retrieved chunk, score, document version, citation position, and chunks used during generation, you can determine which stage needs correction.

Review so far

We separated information needed only for the current conversation from information retained over the long term, divided calculation environments by purpose, and decomposed RAG failures into ingestion, retrieval, and generation. Aya’s fictional support Agent is now less likely to prioritize stale customer notes over official rules or misdiagnose a retrieval failure as a text-generation problem.

The next unresolved problem: Even if internal processing is correct, the boundary still fails if the Agent returns a dangerous UI or reuses developer authentication in production. Next, we close the presentation and Identity boundaries.

9. A2UI, Where an Agent Builds the UI: Create a Presentation Contract, Not Free-Form Generation

A2UI is a mechanism through which an Agent specifies UI components appropriate to its response, which are then transformed into a user-facing interface. If the Agent generates and executes arbitrary HTML or JavaScript, quality, security, and compatibility become difficult to manage. The important idea behind A2UI is that the Agent returns declarative UI data specifying “which components should be arranged and how,” while a trusted renderer displays only approved components.

flowchart LR
    A[Agent] -->|Versioned UI message| V[Schema validation]
    V -->|Approved components| R[Trusted renderer]
    V -->|Unknown or dangerous| X[Reject or render safely]
    R --> U[User interface]

At this boundary, design the schema version, approved components, action allowlist, URL restrictions, input validation, accessibility, and fallback behavior for older clients. Treating the UI as “verifiable data” instead of “code” stabilizes the contract between the Agent and the front end.

10. Authentication Does Not End When “Login Works”

Agents CLI Authentication also explains authentication according to the mode of use. In production, do not conflate the following entities:

• The developer’s Identity for using the CLI
• The Identity used by CI/CD to build and deploy
• The Agent Runtime’s own Identity
• The End user making a request to the Agent
• The delegated Identity used when the Agent accesses an external Tool

Avoid designs that carry credentials that worked locally directly into the Runtime. For every operation, explicitly specify whose permissions were used to read the data, whether the user’s permissions are propagated, and whether the Agent’s own permissions are used.

The point where the demo becomes a product

The support Agent can now continue a conversation, search internal documents, perform necessary calculations in an isolated environment, and return a verifiable UI. At the same time, the Identities of users, developers, the execution environment, and external Tools have been separated. This completes the design needed to make the features work.

From here, we consider whether the same system can be changed safely the next day, by another developer, and in another version. The discussion moves to reproducibility, Tool compatibility, evaluator reliability, Memory and RAG boundaries, and operating costs, but the goal remains one thing: to explain what changed, what broke, and whether the change is safe to release.

11. Can We Build the Same Thing Tomorrow? Expanding the Unit of Reproducibility

“I entered the same AI instruction (Prompt) as yesterday, but it produced different code today.” This is not unusual in Agentic Coding. The cause is not only model variability; the entire world read by the Agent may have changed.

To explain the same development result, consider at least the following context:

Reproducible build context
  = Human request and acceptance criteria
  + repository revision
  + project-specific instruction
  + Skills used and their versions
  + point in time of the official documentation consulted
  + CLI, SDK, and template versions
  + model and toolchain
  + runtime environment configuration

This formula does not mean “freeze everything and eliminate creativity.” Its purpose is to make it possible to explain which change caused the result to change.

In order, the formula lists the request, source code, development rules, reference procedures, official documentation, development tools, model, and environment. It is not a list of things that must remain frozen forever, but a checklist for locating differences when results change.

11.1 Make the Transformation Process a Deliverable, Not Just the Generated Output

A conventional build produces a binary from source and dependencies. In Agentic Coding, plans, answers to questions, reference material, Tool executions, and test results are also important materials for explaining the transformation process.

This does not mean that every detail of internal reasoning must be stored without limit. What should be retained is the minimum information needed to verify decisions.

• Final specifications and unresolved issues
• Changed Files and diffs
• URIs and retrieval times of the official documentation read
• Applied Skill names and versions
• Executed commands and their exit status
• Test, evaluation, and static-analysis results
• Diffs approved by a human

This allows the deliverable to be explained not as “we do not know because AI made it,” but as “an artifact that passed these specifications, versions, and inspections.”

11.2 A Lockfile Is Not Enough

A package lockfile pins dependencies, but it does not pin external API responses, search indexes, model versions, or Skill contents. Conversely, freezing everything completely prevents the adoption of vulnerability fixes and current information.

Policies should therefore be separated by purpose.

12. Evolve Tools Without Breaking Them: Give Them API-Grade Contracts

Carefully written Tool descriptions alone are not sufficient as production contracts. As with an HTTP API—the invocation boundary of a Web service—design compatibility, errors, timeouts, partial success, and versions.

12.1 Argument “Type” and “Meaning” Are Different

amount: number does not reveal the currency, whether tax is included, the smallest unit, or whether negative values are permitted. date: string does not reveal the timezone, whether it contains only a date or also a time, or whether a deadline is inclusive.

Beyond the machine type, include a semantic type that expresses business meaning such as currency or timezone.

In the following table, the rightmost column explains why the middle column alone is insufficient. For example, a number can be used in calculations, but there is no business-correct answer unless you know whether it represents yen or dollars and whether it includes tax.

12.2 Do Not Return Errors Only as Natural Language

If a Tool returns only “The operation could not be completed,” the Agent cannot determine whether it should retry, ask the user, or stop.

The following JSON is an example of a structured failure. Its purpose is to prevent infinite retries and allow the Agent to select a safe next action.

{
  "status": "error",
  "code": "CUSTOMER_NOT_VERIFIED",
  "retryable": false,
  "safe_message": "Identity verification is required",
  "operator_detail": "order lookup blocked before data access",
  "next_allowed_actions": ["request_verification", "handoff_to_human"],
  "trace_id": "trace-..."
}

This format allows the system to show a safe explanation to the user, retain investigative information for operators, and expose only permitted next actions to the Agent.

12.3 Do Not Rely on “the Model Will Handle It” for Version Migration

Changing a Tool’s argument names or return values can break older Agents, old Sessions, evaluation data, and UI renderers.

• Introduce a new version for breaking changes
• Publish the retirement deadline for the old version
• Record the required version in the Agent manifest
• Provide contract tests
• Observe both versions during rollout
• Decide whether unknown fields should be safely ignored or rejected

Natural language is flexible, but that is not a reason to make execution boundaries ambiguous.

13. Is the Scorecard Itself Correct? Inspect the Evaluation System

The evaluation code, dataset, and Judge prompt used to measure Agent quality are also software. If they fail, they can classify a poor Agent as a good one.

13.1 Evaluation Questions Leak into Development (Dataset Leakage)

If evaluation questions become mixed into Instructions, Skills, examples, or training material, the Agent may have seen the answers rather than generalized to unseen work.

Countermeasures include the following:

• Separate development, tuning, and final-decision datasets
• Restrict access to the final-decision dataset
• Add paraphrases with the same meaning and additional boundary values
• Continuously add anonymized long-tail cases from real usage
• Separate question authors from release approvers where necessary

13.2 Regression-Test the Evaluator

Give the evaluator calibration cases whose outcomes are unambiguous.

• Pass an obviously correct answer
• Fail an obvious unauthorized action
• Fail an answer that includes a citation but refers to the wrong company
• Fail a path whose content is correct but which used a prohibited Tool
• Fail a definitive claim in a situation where the answer should be “unknown”

When updating the Judge, rerun not only the Agent tests but also this calibration.

13.3 Is a Difference in Evaluation Scores Meaningful?

An increase from 4.82 to 4.86 may simply reflect sampling variation. Examine not only the difference in averages, but also case-by-case improvements and regressions, critical failures, variance across repeated executions, cost, and latency.

In practice, a release report like the following is easy to interpret.

This table is not intended merely to compare score magnitudes. The upper rows show user value and safety, while the lower rows show regressions in time and cost. p95 latency is the response time within which approximately 95 of 100 requests complete. If even one row is unacceptable, do not decide to release solely because the average improved.

Put latency and cost regressions in the same diff as quality.

14. Separate Information to Remember from Information to Retrieve Again

Memory and RAG are often confused because both retrieve information from the past.

• RAG primarily retrieves evidence from documents or data sources
• Memory primarily reuses previous states or summaries related to a user or case

For example, “the return period is 30 days” should be retrieved through RAG from the official policy document. “This user prefers concise answers” is a candidate for Memory. If official rules are stored only in Memory, stale content may remain after the policy is updated.

14.1 Information That Should Generally Not Be Stored in Memory

• passwords, tokens, and private keys
• Temporary authentication information
• Rules or prices whose original source must be checked
• Personal information that cannot follow deletion requests
• Summaries that turn unsupported inferences into facts
• Raw Tool output containing highly sensitive information

14.2 Precedence of Retrieved Information

Decide what happens when Memory and RAG conflict during answer generation. In general, prioritize official sources with clear versions and authority, and use Memory to supplement preferences and context. The system also needs the option to expose the conflict and ask the user for confirmation instead of concealing it.

14.3 Divide RAG Evaluation into Stages

By preserving evidence from before and after retrieval instead of scoring only the final answer, you can identify where improvement is needed.

15. Do Not Add Observability and Cost Management as Afterthoughts

An Agent may use multiple models, Tools, searches, and retries for a single request. The user sees one answer, but internally, a small distributed system is operating.

15.1 Break Down Where Time Was Spent

Total response time alone does not reveal how to improve performance. Separate the following as spans:

• Input validation and authentication
• Model planning
• Tool selection
• Tool queue, Network, and processing time
• RAG retrieval
• Retries and backoff
• Final generation
• Output validation

The Tool graph and events seen in the Playground help with understanding during development. In production, apply the same idea to traces so you can distinguish whether the model, a Tool, or retries caused a delay.

15.2 Measure Cost per Business Transaction, Not Only Tokens

Agent costs include not only model tokens, but also retrieval, embedding, database queries, Sandboxes, logs, Network usage, and human review.

Total cost per business transaction
  = model
  + Tool and data access
  + retrieval and index
  + sandbox
  + observability
  + human review
  + failures and retries

Switching to a smaller model can increase total cost if it repeatedly selects the wrong Tool and triggers more retries. Conversely, a higher-performing model may reduce total cost if it creates the correct plan in one attempt and reduces the time humans spend reviewing it.

15.3 Examine Quality, Time, and Cost in the Same Change Diff

Optimizing a single number degrades another dimension. In a release review, place quality, critical failures, latency, and cost side by side, and assess them as trade-offs against business value.

15.4 Do Not Create Too Many Types of Monitoring Labels

If user IDs, entire queries, and all Tool arguments are included as metric labels, they create a high-cardinality state in which the number of distinct values grows without limit. This increases monitoring costs and spreads sensitive information.

• Use aggregatable categories in metrics
• Move from a trace ID to secure logs for individual investigations
• Do not retain Prompt or Tool-result bodies by default
• Design sampling and redaction
• Separate viewing scopes for development, operations, and auditing

Observability does not mean retaining everything. It means preserving the minimum signals needed to identify causes.

Review of the design section

We can now examine reproducibility conditions from request to implementation, Tool compatibility, evaluator reliability, Memory and RAG information sources, and even time and cost as parts of a single change diff. This section was not about adding features; it was about continuing to change existing features safely.

The next unresolved problem: To hand design principles over to operations, we must translate “be careful” into rules that must always be upheld, numbers that must be monitored, and review procedures applied to representative cases.

16. Promises to Uphold in Production

Let us restate the design so far not in terms of product names, but as “conditions that must always be upheld.” The left column contains the promise made to users or operators; the right column explains how to verify it.

Metrics to Monitor

• Rate of incorrect Tool selection
• Argument-normalization failure rate
• Missing-provenance rate
• Tool timeouts and retry count
• Duplicate side-effect rate
• Percentage of traces that can be fully reconstructed
• Memory-expiration and deletion-propagation time
• Percentage of RAG queries where the correct document ranks highly
• Number of critical evaluation failures
• Regression differences before and after version updates

17. Return to Aya’s Project and Conduct a Design Review

Let us review the support Agent Aya requested at the beginning against the design principles developed so far. Its task is to “answer order-status questions, draft a proposed shipping-address change when necessary, and apply the change only after approval.”

A poor design passes an order number and free-form text to a single broadly privileged Tool, retries automatically on failure, and returns only a natural-language result.

The improved design looks like this:

1. Separate order lookup and shipping-address changes into different Tools
2. Perform lookups under the user’s permissions and changes under a dedicated execution identity
3. Verify the order number against the customer’s Identity
4. Draft the change and bind approval to the exact content confirmed by a human
5. Finalize it exactly once with an idempotency key
6. Connect the request, approval, Tool arguments, and result in the same trace
7. Test successful lookup, rejection of unauthorized access, duplicate submission, and retries after a timeout

At this point, Agent development is no longer prompt tuning. It has become software engineering that designs contracts, state transitions, permissions, and evidence.

18. Practical Checklist

Specifications and Implementation

• Requirements are written from reader or user scenarios
• Capabilities, prohibited actions, and actions requiring confirmation are distinguished
• Versions of specifications, Skills, templates, and dependencies are recorded
• A human reviews the diff and design after automatic generation

Tool

• Tool name, description, arguments, and return values are consistent
• Input ranges, units, currency, time, and Tenant are explicit
• Real data, cache, fixture, and simulation are distinguished
• Timeout, partial results, and pagination can be represented
• Side effects use idempotency and approval

Test and Eval

• Calculations and policy are inspected deterministically
• The Agent’s Tool selection, arguments, and order are also inspected
• Cases include not only success, but also refusal, missing data, failures, and adversarial input
• LLM Judge version, rubric, and variability are recorded
• Critical failures are separated from averages

State, Runtime, and UI

• Session, State, and Memory have defined scopes and TTLs
• Memory corrections, deletion, and freshness are handled
• Sandbox Identity, Network, Files, and resource limits are defined
• UI messages are validated against a schema
• Developer, deployment, Runtime, and user Identities are separated

Conclusion

The experience of creating an Agent from natural language in a short time is certainly impressive. But the essence of the App Builders Track is not speed itself.

Turn ambiguity in a request into specifications. Design Tools as contracts with the outside world. Attach provenance to values. Separate conversational memory by scope and expiration. Evaluate not only the final answer, but also the execution path. Turn calculations, isolation, UI, and authentication into inspectable boundaries.

Only then can something built by an exceptionally talented new hire be transformed from “a demo that worked” into “software that a company can operate responsibly.”

Official Documentation

• ADK — Coding with AI
• Agents CLI Getting Started
• Agents CLI Skills Reference
• Agents CLI Project Structure
• Agents CLI Manual Workflow
• Agents CLI Evaluation Guide
• Agents CLI Authentication
• Agent Tracing
• Agent Architecture Components
• Memory Bank Setup
• Gemini API Code Execution
• Agent Platform Code Execution Sandbox
• RAG Engine Quickstart