---
season: 3
season_name: "Season 3 — October 2026 (Autumn / Halloween Edition)"
day: 22
title: "Agent Observability: One Audit Trail for Every Agent"
summary: "Capture, bridge, and normalize OpenTelemetry GenAI and OpenInference spans into an immutable, PII-scrubbed BigQuery audit trail."
tags: ["Observability", "OpenTelemetry", "Cloud Trace", "ADK"]
canonical_url: "https://adventofagents.com/2026/10/22"
markdown_url: "https://adventofagents.com/2026/10/22.md"
video_url: "https://www.youtube.com/embed/MvbkDLsIqqk"
primary_video:
  title: "Day 22: One Audit Trail for Every Agent: Google Cloud Kata"
  creator_name: "Dani Zamora, with Jesse Carah"
  duration: "15:59"
  video_url: "https://www.youtube.com/embed/MvbkDLsIqqk"
companion_videos:
  - id: "s3-d22-agent-clinic-evals"
    title: "How to Actually Evaluate (Any) AI Agent: Beyond the Vibe Check"
    creator_name: "Dani Zamora & Matt Feroz"
    creator_handle: "@googlecloudtech"
    creator_url: "https://www.youtube.com/@googlecloudtech"
    duration: "26:07"
    video_url: "https://www.youtube.com/embed/wPdoZRbvaF4"
    watch_url: "https://www.youtube.com/watch?v=wPdoZRbvaF4"
    invitation_headline: "Telemetry is not just for eyeballing traces: turn it into evals that keep downstream processes safe"
---

# ⚡ Day 22: Agent Observability: One Audit Trail for Every Agent

> **Season 3 — October 2026 (Autumn / Halloween Edition)** · [Interactive Web View](https://adventofagents.com/2026/10/22?utm_source=adventofagents&utm_medium=markdown&utm_campaign=adventofagents_s3_2026&utm_content=day22) · [Raw Markdown](https://adventofagents.com/2026/10/22.md?utm_source=adventofagents&utm_medium=markdown&utm_campaign=adventofagents_s3_2026&utm_content=day22)

**Summary:** Capture, bridge, and normalize OpenTelemetry GenAI and OpenInference spans into an immutable, PII-scrubbed BigQuery audit trail.

## 🧭 Community Perspective & Companion Deep Dive

- **[Featured Perspective: AI Agent Clinic, evaluating any AI agent (Google Cloud Tech)](https://www.youtube.com/watch?v=wPdoZRbvaF4)** by **Dani Zamora & Matt Feroz** (@googlecloudtech · 26:07) — In this AI Agent Clinic episode, Dani and Matt build an end-to-end evaluation pipeline for a LangGraph documentation agent, and the scorecard catches a 0.33 documentation quality score that eyeballing missed.

**Day 22 of Google's Advent of Agents - Season 3**

Your auditors want an immutable trail of every tool call and model decision, with timings, for every agent. But your agents were built with ADK, with LangGraph, and in a no-code designer, and each one writes its telemetry in a different dialect.

**How It Works**

- **Same tree, different names**: every framework here emits OpenTelemetry, the same tree of spans with ids, parents and timestamps. ADK writes the OpenTelemetry GenAI conventions (`gen_ai.request.model`, `gen_ai.usage.input_tokens`); LangGraph, through the OpenInference instrumentor, writes OpenInference (`openinference.span.kind`, `llm.token_count.prompt`).

![Same Agent, Two Telemetry Dictionaries: OpenTelemetry GenAI vs. OpenInference](/s03/s3-day22-dict-s4.jpg)

- **Bridging the console gap**: Agent Runtime's Session view groups spans by `gen_ai.conversation.id` and sums `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. Adding a 15-line OpenTelemetry `SpanProcessor` (`genai_bridge.py`) copies `session.id` and token counts onto those `gen_ai` attributes before export so LangGraph sessions light up in the console out of the box.

![Agent Runtime Session View: Out-of-the-Box LangGraph vs. With the 15-Line genai_bridge.py](/s03/s3-day22-session-bridge.jpg)

- **The words are opt-in**: with `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY`, ADK writes prompts and responses as a separate log entry per model call, linked to its span by trace id and span id. OpenInference puts them on the span (`input.value`, `output.value`).
- **One immutable table**: Cloud Trace keeps every span in the `_Trace` log bucket. Link it to BigQuery with zero copy, normalize both dialects in one SQL view, and lock the retention policy of the audit bucket (`--locked`) so nobody can shorten retention or delete records until they expire.

![BigQuery Joined Audit Trail: Cloud Trace Spans Joined with Scrubbed Prompt and Response Logs](/s03/s3-day22-bigquery-words-join.jpg)

**Why It Matters Downstream**

Telemetry is not only for opening a trace when something looks off. The same spans feed the evaluations, quality alerts and downstream governance pipelines that depend on your agent behaving, which is exactly where a silent schema mismatch or token-counting difference costs the most. In the video, Jesse Carah, AI Incubation Lead at Google Cloud, shares what he sees with customers: security, compliance and platform teams read the same telemetry for different reasons, so consolidate it in one warehouse, on one open schema, and let SQL do the translation.

![Why Two Dictionaries Exist: OpenTelemetry GenAI and OpenInference](/s03/s3-day22-two-dictionaries.jpg)

**Resources:**

- [GitHub Repo: agent-telemetry-side-by-side (code, comparison app, 15-line bridge & SQL)](https://g.dev/cloud/adventofagents-season3-day22-demo)
- [Set up tracing for Agent Runtime](https://g.dev/cloud/adventofagents-season3-docs-agent-runtime-tracing)
- [Query and analyze traces](https://g.dev/cloud/adventofagents-season3-docs-trace-analytics)
- [Google Cloud Next: Google Cloud and Arize unify agent observability](https://www.youtube.com/watch?v=nLH0IqHLxaA)

## Code & Commands

### Step 1: Capture an ADK and a LangGraph Agent's Telemetry as JSON

```python
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Run one agent on one question and save its telemetry as plain JSON.

    python capture.py adk                  # ADK, native OpenTelemetry  -> GenAI conventions
    python capture.py langgraph            # LangGraph + OpenInference  -> OpenInference conventions
    python capture.py adk-openinference    # bonus: the SAME ADK agent with the OpenInference instrumentor
    python capture.py adk --question serial
    python capture.py adk --model claude-sonnet-5-5      # same harness, another provider

Writes traces/<agent>__<question>.json with:
    spans   one object per span: name, ids, parent, start/end, duration, attributes
    events  OpenTelemetry log records emitted during the run (where GenAI puts the words)
    session the framework's own record of the conversation (ADK session events,
            LangGraph checkpoint), which is NOT telemetry: it is what the agent
            reads back to continue the conversation

Nothing leaves your machine except the model calls: the exporters write to memory.
"""

import argparse
import asyncio
import json
import os
import pathlib
import sys

# ---- where the words go ------------------------------------------------------
# GenAI conventions (ADK): words go into separate log records ("events"), opt-in.
os.environ.setdefault("OTEL_SEMCONV_STABILITY_OPT_IN", "gen_ai_latest_experimental")
os.environ.setdefault("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "EVENT_ONLY")
# Keep ADK's own copy of the words off the spans, like Agent Runtime does in production.
os.environ.setdefault("ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS", "false")
# OpenInference: words go on the spans (input.value / output.value), on by default.

from opentelemetry import _logs, trace  # noqa: E402
from opentelemetry.sdk._logs import LoggerProvider  # noqa: E402
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor  # noqa: E402
from opentelemetry.sdk._logs.export import InMemoryLogRecordExporter  # noqa: E402
from opentelemetry.sdk.resources import Resource  # noqa: E402
from opentelemetry.sdk.trace import TracerProvider  # noqa: E402
from opentelemetry.sdk.trace.export import SimpleSpanProcessor  # noqa: E402
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter  # noqa: E402

HERE = pathlib.Path(__file__).parent
sys.path.insert(0, str(HERE))
from tools import QUESTIONS  # noqa: E402

span_store = InMemorySpanExporter()
log_store = InMemoryLogRecordExporter()


def setup_otel(service_name: str) -> TracerProvider:
    resource = Resource.create({"service.name": service_name})
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(SimpleSpanProcessor(span_store))
    trace.set_tracer_provider(provider)
    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(SimpleLogRecordProcessor(log_store))
    _logs.set_logger_provider(logger_provider)
    return provider


SESSION = {}


async def run_adk(question: str) -> str:
    from google.adk.runners import InMemoryRunner
    from google.genai import types

    from adk_agent.agent import root_agent

    runner = InMemoryRunner(agent=root_agent, app_name="trip_concierge")
    session = await runner.session_service.create_session(app_name="trip_concierge", user_id="ana")
    answer = ""
    async for event in runner.run_async(
        user_id="ana",
        session_id=session.id,
        new_message=types.Content(role="user", parts=[types.Part(text=question)]),
    ):
        if event.is_final_response() and event.content and event.content.parts:
            answer = "".join(p.text or "" for p in event.content.parts)
    # ADK's session: every event (user message, function calls, function responses,
    # final answer) plus state, as stored by the SessionService.
    stored = await runner.session_service.get_session(
        app_name="trip_concierge", user_id="ana", session_id=session.id
    )
    SESSION.update({"framework": "adk", "kind": "ADK Session (SessionService)",
                    "data": stored.model_dump(mode="json", exclude_none=True)})
    await runner.close()
    return answer.strip()


async def run_langgraph(question: str) -> str:
    from langgraph_agent import agent

    config = {"configurable": {"thread_id": "session-1"}, "metadata": {"session_id": "session-1", "user_id": "ana"}}
    state = await agent.ainvoke({"messages": [("user", question)]}, config=config)
    # LangGraph's session: the checkpoint for this thread (graph state + metadata).
    from langchain_core.messages import messages_to_dict

    snap = await agent.aget_state(config)
    SESSION.update({"framework": "langgraph", "kind": "LangGraph checkpoint (thread state)",
                    "data": {"thread_id": "session-1", "next": list(snap.next), "metadata": snap.metadata,
                             "created_at": snap.created_at,
                             "values": {"messages": messages_to_dict(snap.values["messages"])}}})
    content = state["messages"][-1].content
    if isinstance(content, list):
        content = " ".join(p.get("text", "") if isinstance(p, dict) else str(p) for p in content)
    return str(content).strip()


def hex_id(value, width):
    return format(value, f"0{width}x") if value else None


def simplify(span) -> dict:
    """The parts of a span you actually look at. Everything else is plumbing."""
    return {
        "name": span.name,
        "trace_id": hex_id(span.context.trace_id, 32),
        "span_id": hex_id(span.context.span_id, 16),
        "parent_span_id": hex_id(span.parent.span_id, 16) if span.parent else None,
        "start_time_ns": span.start_time,
        "end_time_ns": span.end_time,
        "duration_ms": round((span.end_time - span.start_time) / 1e6, 1),
        "instrumentation": span.instrumentation_scope.name if span.instrumentation_scope else None,
        "attributes": {k: v for k, v in (span.attributes or {}).items()},
        "status": span.status.status_code.name,
    }


def simplify_log(item) -> dict:
    record = item.log_record
    return {
        "event_name": getattr(record, "event_name", None),
        "trace_id": hex_id(record.trace_id, 32),
        "span_id": hex_id(record.span_id, 16),
        "attributes": {k: v for k, v in (record.attributes or {}).items()},
        "body": record.body,
    }


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("agent", choices=["adk", "langgraph", "adk-openinference"])
    parser.add_argument("--question", choices=list(QUESTIONS), default="parallel")
    parser.add_argument("--model", default=None, help="Gemini (default) or a Claude model")
    args = parser.parse_args()
    if args.model:
        os.environ["MODEL"] = args.model
    model = os.environ.get("MODEL", "gemini-3.8-flash")

    provider = setup_otel(f"trip-concierge-{args.agent}")
    question = QUESTIONS[args.question]

    if args.agent == "adk":
        answer = asyncio.run(run_adk(question))
    elif args.agent == "adk-openinference":
        from openinference.instrumentation.google_adk import GoogleADKInstrumentor

        GoogleADKInstrumentor().instrument(tracer_provider=provider)
        answer = asyncio.run(run_adk(question))
    else:
        from openinference.instrumentation.langchain import LangChainInstrumentor

        LangChainInstrumentor().instrument(tracer_provider=provider)
        answer = asyncio.run(run_langgraph(question))

    spans = sorted((simplify(s) for s in span_store.get_finished_spans()), key=lambda s: s["start_time_ns"])
    events = [simplify_log(r) for r in log_store.get_finished_logs()]
    suffix = "" if model.startswith("gemini") else f"__{model}"
    out = HERE / "traces" / f"{args.agent}__{args.question}{suffix}.json"
    out.parent.mkdir(exist_ok=True)
    out.write_text(json.dumps(
        {"agent": args.agent, "model": model, "question": question,
         "answer": answer, "spans": spans, "events": events, "session": SESSION},
        indent=2, default=str,
    ))
    n_session = len(SESSION.get("data", {}).get("events", SESSION.get("data", {}).get("values", {}).get("messages", [])))
    print(f"{args.agent}: {len(spans)} spans, {len(events)} events, {n_session} session items -> {out.relative_to(HERE)}")
    print(f"answer: {answer}")


if __name__ == "__main__":
    main()
```

### Step 2: 15-Line Bridge So LangGraph Sessions Light Up in Agent Runtime

```python
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Bridge OpenInference spans onto the OpenTelemetry GenAI keys read by Agent Platform.

Out of the box, Agent Runtime's LanggraphAgent(enable_tracing=True) wires
openinference-instrumentation-langchain, which records conversations under
`session.id` and token counts under `llm.token_count.prompt` and `llm.token_count.completion`.
The Agent Platform console's Session view groups spans by `gen_ai.conversation.id` and sums
`gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` on `gen_ai.operation.name = "generate_content"` spans.

Calling `install_genai_bridge()` inside `model_builder` (after Agent Runtime's
default exporter setup has created the `TracerProvider`) keeps all OpenInference
attributes and resource identifiers intact while stamping the four `gen_ai`
keys the console expects, so Session view lights up for LangGraph deployments too.
"""

from opentelemetry.sdk.trace import SpanProcessor
from opentelemetry.trace import Status, StatusCode


class GenAIBridgeSpanProcessor(SpanProcessor):
    """Copies OpenInference span attributes onto their OpenTelemetry GenAI equivalents."""

    def on_end(self, span) -> None:
        a = dict(span.attributes or {})
        if "session.id" in a and "gen_ai.conversation.id" not in a:
            a["gen_ai.conversation.id"] = str(a["session.id"])
        kind = a.get("openinference.span.kind")
        if (span.parent is None or span.name == "LangGraph") and "gen_ai.operation.name" not in a:
            a["gen_ai.operation.name"] = "invoke_agent"
            a.setdefault("gen_ai.agent.name", "trip_concierge_langgraph")
        elif kind == "LLM":
            a.setdefault("gen_ai.operation.name", "generate_content")
            if "llm.model_name" in a:
                a.setdefault("gen_ai.request.model", str(a["llm.model_name"]))
            prompt = int(a.get("llm.token_count.prompt") or 0)
            comp = int(a.get("llm.token_count.completion") or 0)
            reas = int(a.get("llm.token_count.completion_details.reasoning") or 0)
            a.setdefault("gen_ai.usage.input_tokens", prompt)
            a.setdefault("gen_ai.usage.output_tokens", comp + reas)
            if reas:
                a.setdefault("gen_ai.usage.reasoning.output_tokens", reas)
        elif kind == "TOOL":
            a.setdefault("gen_ai.operation.name", "execute_tool")
            a.setdefault("gen_ai.tool.name", str(a.get("tool.name") or span.name))
        if getattr(span, "_attributes", None) is not None:
            span._attributes.update(a)
        # OpenInference marks success as OK; OTel asks instrumentations to leave it UNSET,
        # and the console counts any non-UNSET status as an error.
        if span.status.status_code == StatusCode.OK:
            span._status = Status(StatusCode.UNSET)


def install_genai_bridge() -> None:
    """Prepend GenAIBridgeSpanProcessor to the active TracerProvider before export."""
    import opentelemetry.trace

    tp = opentelemetry.trace.get_tracer_provider()
    multi = getattr(tp, "_active_span_processor", None)
    if multi is not None and hasattr(multi, "_span_processors"):
        if not any(type(p).__name__ == "GenAIBridgeSpanProcessor" for p in multi._span_processors):
            multi._span_processors = [GenAIBridgeSpanProcessor(), *multi._span_processors]
```

### Step 3: One SQL View for Both Dialects (BigQuery)

```sql
-- Day 22 · One schema for every agent (v2, tested on real spans)
--
-- Three agents wrote spans in three shapes, through the same OTLP pipe:
--   ADK native (adk web --otel_to_cloud)   gen_ai.* + gcp.vertex.agent.*
--   ADK + OpenInference                    openinference.span.kind + input/output.value
--   LangGraph + OpenInference              same vocabulary, twice the spans
-- One view makes them one table: one row per span, OpenInference names.
--
-- Where it runs:
--   Logging > Log Analytics:  FROM `PROJECT_ID.global._Trace.Spans._AllSpans`
--   BigQuery (zero copy):     gcloud logging links create agent_spans --bucket=_Trace --location=global
--                             FROM `PROJECT_ID.agent_spans._AllSpans`

CREATE OR REPLACE VIEW `PROJECT_ID.agent_eval.v_spans_openinference` AS
WITH spans AS (
  SELECT
    trace_id, span_id, parent_span_id, name, start_time, end_time,
    duration_nano / 1e6 AS duration_ms,
    JSON_VALUE(resource.attributes, '$."service.name"') AS service_name,
    attributes AS a
  FROM `PROJECT_ID.agent_spans._AllSpans`
),
typed AS (
  SELECT
    *,
    -- 1. What kind of step is this? OpenInference says so; ADK's span names say so.
    CASE
      WHEN JSON_VALUE(a, '$."openinference.span.kind"') IS NOT NULL
        THEN JSON_VALUE(a, '$."openinference.span.kind"')
      WHEN STARTS_WITH(name, 'invoke_agent') THEN 'AGENT'
      WHEN name = 'execute_tool (merged)' THEN 'CHAIN'   -- ADK bookkeeping for parallel calls
      WHEN STARTS_WITH(name, 'execute_tool') THEN 'TOOL'
      WHEN STARTS_WITH(name, 'call_llm') THEN 'LLM'
      WHEN STARTS_WITH(name, 'generate_content') THEN 'LLM'
      ELSE 'CHAIN'
    END AS span_kind,
    -- ADK native writes usage on call_llm AND on its generate_content child; count once.
    STARTS_WITH(name, 'generate_content') AS is_duplicate_usage,
    -- 2. Which model, how many tokens
    COALESCE(JSON_VALUE(a, '$."llm.model_name"'),
             JSON_VALUE(a, '$."gen_ai.request.model"')) AS model,
    COALESCE(SAFE_CAST(JSON_VALUE(a, '$."llm.token_count.prompt"') AS INT64),
             SAFE_CAST(JSON_VALUE(a, '$."gen_ai.usage.input_tokens"') AS INT64)) AS input_tokens,
    -- Output tokens mean different things: gen_ai.usage.output_tokens already includes
    -- thinking; llm.token_count.completion is the visible answer only, with thinking in
    -- llm.token_count.completion_details.reasoning. Normalize to "everything generated".
    COALESCE(SAFE_CAST(JSON_VALUE(a, '$."gen_ai.usage.output_tokens"') AS INT64),
             SAFE_CAST(JSON_VALUE(a, '$."llm.token_count.completion"') AS INT64)
               + IFNULL(SAFE_CAST(JSON_VALUE(a, '$."llm.token_count.completion_details.reasoning"') AS INT64), 0)) AS output_tokens,
    COALESCE(SAFE_CAST(JSON_VALUE(a, '$."gen_ai.usage.reasoning.output_tokens"') AS INT64),
             SAFE_CAST(JSON_VALUE(a, '$."llm.token_count.completion_details.reasoning"') AS INT64)) AS thinking_tokens,
    -- 3. Which tool, with what arguments
    COALESCE(JSON_VALUE(a, '$."tool.name"'),
             JSON_VALUE(a, '$."gen_ai.tool.name"')) AS tool_name,
    COALESCE(JSON_VALUE(a, '$."tool.parameters"'),
             JSON_VALUE(a, '$."gcp.vertex.agent.tool_call_args"')) AS tool_args,
    -- 4. What went in and out (only present when content capture is on)
    COALESCE(JSON_VALUE(a, '$."input.value"'),
             JSON_VALUE(a, '$."gcp.vertex.agent.llm_request"')) AS raw_input,
    COALESCE(JSON_VALUE(a, '$."output.value"'),
             JSON_VALUE(a, '$."gcp.vertex.agent.llm_response"'),
             JSON_VALUE(a, '$."gcp.vertex.agent.tool_response"')) AS raw_output,
    COALESCE(JSON_VALUE(a, '$."session.id"'),
             JSON_VALUE(a, '$."gen_ai.conversation.id"'),
             JSON_VALUE(a, '$."gcp.vertex.agent.session_id"')) AS own_session_id
  FROM spans
)
SELECT
  trace_id, span_id, parent_span_id, service_name,
  name AS span_name, span_kind, start_time, duration_ms,
  model, input_tokens, output_tokens, thinking_tokens, is_duplicate_usage,
  IF(span_kind = 'TOOL', tool_name, NULL) AS tool_name,
  IF(span_kind = 'TOOL', tool_args, NULL) AS tool_args,
  -- 5. Who was talking: tool spans don't carry the session, the root span does.
  COALESCE(own_session_id,
           FIRST_VALUE(own_session_id IGNORE NULLS)
             OVER (PARTITION BY trace_id ORDER BY start_time)) AS session_id,
  -- 6. PII: scrub before anyone reads it. (Production: Sensitive Data Protection
  --    de-identify templates on the ingestion path, not a regex in a view.)
  REGEXP_REPLACE(raw_input,  r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}', '[email]') AS input_value,
  REGEXP_REPLACE(raw_output, r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}', '[email]') AS output_value
FROM typed;
```

## Resources & Links

- **#1 Featured Cross-Promotion: [Featured Perspective: AI Agent Clinic, evaluating any AI agent (Google Cloud Tech)](https://www.youtube.com/watch?v=wPdoZRbvaF4)** — Why telemetry matters downstream: from traces to evals that catch what eyeballing misses.
- **[GitHub Repo: agent-telemetry-side-by-side (code, comparison app, 15-line bridge & SQL)](https://g.dev/cloud/adventofagents-season3-day22-demo)** — Start here: a five-step README walkthrough, runnable ADK and LangGraph agents, the trace comparison web app, the 15-line GenAIBridgeSpanProcessor and the BigQuery SQL views.
- **[Set up tracing for Agent Runtime](https://g.dev/cloud/adventofagents-season3-docs-agent-runtime-tracing)** — Turn on traces for ADK, LangChain, LangGraph, LlamaIndex or custom agents deployed to Agent Runtime.
- **[Query and analyze traces](https://g.dev/cloud/adventofagents-season3-docs-trace-analytics)** — Run SQL over every span in the project, in Log Analytics or through a BigQuery linked dataset.
- **[Google Cloud Next: Google Cloud and Arize unify agent observability](https://www.youtube.com/watch?v=nLH0IqHLxaA)** — Ameer Abbas (Google Cloud) and Jason Lopatecki (Arize AI) on converging OpenTelemetry GenAI and OpenInference for production agents.