Home / How-to Guides
Edit
Report

Monitor AI Agent traffic with OpenTelemetry

Uses: AI Gateway kongctl
Incompatible with
on-prem
Related Documentation
AI Gateway Documentation kongctl Documentation
Tags
#ai #a2a #tracing #metrics #observability
Related Resources
AI Gateway
AI Agent entity
OpenTelemetry Policy
Gen AI OpenTelemetry metrics reference
Get started with AI Agent
A2A protocol specification
Minimum Version
AI Gateway - 2.0
TL;DR

Attach an OpenTelemetry Policy to an AI Agent entity to export distributed traces and OTLP metrics for A2A traffic to a collector. Configure traces_endpoint and metrics.endpoint on the Policy to send A2A span and metric data to your observability backend.

This tutorial shows you how to create an AI Agent alongside an OpenTelemetry Policy using kongctl, send an A2A request, and validate both the resulting trace span and OTLP metrics in a local OpenTelemetry Collector.

Prerequisites

AI Gateway running

This is a Konnect tutorial and requires a Konnect personal access token.

  1. Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.

  2. Export your token to an environment variable:

    export KONNECT_TOKEN='YOUR_KONNECT_PAT'

  3. Run the AI Gateway quickstart script to automatically provision a control plane and data plane in Kong Konnect, and configure your environment:

    curl -Ls https://get.konghq.com/ai | bash -s -- -k $KONNECT_TOKEN \
   -e KONG_TRACING_INSTRUMENTATIONS=all \
   -e KONG_TRACING_SAMPLING_RATE=1.0

This sets up a AI Gateway control plane named ai-quickstart, provisions a local data plane, and prints out the following environment variables export:

export AI_GATEWAY_ID=your-gateway-id
export KONNECT_TOKEN=$KONNECT_TOKEN
export KONNECT_CONTROL_PLANE_NAME=ai-quickstart
export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
export KONNECT_PROXY_URL='http://localhost:8000'

Copy and paste these into your terminal to configure your session.

kongctl   v1.13.0+

This tutorial uses kongctl to manage Konnect resources programmatically. We recommend keeping kongctl up to date with the latest version (1.13.0).

  1. Install kongctl from developer.konghq.com/kongctl.

  2. Verify the installation:

    kongctl version

OpenAI API key
  1. Create an OpenAI account.
  2. Get an API key.
  3. Export your key: 
    export OPENAI_API_KEY='OPENAI_API_KEY'

A2A agent

You need a running A2A-compliant agent. This guide uses a sample KongAir travel agent that uses OpenAI and LangGraph to answer flight route queries.

Create a docker-compose.yaml file:

cat <<'EOF' > docker-compose.yaml
services:
  a2a-agent:
    container_name: a2a-kongair-agent
    image: ghcr.io/tomek-labuk/a2a-kongair-openai-agent:1.0.0
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - OPENAI_MODEL=gpt-5-mini
      - KONGAIR_BASE_URL=https://api.kong-air.com
      - PUBLIC_AGENT_URL=http://localhost:10000
    ports:
      - "10000:10000"
EOF

Start the agent:

docker compose up -d --wait

The agent listens on port 10000 and uses the A2A JSON-RPC protocol to handle flight route queries. In this guide, the gateway service points to host.docker.internal:10000 instead of the container name because Kong Gateway runs in its own container with a separate DNS resolver.

OpenTelemetry Collector

Launch a local OpenTelemetry Collector in the background, listening on port 4318:

docker run -d \
  --name otel-collector \
  -p 127.0.0.1:4318:4318 \
  otel/opentelemetry-collector:0.141.0

Create an AI Agent and OpenTelemetry Policy

Create an OpenTelemetry Policy that exports traces and metrics to your collector, and an AI Agent that attaches it. Setting global to false on the Policy means it only applies to entities that reference it instead of every resource on your AI Gateway, so the kongair-flight-booking-agent entity lists otel-a2a in its policies field to opt in. The service.name value under resource_attributes labels the exported data, which is useful if multiple gateways or services send to the same collector.

kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" << 'EOF'
_defaults:
  kongctl:
    namespace: ai-gateway-get-started

ai_gateway_policies:
  - ref: otel-a2a
    name: otel-a2a
    display_name: "otel-a2a"
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    type: opentelemetry
    enabled: true
    global: false
    config:
      traces_endpoint: http://host.docker.internal:4318/v1/traces
      metrics:
        endpoint: http://host.docker.internal:4318/v1/metrics
        enable_ai_metrics: true
      resource_attributes:
        service.name: kong-a2a

ai_gateway_agents:
  - ref: kongair-flight-booking-agent
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: kongair-flight-booking-agent
    display_name: "Kong Air Flight Booking Agent"
    type: a2a
    enabled: true
    policies: [ !ref otel-a2a#name ]
    config:
      url: http://host.docker.internal:10000
      route:
        paths:
          - /a2a
        methods:
          - GET
          - POST
        protocols:
          - http
          - https
        strip_path: true
      max_request_body_size: 8388608
EOF

Send an A2A request

Send a message/send JSON-RPC request to test the agent:

curl -i -X POST "$KONNECT_PROXY_URL/a2a/" \
     --no-progress-meter --fail-with-body  \
     -H "Content-Type: application/json" \
     --json '{
       "jsonrpc": "2.0",
       "id": "1",
       "method": "message/send",
       "params": {
         "message": {
           "kind": "message",
           "messageId": "msg-001",
           "role": "user",
           "parts": [
             {
               "kind": "text",
               "text": "What flights are available on route KA-123?"
             }
           ]
         }
       }
     }'

A successful response (status 200) contains the agent’s reply.

Validate traces

Search the collector’s logs for kong.a2a to find the emitted span:

docker logs otel-collector 2>&1 | grep -A 15 kong.a2a

You should see a kong.a2a span with the same shape. The Trace ID, Parent ID, ID, Start time, End time, kong.a2a.task.id, and kong.a2a.context.id values are generated per request, so yours will differ from the example:

Span #3
    Trace ID       : 1bfc19e17dd9121769882cd9b8bf5de1
    Parent ID      : de4e6ed2c16a2dd3
    ID             : 240b2b9ac3ac9e38
    Name           : kong.a2a
    Kind           : Internal
    Start time     : 2026-04-03 06:48:41.44707456 +0000 UTC
    End time       : 2026-04-03 06:48:47.140356608 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> kong.a2a.protocol.version: Str(unknown)
     -> rpc.system: Str(jsonrpc)
     -> rpc.method: Str(message/send)
     -> kong.a2a.task.id: Str(8a98bbbf-7d09-4336-b3aa-afe73e3a38d3)
     -> kong.a2a.task.state: Str(completed)
     -> kong.a2a.context.id: Str(df2e34aa-27ce-44ee-b5d3-3130b4f10985)
     -> kong.a2a.operation: Str(message/send)

The remaining attributes are fixed values you can match against directly. rpc.system is always jsonrpc, and rpc.method and kong.a2a.operation reflect the JSON-RPC method you sent, message/send in this example. kong.a2a.task.state reflects the task’s outcome, completed for a successful response. kong.a2a.protocol.version is unknown because the request didn’t carry an A2A-Version header. See AI Agent OpenTelemetry span attributes for the full attribute list.

Validate metrics

Search the collector’s logs for kong.gen_ai.a2a to find the emitted metrics:

docker logs otel-collector 2>&1 | grep -A 15 kong.gen_ai.a2a

You should see metrics with the same shape. kong.konnect.cp.id identifies your AI Gateway control plane and is unique to your environment, and the Count, Sum, and Value fields depend on how many requests you sent and their actual duration or size, so match on the overall structure rather than the exact numbers:

Metric #0
Descriptor:
     -> Name: kong.gen_ai.a2a.request.count
     -> Description: Counts A2A requests.
     -> Unit: {request}
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> kong.gen_ai.a2a.binding: Str(jsonrpc)
     -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0)
     -> kong.service.name: Str(kongair-flight-booking-agent)
     -> kong.route.name: Str(kongair-flight-booking-agent-route)
     -> kong.workspace.name: Str(default)
     -> kong.gen_ai.a2a.method: Str(message/send)
Value: 1

Metric #1
Descriptor:
     -> Name: kong.gen_ai.a2a.request.duration
     -> Description: Measures A2A request duration in seconds.
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> kong.gen_ai.a2a.binding: Str(jsonrpc)
     -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0)
     -> kong.service.name: Str(kongair-flight-booking-agent)
     -> kong.route.name: Str(kongair-flight-booking-agent-route)
     -> kong.workspace.name: Str(default)
     -> kong.gen_ai.a2a.method: Str(message/send)
Count: 1
Sum: 22.988000

Metric #2
Descriptor:
     -> Name: kong.gen_ai.a2a.response.size
     -> Description: Measures A2A response body size in bytes.
     -> Unit: By
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> kong.gen_ai.a2a.binding: Str(jsonrpc)
     -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0)
     -> kong.service.name: Str(kongair-flight-booking-agent)
     -> kong.route.name: Str(kongair-flight-booking-agent-route)
     -> kong.workspace.name: Str(default)
     -> kong.gen_ai.a2a.method: Str(message/send)
Count: 1
Sum: 1496.000000

Metric #3
Descriptor:
     -> Name: kong.gen_ai.a2a.task.state.count
     -> Description: Counts A2A task state transitions.
     -> Unit: {state}
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> kong.konnect.cp.id: Str(e221e0b2-f56d-4f30-871b-183d5c4146a0)
     -> kong.service.name: Str(kongair-flight-booking-agent)
     -> kong.route.name: Str(kongair-flight-booking-agent-route)
     -> kong.workspace.name: Str(default)
     -> kong.gen_ai.a2a.task.state: Str(completed)
Value: 1

The rest of the attributes are fixed values you can match against directly. kong.service.name and kong.route.name match the entity names you created (kong.route.name carries a -route suffix because AI Gateway auto-generates a Route for the agent). kong.workspace.name, kong.gen_ai.a2a.binding, and kong.gen_ai.a2a.method reflect your configuration and the request you sent.

See A2A metrics for the full metric reference, including kong.gen_ai.a2a.request.duration, kong.gen_ai.a2a.response.size, kong.gen_ai.a2a.ttfb, and kong.gen_ai.a2a.request.error.count.

You can also view A2A traffic metrics without setting up a collector, using Konnect Analytics:

  1. Go to Observability > Dashboards.
  2. Click Create dashboard > Create from template.
  3. Select the Agentic analytics dashboard. This dashboard highlights which tools are called most frequently, breaks down tool usage by consumer, and tracks average latency per tool over time, helping teams operating agentic services understand usage patterns and identify performance bottlenecks.
  4. Click Use template to see agent traffic volume, error rates, and other stats.

Cleanup

Stop the A2A agent and OpenTelemetry Collector

 
docker compose down
docker rm -f a2a-kongair-agent otel-collector

Clean up AI Gateway resources

To clean up all AI Gateway resources created in this guide, run:

curl -Ls https://get.konghq.com/ai | bash -s -- -d

FAQs

What logging configuration does the AI Agent need to emit A2A traces and metrics?

A2A span and metric emission itself doesn’t depend on a separate flag on the agent. The OpenTelemetry Policy’s traces_endpoint and metrics.endpoint control where that data is exported. Set config.logging.payloads to true on the AI Agent if you also want request and response bodies captured alongside the telemetry.

Can one OpenTelemetry Policy cover multiple AI Agents or AI Models?

Yes. Set global: true on the Policy to apply it to every resource on the AI Gateway instead of listing it in each entity’s policies field. Keep global: false and reference the Policy by name (or !ref) from each entity’s policies array to scope export to specific agents or models.

What span attributes does AI Gateway emit for A2A traffic?

A kong.a2a child span carries attributes like kong.a2a.operation, kong.a2a.task.id, kong.a2a.task.state, and kong.a2a.context.id. See the full list in AI Agent OpenTelemetry span attributes.

What metrics are available for A2A traffic?

Counters and histograms in the kong.gen_ai.a2a.* namespace cover request volume, duration, response size, time to first byte, errors, and task state transitions. See A2A metrics for the full reference.

Something wrong?
Report an Issue | Edit this Page
View all Kong documentation

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!
Contribute

Still need help?

Ask in our Slack Community
Contact Support