OpenTelemetry Policy

The OpenTelemetry AI Policy provides metrics, traces, and logs in the OpenTelemetry format and can be used with any OpenTelemetry compatible backend.

You can configure a OpenTelemetry AI Policy to apply globally across every resource on your AI Gateway or only on resources where it is referenced.

The OpenTelemetry AI Policy allows you to collect data for the following signals:

• Metrics
• Tracing
• Logging

Common use cases for the OpenTelemetry AI Policy:

To set up an OpenTelemetry backend for AI Gateway, you need support for OTLP over HTTP with Protobuf encoding. You can:

• Send data directly to an OpenTelemetry-compatible backend that natively supports OTLP over HTTP with Protobuf encoding, like Jaeger (v1.35.0+).

  This is the simplest setup, since it doesn’t require any additional components between the data plane and the backend.

• Use the OpenTelemetry Collector, which acts as an intermediary between the data plane and one or more backends.

  OTEL Collector can receive all OpenTelemetry signals supported by the AI Gateway OpenTelemetry Policy, including traces, metrics, and logs, and then process, transform, or route that data before exporting it to a compatible backend.

  This option is useful when you need capabilities such as signal fan-out, filtering, enrichment, batching, or exporting to multiple backends. The OpenTelemetry Collector supports a wide range of exporters, available at open-telemetry/opentelemetry-collector-contrib.

See the Kong Gateway tracing documentation for more details about OpenTelemetry and tracing in AI Gateway.

Resource attributes describe the entity that produced the telemetry and are shared across all signals.

You can add or override resource attributes by configuring the config.resource_attributes parameter. Custom resource attributes are merged with the default attributes and are included with all exported telemetry data. Some metric backends, such as Prometheus, apply resource attributes to every metric. Be mindful of the impact on cardinality.

Metrics are natively supported by the OpenTelemetry Policy. To send metrics, set the following config.metrics parameters:

Some metrics have additional requirements:

• gen_ai.server.request.duration and mcp.client.operation.duration require config.metrics.enable_latency_metrics set to true in the OpenTelemetry AI Policy.
• The error.type attribute on duration metrics requires config.metrics.enable_request_metrics set to true in the OpenTelemetry AI Policy.

For all available metrics, see the OpenTelemetry metrics reference.

Built-in tracing instrumentations

AI Gateway has a series of built-in tracing instrumentations which are configured by the tracing_instrumentations parameter. AI Gateway creates a top-level span for each request by default when tracing_instrumentations is enabled.

The top level span has the following attributes:

• http.method: HTTP method
• http.url: HTTP URL
• http.host: HTTP host
• http.scheme: HTTP scheme (http or https)
• http.flavor: HTTP version
• net.peer.ip: Client IP address

For more information, see the Tracing reference.

Note: When the OpenTelemetry Policy is used together with the Proxy Cache Advanced Policy, cache-HIT responses are not traced. This is expected behavior. When a request results in a cache-HIT, the response is served before the request lifecycle reaches the phase where the OpenTelemetry Policy executes. As a result, no spans are generated for cache-HIT requests. Cache-MISS requests continue through the full request lifecycle and are traced normally.

Gen AI tracing attributes

AI-specific span attributes are emitted following the OpenTelemetry Gen AI semantic conventions. These attributes capture model parameters, token usage, and tool-call metadata.

Propagation

The OpenTelemetry AI Policy supports propagation of the following header formats:

• w3c: W3C trace context
• b3 and b3-single: Zipkin headers
• jaeger: Jaeger headers
• ot: OpenTracing headers
• datadog: Datadog headers
• aws: AWS X-Ray header
• gcp: GCP X-Cloud-Trace-Context header

This AI Policy offers extensive options for configuring tracing header propagation, providing a high degree of flexibility. You can customize which headers are used to extract and inject tracing context. Additionally, you can configure headers to be cleared after the tracing context extraction process, enabling a high level of customization.

 
flowchart LR
   id1(Original Request) --> Extract
   id1(Original Request) -->|"headers (original)"| Extract
   id1(Original Request) --> Extract
   subgraph ide1 [Headers Propagation]
   Extract --> Clear
   Extract -->|"headers (original)"| Clear
   Extract --> Clear
   Clear -->|"headers (filtered)"| Inject
   end
   Extract -.->|extracted ctx| id2((tracing logic))
   id2((tracing logic)) -.->|updated ctx| Inject
   Inject -->|"headers (updated ctx)"| id3(Updated request)
  

See the Policy’s configuration reference for a complete overview of the available options and values.

Note: If any of the config.propagation.* configuration options (extract, clear, or inject) are configured, the config.propagation configuration takes precedence over the deprecated header_type parameter. If none of the config.propagation.* configuration options are set, the header_type parameter is still used to determine the propagation behavior.

OTLP exporter

The OpenTelemetry AI Policy implements the OTLP/HTTP exporter, which uses Protobuf payloads encoded in binary format and is sent via an HTTP/1.1.

config.connect_timeout, config.read_timeout, and config.send_timeout are used to set the timeouts for the HTTP request.

config.batch_span_count and config.batch_flush_delay are used to set the maximum number of spans and the delay between two consecutive batches.

Create a custom span

The OpenTelemetry AI Policy is built on top of the AI Gateway tracing PDK. You can customize the spans and add your own spans through the universal tracing PDK.

1. Create a file named custom-span.lua with the following content:

   -- Modify the root span
   local root_span = kong.tracing.get_root_span()
   root_span:set_attribute("custom.attribute", "custom value")
   
   -- Modify the active span
   local active_span = kong.tracing.active_span()
   active_span:set_attribute("custom.attribute", "custom value")
   
   -- Create a custom span
   local span = kong.tracing.start_span("custom-span")
   
   -- Append attributes
   span:set_attribute("custom.attribute", "custom value")
   
   -- Close the span
   span:finish()

   Copied!

2. Load the file into an environment variable:

   export CUSTOM_SPAN_LUA=$(cat custom-span.lua)

   Copied!

3. Apply the Lua code with the Post-function Policy:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Post Function - Custom Span",
      "name": "post-function",
      "type": "post-function",
      "config": {
        "access": [
          "'$CUSTOM_SPAN_LUA'"
        ]
      }
    }
    '

ai_gateway_policies:
  - ref: post-function
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Post Function - Custom Span
    name: post-function
    type: post-function
    config:
      access:
      - !env CUSTOM_SPAN_LUA

This AI Policy supports OpenTelemetry Logging, which can be configured as described in the configuration reference to export logs in OpenTelemetry format to an OTLP-compatible backend.

Log scopes

Two different kinds of logs are exported:

• API transactional logs (also known as access logs) represent metadata about client requests. These access logs are produced during the request lifecycle. These logs typically don’t have a severity.
• Runtime and error logs aren’t directly associated with a request. They’re produced by the data plane and provide data about its internal execution. For example, they could be logs generated asynchronously (in a timer) or during a worker’s startup.

Log level

Logs are recorded based on the log level that is configured for AI Gateway. If a log is emitted with a level that is lower than the configured log level, it is not recorded or exported.

Note: Not all logs are guaranteed to be recorded. Logs that aren’t recorded include those produced by the Nginx master process and low-level errors produced by Nginx. Operators are expected to still capture the Nginx error.log file (which always includes all such logs) in addition to using this feature, to avoid losing any details that might be useful for deeper troubleshooting.

Runtime and error log entry

Each log entry adheres to the OpenTelemetry Logs Data Model. The available information depends on the log scope and on whether tracing is enabled for this Policy.

Every log entry includes the following fields:

• Timestamp: Time when the event occurred.
• ObservedTimestamp: Time when the event was observed.
• SeverityText: The severity text (log level).
• SeverityNumber: Numerical value of the severity.
• Body: The error log line.
• Resource: Configurable resource attributes.
• InstrumentationScope: Metadata that describes AI Gateway’s data emitter.
• Attributes: Additional information about the event.
  • introspection.source: Full path of the file that emitted the log.
  • introspection.current.line: Line number that emitted the log.

In addition to the above, request-scoped logs include:

• Attributes: Additional information about the event.
  • request.id: AI Gateway’s request ID.

In addition to the above, when tracing is enabled, request-scoped logs include:

• TraceID: Request trace ID.
• SpanID: Request span ID.
• TraceFlags: W3C trace flag.

The OpenTelemetry Policy uses internal queues to decouple the production of log entries from their transmission to the upstream server.

With queuing, entries are placed in a configurable queue before being sent in batches to the upstream server. This has the following benefits:

• Reduces concurrency on the upstream server
• Helps deal with temporary outages of the upstream server due to network or administrative changes
• Can reduce resource usage both in AI Gateway and on the upstream server by collecting multiple entries from the queue in one request

Note: Because queues are structural elements for components in AI Gateway, they only live in the main memory of each worker process and aren’t shared between workers. Therefore, queued content isn’t preserved under abnormal operational situations, like power loss or unexpected worker process shutdown due to memory shortage or program errors.

You can configure several parameters for queuing:

When the OpenTelemetry AI Policy is configured along with a Policy that uses the Log Serializer, the trace ID of each request is added to the key trace_id in the serialized log output.

The value of this field is an object that can contain different formats of the current request’s trace ID. In case there are multiple tracing headers in the same request, the trace_id field includes one trace ID format for each different header format, as in the following example:

"trace_id": {
  "w3c": "4bf92f3577b34da6a3ce929d0e0e4736",
  "datadog": "11803532876627986230"
},

The config.access_logs.custom_attributes_by_lua configuration lets you dynamically modify access log fields using Lua code. This field is only valid when config.access_logs.endpoint is also set. The following example configuration removes the route field from the access logs:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "OpenTelemetry - Remove Field",
      "name": "opentelemetry",
      "type": "opentelemetry",
      "config": {
        "access_logs": {
          "endpoint": "http://my-access-log-endpoint:9999/",
          "custom_attributes_by_lua": {
            "route": "return nil"
          }
        }
      }
    }
    '

ai_gateway_policies:
  - ref: opentelemetry
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: OpenTelemetry - Remove Field
    name: opentelemetry
    type: opentelemetry
    config:
      access_logs:
        endpoint: http://my-access-log-endpoint:9999/
        custom_attributes_by_lua:
          route: return nil

New fields can be added the same way:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "OpenTelemetry - Add Field",
      "name": "opentelemetry",
      "type": "opentelemetry",
      "config": {
        "access_logs": {
          "endpoint": "http://my-access-log-endpoint:9999/",
          "custom_attributes_by_lua": {
            "header": "return kong.request.get_header('h1')"
          }
        }
      }
    }
    '

ai_gateway_policies:
  - ref: opentelemetry
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: OpenTelemetry - Add Field
    name: opentelemetry
    type: opentelemetry
    config:
      access_logs:
        endpoint: http://my-access-log-endpoint:9999/
        custom_attributes_by_lua:
          header: return kong.request.get_header('h1')

Dot characters (.) in the field key create nested fields. Use a backslash \ to escape a dot if you want to keep it as part of a flat field name instead of nesting it. For example, [my_entry.log\.field] produces a my_entry object with a single log.field key, instead of nesting into log and field.

Targeting AI Gateway fields

AI Gateway logs the outcome of an LLM request under a nested ai object, for example ai.$POLICY_NAME.meta, ai.$POLICY_NAME.usage, and, when payload logging is enabled, ai.$POLICY_NAME.payload.request and ai.$POLICY_NAME.payload.response. Because custom_attributes_by_lua keys are split into nested table accesses the same way, you can use the same unescaped, dotted-key syntax to remove or override those fields.

For example, to stop logging LLM request and response payloads:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "OpenTelemetry - Disable Payload Logging",
      "name": "opentelemetry",
      "type": "opentelemetry",
      "config": {
        "access_logs": {
          "endpoint": "http://my-access-log-endpoint:9999/",
          "custom_attributes_by_lua": {
            "ai.opentelemetry.payload.request": "return nil"
          }
        }
      }
    }
    '

ai_gateway_policies:
  - ref: opentelemetry
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: OpenTelemetry - Disable Payload Logging
    name: opentelemetry
    type: opentelemetry
    config:
      access_logs:
        endpoint: http://my-access-log-endpoint:9999/
        custom_attributes_by_lua:
          ai.opentelemetry.payload.request: return nil

Note: Escaping the dots (for example, ai\.opentelemetry\.payload\.request) targets a literal flat key instead of the nested ai.opentelemetry.payload.request field, so it won’t match. Use unescaped dots to target AI Gateway fields.

Because the OpenTelemetry Policy applies custom_attributes_by_lua in its own log phase, which runs after AI Gateway sets the ai.* fields on the request, it can override or remove any ai.* field. The reverse isn’t possible, since AI Gateway can’t run after the Policy’s log phase to override a field the Policy already set.

The OpenTelemetry Policy doesn’t use the same table for logging as the other logging Policies. This is because it uses config.access_logs.custom_attributes_by_lua instead of config.custom_fields_by_lua.

Limitations

Lua code runs in a restricted sandbox environment, whose behavior is governed by the untrusted_lua configuration.

As this code runs in the log phase, only PDK methods that can run in that phase can be used.

The OpenTelemetry spans are printed to the console when the log level is set to debug in the AI Gateway configuration file.

The following is an example of the debug logs output:

2022/06/02 15:28:42 [debug] 650#0: *111 [lua] instrumentation.lua:302: runloop_log_after(): [tracing] collected 6 spans:
Span #1 name=GET /wrk duration=1502.994944ms attributes={"http.url":"/wrk","http.method":"GET","http.flavor":1.1,"http.host":"127.0.0.1","http.scheme":"http","net.peer.ip":"172.18.0.1"}
Span #2 name=rewrite phase: opentelemetry duration=0.391936ms
Span #3 name=router duration=0.013824ms
Span #4 name=access phase: cors duration=1500.824576ms
Span #5 name=cors: heavy works duration=1500.709632ms attributes={"username":"kongers"}
Span #6 name=balancer try #1 duration=0.99328ms attributes={"net.peer.ip":"104.21.11.162","net.peer.port":80}