Kong AI Gateway with Kong Operator

Kong Operator manages Kong AI Gateway using a set of Kubernetes Custom Resource Definitions (CRDs). Each CRD maps to a concept in the AI Gateway control plane; you declare the desired state in Kubernetes, and Kong Operator reconciles it with Konnect.

Kong AI Gateway features ship in Kong Operator rapid releases. Use Kong Operator 2.4.0-rapid.3.1, chart version 1.5.0-rapid.3.1, or later. Rapid releases are fully supported production releases. For more information, see Rapid releases.

Kong Operator manages three distinct layers:

• Control plane: You can either use KonnectAIGateway (which provisions and owns the AI Gateway control plane in Konnect) or OnPremAIGateway in tech preview (which provisions a AI Gateway 2.x on-prem). All other resources reference it as their parent.
• Configuration resources: AIGatewayModelProvider, AIGatewayModel, AIGatewayPolicy, AIGatewayAuthStrategy, AIGatewayConsumer, AIGatewayConsumerCredential, AIGatewayConsumerGroup, and AIGatewayAgent declare what the gateway does: which LLM providers to connect to, which model routes to expose, what Policies to enforce, which authentication schemes to accept, and which clients may access it.
• Data plane: AIGatewayDataPlaneCertificate and AIGatewayDataPlane run the traffic-handling binary inside your cluster. When you create an AIGatewayDataPlane, Kong Operator automatically provisions the mTLS certificate and registers it with the control plane.

The following table describes the resource model:

All configuration resources anchor to the KonnectAIGateway as their root via spec.aiGatewayRef. Consumer credentials attach to the AI Consumer entities, not directly to the control plane.

1. AIGatewayModelProvider.spec.aiGatewayRef → KonnectAIGateway
2. AIGatewayModel.spec.aiGatewayRef → KonnectAIGateway
3. AIGatewayModel.spec.apiSpec.model.targets[].provider → AIGatewayModelProvider (by name)
4. AIGatewayPolicy.spec.aiGatewayRef → KonnectAIGateway
5. AIGatewayAuthStrategy.spec.aiGatewayRef → KonnectAIGateway
6. AIGatewayConsumer.spec.aiGatewayRef → KonnectAIGateway
7. AIGatewayConsumerCredential.spec.aiGatewayConsumerRef → AIGatewayConsumer
8. AIGatewayConsumerGroup.spec.aiGatewayRef → KonnectAIGateway
9. AIGatewayAgent.spec.aiGatewayRef → KonnectAIGateway
10. AIGatewayMCPServer.spec.aiGatewayRef → KonnectAIGateway
11. AIGatewayMCPServer.spec.apiSpec.listener.sources[] → other AIGatewayMCPServer resources (by name; conversion-only or upstream-server type only)
12. AIGatewayDataPlaneCertificate.spec.aiGatewayRef → KonnectAIGateway
13. AIGatewayDataPlane.spec.controlPlaneRef → KonnectAIGateway

With an on-premises control plane, every spec.aiGatewayRef above points at an OnPremAIGateway instead, and AIGatewayDataPlane.spec.controlPlaneRef uses type: onpremNamespacedRef. See On-premises control plane.

Deploying on-prem Kong AI Gateway 2.x with Kong Operator is in Tech Preview. Running Kong AI Gateway with an OnPremAIGateway control plane is not supported for production use, and its resources and behavior may change between releases. It requires Kong Operator 2.4.0-rapid.3.0 or later and a Kong Gateway Enterprise license.

An OnPremAIGateway replaces the Konnect-hosted control plane with one that runs inside Kong Operator. It doesn’t create a Deployment or Pods of its own. For each OnPremAIGateway, Kong Operator runs a control plane instance that:

1. Watches the AI Gateway configuration resources whose spec.aiGatewayRef points at the OnPremAIGateway, and the Secrets they reference.
2. Translates them into AI Gateway configuration.
3. Pushes that configuration to the Admin API of every AIGatewayDataPlane attached to the OnPremAIGateway.

Nothing is sent to Konnect, so Konnect features such as analytics and the Konnect UI aren’t available. To follow the getting started series with an on-premises control plane, see Install Kong Operator for Kong AI Gateway.

Enable the on-premises controller

The OnPremAIGateway controller is disabled by default. Enable it alongside the AIGatewayDataPlane controller when installing Kong Operator:

--set env.ENABLE_CONTROLLER_AIGATEWAYDATAPLANE=true \
--set env.ENABLE_CONTROLLER_ONPREMAIGATEWAY=true

Apply a KongLicense to the cluster. Kong Operator injects the license into every AIGatewayDataPlane that references an OnPremAIGateway, and reports it in a LicenseValid condition on both resources. A missing license doesn’t block Ready: the data plane runs without a license and reports LicenseValid=False.

Reference an OnPremAIGateway

A configuration resource must set both group and kind in spec.aiGatewayRef:

spec:
  aiGatewayRef:
    group: aigateway.konghq.com
    kind: OnPremAIGateway
    namespacedRef:
      name: my-ai-gateway-cp

When group and kind are omitted, the reference defaults to group: konnect.konghq.com and kind: KonnectAIGateway. The API server accepts the resource, but no on-premises control plane picks it up: it never gets a status, and its configuration is never pushed. Check kubectl get <resource> -o jsonpath='{.spec.aiGatewayRef.kind}' if a resource has no Programmed condition at all.

You can’t change kind on an existing resource to move it between a KonnectAIGateway and an OnPremAIGateway. Delete and recreate it instead.

An AIGatewayDataPlane attaches to an OnPremAIGateway with spec.controlPlaneRef.type: onpremNamespacedRef:

spec:
  controlPlaneRef:
    type: onpremNamespacedRef
    onpremNamespacedRef:
      name: my-ai-gateway-cp

For an on-premises data plane, Kong Operator creates a ClusterIP Service named <data-plane-name>-admin on port 8444, and a TLS certificate for it. The control plane instance pushes configuration to the data plane’s Admin API through this Service using mTLS. The AIGatewayDataPlane reports the certificate in an AdminCertificateProvisioned condition. You don’t need to expose this Service outside the cluster.

Order of operations

Deploy at least one AIGatewayDataPlane before you wait on configuration resources. Until a data plane is running, configuration resources report Programmed=False with the message no Admin API endpoints discovered; configuration not pushed. Once a data plane is Ready, pending configuration is pushed and the resources become Programmed without any further action.

OnPremAIGateway settings

All spec fields are optional. An empty spec watches only the OnPremAIGateway’s own namespace and uses the operator’s default sync settings.

Watch configuration in other namespaces

To keep configuration resources in a different namespace from the OnPremAIGateway, list the namespace in spec.watchNamespaces, and create a WatchNamespaceGrant in that namespace allowing the OnPremAIGateway’s namespace to watch it:

apiVersion: aigateway.konghq.com/v1alpha1
kind: OnPremAIGateway
metadata:
  name: my-ai-gateway-cp
  namespace: kong
spec:
  watchNamespaces:
    type: list
    list:
      - ai-team-a
---
apiVersion: gateway-operator.konghq.com/v1alpha1
kind: WatchNamespaceGrant
metadata:
  name: my-ai-gateway-cp
  namespace: ai-team-a
spec:
  from:
    - group: aigateway.konghq.com
      kind: OnPremAIGateway
      namespace: kong

Resources are in ai-team-a, then set spec.aiGatewayRef.namespacedRef.namespace: kong. Without the grant, the OnPremAIGateway reports WatchNamespaceGrantValid=False and Ready=False, and its control plane instance doesn’t run.

Status conditions

OnPremAIGateway reports these conditions:

Configuration resources report a Programmed condition, as they do with Konnect. Their ID and ORGID columns stay empty because they’re never created in Konnect.

Tech Preview limitations

The following behavior was observed with Kong Operator 2.4.0-rapid.3.0 and may change:

• A single invalid resource marks the whole gateway as not ready. For example, an AIGatewayModelProvider that references a missing Secret sets ConfigurationTranslated=False and Ready=False on the OnPremAIGateway. The invalid resource is left out, and the rest of the configuration is still pushed to the data planes, so traffic for other AI Models keeps working.
• Configuration is pushed, not pulled, so new data plane Pods depend on Kong Operator. Running data plane Pods keep serving their last pushed configuration if Kong Operator becomes unavailable. A data plane Pod that starts while Kong Operator is unavailable has no configuration, isn’t ready, and doesn’t receive traffic until Kong Operator recovers and pushes configuration to it. To work around this issue, run more than one data plane replica, and avoid restarting data planes and Kong Operator at the same time.

AIGatewayModelProvider supports the following upstream LLM providers via spec.apiSpec.type:

Each resource type is covered end-to-end in the getting started series:

• AI Providers and AI Models: Deploy Kong AI Gateway covers AIGatewayModelProvider, AIGatewayModel, and AIGatewayDataPlane.
• AI Policies: Apply AI Policies covers AIGatewayPolicy, including global and model-scoped enforcement.
• AI Auth Strategies and AI Consumers: Add AI Consumers covers AIGatewayAuthStrategy, AIGatewayConsumer, AIGatewayConsumerCredential, and AIGatewayConsumerGroup.
• AI Agents: AIGatewayAgent supports a2a and http agent types. Set spec.apiSpec.type to the agent protocol and spec.apiSpec.config.url to the upstream agent URL.
• Routing and load balancing: Route to a model using a selector and Load balancing across targets cover config.route.model and config.balancer on AIGatewayModel.
• MCP tools: Exposing tools with AIGatewayMCPServer covers the five AIGatewayMCPServer types.

Kong Operator always places a model selector in front of an AI Model’s targets, even when a single AIGatewayModel is the only one on its route. By default, the selector matches the request body’s model field against the AI Model’s own name (spec.apiSpec.model.name). A request whose model value doesn’t match never reaches any target; it falls through instead of being proxied. This is why every request in the deploy and policy guides includes "model": "gpt-4o-mini": that value is what the selector uses to activate this AI Model, not just metadata for the upstream provider.

To accept a client-side alias instead of the AI Model’s own name, set config.route.model:

• values: the alias to match, as a single-item list. When omitted, the AI Model’s own name is used.
• bodyParam, headerParam, or pathParam: which part of the request to match the alias against (a JSON body property, a header, or a named regex capture group in paths, respectively). Set at most one of these; when none are set, the request body’s model property is matched by default.

The following configuration accepts a client-side alias in the request body’s model field:

config:
  route:
    paths:
      - /v1
    model:
      values:
        - my-gpt-4o

With this configuration, a client that sends "model": "my-gpt-4o" in the request body is routed to this AI Model, regardless of the upstream provider model name on its targets.

To match on a header instead of the body, set headerParam:

config:
  route:
    paths:
      - /v1
    model:
      headerParam: x-model-route
      values:
        - my-gpt-4o

A client that sends the header x-model-route: my-gpt-4o is routed to this AI Model.

config.route.model is a different mechanism from config.route.headers and config.route.paths. The latter are Kong’s native route-matching criteria. They decide whether a request reaches this AI Model’s route at all. config.route.model runs after the route matches, to select which AI Model’s targets handle the request when multiple AI Models share a route. Setting config.route.headers to gate a route by header presence does not make that header usable as a model alias source, and combining the two produces a proxy error instead of the expected routing behavior.

An AIGatewayModel can define multiple targets, and config.balancer controls how requests are distributed across them once the AI Model is selected. Set config.balancer.algorithm to one of:

See Load balancing with AI Proxy Advanced for a full description of each algorithm’s behavior and tradeoffs. The algorithms and their semantics are the same; only the configuration surface differs.

AIGatewayMCPServer exposes tools over the Model Context Protocol (MCP), either by converting a REST API’s endpoints into MCP tools, or by proxying an existing MCP server. Set spec.apiSpec.type to one of five types:

Every type shares name, displayName, enabled, labels, policies, and a config.route.paths list, alongside a type-specific tools list and config.url (the upstream to convert or proxy). listener, conversion-listener, and passthrough-listener also accept an access block, gating the endpoint by AIGatewayConsumerGroup ACLs or auth strategy, the same way spec.apiSpec.model.access does on AIGatewayModel.

The following configuration converts two REST endpoints into MCP tools, then exposes them behind an ACL-gated listener:

kind: AIGatewayMCPServer
apiVersion: aiconfiguration.konghq.com/v1alpha1
metadata:
  name: flights-tools
  namespace: kong
spec:
  aiGatewayRef:
    type: namespacedRef
    namespacedRef:
      name: my-ai-gateway-cp
  apiSpec:
    type: conversion-only
    conversion-only:
      name: flights-tools
      displayName: Flights API tools
      enabled: Enabled
      config:
        url: https://flights-api.example.com/openapi.json
        route:
          paths:
            - /mcp/flights
      tools:
        - name: search_flights
          description: Search available flights by origin, destination, and date
          method: GET
          path: /flights
        - name: get_flight_status
          description: Get real-time status for a given flight number
          method: GET
          path: /flights/{flightNumber}/status
---
kind: AIGatewayMCPServer
apiVersion: aiconfiguration.konghq.com/v1alpha1
metadata:
  name: flights-mcp-listener
  namespace: kong
spec:
  aiGatewayRef:
    type: namespacedRef
    namespacedRef:
      name: my-ai-gateway-cp
  apiSpec:
    type: listener
    listener:
      name: flights-mcp-listener
      displayName: Flights MCP listener
      enabled: Enabled
      sources:
        - flights-tools
      access:
        aclAttributeType: consumer
      config:
        route:
          paths:
            - /mcp/flights

A passthrough-listener or upstream-server follows the same shape, but its config.url points at a real MCP server instead of a REST API, and its tools entries don’t need method/path (those are inherent to the upstream server’s own tool definitions).

AIGatewayAuthStrategy configures the authentication scheme the gateway uses to verify downstream clients. Two types are supported: key-auth (API key) and openid-connect (OIDC).

The following is an example key auth AI Auth Strategy configuration:

apiVersion: aiconfiguration.konghq.com/v1alpha1
kind: AIGatewayAuthStrategy
metadata:
  name: key-auth-provider
  namespace: kong
spec:
  aiGatewayRef:
    type: namespacedRef
    namespacedRef:
      name: my-ai-gateway-cp
  apiSpec:
    type: key-auth
    key-auth:
      name: key-auth-provider
      displayName: API Key Authentication
      config:
        hideCredentials: Enabled

OIDC client secrets use a SensitiveDataSource value in a list. Store the secret in Kubernetes and reference it:

kubectl create secret generic oidc-client-secret \
  --from-literal=clientSecret=<your-client-secret> \
  -n kong
kubectl label secret oidc-client-secret konghq.com/secret=true -n kong

The following is an example OpenID Connect AI Auth Strategy configuration:

apiVersion: aiconfiguration.konghq.com/v1alpha1
kind: AIGatewayAuthStrategy
metadata:
  name: oidc-provider
  namespace: kong
spec:
  aiGatewayRef:
    type: namespacedRef
    namespacedRef:
      name: my-ai-gateway-cp
  apiSpec:
    type: openid-connect
    openid-connect:
      name: oidc-provider
      displayName: OpenID Connect Authentication
      config:
        issuer: https://your-idp.example.com/.well-known/openid-configuration
        clientID:
          - your-client-id
        clientSecret:
          - type: secretRef
            secretRef:
              name: oidc-client-secret
              key: clientSecret

Provider API keys must not appear as plain text in manifests committed to source control. The AIGatewayModelProvider config.auth fields accept a SensitiveDataSource value with two modes:

# Inline (development only — avoid committing)
value:
  type: inline
  value: "Bearer sk-xxxx"

# Secret reference (recommended for production)
value:
  type: secretRef
  secretRef:
    name: my-secret
    key: token

For teams already using a secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager), External Secrets Operator syncs secrets into Kubernetes automatically and rotates them without redeploying the AIGatewayModelProvider.

All AI Gateway CRDs expose a Programmed status condition. Check the reconciliation state of all resources at once:

kubectl get \
  konnectaigateway,aigatewaymodelprovider,aigatewaymodel,aigatewaypolicy,aigatewayauthstrategy,aigatewaydataplane \
  -n kong

With an on-premises control plane, replace konnectaigateway with onpremaigateway:

kubectl get \
  onpremaigateway,aigatewaymodelprovider,aigatewaymodel,aigatewaypolicy,aigatewayauthstrategy,aigatewaydataplane \
  -n kong

Describe any resource to see the full status and any operator error messages:

kubectl describe konnectaigateway/my-ai-gateway-cp -n kong

AI Provider not reconciling

The provider depends on the KonnectAIGateway being Programmed=True first. Check the control plane status, then verify the Konnect auth Secret it references is correctly formed.

AI Model Route unreachable

Confirm the AIGatewayDataPlane pod is running and the LoadBalancer address is assigned:

kubectl get pods,svc -n kong -l gateway-operator.konghq.com/managed-by-name=my-ai-gateway-dp

AI Policy not taking effect

Verify spec.aiGatewayRef.namespacedRef.name matches your KonnectAIGateway name exactly. Describe the AI Policy to surface any reconciliation errors:

kubectl describe aigatewaypolicy -n kong

On-premises resource has no status

A configuration resource with no conditions at all is usually missing group: aigateway.konghq.com and kind: OnPremAIGateway in spec.aiGatewayRef, so it targets a KonnectAIGateway instead. See Reference an OnPremAIGateway.

On-premises resource not programmed

Read the condition message:

kubectl get aigatewaymodelprovider/openai-provider -n kong \
  -o jsonpath='{.status.conditions[?(@.type=="Programmed")].message}'

• no Admin API endpoints discovered; configuration not pushed: An AIGatewayDataPlane that is attached to the OnPremAIGateway isn’t running yet. Deploy one, or check its Ready condition.
• failed to fetch Secret ... not found (note: the operator only reads Secrets carrying its Secret label selector, konghq.com/secret=true by default): The Secret is missing or lacks the konghq.com/secret=true label.

On-premises request returns a 500 DNS or balancer error

A request returns 500 with a message like failed the initial dns/balancer resolve for 'ai-gateway.upstream.local' with: dns server error.

The request’s model value doesn’t match any AI Model on the route, so no target was selected. Send the AI Model’s spec.apiSpec.model.name, or an alias from config.route.model.values. See Route to a model using a selector.

Kong Operator LOGS

For any resource stuck in a non-Programmed state, check the Kong Operator logs:

kubectl logs -n kong-system \
  -l app.kubernetes.io/name=kong-operator \
  --since=10m