Configure Kong AI Gateway on-prem

The existing AI Gateway documentation follows the Konnect model: AI Models, AI Model Providers, AI MCP Servers, AI Agents, and more, that you configure in Konnect. See AI Gateway entities for the full list. Self-hosted Kong Gateway doesn’t have these objects. This page shows you how to configure the same capabilities with AI plugins on Services and Routes.

There are two ways to bridge that gap:

• Read How entities translate to Kong Gateway configuration to translate what you read in AI Gateway docs into the plugin fields you configure by hand.
• Convert an existing AI Gateway 2.0 decK configuration into the equivalent self-hosted config, instead of hand-writing the plugins.

Both deployments run the same Kong Gateway Services, Routes, Plugins, and Consumers (what we call primitives). When you save an AI entity in Konnect, AI Gateway generates the primitives that data planes run. On-prem, you produce those same primitives yourself, either by writing the plugin config directly or by converting an entity-model file.

AI entities are a high-level abstraction. When you create one, a dedicated conversion step translates it into the same Routes, Services, plugins, and Consumers that self-hosted Kong Gateway uses, and that’s what data plane nodes actually run.

This mapping isn’t always 1:1. Some entities carry over almost directly, others become more than one Kong Gateway object, and a few have no self-hosted equivalent at all.

The following table shows how Konnect AI Gateway entities map to on-prem Kong Gateway configuration.

On-prem, a client sends requests to Kong Gateway, where a Service and Route carry the AI plugin. The plugin applies the AI behavior and proxies the request to the upstream, whether that’s an LLM provider, an MCP server, or an agent.

 
flowchart LR
  Client(Client)
  subgraph Gateway["Kong Gateway"]
    P1[AI Proxy Advanced]
    P2[AI MCP Proxy]
    P3[AI A2A Proxy]
  end
  LLM(LLM providers)
  MCP(MCP servers)
  Agent(Agents)
  Client --> P1 --> LLM
  Client --> P2 --> MCP
  Client --> P3 --> Agent
  

Figure 1: On-prem, AI Gateway capabilities are delivered by plugins on Kong Gateway, each proxying to its upstream.

Use the AI Proxy Advanced (ai-proxy-advanced) plugin to transform and proxy requests to multiple AI providers and models at the same time, and to load balance across targets.

• On Konnect: An AI Model generates a Service, one Route per capability it serves (chat completions, embeddings, and so on), and the plugins on each Route (ai-model-selector and ai-proxy-advanced).
• On-prem: You create one Route per capability you want to expose, each carrying its own ai-proxy-advanced plugin.

The AI Model Provider referenced by a target has no plugin of its own. Set its type and auth in the corresponding targets entry of the ai-proxy-advanced plugin.

Use the AI MCP Proxy (ai-mcp-proxy) plugin to convert APIs into MCP tools, proxy MCP servers, expose MCP tools to AI clients, and observe MCP traffic.

• On Konnect: An AI MCP Server generates one or more Routes, each carrying an ai-mcp-proxy plugin. The number of Routes, and whether the plugin converts a REST API into MCP tools or proxies an existing MCP server, depends on the server mode.
• On-prem: You configure the plugin’s mode and Routes to match the topology you want.

Use the AI A2A Proxy (ai-a2a-proxy) plugin to add observability and gateway control to Agent-to-Agent (A2A) protocol traffic. The plugin supports both JSON-RPC and REST bindings, and always applies A2A protocol handling, agent card URL rewriting, and A2A telemetry. It has no mode to turn that off.

On Konnect, an AI Agent with type: a2a generates:

• One Service
• One Route
• One ai-a2a-proxy plugin

This matches what you configure on-prem. An AI Agent with type: http generates a Service and Route with no AI plugin, since type: http is a plain transparent proxy. Replicate it on-prem with a Service and Route and no plugin attached.

Use the key-auth or openid-connect auth strategy, backed by the key-auth and openid-connect Kong Gateway plugins, to authenticate the clients calling your AI routes.

• On Konnect: An AI Auth Strategy generates a key-auth or openid-connect plugin on the Route of every AI Model or AI Agent that references it, plus a shared anonymous Consumer carrying a request-termination plugin that returns 401 for requests that don’t authenticate.
• On-prem: You attach the corresponding plugin to each Route yourself, and configure the anonymous Consumer and request-termination plugin to match.

An AI Policy applies a specific behavior, named by its type (for example, ai-prompt-guard or ai-rate-limiting-advanced), either globally or scoped to whatever it’s attached to: an AI Model, AI Agent, AI MCP Server, AI Consumer, or AI Consumer Group. On Konnect, an AI Policy generates the Kong Gateway plugin named by that type, applied at the same scope.

On-prem, each Policy-backed plugin’s configuration maps 1:1 to an AI Gateway Policy. The AI Policy fields and the plugin fields are the same.

Access control and secret management on-prem use the same Kong Gateway objects as any other Gateway configuration, so the objects themselves need no AI-specific setup. Use the existing Kong Gateway documentation:

• Consumers: Authenticate the clients that call your AI routes. For Consumer-scoped behavior, such as OpenID Connect authentication, attach the plugin directly to that Consumer.
• Consumer Groups: Apply shared rate limits and policies to groups of Consumers by attaching the plugin to the Consumer Group.
• Vaults: Store and reference provider credentials.

AI Data Plane Certificates have no on-prem equivalent. They authorize Konnect-managed data planes to connect to a specific AI Gateway. Self-hosted hybrid mode deployments use the standard Certificate entity and hybrid mode node configuration instead.

AI Model, AI MCP Server, AI Agent, and AI Policy entities need conversion, since each one generates Kong Gateway Services, Routes, or plugins. AI Consumers, AI Consumer Groups, and AI Vaults convert with only field renames (for example, a Consumer’s name becomes username), since they’re already shaped like their native Kong Gateway equivalents.

decK’s file ai2kong, ai sync, and ai dump all translate the same AI Gateway 2.x configuration between its entity-model representation and its self-hosted plugin representation. They don’t change versions, only where the config runs. kongctl convert ai-gateway does something different: it upgrades an older AI Gateway running on Kong Gateway plugin configuration to the 2.x entity model, for Konnect.

The rest of this section uses deck file ai2kong to show exactly what config each entity generates.

Conversion workflow

Every example that follows uses the same two-step conversion, shown once here. Write your AI Gateway 2.0 entity configuration to ai.yaml, then:

1. Convert it to Kong Gateway 3.x config:

   deck file ai2kong --source ai.yaml --output-file kong.yaml

   Copied!

2. Sync the converted config to your self-hosted Kong Gateway:

   deck gateway sync kong.yaml

   Copied!

To skip the intermediate file, run deck ai sync ai.yaml instead of both steps.

The following sections show what ai.yaml and the resulting kong.yaml look like for each entity type.

Convert an AI Model

The following AI Model, gpt-5-2, exposes the generate capability on /ai and routes to a single target backed by the openai-prod AI Model Provider:

# ai.yaml (AI Gateway 2.0 entity model)
models:
  - name: gpt-5-2
    capabilities:
      - generate
    formats:
      - type: openai
    config:
      route:
        paths:
          - /ai
        model:
          body_param: model
          values: ["gpt-5.2"]
    targets:
      - name: gpt-5.2
        provider: openai-prod
        config:
          type: openai
          temperature: 1.0
          max_tokens: 1024
model_providers:
  - name: openai-prod
    type: openai
    config:
      auth:
        type: basic
        headers:
          - name: Authorization
            value: "{vault://ai/openai-token}"

Converting it generates a Service, a Route, and an ai-proxy-advanced plugin on that Route:

# kong.yaml (self-hosted Kong Gateway 3.x)
_format_version: "3.0"
_info:
  select_tags:
  - 'managed_by:deck-ai'
ai_models:
- name: gpt-5-2
plugins:
- config:
    body_path: model
    max_request_body_size: 8388608
    source: body
  name: ai-model-selector
  route: openai-chat
- config:
    balancer:
      algorithm: round-robin
    genai_category: text/generation
    llm_format: openai
    targets:
    - auth:
        header_name: Authorization
        header_value: '{vault://ai/openai-token}'
      description: gpt-5.2
      logging:
         log_payloads: false
         log_statistics: true
      model:
        model_alias: 'gpt-5.2'
        name: gpt-5.2
        options:
          max_tokens: 1024
          temperature: 1
        provider: openai
      route_type: llm/v1/chat
  model: gpt-5-2
  name: ai-proxy-advanced
  route: openai-chat
services:
- name: ai-gateway
  routes:
  - methods:
    - POST
    name: openai-chat
    paths:
    - /ai/chat/completions
    strip_path: false
  url: http://ai-gateway.upstream.local

The AI Model Provider generates no object of its own. Its type becomes the target’s model.provider, and its auth is materialized into the same ai-proxy-advanced target.

The converted output uses the model_alias field name from the self-hosted ai-proxy-advanced plugin schema. This is distinct from config.route.model on the AI Gateway entity shown in the input; deck file ai2kong handles the translation between the two.

Convert an AI MCP Server

The following AI MCP Server, demo-mcp, uses conversion-listener mode to expose a single get-item tool that proxies a REST GET request:

# ai.yaml (AI Gateway 2.0 entity model)
mcp_servers:
  - ref: demo-mcp
    name: demo-mcp
    display_name: "Demo MCP Server"
    type: conversion-listener
    enabled: true
    access:
      acl_attribute_type: consumer
    config:
      url: https://mock.example.com
      route:
        paths:
          - /demo-mcp
    tools:
      - name: get-item
        description: Fetch an item by id
        annotations:
          title: Fetch an item by id
        method: GET
        path: /items/{itemId}
        parameters:
          - name: itemId
            in: path
            description: The item id
            required: true
            schema:
              type: string

Converting it generates a Service, a Route, and an ai-mcp-proxy plugin carrying the tool definition:

# kong.yaml (self-hosted Kong Gateway 3.x)
_format_version: "3.0"
_info:
  select_tags:
  - 'managed_by:deck-ai'
services:
- host: localhost
  name: demo-mcp
  routes:
  - name: demo-mcp-route
    paths:
    - /demo-mcp
    plugins:
    - config:
        acl_attribute_type: consumer
        include_consumer_groups: true
        logging:
          log_audits: false
          log_payloads: false
          log_statistics: true
        mode: conversion-listener
        tools:
        - annotations:
            title: Fetch an item by id
          description: Fetch an item by id
          method: GET
          name: get-item
          parameters:
          - description: The item id
            in: path
            name: itemId
            required: true
            schema:
              type: string
          path: /items/{itemId}
      name: ai-mcp-proxy

Convert an AI Agent

The following AI Agent, demo-agent, proxies Agent-to-Agent traffic to an upstream agent:

# ai.yaml (AI Gateway 2.0 entity model)
agents:
  - ref: demo-agent
    name: demo-agent
    type: a2a
    display_name: "Demo Agent"
    enabled: true
    labels:
      team: demo
    config:
      url: https://agent.example.com
      route:
        paths:
          - /agents/demo
      logging:
        max_payload_size: 524288

Converting it generates a Service, a Route, and an ai-a2a-proxy plugin on that Route:

# kong.yaml (self-hosted Kong Gateway 3.x)
_format_version: "3.0"
_info:
  select_tags:
  - 'managed_by:deck-ai'
services:
- name: demo-agent
  routes:
  - name: demo-agent-route
    paths:
    - /agents/demo
    plugins:
    - config:
        logging:
          log_payloads: false
          log_statistics: true
          max_payload_size: 524288
      name: ai-a2a-proxy
  tags:
  - team:demo
  url: https://agent.example.com

An AI Agent with type: http converts the same way, minus the ai-a2a-proxy plugin: just the Service and Route.

Convert an AI Policy

An AI Policy generates no object of its own. It must be attached to another entity, such as an AI Model, AI Agent, or AI MCP Server, and converts into the Kong Gateway plugin named by the policy type, scoped to that entity’s Route.

The following ai.yaml defines a minimal chat model, gpt-5-2, with an ai-gw-prompt-guard policy attached:

# ai.yaml (AI Gateway 2.0 entity model)
models:
  - name: gpt-5-2
    capabilities:
      - generate
    formats:
      - type: openai
    config:
      route:
        paths:
          - /ai
        model:
          body_param: model
          values: ["gpt-5.2"]
    policies:
      - ai-gw-prompt-guard
    targets:
      - name: gpt-5.2
        provider: openai-prod
        config:
          type: openai
model_providers:
  - name: openai-prod
    type: openai
    config:
      auth:
        type: basic
        headers:
          - name: Authorization
            value: "{vault://ai/openai-token}"
policies:
  - ref: ai-gw-prompt-guard
    name: ai-gw-prompt-guard
    display_name: "AI Prompt Guard"
    type: ai-prompt-guard
    enabled: true
    config:
      deny_patterns:
        - ".*(W|w)ar.*"

Converting it generates the same Service, Route, ai-model-selector, and ai-proxy-advanced plugins as converting an AI Model, plus an ai-prompt-guard plugin on the same Route, scoped to the gpt-5-2 model:

# kong.yaml (self-hosted Kong Gateway 3.x)
- config:
    deny_patterns:
    - .*(W|w)ar.*
  model: gpt-5.2
  name: ai-prompt-guard
  route: openai-chat

The model and route fields on the plugin scope it to that AI Model’s traffic instead of applying globally.