AI Models

The AI Model entity gives clients a single endpoint for one or more upstream LLMs through AI Gateway. Callers don’t need to know which AI Provider or model handles the request. Use AI Models to:

• Expose multiple LLM providers behind one endpoint
• Load-balance traffic across them
• Add observability to model traffic
• Attach policies for security and transformation

An AI Model declares which capabilities it exposes (like generate or embeddings), which upstream LLM models it routes to via AI Model Providers, and how requests are distributed and logged. Consumer authentication is configured through AI Auth Strategies on the model.

AI Models can be created and managed through:

• Konnect UI
• AI Gateway API: /v1/ai-gateways/{aiGatewayId}/models
• kongctl

For configuration examples and step-by-step setup instructions, see Set up an AI Model.

At request time, the AI Model mediates traffic between clients and upstream AI Provider APIs:

1. Translates between the request and response format chosen for the AI Model and the upstream AI Provider’s native format.
2. Resolves upstream connection coordinates (protocol, host, port, path, HTTP method) from the selected target and its AI Model Provider, unless the target is a self-hosted model.
3. Authenticates to the upstream LLM service using credentials stored on the AI Model Provider entity.
4. Decorates the upstream request with per-target configuration (such as temperature or token-limit overrides) declared on targets[].config.
5. Records usage statistics (tokens, cost, latency) for attached log AI Policies, and optionally the full request and response when payload logging is enabled.
6. Fulfills requests to self-hosted models using the supported native format transformations.

A single AI Model can expose multiple upstream AI Providers behind a consistent client-facing format, so callers don’t change their request shape when the underlying AI Provider changes.

When you create or update an AI Model, AI Gateway provisions the necessary runtime resources and applies the configuration atomically. Credentials are sourced from the AI Model Provider entity that the AI Model’s targets reference at model creation time. If you update the AI Model Provider’s credentials later, those changes automatically propagate to all AI Models that use it.

An AI Model is a managed entity. AI Gateway owns its runtime configuration. Direct modifications through other APIs are not supported. To change an AI Model’s configuration, update the AI Model entity directly.

When you expose an AI Model, you choose which AI capabilities it provides through the capabilities field. The type you select determines which capabilities are available:

• model type: for synchronous request/response workloads. Available capabilities: generate, agentic, embeddings, audio/speech, audio/transcription, audio/translation, image, video, realtime, rerank.
• api type: for asynchronous batch processing. Available capabilities: batches, files.

Not every LLM service supports every capability. The set of capabilities you can declare on an AI Model depends on what the AI Model Provider in targets exposes. See AI Gateway providers for per-provider details.

By default, AI Models expose all endpoints using OpenAI-compatible format. AI Gateway provides a single, standardized interface across all providers, so you can swap providers (OpenAI, Anthropic, self-hosted, etc.) without changing client code or integration logic.

The formats array lets you control the request and response format. Each entry has a type that selects the format. The default openai format translates upstream provider responses into the OpenAI shape, so clients use one API format regardless of provider.

If you need the provider’s native format instead, set formats[].type to a non-OpenAI value. The AI Model passes requests upstream without conversion, while AI Gateway continues to provide analytics, logging, and cost calculation. You can also customize the endpoint paths through config.route.paths if needed.

When a native format is set, only the corresponding provider is supported with its specific APIs.

An AI Model is a virtual model: it exposes a single config.route configuration and one set of capabilities, then distributes requests across one or more concrete upstream models declared in its targets array. Each entry represents a single upstream model instance with one URL.

For each target, you provide the upstream model name (for example, gpt-4o) and reference the AI Model Provider to use by its name. Each target can also override settings such as temperature, max_tokens, input_cost, and output_cost. For providers with cache, context-window, or service-tier pricing, a target also accepts cache_read_cost, cache_write_cost, cache_write_cost_list, context_window_factor, and service_tier_factor. See Model cost management for how these fields combine to calculate a request’s cost.

There’s no separate target entity or endpoint. Targets are managed only as nested data inside an AI Model, through the same AI Model API surface used to create, update, and delete the parent. Adding, removing, or modifying a target is an update to the AI Model itself.

An AI Model routes to a single target by default. You can add more than one target when you want redundancy, fallback between providers, or cost and latency optimization. When you have multiple targets, configure config.balancer to distribute requests according to a load balancing algorithm.

When an AI Model has more than one target, the load balancer sits between the virtual model and its targets, distributing requests according to config.balancer. For algorithm details, selection guidance, and tuning, see Load balancing.

Algorithms

The algorithm field lets you choose how to distribute requests across target models based on your priorities. Select a strategy to optimize for cost, latency, even distribution, intelligent routing, or failover behavior.

Retry and fallback

To add redundancy and failover, the load balancer supports configurable retries, timeouts, and failover to different targets when one is unavailable. Fallback works across targets with any supported format, so you can mix providers freely (for example, OpenAI and Mistral). For configuration details, see Retry and fallback configuration.

Client errors don’t trigger failover. To fail over on additional error types, set failover_criteria to include HTTP codes like http_429 or http_502, and non_idempotent for POST requests.

Forward proxy support

Set config.proxy on this entity to route its outbound requests through an HTTP forward proxy. Use this in network-isolated deployments where AI Gateway cannot open direct connections to LLM providers or auxiliary services.

The proxy record is identical for AI Model, AI MCP Server, and supported AI Policy entities. Existing capabilities such as load balancing, health checking, streaming, WebSocket, and HTTP/2 continue to work when the proxy is active.

For the full field reference, traffic flow, and limitations, see Forward proxy support.

Health check and circuit breaker

To improve reliability under sustained failures, the load balancer includes a circuit breaker. When a target reaches the failure threshold set by max_fails, the load balancer stops routing requests to it until the fail_timeout period elapses. For behavior examples and tuning, see Circuit breaker.

Vector store

To route requests based on semantic similarity and keep similar requests on the same model instance, you can use a vector store. This is useful for caching consistency, routing to specialized model variants, or matching requests against historical patterns.

A vector store holds numerical representations (embeddings) of requests and responses so the runtime can match new requests against stored vectors. It powers the semantic algorithm and any similarity-matching workflow on the AI Model. Configure storage through config.balancer.vectordb by selecting a type:

For deeper background on vector storage and similarity matching, see Embedding-based similarity matching.

Embeddings

Configure an embedding model to enable semantic routing. This lets AI Gateway route requests based on meaning and content similarity rather than just cost or latency. For example, route domain-specific queries to specialized providers or keep similar requests on the same provider for consistency.

Set config.balancer.embeddings to reference an AI Model Provider and embedding model name. Supported provider types: azure, bedrock, gemini, huggingface, mistral, ollama, openai. The embedding model also powers the semantic load balancing algorithm.

The AI Model resolves runtime values from request data using placeholder substitution. This lets you select the target model dynamically per request, route to per-deployment Azure endpoints, or fan out to multiple providers from a single AI Model.

Substitution applies to the name of each target model and to any per-target config option. Three placeholders are available:

• $(headers.header_name): the value of a request header.
• $(uri_captures.path_parameter_name): the value of a captured URI path parameter.
• $(query_params.query_parameter_name): the value of a query string parameter.

For examples of using templating, consult the AI Gateway documentation and API reference.

By default, applications or services making requests to the AI Model endpoint must specify the actual upstream model name (like gpt-4o) in the model field. If you want to allow them to use a different name, for abstraction, stability, or to hide implementation details, configure config.route.model with a routing rule.

config.route.model accepts one of the following match strategies:

• body_param: matches a value in the request body, indexed by property name (for example, body_param: model).
• header_param: matches a value in the request headers, indexed by header name.
• path_param: matches a value present in the request path, indexed by path parameter name.

Each strategy pairs with a values array (currently limited to one value) listing the alias that routes to this AI Model. When a routing rule is set, clients can send the alias value (in the body, a header, or the path, depending on the strategy chosen) instead of the upstream model name. This is useful when you want to decouple your client API from upstream provider changes. For example, you could expose an alias like production-chat-model while swapping the underlying upstream model from gpt-4o to claude-3-sonnet without your clients noticing.

When config.route.model isn’t set, AI Gateway creates a default model selector using the AI Model’s name and format.

To limit which teams or applications can call an AI Model, use the access.acls field to set an allow list or a deny list. Reference AI Consumers (individual applications), AI Consumer Groups (teams), or Authenticated Groups (all consumers authenticated via a specific OAuth2 scope or claim) by name.

To control how consumers authenticate before their access is evaluated, configure the access.auth_strategies array with one or more AI Auth Strategy references. Each AI Model supports one key-auth auth strategy and one openid-connect auth strategy simultaneously.

Attach an AI Policy to an AI Model to add security, observability, governance, rate limiting, and cost optimization to all requests through that model. For example, you can add guardrails (AI Prompt Guard, AI Lakera Guard), enable logging and metrics, audit and compliance controls, cache responses, or rate-limit LLM traffic.

Reference AI Policies through the policies field, which accepts AI Policy names or IDs. You can attach multiple AI Policies to a single AI Model; each applies independently, and the same AI Policy type can be attached with different configurations.

AI Policy execution order

Authentication runs first, through the AI Model’s assigned AI Auth Strategy, not through an attached AI Policy. Attached AI Policies then execute in a defined order based on policy type. If execution order matters for your use case, refer to the plugin priority documentation.

When your data plane sits behind a corporate firewall or security boundary, configure a forward proxy to route all outbound AI provider requests through your organization’s proxy. This is required when direct internet access is restricted and all external traffic must pass through a bastion host or inspection gateway.

Use the config.proxy object to specify the proxy endpoint with http_proxy or https_proxy, and optionally add auth credentials if the proxy requires authentication. Use no_proxy to bypass the proxy for specific hosts that are already inside your trusted network.

AI Gateway automatically records token consumption, request latency, and per-provider costs for every AI Model, with no separate toggle required. This data flows into Konnect analytics and any attached logging AI Policies, letting you monitor API spend, identify slow providers, and audit which AI Models drive the most usage.

Optionally enable payloads to capture full request and response bodies. This is useful for debugging model responses, auditing sensitive operations, or replaying requests.

Payload logging may expose sensitive data in your logging destination. Only enable it when your logging pipeline is prepared to handle request and response bodies, and verify that logging destinations comply with your data residency and privacy policies.

For response streaming behavior, see Streaming.

Before creating an AI Model, first create an AI Model Provider to store credentials for the upstream LLM service.

The following example:

• Creates an OpenAI Model that exposes the generate capability, routed through a single OpenAI Provider, with token usage logging enabled.
• References a provider named my-openai-account. Either see Set up an AI Model Provider to create it, or substitute the name of your own AI Model Provider in targets[].provider.

This AI Model proxies client requests to /v1/chat/completions. The base path /v1 comes from config.route.paths, and /chat/completions is appended by the generate capability automatically.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "my-gpt-4o",
      "name": "my-gpt-4o",
      "type": "model",
      "capabilities": [
        "generate"
      ],
      "formats": [
        {
          "type": "openai"
        }
      ],
      "policies": [],
      "targets": [
        {
          "name": "gpt-4o",
          "provider": "my-openai-account",
          "config": {
            "type": "openai"
          }
        }
      ],
      "config": {
        "route": {
          "paths": [
            "/v1"
          ],
          "model": {
            "body_param": "model",
            "values": [
              "my-gpt-4o"
            ]
          }
        },
        "logging": {
          "payloads": false
        }
      }
    }
    '

ai_gateway_models:
  - ref: my-gpt-4o
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: my-gpt-4o
    name: my-gpt-4o
    type: model
    capabilities:
    - generate
    formats:
    - type: openai
    policies: []
    targets:
    - name: gpt-4o
      provider: my-openai-account
      config:
        type: openai
    config:
      route:
        paths:
        - "/v1"
        model:
          body_param: model
          values:
          - my-gpt-4o
      logging:
        payloads: false

Because config.route.model is set here with body_param: model, requests through this AI Model must send "model": "my-gpt-4o" (the alias) in the request body instead of the upstream target name (gpt-4o). Sending the upstream target name instead of the alias fails.