AI Gateway
Introducing AI Gateway
As AI adoption accelerates, applications are evolving beyond basic LLM calls into complex, multi-actor systems, including user apps, agents, orchestration layers, and context servers that interact with foundation models in real time.
To support this shift, developers are adopting protocols like Model Context Protocol (MCP) and Agent2Agent (A2A) to standardize how components exchange tools, data, and decisions.
But infrastructure often falls behind, with challenges around authentication, rate limiting, data security, observability, and constant provider changes.
AI Gateway addresses these challenges with a high-performance control plane that secures, governs, and observes AI-native systems end to end. Whether serving LLM traffic, exposing structured context via MCP, or coordinating agents through A2A, AI Gateway ensures scalable, secure, and reliable AI infrastructure.
Sign up for Konnect to configure AI Gateway using Konnect.
Three traffic types, unified control
Define a single endpoint for any traffic type: LLM, MCP, or A2A. Configure these entities once, then govern them from a unified control plane with built-in auth, policy enforcement, and observability:
- Routing and load balancing across AI providers
- Streaming and authentication
- Control user access and ACLs
- See usage analytics for requests, tokens, errors, and latency
Tools to manage AI Gateway
You can manage these resources through multiple interfaces:
- AI Gateway manager: Manage all your AI Gateway resources from Konnect.
- AI Gateway API: Manage resources in Konnect AI Gateways via the API.
- kongctl: Manage resources and configuration from the command line.
AI Gateway providers
Connect to any major LLM provider through one consistent API, and switch or combine providers without rewriting your integration.
Use cases for AI Gateway
|
Use case
(You’re trying to…) |
Outcome
(…so that you can…) |
Feature
(…using these solutions…) |
|---|---|---|
| Give every team governed access to AI, without each team managing its own vendor relationship | Centralize LLM provider credentials in one place and rotate them without touching every team’s configuration | AI Model Provider |
| Scope model access and token budgets by team or department instead of managing permissions per person | AI Consumer Group | |
| Standardize prompts and default instructions across every team instead of each team writing their own | AI Prompt Template | |
| Package and bill AI access like a product, for customers paying for usage | Create pricing tiers (like Free, Pro, and Enterprise) with different token quotas and cost budgets | AI Consumer Group |
| Track token usage and automate invoicing based on prompt and response volume | Metering and billing | |
| Enforce per-tier spend caps driven by identity claims instead of managing group membership by hand | Set up a Kong Identity auth server for tiered AI budgets and Enforce tiered AI budgets with Kong Identity | |
| Cut LLM spend at scale on high-volume or repetitive traffic | Serve repeated or near-duplicate prompts from cache instead of calling the upstream LLM again | AI Semantic Cache |
| Shrink prompts before they reach the upstream provider to reduce token costs | AI Prompt Compressor | |
| Calculate the true cost of each request, including cache, context-window, and service-tier pricing, and enforce spend limits against it | Model cost management and AI Rate Limiting Advanced | |
| Expose your APIs as tools an AI agent can call, governed like any other API product | Turn existing APIs into tools an AI agent can discover and call over Model Context Protocol | AI MCP Server |
| Scope agent tool calls with OAuth2, so write actions can require tighter access than read actions | AI Auth Strategies | |
| Enforce governance on outgoing AI prompts by protecting against unsafe prompts and data leaks | Block requests that match disallowed topics or keywords | AI Prompt Guard |
| Catch jailbreak and prompt-injection attempts phrased in natural language, not just blocked keywords | AI Semantic Prompt Guard | |
| Redact PII before it reaches the upstream LLM provider | AI Sanitizer | |
| Catch unsafe or noncompliant content in the LLM’s response before it reaches the user | AI Semantic Response Guard | |
| Enforce guardrails through a cloud vendor’s own trust and safety service if you’re already using one | AI AWS Guardrails, AI Azure Content Safety, or AI GCP Model Armor | |
| Keep AI traffic online when a provider fails | Automatically fail over to a different provider or model when one is slow or unavailable | Load balancing |
Implement common scenarios
Explore end-to-end recipes for building real-world AI scenarios with AI Gateway, or check our AI Cookbooks to discover more.
Proxy AI CLI tools
AI Gateway can proxy requests from AI command-line tools to LLM providers. This gives you centralized control over AI traffic, including authentication, governance, and observability.
Observability and metrics
Track token usage, latency, and costs through audit logs, metrics exporters, and OpenTelemetry, or use Konnect Observability for pre-built dashboards.
AI Gateway guides
- Aggregate MCP tools from multiple AI MCP Server entities View →
- Enforce tiered AI budgets on AI Models with Kong Identity View →
- Route A2A agent traffic through AI Gateway View →
- Get started with AI Gateway View →
- Map the WeatherAPI to an MCP Server View →
- Limit A2A request body size View →
- Map a RESTful API to MCP tools View →
- Monitor AI Agent traffic with OpenTelemetry View →
- View More →
Frequently Asked Questions
How do I deploy AI Gateway?
AI Gateway is managed through Konnect. Data plane nodes run in your environment (self-hosted, cloud, or Kubernetes) and connect to Konnect for configuration and observability.
Why should I use AI Gateway instead of adding the LLM’s API behind Kong Gateway?
If you just add an LLM’s API behind Kong Gateway, you can only interact at the API level with internal traffic. With AI Gateway AI Policies and runtime components, Kong Gateway can understand the prompts that are being sent through the gateway. AI Policies can inspect the body and provide more specific AI capabilities to your traffic.
I’m migrating from AI Gateway running on Kong Gateway. Where do I start?
AI Gateway 2.x replaces Kong Gateway’s plugin-centric model (V1) with an entity model (V2). Instead of configuring the AI Proxy/AI Proxy Advanced plugin directly, you now create AI Model Provider and AI Model entities to manage upstream connectivity and routing. See AI Gateway 2.x concepts for a mapping of V1 concepts to V2 entities, and Migrate to AI Gateway 2.x for step-by-step instructions.
I was using AI Proxy or AI Proxy Advanced. Where did those plugins go?
In AI Gateway 2.x, the proxy is configured through the AI Model entity rather than the AI Proxy plugin. An AI Model defines the upstream provider, model name, and routing behavior that the old AI Proxy plugin handled. You can attach AI Policies for things like guardrails, transformations, and rate limiting as before.
Can I run AI Gateway on-prem?
Yes. AI Gateway supports both Konnect and self-hosted deployments. See Configure Kong AI Gateway on-prem for setup instructions.