--- title: "Kong AI Gateway" description: This page is an introduction to AI Gateway. url: "/ai-gateway/" canonical_url: "/ai-gateway/" content_type: landing_page min_version: ai-gateway: '2.0' products: - AI Gateway tags: - ai canonical: true works_on: - konnect --- # AI Gateway Connectivity and governance layer for modern AI-native applications ## Introducing AI Gateway As AI adoption accelerates, applications are evolving beyond basic LLM calls into complex, multi-actor systems, including user apps, agents, orchestration layers, and context servers that interact with foundation models in real time. To support this shift, developers are adopting protocols like Model Context Protocol (MCP) and Agent2Agent (A2A) to standardize how components exchange tools, data, and decisions. But infrastructure often falls behind, with challenges around authentication, rate limiting, data security, observability, and constant provider changes. AI Gateway addresses these challenges with a high-performance control plane that secures, governs, and observes AI-native systems end to end. Whether serving LLM traffic, exposing structured context via MCP, or coordinating agents through A2A, AI Gateway ensures scalable, secure, and reliable AI infrastructure. [Sign up for Konnect](https://konghq.com/products/kong-konnect/register?utm_medium=referral&utm_source=docs&utm_content=ai-gateway) to configure AI Gateway using Konnect. ![Overview of AI gateway](/assets/images/gateway/ai-gateway-overview.svg) ## Get started with AI Gateway #### Quickstart You can use the [quickstart script](https://get.konghq.com/ai) to get a demo instance of AI Gateway running almost instantly. This command requires a [Konnect Access Token](https://cloud.konghq.com/global/account/tokens). ```sh curl -Ls https://get.konghq.com/ai | bash -s -- -k $KONNECT_TOKEN ``` The script creates an AI Gateway control plane in Konnect and deploys a local data plane using Docker. All licensing is handled automatically by Konnect. #### Konnect UI To set up AI Gateway using the Konnect UI, sign in to your Konnect account at [cloud.konghq.com](https://cloud.konghq.com/), then navigate to **AI Gateway** and click **New AI Gateway**. ## LLM quickstart Proxy your first model through AI Gateway with a guided setup. ## MCP quickstart Expose and observe your first tool server over Model Context Protocol. ## A2A quickstart Route and secure agent-to-agent traffic with protocol-aware observability. ## Three traffic types, unified control Define a single endpoint for any traffic type: LLM, MCP, or A2A. Configure these entities once, then govern them from a unified control plane with built-in auth, policy enforcement, and observability: * [Routing and load balancing](/ai-gateway/load-balancing/) across AI providers * [Streaming and authentication](/ai-gateway/streaming/) * Control user access and ACLs * See [usage analytics](/ai-gateway/monitor-ai-llm-metrics/) for requests, tokens, errors, and latency ## Tools to manage AI Gateway You can manage these resources through multiple interfaces: - [AI Gateway manager](https://cloud.konghq.com/ai-manager): Manage all your AI Gateway resources from Konnect. - [AI Gateway API](/api/konnect/ai-gateway/): Manage resources in Konnect AI Gateways via the API. - [kongctl](/kongctl/): Manage resources and configuration from the command line. ## AI Gateway providers Connect to any major LLM provider through one consistent API, and switch or combine providers without rewriting your integration. [OpenAI](/ai-gateway/ai-providers/openai/) [Anthropic](/ai-gateway/ai-providers/anthropic/) [Azure AI](/ai-gateway/ai-providers/azure/) [Amazon Bedrock](/ai-gateway/ai-providers/bedrock/) [Gemini](/ai-gateway/ai-providers/gemini/) [Vercel](/ai-gateway/ai-providers/vercel/) [More...](/ai-gateway/ai-providers/) ## Use cases for AI Gateway ### **Give every team governed access to AI**, without each team managing its own vendor relationship - **Outcome**: Centralize LLM provider credentials in one place and rotate them without touching every team's configuration - **Feature**: [AI Model Provider](/ai-gateway/entities/ai-model-provider/) - **Outcome**: Scope model access and token budgets by team or department instead of managing permissions per person - **Feature**: [AI Consumer Group](/ai-gateway/entities/ai-consumer-group/) - **Outcome**: Standardize prompts and default instructions across every team instead of each team writing their own - **Feature**: [AI Prompt Template](/ai-gateway/policies/ai-prompt-template/) ### **Package and bill AI access like a product**, for customers paying for usage - **Outcome**: Create pricing tiers (like Free, Pro, and Enterprise) with different token quotas and cost budgets - **Feature**: [AI Consumer Group](/ai-gateway/entities/ai-consumer-group/) - **Outcome**: Track token usage and automate invoicing based on prompt and response volume - **Feature**: [Metering and billing](/metering-and-billing/) - **Outcome**: Enforce per-tier spend caps driven by identity claims instead of managing group membership by hand - **Feature**: [Set up a Kong Identity auth server for tiered AI budgets](/ai-gateway/set-up-kong-identity-for-tiered-ai-budgets/) and [Enforce tiered AI budgets with Kong Identity](/ai-gateway/enforce-tiered-ai-budgets-with-kong-identity/) ### **Cut LLM spend at scale** on high-volume or repetitive traffic - **Outcome**: Serve repeated or near-duplicate prompts from cache instead of calling the upstream LLM again - **Feature**: [AI Semantic Cache](/ai-gateway/policies/ai-semantic-cache/) - **Outcome**: Shrink prompts before they reach the upstream provider to reduce token costs - **Feature**: [AI Prompt Compressor](/ai-gateway/policies/ai-prompt-compressor/) - **Outcome**: Calculate the true cost of each request, including cache, context-window, and service-tier pricing, and enforce spend limits against it - **Feature**: [Model cost management](/ai-gateway/model-cost-management/) and [AI Rate Limiting Advanced](/ai-gateway/policies/ai-rate-limiting-advanced/) ### **Expose your APIs as tools an AI agent can call**, governed like any other API product - **Outcome**: Turn existing APIs into tools an AI agent can discover and call over Model Context Protocol - **Feature**: [AI MCP Server](/ai-gateway/entities/ai-mcp-server/) - **Outcome**: Scope agent tool calls with OAuth2, so write actions can require tighter access than read actions - **Feature**: [AI Auth Strategies](/ai-gateway/entities/ai-auth-strategy/) ### **Enforce governance on outgoing AI prompts** by protecting against unsafe prompts and data leaks - **Outcome**: Block requests that match disallowed topics or keywords - **Feature**: [AI Prompt Guard](/ai-gateway/policies/ai-prompt-guard/) - **Outcome**: Catch jailbreak and prompt-injection attempts phrased in natural language, not just blocked keywords - **Feature**: [AI Semantic Prompt Guard](/ai-gateway/policies/ai-semantic-prompt-guard/) - **Outcome**: Redact PII before it reaches the upstream LLM provider - **Feature**: [AI Sanitizer](/ai-gateway/policies/ai-sanitizer/) - **Outcome**: Catch unsafe or noncompliant content in the LLM's response before it reaches the user - **Feature**: [AI Semantic Response Guard](/ai-gateway/policies/ai-semantic-response-guard/) - **Outcome**: Enforce guardrails through a cloud vendor's own trust and safety service if you're already using one - **Feature**: [AI AWS Guardrails](/ai-gateway/policies/ai-aws-guardrails/), [AI Azure Content Safety](/ai-gateway/policies/ai-azure-content-safety/), or [AI GCP Model Armor](/ai-gateway/policies/ai-gcp-model-armor/) ### **Keep AI traffic online when a provider fails** - **Outcome**: Automatically fail over to a different provider or model when one is slow or unavailable - **Feature**: [Load balancing](/ai-gateway/load-balancing/) ## See all AI Policies Attach governance behavior for authentication, guardrails, transformations, and more. ## See all AI entities Entities are the building blocks that make up the AI Gateway ecosystem. This includes AI Models, AI Model Providers, AI Auth Strategies, AI Agents, AI MCP Servers, and AI Consumers. ## Implement common scenarios Explore end-to-end recipes for building real-world AI scenarios with AI Gateway, or check our [AI Cookbooks](/cookbooks/) to discover more. ### Claude SSO integration Secure Claude with single sign-on authentication through AI Gateway. ### Basic LLM routing Route requests across multiple LLM providers with failover and load balancing. ### External MCP servers Expose and govern tools from external Model Context Protocol servers. ### Monetize AI Gateway traffic Meter LLM token usage and turn it into billable revenue with Metering & Billing. ## Proxy AI CLI tools AI Gateway can proxy requests from AI command-line tools to LLM providers. This gives you centralized control over AI traffic, including authentication, governance, and observability. [Claude Code](/ai-gateway/ai-clis/#claude-code) [Codex CLI](/ai-gateway/ai-clis/#codex-cli) [Qwen Code CLI](/ai-gateway/ai-clis/#qwen-code-cli) [More...](/ai-gateway/ai-clis/) ## Observability and metrics Track token usage, latency, and costs through audit logs, metrics exporters, and OpenTelemetry, or use Konnect Observability for pre-built dashboards. ### Audit log Learn about AI Gateway logging capabilities. ### Konnect Observability Visualize LLM metrics in Konnect. ### LLM metrics Expose and visualize LLM metrics. ### Gen AI OTLP span attributes Per-request OpenTelemetry span attributes for AI traffic. ### Gen AI OTLP metrics Aggregated OpenTelemetry metrics for AI, MCP, and A2A traffic. ## Core concepts in AI Gateway ### Architecture Understand how AI Gateway works, including its control plane, data plane, and deployment topologies. ### Manage with kongctl Use kongctl to create and manage AI Gateway resources declaratively or with imperative commands. ### AI Gateway 2.x concepts Understand how v1 concepts like AI Proxy map to v2 entities such as AI Models and AI Model Providers. ### Migrate to AI Gateway 2.x Step-by-step guide for migrating from v1 AI Proxy and AI Proxy Advanced to the v2 entity model. ### Run AI Gateway on-prem Configure AI Gateway on self-hosted Kong Gateway using the Kong Gateway data model and AI plugins. ### Model cost management See how AI Gateway prices models and calculates the cost of a request. ## Key references ### AI Gateway version support policy The AI Gateway version support policy outlines the AI Gateway versioning scheme and version lifecycle, from release to sunset support. [View the version support policy](/ai-gateway/version-support-policy/) ### Changelog Changelog for released AI Gateway versions [See changelog](/ai-gateway/changelog/) ## AI Gateway guides * [Aggregate MCP tools from multiple AI MCP Server entities](/ai-gateway/aggregate-mcp-tools/) * [Enforce tiered AI budgets on AI Models with Kong Identity](/ai-gateway/enforce-tiered-ai-budgets-with-kong-identity/) * [Route A2A agent traffic through AI Gateway](/ai-gateway/get-started-with-ai-agent/) * [Get started with AI Gateway](/ai-gateway/get-started/) * [Map the WeatherAPI to an MCP Server](/ai-gateway/get-started-with-mcp-server/) * [Identify AI Consumers on AI Model traffic with Kong Identity](/ai-gateway/identify-ai-consumers-with-kong-identity/) * [Limit A2A request body size](/ai-gateway/limit-a2a-body-size/) * [Map a RESTful API to MCP tools](/ai-gateway/map-api-to-mcp-tools/) ## Frequently Asked Questions - How do I deploy AI Gateway? AI Gateway is managed through Konnect. Data plane nodes run in your environment (self-hosted, [cloud](/dedicated-cloud-gateways/), or Kubernetes) and connect to Konnect for configuration and observability. - Why should I use AI Gateway instead of adding the LLM's API behind Kong Gateway? If you just add an LLM's API behind Kong Gateway, you can only interact at the API level with internal traffic. With AI Gateway AI Policies and runtime components, Kong Gateway can understand the prompts that are being sent through the gateway. AI Policies can inspect the body and provide more specific AI capabilities to your traffic. - I'm migrating from AI Gateway running on Kong Gateway. Where do I start? AI Gateway 2.x replaces Kong Gateway's plugin-centric model (V1) with an entity model (V2). Instead of configuring the AI Proxy/AI Proxy Advanced plugin directly, you now create AI Model Provider and AI Model entities to manage upstream connectivity and routing. See [AI Gateway 2.x concepts](/ai-gateway/ai-gateway-v2-concepts/) for a mapping of V1 concepts to V2 entities, and [Migrate to AI Gateway 2.x](/ai-gateway/v2-migration-guide/) for step-by-step instructions. - I was using AI Proxy or AI Proxy Advanced. Where did those plugins go? In AI Gateway 2.x, the proxy is configured through the [AI Model entity](/ai-gateway/entities/ai-model/) rather than the AI Proxy plugin. An AI Model defines the upstream provider, model name, and routing behavior that the old AI Proxy plugin handled. You can attach AI Policies for things like guardrails, transformations, and rate limiting as before. - Can I run AI Gateway on-prem? Yes. AI Gateway supports both Konnect and self-hosted deployments. See [Configure Kong AI Gateway on-prem](/ai-gateway/configure-on-prem/) for setup instructions. - How do I manage AI Gateway resources? For Konnect-managed deployments, use [kongctl](/ai-gateway/kongctl/) to create and manage resources declaratively or with imperative commands. For on-prem deployments, use [decK](https://docs.konghq.com/deck/) to manage gateway configuration.