All AI Gateway documentation

Overview of AI gateway capabilities

Get started quickly with AI Gateway setup and usage.

Learn about the core capabilities of AI Gateway.

Learn about the various providers supported by AI Gateway.

Understand how to manage and govern AI usage effectively.

Explore how AI Gateway helps enforce data governance policies.

Best practices and tools for designing effective prompts.

Implement safeguards to ensure safe and compliant AI outputs.

Customize and transform AI requests with Gateway features.

Learn how AI Proxy streaming works.

Learn about AI Gateway logging capabilities.

Explore how to monitor AI LLM metrics in AI Gateway.

Access advanced analytics features in AI Gateway.

Use Azure AI Content Safety to check and audit AI Proxy plugin messages before proxying them to an upstream LLM

Prepend or append an array of llm/v1/chat messages to a user's chat history

Check llm/v1/chat or llm/v1/completions requests against a list of allowed or denied expressions

Provide fill-in-the-blank AI prompts to users

The AI Proxy plugin lets you transform and proxy requests to a number of AI providers and models.

The AI Proxy Advanced plugin lets you transform and proxy requests to multiple AI providers and models at the same time. This lets you set up load balancing between targets.

Create RAG pipelines by automatically injecting content from a vector database

Provides rate limiting for the providers used by any AI plugins.

Use an LLM service to transform a client request body prior to proxying the request to the upstream server

Use an LLM service to transform the upstream HTTP(S) prior to forwarding it to the client

Semantically and intelligently create allow and deny lists of topics that can be requested across every LLM.

Protect sensitive information in client request bodies before they reach upstream services

Compress prompts before they are sent to LLMs to reduce costs, and improve latency

Use AWS Guardrails to validate requests and/or responses in the AI Proxy plugin before forwarding them between clients and upstream LLMs.

This page is an introduction to MCP Traffic Gateway capabilites in Kong AI Gateway.

Secure GitHub MCP Server traffic with Kong Gateway and Kong AI Gateway

Use Kong AI Gateway to govern GitHub MCP traffic

Observe GitHub MCP traffic with Kong AI Gateway

Get started with Kong Konnect MCP Server

Check available Kong Konnect MCP Server tools

Overview of load balancing and retry and fallback strategies in the AI Proxy Advanced plugin.

Set up consistent hashing for load balancing.

Configure load balancing based on the lowest latency.

Set up load balancing based on the lowest usage.

Configure priority-based load balancing.

Set up round-robin load balancing.

Set up semantic load balancing.

Learn how to configure you SIEM provider to collect Konnect logs and configure a Konnect audit log webhook.

Use HashiCorp Vault to securely store and reference API keys for OpenAI, Mistral, and other LLM providers in Kong AI Gateway.

Learn how to use the AI Compressor plugin alongside the RAG Injector and AI Prompt Decorator plugins to keep prompts lean, reduce latency, and optimize LLM usage for cost efficiency

Learn how to configure the AI RAG Injector plugin.

Learn how to quickly get started with AI Gateway

Learn how to observe MCP traffic within GitHub remote MCP server with the AI Proxy Advanced and Kong Gateway Prometheus plugin

Configure the AI Proxy plugin to route requests to a model provider like Mistral, then define reusable templates with the AI Prompt Template plugin to enforce consistent prompt formatting for tasks like summarization, code explanation, and Q&A.

Use the AI Proxy Advanced plugin to route chat requests to OpenAI models based on semantic intent, secured with API keys stored in HashiCorp Vault.

Configure the AI Proxy Advanced plugin to optimize LLM usage and reduce costs by intelligently routing chat requests across multiple OpenAI models based on semantic similarity.

Reduce costs by using llm/v1/files and llm/v1/batches route_types to send asynchronous batched requests to LLMs.

Configure the AI Proxy Advanced plugin to create a chat route using Anthropic.

Configure the AI Proxy Advanced plugin to create a chat route using Ollama.

Configure the AI Proxy Advanced plugin to create a chat route using OpenAI.

Configure the AI Proxy plugin to create a chat route using Anthropic.

Configure the AI Proxy Advanced plugin to create a chat route using Ollama.

Configure the AI Proxy plugin to create a chat route using OpenAI.

Learn how to set up Konnect Config Store as a Vault backend and store a Mistral API key.

Learn how to store and rotate secrets in Google Cloud with Kong Gateway, Mistral, and the AI Proxy plugin.

Use the AI Request Transformer plugin with OpenAI to transform a client request body before proxying it.

Use the AI Response Transformer plugin with OpenAI to transform a response before returning it to the client.

Use the AI Prompt Guard plugin to filter LLM traffic based on regex rules that allow general IT questions and deny unsafe or off-topic content.

Use the AI Semantic Prompt Guard plugin to enforce topic-level guardrails for LLM traffic, filtering prompts based on meaning.

Use the AI Sanitizer plugin to protect sensitive information in requests.

Learn how to use the Azure AI Content Safety plugin.

Learn how to govern MCP traffic within GitHub remote MCP server with the AI Proxy Advanced and AI Prompt Guard plugins

Connect your LangChain integrations with Kong Gateway with no code changes.

Learn how to use the AI AWS Guardrails plugin.

Use a sample Elasticsearch, Logstash, and Kibana stack to visualize data from the AI Proxy plugin.