Mistral AI with Kong AI Gateway

Mistral’s API is OpenAI-compatible, which makes the first integration trivial: change base_url, change the key, ship it. That same key is what gets copied into every service that needs it.

• One key, every service. A Mistral API key is a single bearer token with account-wide scope. Once three services hold it, you can’t tell which one caused a spike, and you can’t revoke access for one of them without rotating the key for all three.
• No identity at the edge. Usage reporting is per key, not per caller. When a key is shared, per-team attribution has to be reconstructed from application logs that may not exist.
• Model names leak into application code. mistral-small-latest, mistral-medium-latest, and codestral-latest differ in price and capability. When those strings are hardcoded across services, moving a workload to a cheaper model becomes a coordinated redeploy.
• Chat and embeddings drift apart. A retrieval service needs an embedding model while a chat service needs a generative one. Wired directly, they become two independent integrations, with two copies of the credential and two places to change.

The root issue is coupling. Application code holds the provider credential, names the provider’s models directly, and repeats that wiring per capability, so there’s no single place to enforce identity or change routing.

Kong AI Gateway splits this into entities that map onto the problem. One AI Model Provider holds the Mistral credential. Two AI Models expose it, one for generate and one for embeddings, sharing a single base path and each answering to its own client-facing alias. An AI Auth Strategy with an AI Consumer issues a separate key per application, so revoking one app is a single delete and the Mistral key never leaves Konnect.

 
sequenceDiagram
    participant C as Client app
    participant K as Kong AI Gateway
    participant M as Mistral API

    C->>K: POST /mistral/chat/completions (apikey, model: mistral-chat)
    activate K
    K->>K: AI Auth Strategy - validate apikey, attach AI Consumer (else 401)
    K->>K: AI Model - resolve alias mistral-chat to mistral-small-latest
    K->>K: AI Model Provider - inject Authorization header
    K->>M: POST /v1/chat/completions (Bearer MISTRAL_API_KEY)
    M-->>K: completion and token usage
    K->>K: record tokens, latency, cost
    K-->>C: OpenAI-format response
    deactivate K
  

A chat request through this recipe passes four stages:

1. Route match. The data plane matches POST /mistral/chat/completions. The /mistral prefix comes from the AI Model’s config.route.paths, and /chat/completions is the endpoint the generate capability exposes. An embeddings request matches /mistral/embeddings on the second AI Model, which shares the same prefix.
2. Authentication. The AI Auth Strategy reads the apikey header and matches it to an AI Consumer credential. An unknown key terminates with 401 here, before Kong opens a connection to Mistral, so no Mistral quota is spent on rejected traffic.
3. Model selection. The AI Model reads the model field from the request body. mistral-chat is an alias declared in config.route.model.values; Kong resolves it to the target’s upstream model name and reports the result in the X-Kong-LLM-Model response header.
4. Upstream call and translation. The AI Model Provider attaches the stored Authorization header, Kong translates the OpenAI-shaped request into Mistral’s contract, and translates the response back. Token counts, latency, and cost are recorded on the way through.

AI Model Provider: one place for the Mistral credential

The AI Model Provider is the only entity that knows the Mistral key. Its type: mistral selects Mistral’s API surface and resolves the upstream base URL, so the recipe never configures a hostname. Because the credential lives on the provider rather than on each model, rotating it is a single apply, and every AI Model whose targets reference this provider picks up the new value.

ai_gateway_model_providers:
- ref: mistral-ai-with-kong-ai-gateway-provider
  ai_gateway: !lookup {id: !env 'AI_GATEWAY_ID'}
  name: mistral-ai-with-kong-ai-gateway-provider
  display_name: mistral-ai-with-kong-ai-gateway
  type: mistral
  config:
    auth:
      type: basic
      headers:
      - name: Authorization
        value: !secret {source: !env 'MISTRAL_TOKEN'}

AI Model: two capabilities behind one path

An AI Model is a virtual model: it declares the wire format it speaks, the base path it answers on, the capabilities it exposes, and the upstream targets it routes to. Mistral supports two capabilities through Kong AI Gateway, generate and embeddings, and each needs its own AI Model because the capability determines the endpoint. Both models in this recipe declare the same /mistral base path, so one base_url in the client covers chat and embeddings.

formats: [{type: openai}] is what lets the stock OpenAI SDK work against Mistral unchanged, and config.route.model decouples the client from the upstream model name.

Some providers also offer a native wire format, which passes requests upstream without translation at the cost of locking the AI Model to that provider. Mistral has no native format, so openai is the format this recipe uses. See AI Model for the formats each provider supports.

ai_gateway_models:
- ref: mistral-ai-with-kong-ai-gateway-chat
  name: mistral-ai-with-kong-ai-gateway-chat
  type: model
  formats:
  - type: openai
  capabilities:
  - generate
  access:
    auth_strategies:
    - mistral-ai-with-kong-ai-gateway-auth
  config:
    max_request_body_size: 10485760
    response_streaming: allow
    route:
      paths:
      - /mistral
      protocols: [http, https]
      model:
        body_param: model
        values: [mistral-chat]
  targets:
  - name: !env 'CHAT_MODEL'
    provider: mistral-ai-with-kong-ai-gateway-provider
    config:
      format: openai
      type: mistral

AI Auth Strategy and AI Consumers: per-application keys

Inbound authentication is separate from the outbound credential on the provider. A key-auth strategy makes Kong look for an apikey header, match it to an AI Consumer’s credential, and reject anything else with 401. Because hide_credentials: true, Kong strips that header before forwarding upstream, so the consumer key never reaches Mistral. Each application gets its own AI Consumer and its own credential, and revoking one application is a delete on that credential rather than a rotation of the Mistral key.

ai_gateway_auth_strategies:
- ref: mistral-ai-with-kong-ai-gateway-auth
  name: mistral-ai-with-kong-ai-gateway-auth
  type: key-auth
  config:
    key_names: [apikey]
    key_in_header: true
    key_in_query: false
    key_in_body: false
    hide_credentials: true

ai_gateway_consumers:
- ref: mistral-ai-with-kong-ai-gateway-consumer
  name: rag-service
  type: api-key
  credentials:
  - ref: mistral-ai-with-kong-ai-gateway-consumer-key
    name: rag-service-key
    type: api-key
    api_key: !secret {source: !env 'CONSUMER_API_KEY'}

This section configures the AI Gateway in two parts. First, adopt the quickstart AI Gateway into a kongctl namespace so the apply command below can manage it. The recipe’s mistral-ai-with-kong-ai-gateway-recipe namespace scopes the gateway so teardown removes only this recipe’s configuration.

kongctl adopt ai-gateway "${KONNECT_CONTROL_PLANE_NAME}" \
  --namespace "${KONNECT_CONTROL_PLANE_NAME}" \
  --pat "${KONNECT_TOKEN}"

Adoption stamps the KONGCTL-namespace label on the AI Gateway.

export CHAT_MODEL='mistral-small-latest'   # served as the mistral-chat alias
export EMBEDDINGS_MODEL='mistral-embed'    # served as the mistral-embeddings alias

cat <<'EOF' > kong-recipe.yaml
_defaults:
  kongctl:
    namespace: mistral-ai-with-kong-ai-gateway-recipe
ai_gateway_model_providers:
- ref: mistral-ai-with-kong-ai-gateway-provider
  ai_gateway: !lookup {id: !env 'AI_GATEWAY_ID'}
  name: mistral-ai-with-kong-ai-gateway-provider
  display_name: mistral-ai-with-kong-ai-gateway
  type: mistral
  config:
    auth:
      type: basic
      headers:
      - name: Authorization
        value: !secret {source: !env 'MISTRAL_TOKEN'}
ai_gateway_auth_strategies:
- ref: mistral-ai-with-kong-ai-gateway-auth
  ai_gateway: !lookup {id: !env 'AI_GATEWAY_ID'}
  name: mistral-ai-with-kong-ai-gateway-auth
  display_name: mistral-ai-with-kong-ai-gateway key auth
  type: key-auth
  config:
    key_names:
    - apikey
    key_in_header: true
    key_in_query: false
    key_in_body: false
    hide_credentials: true
ai_gateway_consumers:
- ref: mistral-ai-with-kong-ai-gateway-consumer
  ai_gateway: !lookup {id: !env 'AI_GATEWAY_ID'}
  name: rag-service
  display_name: RAG service
  type: api-key
  credentials:
  - ref: mistral-ai-with-kong-ai-gateway-consumer-key
    name: rag-service-key
    display_name: RAG service key
    type: api-key
    api_key: !secret {source: !env 'CONSUMER_API_KEY'}
ai_gateway_models:
- ref: mistral-ai-with-kong-ai-gateway-chat
  ai_gateway: !lookup {id: !env 'AI_GATEWAY_ID'}
  name: mistral-ai-with-kong-ai-gateway-chat
  display_name: mistral-ai-with-kong-ai-gateway (chat)
  type: model
  formats:
  - type: openai
  capabilities:
  - generate
  access:
    auth_strategies:
    - mistral-ai-with-kong-ai-gateway-auth
  policies: []
  config:
    max_request_body_size: 10485760
    response_streaming: allow
    logging:
      payloads: true
    route:
      paths:
      - /mistral
      protocols:
      - http
      - https
      methods:
      - POST
      - OPTIONS
      strip_path: true
      model:
        body_param: model
        values:
        - mistral-chat
  targets:
  - name: !env 'CHAT_MODEL'
    provider: mistral-ai-with-kong-ai-gateway-provider
    config:
      format: openai
      type: mistral
- ref: mistral-ai-with-kong-ai-gateway-embeddings
  ai_gateway: !lookup {id: !env 'AI_GATEWAY_ID'}
  name: mistral-ai-with-kong-ai-gateway-embeddings
  display_name: mistral-ai-with-kong-ai-gateway (embeddings)
  type: model
  formats:
  - type: openai
  capabilities:
  - embeddings
  access:
    auth_strategies:
    - mistral-ai-with-kong-ai-gateway-auth
  policies: []
  config:
    max_request_body_size: 10485760
    response_streaming: deny
    logging:
      payloads: true
    route:
      paths:
      - /mistral
      protocols:
      - http
      - https
      methods:
      - POST
      - OPTIONS
      strip_path: true
      model:
        body_param: model
        values:
        - mistral-embeddings
  targets:
  - name: !env 'EMBEDDINGS_MODEL'
    provider: mistral-ai-with-kong-ai-gateway-provider
    config:
      format: openai
      type: mistral
EOF

kongctl apply -f kong-recipe.yaml -o text --auto-approve --pat "${KONNECT_TOKEN}"

rm -f kong-recipe.yaml

The demo script exercises all four behaviors through the OpenAI SDK: a chat call, the same call streamed, an embeddings call, and a call with an invalid key.

The script passes the key through default_headers because the OpenAI SDK reserves api_key for the Authorization: Bearer header, while this auth strategy reads the apikey header. The SDK still requires api_key to be set to something, so the demo passes the literal string unused.

A chat request through Kong looks like this:

curl -s -X POST "$KONNECT_PROXY_URL/mistral/chat/completions" \
  -H "Content-Type: application/json" \
  -H "apikey: $CONSUMER_API_KEY" \
  --json '{"model":"mistral-chat","messages":[{"role":"user","content":"Reply with exactly: pong"}]}'

{"id":"54c390f45d7343e994cd0f648121795f","created":1789107525,"model":"mistral-small-latest","usage":{"prompt_tokens":21,"total_tokens":24,"completion_tokens":3},"object":"chat.completion","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"pong"}}]}

The response carries X-Kong-LLM-Model: mistral/mistral-small-latest, the upstream model Kong resolved the mistral-chat alias to.

Save the demo script as demo.py:

"""Mistral chat and embeddings through Kong AI Gateway.

Demonstrates that a single Kong endpoint fronts two Mistral capabilities while the
Mistral API key stays on the AI Model Provider, never in this script:

  1. Chat completion via the `mistral-chat` alias, showing the upstream model Kong
     resolved it to in the X-Kong-LLM-Model response header.
  2. The same endpoint streamed, proving response_streaming: allow.
  3. Embeddings via the `mistral-embeddings` alias on the same base path.
  4. A call with an invalid key, proving the key-auth AI Auth Strategy rejects it
     before any Mistral quota is spent.

Run:
    export PROXY_URL="$KONNECT_PROXY_URL"
    export CONSUMER_API_KEY='demo-api-key'
    python demo.py
"""

import os
import sys
import time

from openai import APIStatusError, OpenAI

PROXY_URL = os.getenv("PROXY_URL", "http://localhost:8000")
API_KEY = os.getenv("CONSUMER_API_KEY", "demo-api-key")

# Client-facing aliases declared in each AI Model's config.route.model.values.
# Kong resolves these to the upstream Mistral model names.
CHAT_MODEL = "mistral-chat"
EMBEDDINGS_MODEL = "mistral-embeddings"

# ANSI color codes. Disabled when stdout isn't a TTY or NO_COLOR is set.
_USE_COLOR = sys.stdout.isatty() and "NO_COLOR" not in os.environ


def _c(code: str, s: str) -> str:
    return f"\033[{code}m{s}\033[0m" if _USE_COLOR else s


def BOLD(s: str) -> str:
    return _c("1", s)


def DIM(s: str) -> str:
    return _c("2", s)


def GREEN(s: str) -> str:
    return _c("32", s)


def CYAN(s: str) -> str:
    return _c("36", s)


def RED(s: str) -> str:
    return _c("31", s)


def make_client(api_key: str) -> OpenAI:
    """Kong reads the apikey header; the OpenAI SDK reserves api_key for Authorization."""
    return OpenAI(
        base_url=f"{PROXY_URL}/mistral",
        api_key="unused",
        default_headers={"apikey": api_key},
    )


def section(title: str) -> None:
    bar = "=" * 70
    print(f"\n{bar}\n{BOLD(title)}\n{bar}")


def chat(client: OpenAI, prompt: str) -> None:
    print(f"\n{BOLD('[REQUEST]')} model={CHAT_MODEL!r} prompt={prompt!r}")
    start_ms = round(time.time() * 1000)
    try:
        raw = client.chat.completions.with_raw_response.create(
            model=CHAT_MODEL,
            messages=[{"role": "user", "content": prompt}],
        )
    except APIStatusError as e:
        elapsed_ms = round(time.time() * 1000) - start_ms
        print(f"{RED(BOLD('[BLOCKED]'))} {RED(BOLD(str(e.status_code)))} {e.message}  ({elapsed_ms}ms)")
        return

    elapsed_ms = round(time.time() * 1000) - start_ms
    completion = raw.parse()
    usage = completion.usage
    upstream_model = raw.headers.get("x-kong-llm-model", ".")
    upstream_latency = raw.headers.get("x-kong-upstream-latency", ".")
    proxy_latency = raw.headers.get("x-kong-proxy-latency", ".")

    print(f"[RESPONSE] {DIM(completion.choices[0].message.content)}")
    print(f"{GREEN(BOLD('[ROUTED TO]'))} alias={CHAT_MODEL!r} -> upstream model={CYAN(BOLD(upstream_model))}")
    print(f"[TOKENS] {DIM(f'prompt={usage.prompt_tokens} completion={usage.completion_tokens} total={usage.total_tokens}')}")
    print(f"[LATENCY] {DIM(f'upstream={upstream_latency}ms  proxy={proxy_latency}ms  total={elapsed_ms}ms')}")


def stream(client: OpenAI, prompt: str) -> None:
    print(f"\n{BOLD('[REQUEST]')} model={CHAT_MODEL!r} stream=True prompt={prompt!r}")
    start_ms = round(time.time() * 1000)
    first_token_ms = None
    chunks = 0
    text = []
    for chunk in client.chat.completions.create(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    ):
        piece = chunk.choices[0].delta.content
        if piece:
            if first_token_ms is None:
                first_token_ms = round(time.time() * 1000) - start_ms
            chunks += 1
            text.append(piece)
    elapsed_ms = round(time.time() * 1000) - start_ms
    print(f"[RESPONSE] {DIM(''.join(text))}")
    print(f"{GREEN(BOLD('[STREAMED]'))} {chunks} chunks  {DIM(f'first token={first_token_ms}ms  total={elapsed_ms}ms')}")


def embed(client: OpenAI, texts) -> None:
    print(f"\n{BOLD('[REQUEST]')} model={EMBEDDINGS_MODEL!r} inputs={len(texts)}")
    start_ms = round(time.time() * 1000)
    raw = client.embeddings.with_raw_response.create(model=EMBEDDINGS_MODEL, input=texts)
    elapsed_ms = round(time.time() * 1000) - start_ms
    result = raw.parse()
    upstream_model = raw.headers.get("x-kong-llm-model", ".")

    print(f"[RESPONSE] {DIM(f'{len(result.data)} vectors of {len(result.data[0].embedding)} dimensions')}")
    print(f"{GREEN(BOLD('[ROUTED TO]'))} alias={EMBEDDINGS_MODEL!r} -> upstream model={CYAN(BOLD(upstream_model))}")
    print(f"[TOKENS] {DIM(f'prompt={result.usage.prompt_tokens} total={result.usage.total_tokens}')}")
    print(f"[LATENCY] {DIM(f'total={elapsed_ms}ms')}")


def main() -> None:
    client = make_client(API_KEY)

    section("1. Chat, in OpenAI format, answered by Mistral")
    chat(client, "In one sentence, what is an API gateway?")

    section("2. The same endpoint, streamed")
    stream(client, "Name three French cities, comma separated.")

    section("3. Embeddings, same credential, same base path")
    embed(client, [
        "Kong AI Gateway sits between clients and Mistral.",
        "The AI Model Provider holds the Mistral API key.",
    ])

    bad_key = "not-a-real-key"
    section("4. Revoked or unknown key. Kong rejects before calling Mistral")
    print(f"{DIM(f'attempting with apikey={bad_key!r}')}")
    chat(make_client(bad_key), "In one sentence, what is an API gateway?")

    section("Done.")


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        sys.exit(130)

Point the script at the gateway and run it:

export PROXY_URL="$KONNECT_PROXY_URL"
python demo.py

======================================================================
1. Chat, in OpenAI format, answered by Mistral
======================================================================

[REQUEST] model='mistral-chat' prompt='In one sentence, what is an API gateway?'
[RESPONSE] An **API gateway** is a server that acts as an intermediary between clients and backend services, routing requests, aggregating responses, and enforcing policies like authentication, rate limiting, and load balancing.
[ROUTED TO] alias='mistral-chat' -> upstream model=mistral/mistral-small-latest
[TOKENS] prompt=25 completion=41 total=66
[LATENCY] upstream=881ms  proxy=6ms  total=1171ms

======================================================================
2. The same endpoint, streamed
======================================================================

[REQUEST] model='mistral-chat' stream=True prompt='Name three French cities, comma separated.'
[RESPONSE] Paris, Lyon, Marseille
[STREAMED] 3 chunks  first token=395ms  total=424ms

======================================================================
3. Embeddings, same credential, same base path
======================================================================

[REQUEST] model='mistral-embeddings' inputs=2
[RESPONSE] 2 vectors of 1024 dimensions
[ROUTED TO] alias='mistral-embeddings' -> upstream model=mistral/mistral-embed
[TOKENS] prompt=27 total=27
[LATENCY] total=1167ms

======================================================================
4. Revoked or unknown key. Kong rejects before calling Mistral
======================================================================
attempting with apikey='not-a-real-key'

[REQUEST] model='mistral-chat' prompt='In one sentence, what is an API gateway?'
[BLOCKED] 401 Error code: 401 - {'message': 'Unauthorized'}  (6ms)

======================================================================
Done.
======================================================================

What happened

1. The OpenAI SDK talked to Mistral without knowing it. The client set base_url to Kong and sent an OpenAI-shaped request. formats: [{type: openai}] on the AI Model translated it into Mistral’s contract and translated the reply back. No Mistral SDK is installed; the only dependency is openai.
2. The client sent an alias, not a model name. Every call used model: mistral-chat, which config.route.model resolved to the mistral-small-latest target. X-Kong-LLM-Model reports what actually served the request. Re-pointing that target is a one-line change, invisible to callers.
3. Streaming worked over the same endpoint. Adding stream=True produced incremental chunks with a first token at 395 ms, with no separate route or model.
4. Chat and embeddings shared one credential and one provider. The embeddings call went to /mistral/embeddings and returned 1024-dimension vectors from mistral-embed, authenticated by the same AI Consumer key and backed by the same AI Model Provider. Two capabilities, one base path, one credential.
5. The invalid key never reached Mistral. Kong rejected it in 6 ms, against the 881 ms the real call above took once it reached Mistral. No Mistral quota was spent, and no token was billed.
6. The Mistral key stayed in Konnect. The script only ever held the AI Consumer credential. The Mistral key lived on the AI Model Provider as a write-only field and was injected server-side.

Explore in Konnect

Open Konnect and select the mistral-ai-with-kong-ai-gateway-recipe AI Gateway. It holds five resources created by this recipe:

• One AI Model Provider, mistral-ai-with-kong-ai-gateway-provider, holding the Mistral credential. The key is write-only and is never displayed after creation.
• Two AI Models, mistral-ai-with-kong-ai-gateway-chat and mistral-ai-with-kong-ai-gateway-embeddings, each with its own capability, alias rule, and target.
• One AI Auth Strategy, mistral-ai-with-kong-ai-gateway-auth, referenced by both AI Models.
• One AI Consumer, rag-service, and its credential. Deleting that credential revokes the application immediately, without touching the Mistral key.

You can list the same resources from the API:

curl -s "$KONNECT_CONTROL_PLANE_URL/v1/ai-gateways/$AI_GATEWAY_ID/models" \
  -H "Authorization: Bearer $KONNECT_TOKEN" \
  | jq -r '.data[] | "\(.name)  \(.capabilities)  \(.config.route.paths)"'

mistral-ai-with-kong-ai-gateway-embeddings  ["embeddings"]  ["/mistral"]
mistral-ai-with-kong-ai-gateway-chat  ["generate"]  ["/mistral"]

Kong AI Gateway records token consumption, request latency, and per-provider cost for every AI Model with no extra configuration, so you can monitor spend and see which AI Models drive the most usage. Platform-wide telemetry lives under the Observability menu.

• Add a second tier. Add a third AI Model on the same provider with its own alias and a mistral-medium-latest target, and let callers pick per request. Set input_cost and output_cost on each target so Konnect reports spend per tier. See Model cost management.
• Budget each application. Attach an ai-rate-limiting-advanced AI Policy to cap tokens per AI Consumer, so one misbehaving application can’t drain the account. The AI policy counts tokens rather than requests, which is the correct unit for LLM cost control.
• Add guardrails. Attach ai-prompt-guard to filter prompts by pattern, or ai-sanitizer to strip PII before it reaches Mistral.
• Fail over to another provider. An AI Model can hold targets from different AI Model Providers. Add a second provider and target alongside the Mistral one, then set config.balancer to distribute or fail over between them. See Load balancing.
• Swap keys for tokens. Replace the key-auth AI Auth Strategy with openid-connect to accept tokens your identity provider already issues. An AI Model supports one of each simultaneously.

The recipe’s kongctl namespace scoped all resources, so this teardown removes only this recipe’s configuration. Tear down the local Data Plane and delete the AI Gateway from Konnect:

export KONNECT_CONTROL_PLANE_NAME='mistral-ai-with-kong-ai-gateway-recipe' && curl -Ls https://get.konghq.com/ai | bash -s -- -d -k $KONNECT_TOKEN

Then delete the Mistral API key in the Mistral console if it was created only for this recipe.