curl -X POST "$KONNECT_PROXY_URL/v1/chat/completions" \
--no-progress-meter --fail-with-body \
-H "Accept: application/json"\
-H "Content-Type: application/json" \
--json '{
"messages": [
{
"role": "user",
"content": "Say this is a test!"
}
],
"model": "my-mantle-model"
}'Route OpenAI traffic to Amazon Bedrock Mantle with AI Gateway
Incompatible with
on-prem
Related Documentation
Related Resources
Minimum Version
AI Gateway - 2.3
TL;DR
Create an AI Model Provider of type bedrock with your AWS credentials, then create an AI Model with the openai format and a Bedrock target that sets endpoint_type to mantle. AI Gateway sends OpenAI requests to bedrock-mantle.{region}.api.aws in OpenAI format, without translating them to Bedrock Converse, and signs them with SigV4.
Prerequisites
AI Gateway running
This is a Konnect tutorial and requires a Konnect personal access token.
-
Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
-
Export your token to an environment variable:
export KONNECT_TOKEN='YOUR_KONNECT_PAT' -
Run the AI Gateway quickstart script to automatically provision a control plane and data plane in Kong Konnect, and configure your environment:
curl -Ls https://get.konghq.com/ai | bash -s -- -k $KONNECT_TOKEN
This sets up a AI Gateway control plane named ai-quickstart, provisions a local data plane, and prints out the following environment variables export:
export AI_GATEWAY_ID=your-gateway-id
export KONNECT_TOKEN=$KONNECT_TOKEN
export KONNECT_CONTROL_PLANE_NAME=ai-quickstart
export KONNECT_CONTROL_PLANE_URL=https://us.api.konghq.com
export KONNECT_PROXY_URL='http://localhost:8000'Copy and paste these into your terminal to configure your session.
Want a fully-hosted AI Gateway? See AI Gateway 2.0 on Dedicated Cloud Gateways.
kongctl v1.24.0+
This tutorial uses kongctl to manage Konnect resources programmatically. We recommend keeping kongctl up to date with the latest version (1.24.0).
- Install kongctl from developer.konghq.com/kongctl.
-
Verify the installation:
kongctl version
Amazon Bedrock Mantle
- In the AWS Management Console, confirm that Amazon Bedrock Mantle is available in your region and that your account has access to the model you want to use.
- Create an IAM user or role with permission to call Bedrock Mantle, then create access keys for it. The following is an example security policy you can apply to your IAM user or role:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "bedrock-mantle:GetModel", "bedrock-mantle:ListFiles", "bedrock:InvokeModel", "bedrock-mantle:CreateFile", "bedrock-mantle:ListModels", "bedrock-mantle:CancelInference", "bedrock-mantle:GetFile", "bedrock:InvokeModelWithResponseStream", "bedrock-mantle:CreateInference", "bedrock-mantle:CountTokens" ], "Resource": [ "arn:aws:bedrock-mantle:{region}:{account_id}:project/{project_name_or_default}", "arn:aws:bedrock-mantle:{region}:{account_id}:project/{project_name_or_default}/{model_name_1}", "arn:aws:bedrock-mantle:{region}:{account_id}:project/{project_name_or_default}/{model_name_2}" ] } ] } - Export your AWS credentials, region, and the Mantle model ID:
export AWS_ACCESS_KEY_ID='YOUR_AWS_ACCESS_KEY_ID' export AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_ACCESS_KEY' export AWS_REGION='YOUR_AWS_REGION' export MANTLE_MODEL_ID='YOUR_MANTLE_MODEL_ID'
Create an AI Model Provider
Create a bedrock type AI Model Provider that stores your AWS credentials. The same provider works for Bedrock Runtime and Bedrock Mantle targets:
kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" << 'EOF'
ai_gateway_model_providers:
- ref: my-aws-account
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: my-aws-account
display_name: "AWS Production"
type: bedrock
config:
auth:
type: aws
access_key_id: !env AWS_ACCESS_KEY_ID
secret_access_key: !secret {source: !env AWS_SECRET_ACCESS_KEY}
EOFTo authenticate with an Amazon Bedrock API key instead of SigV4, set config.auth.type to basic and add an Authorization header with the value Bearer <BEDROCK_API_KEY> in config.auth.headers.
Create an AI Model with a Mantle target
Create an AI Model that accepts OpenAI requests and routes them to Bedrock Mantle:
kongctl apply -f - --auto-approve --pat "$KONNECT_TOKEN" << 'EOF'
ai_gateway_models:
- ref: my-mantle-model
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: my-mantle-model
display_name: "my-mantle-model"
type: model
formats:
- type: openai
config:
route:
paths:
- /
model:
body_param: model
values:
- my-mantle-model
targets:
- name: !env MANTLE_MODEL_ID
provider: my-aws-account
config:
type: bedrock
region: !env AWS_REGION
endpoint_type: mantle
capabilities:
- generate
EOFThe AI Model uses the following settings:
formats: [type: openai]: Accepts OpenAI Chat Completions requests. A Mantle target can’t be paired with thebedrockformat, because Mantle has no Converse or InvokeModel API, and Konnect rejects that configuration.targets[].config.endpoint_type: mantle: Sends requests tohttps://bedrock-mantle.{region}.api.awsinstead of the default Bedrock Runtime endpoint. AI Gateway keeps the OpenAI request format instead of translating it to Bedrock Converse, and changes only the host, path, and authentication headers.config.route.model: Lets clients send the aliasmy-mantle-modelin themodelfield instead of the upstream model ID.
Validate
Send a chat request to the AI Model:
A 200 response with an OpenAI Chat Completions body confirms that Bedrock Mantle served the request.
Cleanup
Clean up AI Gateway resources
To clean up all AI Gateway resources created in this guide, run:
curl -Ls https://get.konghq.com/ai | bash -s -- -d