Check requests with an inline guardrail configuration
Configure the AI NVIDIA NeMo Guardrail policy to send a complete guardrail configuration to NeMo with every check.
AI Gateway forwards config.guardrails.config to NeMo as-is and doesn’t validate it, so configuration errors appear only when NeMo rejects the check.
The model named in guardrails.config.models must be available to the NeMo microservice.
Prerequisites
-
A running NeMo Guardrails microservice that AI Gateway can reach.
-
An AI Model.
-
An OpenAI API key.
Environment variables
OPENAI_API_KEY: The OpenAI API key that the NeMo Guardrails microservice uses to evaluate your rails.
ai_gateway_policies:
- ref: check-requests-with-an-inline-configuration
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
name: check-requests-with-an-inline-configuration
display_name: Check requests with an inline guardrail configuration
type: ai-nvidia-nemo-guardrail
enabled: true
config:
nemo_endpoint: http://nemo:7331/v1/guardrail/checks
model: gpt-5.1
guarding_mode: INPUT
timeout: 30000
auth:
api_key: !env OPENAI_API_KEY
prefix: ''
guardrails:
config:
models:
- type: content_safety
engine: openai
model: gpt-5.1
rails:
input:
flows:
- content safety check input $model=content_safety
prompts:
- task: content_safety_check_input $model=content_safety
models:
- openai/gpt-5.1
content: 'Task: Check if there is unsafe content in the user message.
'
output_parser: nemoguard_parse_prompt_safety
max_tokens: 50
global: falseMake sure to replace the following placeholders with your own values:
AI_GATEWAY_ID: Theidof your AI Gateway.
curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"name": "check-requests-with-an-inline-configuration",
"display_name": "Check requests with an inline guardrail configuration",
"type": "ai-nvidia-nemo-guardrail",
"enabled": true,
"config": {
"nemo_endpoint": "http://nemo:7331/v1/guardrail/checks",
"model": "gpt-5.1",
"guarding_mode": "INPUT",
"timeout": 30000,
"auth": {
"api_key": "'$OPENAI_API_KEY'",
"prefix": ""
},
"guardrails": {
"config": {
"models": [
{
"type": "content_safety",
"engine": "openai",
"model": "gpt-5.1"
}
],
"rails": {
"input": {
"flows": [
"content safety check input $model=content_safety"
]
}
},
"prompts": [
{
"task": "content_safety_check_input $model=content_safety",
"models": [
"openai/gpt-5.1"
],
"content": "Task: Check if there is unsafe content in the user message.\n",
"output_parser": "nemoguard_parse_prompt_safety",
"max_tokens": 50
}
]
}
}
},
"global": false
}
'Make sure to replace the following placeholders with your own values:
-
region: Geographic region where your Kong Konnect is hosted and operates. -
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account. -
AIGatewayId: Theidof the AI Gateway.
See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.
Prerequisite: Configure your Personal Access Token
terraform {
required_providers {
konnect = {
source = "kong/konnect"
}
}
}
provider "konnect" {
personal_access_token = "$KONNECT_TOKEN"
server_url = "https://us.api.konghq.com/"
}Add the following to your Terraform configuration:
resource "konnect_ai_gateway_policy" "my_policy" {
name = "check-requests-with-an-inline-configuration"
display_name = "Check requests with an inline guardrail configuration"
type = "ai-nvidia-nemo-guardrail"
enabled = true
global = false
config = jsonencode({
nemo_endpoint = "http://nemo:7331/v1/guardrail/checks"
model = "gpt-5.1"
guarding_mode = "INPUT"
timeout = 30000
auth = {
api_key = var.openai_api_key
prefix = ""
}
guardrails = {
config = {
models = [
{
type = "content_safety"
engine = "openai"
model = "gpt-5.1"
} ]
rails = {
input = {
flows = ["content safety check input $model=content_safety"]
}
}
prompts = [
{
task = "content_safety_check_input $model=content_safety"
models = ["openai/gpt-5.1"]
content = <<EOF
Task: Check if there is unsafe content in the user message.
EOF
output_parser = "nemoguard_parse_prompt_safety"
max_tokens = 50
} ]
}
}
})
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "openai_api_key" {
type = string
}After creating the policy, add check-requests-with-an-inline-configuration to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
ai_gateway_models:
- ref: my-gpt-4o
ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
display_name: my-gpt-4o
name: my-gpt-4o
type: model
capabilities:
- generate
formats:
- type: openai
targets:
- name: gpt-4o
provider: my-openai-account
config:
type: openai
config:
route:
paths:
- "/v1"
model:
body_param: model
values:
- my-gpt-4o
logging:
payloads: false
policies:
- !ref check-requests-with-an-inline-configuration#nameAfter creating the policy, add check-requests-with-an-inline-configuration to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models/my-gpt-4o \
--header "accept: application/json" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $KONNECT_TOKEN" \
--data '
{
"display_name": "my-gpt-4o",
"name": "my-gpt-4o",
"type": "model",
"capabilities": [
"generate"
],
"formats": [
{
"type": "openai"
}
],
"policies": [
"check-requests-with-an-inline-configuration"
],
"targets": [
{
"name": "gpt-4o",
"provider": "my-openai-account",
"config": {
"type": "openai"
}
}
],
"config": {
"route": {
"paths": [
"/v1"
],
"model": {
"body_param": "model",
"values": [
"my-gpt-4o"
]
}
},
"logging": {
"payloads": false
}
}
}
'After creating the policy, add check-requests-with-an-inline-configuration to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.
resource "konnect_ai_gateway_model" "my_model" {
display_name = "my-gpt-4o"
name = "my-gpt-4o"
type = "model"
capabilities = ["generate"]
formats = [
{
type = "openai"
} ]
targets = [
{
name = "gpt-4o"
provider = "my-openai-account"
config = {
type = "openai"
}
} ]
config = jsonencode({
route = {
paths = ["/v1"]
model = {
body_param = "model"
values = ["my-gpt-4o"]
}
}
logging = {
payloads = false
}
})
policies = [konnect_ai_gateway_policy.my_policy.name]
gateway_id = konnect_ai_gateway.my_ai_gateway.id
}This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.
variable "openai_api_key" {
type = string
}