AI AWS Guardrails: Guard streamed responses - Policy

Guard streamed responses

Configure the AI AWS Guardrails policy to evaluate streamed model responses. The policy buffers the response in chunks of the configured size and sends each chunk to the guardrail. A larger buffer gives the guardrail more context per check, and delays the first bytes the client receives.

Prerequisites

An AWS Bedrock Guardrails policy and access to the AWS Bedrock Guardrails service.
An existing AI Model.

Environment variables

GUARDRAILS_ID: The guardrail identifier to apply.
GUARDRAILS_VERSION: The guardrail version to apply.
AWS_REGION: The AWS region where the guardrail is deployed.
AWS_ACCESS_KEY_ID: The AWS access key ID used to authenticate the request.
AWS_SECRET_ACCESS_KEY: The AWS secret access key used to authenticate the request.

Set up the policy

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: aws-guardrails-guard-streamed-responses
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: aws-guardrails-guard-streamed-responses
    display_name: Guard streamed responses
    type: ai-aws-guardrails
    enabled: true
    config:
      guardrails_id: !env GUARDRAILS_ID
      guardrails_version: !env GUARDRAILS_VERSION
      aws_region: !env AWS_REGION
      aws_access_key_id: !env AWS_ACCESS_KEY_ID
      aws_secret_access_key: !env AWS_SECRET_ACCESS_KEY
      guarding_mode: OUTPUT
      response_buffer_size: 500
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "aws-guardrails-guard-streamed-responses",
      "display_name": "Guard streamed responses",
      "type": "ai-aws-guardrails",
      "enabled": true,
      "config": {
        "guardrails_id": "'$GUARDRAILS_ID'",
        "guardrails_version": "'$GUARDRAILS_VERSION'",
        "aws_region": "'$AWS_REGION'",
        "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
        "aws_secret_access_key": "'$AWS_SECRET_ACCESS_KEY'",
        "guarding_mode": "OUTPUT",
        "response_buffer_size": 500
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "aws-guardrails-guard-streamed-responses"
  display_name = "Guard streamed responses"
  type = "ai-aws-guardrails"
  enabled = true
  global = false
  config = jsonencode({
    guardrails_id = var.guardrails_id
    guardrails_version = var.guardrails_version
    aws_region = var.aws_region
    aws_access_key_id = var.aws_access_key_id
    aws_secret_access_key = var.aws_secret_access_key
    guarding_mode = "OUTPUT"
    response_buffer_size = 500
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "guardrails_id" {
  type = string
}
variable "guardrails_version" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_secret_access_key" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: aws-guardrails-guard-streamed-responses
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: aws-guardrails-guard-streamed-responses
    display_name: Guard streamed responses
    type: ai-aws-guardrails
    enabled: true
    config:
      guardrails_id: !env GUARDRAILS_ID
      guardrails_version: !env GUARDRAILS_VERSION
      aws_region: !env AWS_REGION
      aws_access_key_id: !env AWS_ACCESS_KEY_ID
      aws_secret_access_key: !env AWS_SECRET_ACCESS_KEY
      guarding_mode: OUTPUT
      response_buffer_size: 500
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "aws-guardrails-guard-streamed-responses",
      "display_name": "Guard streamed responses",
      "type": "ai-aws-guardrails",
      "enabled": true,
      "config": {
        "guardrails_id": "'$GUARDRAILS_ID'",
        "guardrails_version": "'$GUARDRAILS_VERSION'",
        "aws_region": "'$AWS_REGION'",
        "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
        "aws_secret_access_key": "'$AWS_SECRET_ACCESS_KEY'",
        "guarding_mode": "OUTPUT",
        "response_buffer_size": 500
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "aws-guardrails-guard-streamed-responses"
  display_name = "Guard streamed responses"
  type = "ai-aws-guardrails"
  enabled = true
  global = false
  config = jsonencode({
    guardrails_id = var.guardrails_id
    guardrails_version = var.guardrails_version
    aws_region = var.aws_region
    aws_access_key_id = var.aws_access_key_id
    aws_secret_access_key = var.aws_secret_access_key
    guarding_mode = "OUTPUT"
    response_buffer_size = 500
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "guardrails_id" {
  type = string
}
variable "guardrails_version" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_secret_access_key" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: aws-guardrails-guard-streamed-responses
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: aws-guardrails-guard-streamed-responses
    display_name: Guard streamed responses
    type: ai-aws-guardrails
    enabled: true
    config:
      guardrails_id: !env GUARDRAILS_ID
      guardrails_version: !env GUARDRAILS_VERSION
      aws_region: !env AWS_REGION
      aws_access_key_id: !env AWS_ACCESS_KEY_ID
      aws_secret_access_key: !env AWS_SECRET_ACCESS_KEY
      guarding_mode: OUTPUT
      response_buffer_size: 500
    global: false

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "aws-guardrails-guard-streamed-responses",
      "display_name": "Guard streamed responses",
      "type": "ai-aws-guardrails",
      "enabled": true,
      "config": {
        "guardrails_id": "'$GUARDRAILS_ID'",
        "guardrails_version": "'$GUARDRAILS_VERSION'",
        "aws_region": "'$AWS_REGION'",
        "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
        "aws_secret_access_key": "'$AWS_SECRET_ACCESS_KEY'",
        "guarding_mode": "OUTPUT",
        "response_buffer_size": 500
      },
      "global": false
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "aws-guardrails-guard-streamed-responses"
  display_name = "Guard streamed responses"
  type = "ai-aws-guardrails"
  enabled = true
  global = false
  config = jsonencode({
    guardrails_id = var.guardrails_id
    guardrails_version = var.guardrails_version
    aws_region = var.aws_region
    aws_access_key_id = var.aws_access_key_id
    aws_secret_access_key = var.aws_secret_access_key
    guarding_mode = "OUTPUT"
    response_buffer_size = 500
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "guardrails_id" {
  type = string
}
variable "guardrails_version" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_secret_access_key" {
  type = string
}

Copied!

kongctl

policy.yaml

Copied!

ai_gateway_policies:
  - ref: aws-guardrails-guard-streamed-responses
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    name: aws-guardrails-guard-streamed-responses
    display_name: Guard streamed responses
    type: ai-aws-guardrails
    enabled: true
    config:
      guardrails_id: !env GUARDRAILS_ID
      guardrails_version: !env GUARDRAILS_VERSION
      aws_region: !env AWS_REGION
      aws_access_key_id: !env AWS_ACCESS_KEY_ID
      aws_secret_access_key: !env AWS_SECRET_ACCESS_KEY
      guarding_mode: OUTPUT
      response_buffer_size: 500
    global: true

Make sure to replace the following placeholders with your own values:

AI_GATEWAY_ID: The id of your AI Gateway.

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "aws-guardrails-guard-streamed-responses",
      "display_name": "Guard streamed responses",
      "type": "ai-aws-guardrails",
      "enabled": true,
      "config": {
        "guardrails_id": "'$GUARDRAILS_ID'",
        "guardrails_version": "'$GUARDRAILS_VERSION'",
        "aws_region": "'$AWS_REGION'",
        "aws_access_key_id": "'$AWS_ACCESS_KEY_ID'",
        "aws_secret_access_key": "'$AWS_SECRET_ACCESS_KEY'",
        "guarding_mode": "OUTPUT",
        "response_buffer_size": 500
      },
      "global": true
    }
    '

Copied!

Make sure to replace the following placeholders with your own values:

region: Geographic region where your Kong Konnect is hosted and operates.
KONNECT_TOKEN: Your Personal Access Token (PAT) associated with your Konnect account.
AIGatewayId: The id of the AI Gateway.

See the Konnect AI Gateway API reference to learn about region-specific URLs and personal access tokens.

Prerequisite: Configure your Personal Access Token

terraform {
  required_providers {
    konnect = {
      source  = "kong/konnect"
    }
  }
}

provider "konnect" {
  personal_access_token = "$KONNECT_TOKEN"
  server_url            = "https://us.api.konghq.com/"
}

Copied!

Add the following to your Terraform configuration:

resource "konnect_ai_gateway_policy" "my_policy" {
  name = "aws-guardrails-guard-streamed-responses"
  display_name = "Guard streamed responses"
  type = "ai-aws-guardrails"
  enabled = true
  global = true
  config = jsonencode({
    guardrails_id = var.guardrails_id
    guardrails_version = var.guardrails_version
    aws_region = var.aws_region
    aws_access_key_id = var.aws_access_key_id
    aws_secret_access_key = var.aws_secret_access_key
    guarding_mode = "OUTPUT"
    response_buffer_size = 500
  })


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "guardrails_id" {
  type = string
}
variable "guardrails_version" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_secret_access_key" {
  type = string
}

Copied!

Attach the policy

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

kongctl

model.yaml

Copied!

ai_gateway_models:
  - ref: my-gpt-4o
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: my-gpt-4o
    name: my-gpt-4o
    type: model
    capabilities:
    - generate
    formats:
    - type: openai
    targets:
    - name: gpt-4o
      provider: my-openai-account
      config:
        type: openai
    config:
      route:
        paths:
        - "/v1"
        model:
          body_param: model
          values:
          - my-gpt-4o
      logging:
        payloads: false
    policies:
    - !ref aws-guardrails-guard-streamed-responses#name

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models/my-gpt-4o \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "my-gpt-4o",
      "name": "my-gpt-4o",
      "type": "model",
      "capabilities": [
        "generate"
      ],
      "formats": [
        {
          "type": "openai"
        }
      ],
      "policies": [
        "aws-guardrails-guard-streamed-responses"
      ],
      "targets": [
        {
          "name": "gpt-4o",
          "provider": "my-openai-account",
          "config": {
            "type": "openai"
          }
        }
      ],
      "config": {
        "route": {
          "paths": [
            "/v1"
          ],
          "model": {
            "body_param": "model",
            "values": [
              "my-gpt-4o"
            ]
          }
        },
        "logging": {
          "payloads": false
        }
      }
    }
    '

Copied!

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Model. The tool examples update an existing AI Model with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Model.

resource "konnect_ai_gateway_model" "my_model" {
  display_name = "my-gpt-4o"
  name = "my-gpt-4o"
  type = "model"
  capabilities = ["generate"]
  formats = [
    {
      type = "openai"
    }  ]
  targets = [
    {
      name = "gpt-4o"
      provider = "my-openai-account"

      config = {
        type = "openai"
      }
    }  ]
  config = jsonencode({

    route = {
      paths = ["/v1"]

      model = {
        body_param = "model"
        values = ["my-gpt-4o"]
      }
    }

    logging = {
      payloads = false
    }
  })
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "guardrails_id" {
  type = string
}
variable "guardrails_version" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_secret_access_key" {
  type = string
}

Copied!

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

kongctl

consumer.yaml

Copied!

ai_gateway_consumers:
  - ref: mobile-app-production
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Mobile App - Production
    name: mobile-app-production
    type: api-key
    policies:
    - !ref aws-guardrails-guard-streamed-responses#name

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumers/mobile-app-production \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Mobile App - Production",
      "name": "mobile-app-production",
      "type": "api-key",
      "policies": [
        "aws-guardrails-guard-streamed-responses"
      ]
    }
    '

Copied!

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Consumer. The tool examples update an existing AI Consumer with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer.

resource "konnect_ai_gateway_consumer" "my_consumer" {
  display_name = "Mobile App - Production"
  name = "mobile-app-production"
  type = "api-key"
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "guardrails_id" {
  type = string
}
variable "guardrails_version" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_secret_access_key" {
  type = string
}

Copied!

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

kongctl

consumer_group.yaml

Copied!

ai_gateway_consumer_groups:
  - ref: internal-teams
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Internal Teams
    name: internal-teams
    policies:
    - !ref aws-guardrails-guard-streamed-responses#name

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

curl -X PUT https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/consumer-groups/internal-teams \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Internal Teams",
      "name": "internal-teams",
      "policies": [
        "aws-guardrails-guard-streamed-responses"
      ]
    }
    '

Copied!

After creating the policy, add aws-guardrails-guard-streamed-responses to the policies array of your AI Consumer Group. The tool examples update an existing AI Consumer Group with the policy attached: replace the entity configuration in the example with your own, as the update replaces the entire AI Consumer Group.

resource "konnect_ai_gateway_consumer_group" "my_consumer_group" {
  display_name = "Internal Teams"
  name = "internal-teams"
  policies = [konnect_ai_gateway_policy.my_policy.name]


  gateway_id = konnect_ai_gateway.my_ai_gateway.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "guardrails_id" {
  type = string
}
variable "guardrails_version" {
  type = string
}
variable "aws_region" {
  type = string
}
variable "aws_access_key_id" {
  type = string
}
variable "aws_secret_access_key" {
  type = string
}

Copied!

AI AWS Guardrails Policy

Guard streamed responses

Prerequisites

Environment variables

Set up the policy

Attach the policy

Help us make these docs great!

Still need help?