AI NVIDIA NeMo Guardrail Policy

The AI NVIDIA NeMo Guardrail Policy inspects requests and responses handled by the AI Model entity and checks them against NVIDIA NeMo Guardrails. AI Gateway sends the content it extracts to the NeMo Guardrails microservice, and blocks any request or response that violates the safety rails you configure.

This AI Policy doesn’t evaluate content. It delegates every decision to NeMo Guardrails, so the rails you define in NeMo (content safety, topic control, jailbreak detection, or your own Colang flows) determine what AI Gateway allows through.

Before using the AI NVIDIA NeMo Guardrail Policy, you need:

• A running NeMo Guardrails microservice that AI Gateway can reach. The microservice listens on port 7331 by default, and the Policy calls its /v1/guardrail/checks endpoint.

• At least one guardrail configuration. You can store configurations on the NeMo server and reference them by ID, or send a configuration inline with every check. For more information, see Manage guardrail configurations in the NVIDIA documentation.

The AI NVIDIA NeMo Guardrail Policy can be applied to:

• Input data (requests)
• Output data (responses)
• Both input and output data

Here’s how it works if you apply it to both requests and responses:

1. The AI Policy intercepts the request and sends the extracted text to the NeMo Guardrails microservice.
   • NeMo runs the configured input rails and returns a pass or block status.
2. If NeMo allows the content, AI Gateway forwards the request to the upstream model.
3. The Policy intercepts the response from the upstream provider and sends the extracted text to NeMo Guardrails.
   • NeMo runs the configured output rails and returns a pass or block status.
4. If NeMo allows the content, AI Gateway forwards the response to the client.

 
sequenceDiagram
    autonumber
    participant Client
    participant Policy as AI NVIDIA NeMo Guardrail
    participant NeMo as NeMo Guardrails
    participant AI as Upstream AI Service

    Client->>Policy: Send request
    Policy->>NeMo: Intercept & send extracted request text
    NeMo->>NeMo: Run configured input rails
    NeMo->>Policy: Return pass or block status
    Policy->>AI Gateway: Forward allowed request
    AI Gateway->>AI: Process allowed request
    AI->>AI Gateway: Return AI response
    AI Gateway->>Policy: Forward response
    Policy->>NeMo: Intercept & send extracted response text
    NeMo->>NeMo: Run configured output rails
    NeMo->>Policy: Return pass or block status
    Policy->>Client: Return allowed response
  

Figure 1: Diagram showing the request and response flow with the AI NVIDIA NeMo Guardrail Policy.

Guarding mode

By default, the Policy checks requests only. Use config.guarding_mode to change which phases it inspects:

To control which parts of the conversation the Policy sends for evaluation, use config.text_source. Set it to concatenate_user_content to check only user input, or concatenate_all_content to include the full exchange, including system and assistant messages.

Match the rails defined in your NeMo guardrail configuration to the phases named in config.guarding_mode. A configuration used with guarding_mode: INPUT should define only an input rail; a configuration used with guarding_mode: OUTPUT should define only an output rail. NeMo evaluates whichever rails a configuration defines every time it’s called, regardless of guarding_mode, so an output rail included in an input-only check runs against an empty response.

The config.guardrails field selects which NeMo guardrail configuration to apply.

Reference a single stored configuration

Use config_id to reference a configuration that’s already stored on the NeMo server:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "AI NVIDIA NeMo Guardrail - Stored Configuration",
      "name": "ai-nvidia-nemo-guardrail",
      "type": "ai-nvidia-nemo-guardrail",
      "config": {
        "nemo_endpoint": "http://nemo:7331/v1/guardrail/checks",
        "model": "gpt-4o",
        "guarding_mode": "INPUT",
        "timeout": 30000,
        "auth": {
          "api_key": "'$OPENAI_API_KEY'",
          "header": "X-Model-Authorization",
          "prefix": ""
        },
        "guardrails": {
          "config_id": "content_safety"
        }
      }
    }
    '

ai_gateway_policies:
  - ref: ai-nvidia-nemo-guardrail
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: AI NVIDIA NeMo Guardrail - Stored Configuration
    name: ai-nvidia-nemo-guardrail
    type: ai-nvidia-nemo-guardrail
    config:
      nemo_endpoint: http://nemo:7331/v1/guardrail/checks
      model: gpt-4o
      guarding_mode: INPUT
      timeout: 30000
      auth:
        api_key: !env OPENAI_API_KEY
        header: X-Model-Authorization
        prefix: ''
      guardrails:
        config_id: content_safety

Send an inline configuration

Use config to send a complete NeMo guardrail configuration with every check, without storing anything on the NeMo server:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/policies \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "AI NVIDIA NeMo Guardrail - Inline Configuration",
      "name": "ai-nvidia-nemo-guardrail",
      "type": "ai-nvidia-nemo-guardrail",
      "config": {
        "nemo_endpoint": "http://nemo:7331/v1/guardrail/checks",
        "model": "gpt-5.1",
        "guarding_mode": "INPUT",
        "timeout": 30000,
        "auth": {
          "api_key": "'$OPENAI_API_KEY'",
          "header": "X-Model-Authorization",
          "prefix": ""
        },
        "guardrails": {
          "config": {
            "models": [
              {
                "type": "content_safety",
                "engine": "openai",
                "model": "gpt-5.1"
              }
            ],
            "rails": {
              "input": {
                "flows": [
                  "content safety check input $model=content_safety"
                ]
              }
            },
            "prompts": [
              {
                "task": "content_safety_check_input $model=content_safety",
                "models": [
                  "openai/gpt-5.1"
                ],
                "content": "Task: Check if there is unsafe content in the user message.\n",
                "output_parser": "nemoguard_parse_prompt_safety",
                "max_tokens": 50
              }
            ]
          }
        }
      }
    }
    '

ai_gateway_policies:
  - ref: ai-nvidia-nemo-guardrail
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: AI NVIDIA NeMo Guardrail - Inline Configuration
    name: ai-nvidia-nemo-guardrail
    type: ai-nvidia-nemo-guardrail
    config:
      nemo_endpoint: http://nemo:7331/v1/guardrail/checks
      model: gpt-5.1
      guarding_mode: INPUT
      timeout: 30000
      auth:
        api_key: !env OPENAI_API_KEY
        header: X-Model-Authorization
        prefix: ''
      guardrails:
        config:
          models:
          - type: content_safety
            engine: openai
            model: gpt-5.1
          rails:
            input:
              flows:
              - content safety check input $model=content_safety
          prompts:
          - task: content_safety_check_input $model=content_safety
            models:
            - openai/gpt-5.1
            content: 'Task: Check if there is unsafe content in the user message.
                 '
            output_parser: nemoguard_parse_prompt_safety
            max_tokens: 50

AI Gateway forwards the contents of config.guardrails.config to NeMo as-is and performs no validation on it. This keeps the Policy compatible with future NeMo releases, but it also means configuration errors surface only when NeMo rejects the check. NeMo returns the error message, and AI Gateway passes it to the client.

For the full set of supported keys, see the NeMo Guardrails configuration reference in the NVIDIA documentation.

When you use an inline configuration, make sure the model you name in guardrails.config.models is available to the NeMo microservice. Depending on how you deploy NeMo, you might need to register it first with the POST /v1/guardrail/models endpoint. For more information, see Manage models in the NVIDIA documentation.

The NeMo Guardrails microservice needs credentials for the LLM that evaluates your rails. Set the key in config.auth.api_key, and AI Gateway forwards it to NeMo on every check.

Two fields control how AI Gateway builds the header:

• config.auth.header: The header name. Defaults to X-Model-Authorization.
• config.auth.prefix: The string prepended to the key. Defaults to Bearer .

config.auth.api_key is a referenceable and encrypted field, so you can store the value in a Vault instead of in your configuration.

• Do not change config.auth.header unless your NeMo deployment expects a different header. The NeMo microservice reads X-Model-Authorization to resolve which LLM provider to use for the guardrail check. If the header is missing, NeMo falls back to its configured default provider, and your rails may run against a model you did not intend.
• For a NeMo model configured with engine: openai, set config.auth.prefix to an empty string (""). NeMo passes the X-Model-Authorization header value straight to its OpenAI client as the API key, without stripping a Bearer scheme from it first, so the default prefix becomes part of the credential and the check fails to authenticate.

When NeMo reports a violation, AI Gateway blocks the content and returns a failure message. You can customize the messages for each phase:

• config.request_failure_message: Returned when a request is blocked.
• config.response_failure_message: Returned when a response is blocked.

Two other fields change how the Policy blocks violations:

• config.allow_masking: Masks violating content instead of blocking the request or response. Enabling this field disables streaming, because the Policy needs the complete payload to mask it.
• config.stop_on_error: Controls what happens when the check itself fails, for example when the NeMo service is unreachable or config.timeout expires. When enabled, which is the default, AI Gateway stops processing the request. Disable it to fail open and let traffic through unchecked.

Setting stop_on_error: false means an outage in the NeMo Guardrails service silently disables your safety rails. Only disable it when availability matters more than enforcement.

The AI NVIDIA NeMo Guardrail Policy emits structured log data for every check it runs, under ai.proxy.nvidia-nemo-guardrail in the request log. For the full list of log fields, see the AI Gateway audit log reference.

To log the raw content of blocked requests and responses, enable config.log_blocked_content. This field is disabled by default. When enabled, the blocked prompt or response body appears under ai.proxy.nvidia-nemo-guardrail.input_faulty_prompt and ai.proxy.nvidia-nemo-guardrail.output_faulty_response in each log entry.

Blocked prompts and responses can contain sensitive or unsafe content. Enable config.log_blocked_content only when your logging pipeline is authorized to store that data.

Set config.proxy on this entity to route its outbound requests through an HTTP forward proxy. Use this in network-isolated deployments where AI Gateway cannot open direct connections to LLM providers or auxiliary services.

The proxy record is identical for AI Model, AI MCP Server, and supported AI Policy entities. Existing capabilities such as load balancing, health checking, streaming, WebSocket, and HTTP/2 continue to work when the proxy is active.

For the full field reference, traffic flow, and limitations, see Forward proxy support.