Cohere provider

Uses: Kong Gateway AI Gateway Admin API deck KIC Konnect API Terraform

Upstream paths

AI Gateway automatically routes requests to the appropriate Cohere API endpoints. The following table shows the upstream paths used for each capability.

Capability	Upstream path or API
Chat completions	`/v1/chat`
Completions	`/v1/generate`
Embeddings	`/v2/embed`
Function calling	`/v1/chat`

Supported capabilities

The following tables show the AI capabilities supported by Cohere provider when used with the AI Proxy or the AI Proxy Advanced plugin.

Set the plugin’s route_type based on the capability you want to use. See the tables below for supported route types.

Text generation

Support for Cohere basic text generation capabilities including chat, completions, and embeddings:

Capability	Route type	Model example	Min version
Chat completions	`llm/v1/chat`	command-a-03-2025	3.6
Completions	`llm/v1/completions`	command-r-plus-08-2024	3.6
Embeddings	`llm/v1/embeddings`	embed-english-v3.0	3.11

Advanced text generation

Support for Cohere function calling to allow Cohere models to use external tools and APIs:

Capability	Route type	Model example	Min version
Function calling	`llm/v1/chat`	command-a-03-2025	3.6

Cohere base URL

The base URL is https://api.cohere.com:443/{route_type_path}, where {route_type_path} is determined by the capability.

AI Gateway uses this URL automatically. You only need to configure a URL if you’re using a self-hosted or Cohere-compatible endpoint, in which case set the upstream_url plugin option.

Supported native LLM formats for Cohere

By default, the AI Proxy plugin uses OpenAI-compatible request formats. Set config.llm_format to a native format to use Cohere-specific APIs and features.

The following native Cohere APIs are supported:

LLM format	Supported APIs
`cohere`	`/v1/rerank` `/v2/rerank`

Configure Cohere with AI Proxy

To use Cohere with AI Gateway, configure the AI Proxy or AI Proxy Advanced.

Here’s a minimal configuration for chat completions:

kong.yaml

Copied!

_format_version: "3.0"
plugins:
  - name: ai-proxy
    config:
      route_type: llm/v1/chat
      auth:
        header_name: Authorization
        header_value: Bearer ${{ env "DECK_COHERE_API_KEY" }}
      model:
        provider: cohere
        name: command-a-03-2025
        options:
          max_tokens: 512
          temperature: 1.0

curl -i -X POST http://localhost:8001/plugins/ \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "ai-proxy",
      "config": {
        "route_type": "llm/v1/chat",
        "auth": {
          "header_name": "Authorization",
          "header_value": "Bearer '$COHERE_API_KEY'"
        },
        "model": {
          "provider": "cohere",
          "name": "command-a-03-2025",
          "options": {
            "max_tokens": 512,
            "temperature": 1.0
          }
        }
      }
    }
    '

Copied!

Make the following request:

curl -X POST https://{region}.api.konghq.com/v2/control-planes/{controlPlaneId}/core-entities/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "name": "ai-proxy",
      "config": {
        "route_type": "llm/v1/chat",
        "auth": {
          "header_name": "Authorization",
          "header_value": "Bearer '$COHERE_API_KEY'"
        },
        "model": {
          "provider": "cohere",
          "name": "command-a-03-2025",
          "options": {
            "max_tokens": 512,
            "temperature": 1.0
          }
        }
      }
    }
    '

Copied!

echo "
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: ai-proxy
  namespace: kong
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: 'true'
config:
  route_type: llm/v1/chat
  auth:
    header_name: Authorization
    header_value: Bearer $COHERE_API_KEY
  model:
    provider: cohere
    name: command-a-03-2025
    options:
      max_tokens: 512
      temperature: 1.0
plugin: ai-proxy
" | kubectl apply -f -

Copied!

resource "konnect_gateway_plugin_ai_proxy" "my_ai_proxy" {
  enabled = true

  config = {
    route_type = "llm/v1/chat"

    auth = {
      header_name = "Authorization"
      header_value = "Bearer var.cohere_api_key"
    }

    model = {
      provider = "cohere"
      name = "command-a-03-2025"

      options = {
        max_tokens = 512
        temperature = 1.0
      }
    }
  }

  control_plane_id = konnect_gateway_control_plane.my_konnect_cp.id
}

Copied!

This example requires the following variables to be added to your manifest. You can specify values at runtime by setting TF_VAR_name=value.

variable "cohere_api_key" {
  type = string
}

Copied!

For more configuration options and examples, see:

AI Proxy examples

AI Proxy Advanced examples

FAQs

How do I use Cohere’s document-grounded chat for RAG pipelines?

Configure AI Proxy with the Cohere provider and send queries with candidate documents. The model filters for relevance and returns answers with citations. See Use Cohere rerank API for document-grounded chat.

Cohere provider

Upstream paths

Supported capabilities

Text generation

Advanced text generation

Cohere base URL

Supported native LLM formats for Cohere

Configure Cohere with AI Proxy

Tutorials

FAQs

Help us make these docs great!

Still need help