Baseten provider

You can proxy requests to Baseten AI models through AI Gateway by creating AI Model Provider and AI Model entities. This reference documents all supported AI capabilities, configuration requirements, and provider-specific details needed for proper integration.

AI Gateway automatically routes requests to the appropriate Baseten API endpoints. The following table shows the upstream paths used for each capability.

The following tables show the AI capabilities supported by the Baseten provider when configuring AI Models.

By default, AI Gateway uses the path templates shown in the tables below (e.g., /chat/completions, /embeddings, etc.). To customize these paths, configure the config.paths field in your AI Model entity. Custom paths take the form {configured_path}/{template_path} — for example, if you set a custom path of /v2, requests to /embeddings would be routed to /v2/embeddings.

Text generation

Support for Baseten text generation capabilities:

¹ Model APIs use the org-qualified model slug as the target name. For a dedicated deployment, set the target upstream_url to the deployment endpoint.

The base URL is https://inference.baseten.co/v1/{capability_path} or https://model-{model_id}.api.baseten.co/environments/{environment}/sync/v1/chat/completions. The {capability_path} is determined by the AI capability.

AI Gateway uses this URL automatically. You only need to configure a URL if you’re using a self-hosted or Baseten-compatible endpoint, in which case set the upstream_url option in your AI Model configuration.

Baseten has the following limitations in AI Gateway:

• The Baseten provider proxies chat completions only. It doesn’t support Baseten’s Anthropic-compatible /v1/messages endpoint, embeddings, or the /predict and async inference APIs.

• Baseten rejects some OpenAI request parameters. audio, modalities, prediction, web_search_options, and n greater than 1 return a 400 error, service_tier returns a 500 error, and the deprecated function_call and functions return a 501 error.

• Baseten accepts but ignores logprobs, top_logprobs, user, store, metadata, verbosity, safety_identifier, and the explicit prompt cache controls. Baseten caches prompts automatically.

To use Baseten with AI Gateway, configure a new AI Model Provider. You can then route AI Models to Baseten Model APIs or to your own dedicated Baseten deployments.

Create an AI Model Provider entity to store your Baseten API key. Baseten authenticates with an Authorization header:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/model-providers \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "Baseten Production",
      "name": "my-baseten-account",
      "type": "baseten",
      "config": {
        "auth": {
          "type": "basic",
          "headers": [
            {
              "name": "Authorization",
              "value": "'$BASETEN_API_KEY'"
            }
          ]
        }
      }
    }
    '

ai_gateway_model_providers:
  - ref: my-baseten-account
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: Baseten Production
    name: my-baseten-account
    type: baseten
    config:
      auth:
        type: basic
        headers:
        - name: Authorization
          value: !secret {source: !env BASETEN_API_KEY}

The same API key authenticates to both Model APIs and dedicated deployments in your Baseten workspace. One AI Model Provider can serve both kinds of target.

Baseten supports the generate capability (chat completions), with and without streaming. Each target on the AI Model references the Baseten provider and sets config.type to baseten. How you set the target name and upstream_url depends on whether the model runs on Baseten Model APIs or on a dedicated deployment.

Use Model APIs for the shared, hosted models in the Baseten model library. Use a dedicated deployment when you need dedicated GPUs or control over hardware and scaling.

Model APIs

By default, AI Gateway sends requests to Baseten Model APIs at https://inference.baseten.co/v1. Set the target name to the model’s slug, which includes the organization prefix, from the Baseten model library. For example, use deepseek-ai/DeepSeek-V4-Pro-0813 or openai/gpt-oss-120b:

curl -X POST https://{region}.api.konghq.com/v1/ai-gateways/{AIGatewayId}/models \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONNECT_TOKEN" \
    --data '
    {
      "display_name": "DeepSeek V4 Pro on Baseten",
      "name": "deepseek-v4-pro",
      "type": "model",
      "capabilities": [
        "generate"
      ],
      "config": {
        "route": {
          "paths": [
            "/baseten"
          ]
        }
      },
      "targets": [
        {
          "name": "deepseek-ai/DeepSeek-V4-Pro-0813",
          "provider": "my-baseten-account",
          "config": {
            "type": "baseten"
          }
        }
      ]
    }
    '

ai_gateway_models:
  - ref: deepseek-v4-pro
    ai_gateway: !lookup {id: !env AI_GATEWAY_ID}
    display_name: DeepSeek V4 Pro on Baseten
    name: deepseek-v4-pro
    type: model
    capabilities:
    - generate
    config:
      route:
        paths:
        - "/baseten"
    targets:
    - name: deepseek-ai/DeepSeek-V4-Pro-0813
      provider: my-baseten-account
      config:
        type: baseten

Dedicated deployments

To route to a model you deployed on Baseten, set the target upstream_url to the deployment’s full chat completions endpoint:

https://model-MODEL_ID.api.baseten.co/environments/ENVIRONMENT/sync/v1/chat/completions

Replace MODEL_ID with the model ID from your Baseten dashboard. Replace ENVIRONMENT with the deployment environment, for example production. Set the target name to the model name your deployment serves. For engines that take a --served-model-name flag, such as vLLM, use that value:

targets:
  - name: SERVED_MODEL_NAME
    provider: my-baseten-account
    config:
      type: baseten
      upstream_url: https://model-MODEL_ID.api.baseten.co/environments/production/sync/v1/chat/completions

If your deployment exposes an API that the generate capability doesn’t cover, use the passthrough format instead. Set formats[].type to passthrough on the AI Model.

AI Gateway applies the following to Baseten requests and responses:

• Reasoning output: Baseten models return reasoning text in either reasoning_content or thinking. AI Gateway normalizes both to reasoning, in non-streaming responses and in streaming deltas.
• Reasoning effort: AI Gateway forwards reasoning_effort unchanged. Baseten validates the value against the accepted values of the target model.
• Cached tokens: When Baseten reports usage.prompt_tokens_details.cached_tokens, AI Gateway includes it in the response, the streaming usage event, and the logs.

The model field in responses contains the target name, for example deepseek-ai/DeepSeek-V4-Pro-0813.

To call the AI Model, send its name in the model field of the request. For more information, see Request routing rules.