Deploy Kong AI Gateway with Kong Operator

The KonnectAIGateway resource creates the AI Gateway control plane in Konnect and serves as the parent for all other resources in this guide.

1. Create the KonnectAIGateway resource:

   echo '
   apiVersion: konnect.konghq.com/v1alpha1
   kind: KonnectAIGateway
   metadata:
     name: my-ai-gateway-cp
     namespace: kong
   spec:
     apiSpec:
       name: my-ai-gateway-cp
       displayName: My AI Gateway
       description: AI Gateway control plane managed by Kubernetes
     konnect:
       authRef:
         name: konnect-api-auth
   ' | kubectl apply -f -

   Copied!

2. Wait for the resource to be ready:

   kubectl wait konnectaigateway/my-ai-gateway-cp -n kong \
     --for=condition=Programmed=True \
     --timeout=10m

   Copied!

The AIGatewayModelProvider resource configures authentication and connection details for an upstream LLM provider. This example uses OpenAI.

1. Store your OpenAI API key in a Kubernetes Secret. The value includes the Bearer prefix because it is used directly as an HTTP Authorization header:

   kubectl create secret generic openai-credentials \
     --from-literal=token="Bearer ${OPENAI_API_KEY}" \
     -n kong
   kubectl label secret openai-credentials konghq.com/secret=true -n kong

   Copied!

2. Create the AIGatewayModelProvider resource, referencing the Secret:

   echo '
   apiVersion: aiconfiguration.konghq.com/v1alpha1
   kind: AIGatewayModelProvider
   metadata:
     name: openai-provider
     namespace: kong
   spec:
     aiGatewayRef:
       type: namespacedRef
       namespacedRef:
         name: my-ai-gateway-cp
     apiSpec:
       type: openai
       openai:
         name: openai-provider
         displayName: OpenAI
         config:
           auth:
             headers:
               - name: Authorization
                 value:
                   type: secretRef
                   secretRef:
                     name: openai-credentials
                     key: token
   ' | kubectl apply -f -

   Copied!

3. Wait for the resource to be ready:

   kubectl wait aigatewaymodelprovider/openai-provider -n kong \
     --for=condition=Programmed=True \
     --timeout=10m

   Copied!

The AIGatewayModel resource defines a Route and maps it to one or more provider targets. Clients send inference requests to the path configured here.

1. Create the AIGatewayModel resource:

   echo '
   apiVersion: aiconfiguration.konghq.com/v1alpha1
   kind: AIGatewayModel
   metadata:
     name: gpt-4o-mini
     namespace: kong
   spec:
     aiGatewayRef:
       type: namespacedRef
       namespacedRef:
         name: my-ai-gateway-cp
     apiSpec:
       type: model
       model:
         name: gpt-4o-mini
         displayName: GPT-4o Mini
         enabled: Enabled
         formats:
           - type: openai
         capabilities:
           - generate
         config:
           route:
             paths:
               - /v1
         targets:
           - name: gpt-4o-mini
             provider:
               name: openai-provider
             config:
               type: openai
               openai:
                 upstreamURL: https://api.openai.com/v1/chat/completions
   ' | kubectl apply -f -

   Copied!

2. Wait for the resource to be ready:

   kubectl wait aigatewaymodel/gpt-4o-mini -n kong \
     --for=condition=Programmed=True \
     --timeout=10m

   Copied!

The AIGatewayDataPlane resource runs the AI Gateway binary inside your Kubernetes cluster. It exposes a LoadBalancer Service on port 8000 for inference requests.

1. Deploy the AIGatewayDataPlane:

   echo '
   apiVersion: aigateway.konghq.com/v1alpha1
   kind: AIGatewayDataPlane
   metadata:
     name: my-ai-gateway-dp
     namespace: kong
   spec:
     controlPlaneRef:
       type: konnectNamespacedRef
       konnectNamespacedRef:
         name: my-ai-gateway-cp
     deployment:
       replicas: 1
     network:
       services:
         ingress:
           type: LoadBalancer
           ports:
             - name: http
               port: 8000
               targetPort: 8000
   ' | kubectl apply -f -

   Copied!

2. Wait for the data plane to be ready:

   kubectl wait aigatewaydataplane/my-ai-gateway-dp -n kong \
     --for=condition=Ready=True \
     --timeout=10m

   Copied!

Export the LoadBalancer address:

export AIGW_HOST=$(kubectl get service my-ai-gateway-dp-ingress -n kong \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $AIGW_HOST

Send a request to the AI Model Route you configured:

curl -s http://$AIGW_HOST:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello from Kong AI Gateway!"}]
  }'

You should receive a response from OpenAI routed through the AI Gateway data plane.