Kong AI Gateway resource sizing guidelines

The Kong AI Gateway is designed to handle high‑volume inference workloads and forward requests to large language model (LLM) providers with predictable latency. This guide explains performance dimensions, capacity planning methodology, and baseline sizing guidance for AI inference traffic.

AI inference performance depends on both token streaming latency and sustained token throughput. Unlike traditional API traffic, most latency comes from upstream models, so the gateway must be evaluated on its ability to pass through tokens efficiently.

Model streams output tokens in server‑sent events (SSE). Processing streamed output is more expensive per token than input, so capacity planning must treat input and output tokens differently.

AI Gateway scales primarily through horizontal worker expansion, not vertical tuning. Treat token throughput as the core capacity metric, and validate performance against real LLM latency profiles. Synthetic or low-latency backends will overstate capacity.

Scale horizontally for token throughput

Kong AI Gateway performance is CPU-bound on token processing. Adding workers increases sustained throughput only when concurrency and streaming behavior scale correctly.

• Add workers and nodes to increase throughput
• Validate scaling efficiency as concurrency grows
• Benchmark against real model latency and token cadence

Allocate CPU and memory for LLM workloads

Compute sizing is dictated by token processing, not request count. Memory supports configuration and streaming buffers. Persistent storage demand is minimal.

• CPU determines maximum tokens per second
• Memory must support configuration and in-memory stream buffers

Use dedicated compute instance classes

Consistent CPU performance is critical for LLM token streaming. Burstable or credit-based instances can introduce token delay spikes and unstable throughput.

• Prefer dedicated compute families (for example, AWS c5, c6g)
• Avoid burstable instances (for example, AWS t, GCP e2, Azure B series)

Effective scaling requires testing with realistic model behavior, applying safety margins, and accommodating upstream model differences.

• Benchmark with your model mix and prompt sizes
• Size for token/s, not just RPS
• Apply redundancy factor 2×–4×
• Consider provider differences (OpenAI vs Gemini)
• Test multi‑node scaling before production

These baseline throughput numbers reflect typical single-worker token processing under streaming LLM workloads. Use these numbers as general guidance only. Benchmark performance in your own environment and with your specific model mix.

Throughput depends on the provider, the model, and the size and structure of your prompts and responses. Benchmark with your real workload to measure accurate throughput and avoid relying on synthetic or idealized figures.

equivalent_output_load = I_peak / R + O_peak
required_workers ≈ equivalent_output_load / O_w

Use redundancy factor 2×–4x- to handle burst, tokenization, and provider variability.

Quick estimate rule of thumb

• 4:1 input:output ratio
• ~1M output tokens/s per vCPU worker

(80M / 4 + 10M) / 1M = 30 workers
→ 60–120 workers w/ redundancy

Inference requests often include large prompts and streamed output. Buffer sizing determines whether payloads are processed in memory or spill to disk, so tune memory settings based on prompt size and workload profile.

AI Gateway benefits from high clock speed, dedicated CPU, and non-burstable compute classes. Select instance families optimized for consistent CPU throughput and avoid throttled instance types.

Cluster size depends on configured entities and sustained token throughput. Smaller environments serve team-level workloads; larger footprints handle multi-tenant platforms and enterprise AI adoption at scale.