The AI LLM as Judge Policy enables automated evaluation of prompt-response pairs using a dedicated LLM. The Policy assigns a numerical score to LLM responses from 1 to 100, where:
1: Completely incorrect or irrelevant response100: Perfect or ideal responseThe AI LLM as Judge Policy offers several configurable features that control how the LLM evaluates prompts and responses:
|
Feature |
Description |
|---|---|
| Configurable system prompt | Instructs the LLM to act as a strict evaluator. |
| Numerical scoring | Assigns a score from 1–100 to assess response quality. |
| History depth | Includes previous chat messages for context when scoring. |
| Ignore prompts | Options to ignore system, assistant, or tool prompts. |
| Sampling rate | Controls probabilistic request volume for judging. |
| Native LLM schema | Leverages the LLM schema for seamless integration. |
1 (wrong or irrelevant) and 100 (ideal).The following sequence diagram illustrates this simplified flow:
sequenceDiagram
actor Client
participant AIGW as AI Gateway
participant LLM as LLM Model (A or B)
participant Judge as AI LLM as Judge
participant JudgeLLM as Judge LLM
Client->>AIGW: Send prompt
AIGW->>LLM: Forward prompt (balancer selects model)
LLM-->>AIGW: Response
AIGW ->>Judge: Prompt + response
Judge->>JudgeLLM: Evaluate response
JudgeLLM-->>Judge: Score (1–100)
Judge-->>AIGW: Evaluation result
AIGW-->>Client: Response
To ensure concise, consistent scoring, configure the LLM that acts as the judge with these values:
|
Setting |
Recommended value |
Description |
|---|---|---|
temperature
|
2
|
Controls randomness. A lower value leads to a more deterministic output. |
max_tokens
|
5
|
Maximum tokens for the LLM response. |
top_p
|
1
|
Nucleus sampling probability; limits token selection. |
These settings produce short, precise numeric scores without extra text or verbosity.