Allow more time for a judged Switchyard route
Configure the AI Routing Provider policy for a Switchyard route that calls a model to decide, such as llm_classifier.
The decision includes a model call, so the default timeout_ms of 300 would send most requests to default_target.
Set timeout_ms to cover the judge call, and keep it at or below 60000.
Prerequisites
-
A running NVIDIA Switchyard decision service that the data plane can reach.
-
The AI Routing Provider custom policy registered, and the data plane running with
KONG_UNTRUSTED_LUA=on. -
An existing AI Model for each target, with no policies attached, and an AI Model that clients call.