AI Proxy Advanced

Release date 2025/07/03

Deprecation

• ai-proxy, ai-proxy-advanced: Deprecated the preserve route_type. You are encouraged to use new route_types added in version 3.11.x.x and onwards.

Bugfix

• Add tried_targets field in serialized analytics logs for record of all tried ai targets.

• Fixed an issue where some of ai metrics was missed in analytics

• Fixed an issue where AI Proxy Advanced can’t failover from other provider to Bedrock.

• Fix consistent-hashing algorithm not using correct value to hash.

• Fixed an issue where the stale semantic key vector may not be refreshed after the plugin config is updated.

• Fixed an issue where AI Proxy and AI Proxy Advanced would use corrupted plugin config.

Known Issues

• If any AI Gateway plugin has been enabled in a self-managed Kong Gateway deployment for more than a week, upgrades from 3.10 versions to 3.11.0.0 will fail due to a license migration issue. This does not affect Konnect deployments.

  A fix will be provided in 3.11.0.1.

  See breaking changes in 3.11 for a temporary workaround.

Release date 2025/07/06

Bugfix

• Fixed an issue where the stale semantic key vector may not be refreshed after the plugin config is updated.

Release date 2025/04/15

Bugfix

• Fixed an issue where AI Proxy and AI Proxy Advanced would use corrupted plugin config.

Release date 2025/03/27

Breaking Change

• Changed the serialized log key of AI metrics from ai.ai-proxy to ai.proxy to avoid conflicts with metrics generated from plugins other than AI Proxy and AI Proxy Advanced. If you are using logging plugins (for example, File Log, HTTP Log, etc.), you will have to update metrics pipeline configurations to reflect this change.

Deprecation

• Deprecated preserve mode in config.route_type. Use config.llm_format instead. The preserve mode setting will be removed in a future release.

Feature

• Added support for boto3 SDKs for Bedrock provider, and for Google GenAI SDKs for Gemini provider.

• Added new priority balancer algorithm, which allows setting apriority group for each upstream model.

• Added the failover_criteria configuration option, which allows retrying requests to the next upstream server in case of failure.

• Added cost to tokens_count_strategy when using the lowest-usage load balancing strategy.

• Added the huggingface, azure, vertex, and bedrock providers to embeddings. They can be used by the ai-proxy-advanced, ai-semantic-cache, ai-semantic-prompt-guard, and ai-rag-injector plugins.

• Allow authentication to Bedrock services with assume roles in AWS.

• Added the ability to set a catch-all target in semantic routing.

Bugfix

• Fixed an issue where AI upstream URL trailing would be empty.

• Fixed an issue where the ai-proxy-advanced plugin failed to failover between providers of different formats.

• Fixed an issue where the ai-proxy-advanced plugin identity running failed in retry scenarios.

Release date 2025/07/07

Bugfix

• Fixed an issue where AI Proxy and AI Proxy Advanced would use corrupted plugin config.

Release date 2024/12/12

Feature

• Added support for streaming responses to the AI Proxy Advanced plugin.

• Made the embeddings.model.name config field a free text entry, enabling use of a self-hosted (or otherwise compatible) model.

Bugfix

• Fixed an issue where stale plugin config was not updated in dbless and hybrid mode.

• Fixed an issue where lowest-usage and lowest-latency strategy did not update data points correctly.

Release date 2024/11/04

Bugfix

• Fixed an issue where stale plugin config was not updated in dbless and hybrid mode.

• Fixed an issue where lowest-usage and lowest-latency strategy did not update data points correctly.

Release date 2024/09/11

Feature

• allow AI plugin to read request from buffered file

• Added the ai-proxy-advanced plugin that supports advanced load balancing between LLM services.