AI Semantic Cache

Release date 2025/03/27

Feature

• Added support for boto3 SDKs for Bedrock provider, and for Google GenAI SDKs for Gemini provider.

• Added the huggingface, azure, vertex, and bedrock providers to embeddings. They can be used by the ai-proxy-advanced, ai-semantic-cache, ai-semantic-prompt-guard, and ai-rag-injector plugins.

• Allow authentication to Bedrock services with assume roles in AWS.

Bugfix

• Fixed an issue where AI upstream URL trailing would be empty.

• Fixed an issue where the Refresh header wasn’t properly sent to the client.

• Fixed issue where the SSE body may have extra trailing.

Release date 2025/03/20

Bugfix

• Fixed issue of SSE body may have extra trailing in some cases.

Release date 2024/12/12

Feature

• Made the embeddings.model.name config field a free text entry, enabling use of a self-hosted (or otherwise compatible) model.

• Added ignore_tool configuration option to discard tool role prompts from the input text.

• Plugin can now be enabled on Consumer Groups.

Bugfix

• Fixed an bug that AI semantic cache can’t use request provided models

• Fixed the exact matching to catch everything including embeddings.

• Fixed an issue where the ai-semantic-cache plugin would abort in stream mode when another plugin enable the buffering proxy mode.

• Fixed an issue where the ai-semantic-cache plugin put the wrong type value in the metrics when using the prometheus plugin.

• Fixed an issue where the plugin failed when handling requests with multiple models.

Release date 2024/11/04

Bugfix

• Fixed an bug that AI semantic cache can’t use request provided models

• Fixed an issue where the ai-semantic-cache plugin would abort in stream mode when another plugin enable the buffering proxy mode.

• Fixed an issue where the ai-semantic-cache plugin put the wrong type value in the metrics when using the prometheus plugin.

Release date 2024/09/11

Feature

• allow AI plugin to read request from buffered file

• Introduced AI Semantic Caching plugin, enabling you to configure an embeddings-based caching system for Large Language Model responses.

Bugfix

• Fix the ai-semantic-caching plugin with a condition for calculating latencies when no embeddings, add deep copy for the request table and fix countback.