Reduce the number of Nginx timers used by the Rate Limiting Advanced plugin

If you are encountering issues with log alerts along the lines of “too many pending timers” or “too many running timers”, we will want to identify the cause of this large use of timers as the default values should be sufficient in most cases. One of the plugins that can use a lot of timers is the Rate Limiting Advanced plugin. This is especially true when there are a large number of plugin instances and they span a number of different workspaces. Timers are essentially ‘connections’ in nginx terms, and therefore using too many timers can reduce Kong’s proxy performance and cause other unintended behavior around vitals, configuration propagation etc.

The namespace parameter of the Rate Limiting Advanced plugin is auto generated by default. Counter data and sync configuration are shared in a namespace, each namespace will loosely equate to one timer instance. Therefore if we allow these to be auto generated we will end up with a large number of namespaces and timers.

In most cases we can reduce the number of timers to just 1 by sharing the same namespace string across all our plugin instances. The only consideration here is that we do not have conflicting keys within the namespace so that our counters are not sharing a key and behaving differently than expected. Therefore, it is best to test this change in a development environment by setting every namespace to a string of your choosing and testing various rate limiting scenarios to confirm that the keys are not conflicting in the namespace and that you maintain the same limiting behavior you had prior to the change.

For example:

1. If we have 3 Rate Limiting Advanced plugins with a unique string namespace value you will see the following keys in Redis:

redis-cli keys *
1) "1601596980:60:mynamespace3"
2) "1601596980:60:mynamespace1"
3) "1601596980:60:mynamespace2"

hgetall 1601597280:60:mynamespace1
1) "2fc35a0d-1821-460e-a6a5-2514e888c37b"
2) "1"

1. If you change the namespace parameter of each plugin to the same string mynamepace you will see the following data in Redis:

redis-cli keys *
1) "1601597400:60:mynamespace"

hgetall 1601597400:60:mynamespace
1) "157ad987-9d50-41d9-9ec9-a242e8e56320"
2) "1"
3) "81414cbd-7ee3-4eb5-bde0-2c638ebb381e"
4) "1"
5) "2fc35a0d-1821-460e-a6a5-2514e888c37b"
6) "1"

In this case the key is made up of the window_start:window_size:namespace, from there the counts are based on the identifier id, in this case consumer id.