When Kong Ingress Controller fails to apply translated Kong Gateway configuration to Kong Gateway, Kong Ingress Controller will try to recover from the failure and record the failure into logs, Kubernetes events, and Prometheus metrics. Recovery usually fails because the translated configuration is rejected by Kong Gateway.
If Kong Ingress Controller fails to apply the translated configuration, it then tries to apply the last successful Kong Gateway configuration to new instances of Kong Gateway to attempt a best effort at making them available.
If the FallbackConfiguration
feature gate is enabled, Kong Ingress Controller discovers the Kubernetes objects that caused the invalid configuration, and tries to build a fallback configuration from valid objects and parts of the last valid configuration that are built from the broken objects. See fallback configuration for more information.
You can observe failures in applying configuration from Kubernetes events and Prometheus metrics:
- Kong Ingress Controller generates an event with the
Warning
type and the KongConfigurationApplyFailed
reason attached to the pod itself when it fails to apply the configuration.
- For each object that causes the invalid configuration, Kong Ingress Controller generates a
Warning
event type and the KongConfigurationApplyFailed
reason attached to the object.
- The Prometheus metric
ingress_controller_configuration_push_count
with the success=false
label shows the total number of failures from applying the configuration by reason and URL of Kong Gateway Admin API.
- The Prometheus metric
ingress_controller_configuration_push_broken_resource_count
reflects the number of Kubernetes resources that caused the error in the last configuration push.
For example, let’s say you create an Ingress
with the ImplementationSpecific
path type and an invalid regex in Path
(which can only be only be done when the validating webhook is disabled, otherwise it will be rejected by the webhook):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
konghq.com/strip-path: "true"
name: ingress-invalid-regex
namespace: default
spec:
ingressClassName: kong
rules:
- http:
paths:
- backend:
service:
name: httpbin-deployment
port:
number: 80
path: /~^^/a$
pathType: ImplementationSpecific
You can get the Kubernetes events:
kubectl get events --all-namespaces --field-selector reason=KongConfigurationApplyFailed
Both the events attached to the invalid ingress and attached to the Kong Ingress Controller pod are recorded:
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
default 2m9s Warning KongConfigurationApplyFailed ingress/ingress-invalid-regex invalid paths.1: should start with: / (fixed path) or ~/ (regex path)
kong 15s Warning KongConfigurationApplyFailed pod/kong-controller-779cb796f4-7q7c2 failed to apply Kong configuration to https://10.244.1.43:8444: HTTP status 400 (message: "failed posting new config to /config")
To see more details about the HTTP 400
error, enable the dump config setting on the controller.