05 Feb 2021

Set Default Annotations with Kyverno

Kyverno is a policy engine for kubernetes. It allows you to implement ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks by creating (Cluster)Policy custom resources. Kyverno is less powerful than rego powered OpenPolicyAgent, but it is also easier to write.

Today I had to set an annotation to an Ingress resource if the user does not provide any value for this annotation.

This is very easy using the Add if not present anchor in a mutate rule. In our use case we want to set the alb.ingress.kubernetes.io/certificate-arn to the value arn:aws:acm:eu-central-1:XXXXXX:certificate/12345 if the user does not provide a value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-default-arn
spec:
  rules:
    - name: set-default-arn
      match:
        resources:
          kinds:
            - Ingress
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              +(alb.ingress.kubernetes.io/certificate-arn): "arn:aws:acm:eu-central-1:XXXXXX:certificate/12345"

After applying the above ClusterPolicy, we can create this ingress:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: helloworld
spec:
  rules:
    - host: helloworld.example.com
      http:
        paths:
          - backend:
              serviceName: helloworld
              servicePort: http
            path: /
            pathType: ImplementationSpecific

The resulting Ingress will have the annotation alb.ingress.kubernetes.io/certificate-arn set to the value arn:aws:acm:eu-central-1:XXXXXX:certificate/12345:

1
2
3
4


% kubectl apply -f ingress.yaml               
ingress.extensions/helloworld created
% kubectl get ingress helloworld -o json | jq '.metadata.annotations["alb.ingress.kubernetes.io/certificate-arn"]'
"arn:aws:acm:eu-central-1:XXXXXX:certificate/12345"

If we create the following Ingress, the kyverno policy will not change the annotation at all:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: "dont-touch-this"
  name: helloworld-dont-touch
spec:
  rules:
    - host: helloworld.example.com
      http:
        paths:
          - backend:
              serviceName: helloworld
              servicePort: http
            path: /
            pathType: ImplementationSpecific

Create the resource and observe the value of the annotation:

1
2
3
4


% kubectl apply -f ingress-dont-touch.yaml               
ingress.extensions/helloworld-dont-touch created
% kubectl get ingress helloworld -o json | jq '.metadata.annotations["alb.ingress.kubernetes.io/certificate-arn"]'
"dont-touch-this"

Combining this with the feature of matching rules on annotation, we can apply this annotation only to certain ALB ingress groups. The AWS loadbalancer controller, will create an ALB ingress for every value of alb.ingress.kubernetes.io/group.name. Certain settings must be equal across all ingresses from the same group. The following kyverno policy allows us to set the alb.ingress.kubernetes.io/scheme annotation to the same default value in the ingress group internal:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-default-internal-scheme
spec:
  rules:
    - name: set-default-internal-scheme
      match:
        resources:
           annotations:
             alb.ingress.kubernetes.io/group.name: "internal"
             kubernetes.io/ingress.class: "alb"
          kinds:
            - Ingress
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              +(alb.ingress.kubernetes.io/scheme): "internal"

After applying this ClusterPolicy every Ingress with the ingress class alb annotation, the ALB group internal will have the alb.ingress.kubernetes.io/scheme annotation set to internal if not set. If we want to enforce the value ìnternal for the annotation alb.ingress.kubernetes.io/scheme we can apply the following policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-internal-scheme
spec:
  rules:
    - name: enforce-internal-scheme
      match:
        resources:
           annotations:
             alb.ingress.kubernetes.io/group.name: "internal"
             kubernetes.io/ingress.class: "alb"
          kinds:
            - Ingress
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              alb.ingress.kubernetes.io/scheme: "internal"

The difference is the missing +() anchor around the annotation key. This policy will overwrite any value of alb.ingress.kubernetes.io/scheme a created or changed ingress will have.