Custom cache for Kubernetes Controllers
When writing a Kubernetes controller using controller runtime, you can configure the cache that controller runtime uses to back the integrated client.
When providing options to create the manager, we can configure how the cache operates by configuring the cache.Options.
options := ctrl.Options{Cache: customCache()}
Specifically, we are interested in the ByObject functionality.
ByObjectrestricts the cache’s ListWatch to the desired fields per GVK at the specified object.
This allows us to provide a map of types we want to filter along with a cache.TransformFunc.
In the example below, we’re going to transform pod objects to drop some Object Metadata fields as well as clear out the status.
// customCache provides the ByObject map to provide a transform function
// the objects we want to transform.
func customCache() cache.Options {
return cache.Options{
ByObject: map[client.Object]cache.ByObject{
&corev1.Pod{}: cache.ByObject{Transform: filterPod},
}
}
}
// filterPod is used to mutate the pod prior to inserting it into the
// cache.
func filterPod(p any) (any, error) {
obj, ok := p.(*corev1.Pod)
if !ok {
return p, nil
}
pod := obj.DeepCopy()
// Ignore labels, annotations, managedby fields
pod.ObjectMeta = metav1.ObjectMeta{
Name: pod.Name,
Namespace: pod.Namespace,
UID: pod.UID,
OwnerReferences: pod.OwnerReferences,
Generation: pod.Generation,
ResourceVersion: pod.ResourceVersion,
}
// Clear out the status
pod.Status = corev1.PodStatus{}
return pod, nil
}
By explicitly setting the ObjectMeta fields, we have more control over the information we care about.
This allows us to reduce the memory footprint of the items in our cache.
This includes objects we may have explicitly configured the controller to watch or objects that we have used the controller runtime client to fetch from Kubernetes.
Additionally, there are some instances where we are only interested in the metav1.PartialObjectMetadata and the status of a resource, this provides an opportunity to strip out the rest of the data.
// filterDeployment preserves only the ObjectMeta of a deployment while
// clearing out the rest.
func filterDeployment(d any) (any, error) {
obj, ok := p.(*appsv1.Deployment)
if !ok {
return d, nil
}
deployment := obj.DeepCopy()
deployment.Spec = appsv1.DeploymentSpec{}
return deployment, nil
}
How neat is that?