cert-manager Self-Assessment

June 2024

Primary Authors: Maël Valais, Ashley Davis Security reviewers: Maël Valais, Ashley Davis, Tim Ramlot

This document is intended to aid in roadmapping, help with the onboarding of new maintainers and provide a good introduction to the security model of cert-manager.

Metadata

Assessment StageComplete
Softwarehttps://github.com/cert-manager/cert-manager
Security ProviderYes, the primary function of cert-manager is to support the security of an integrating system.
LanguagesGo (Make + bash used for build system)
SBOMNot generated for cert-manager, but can be built for for sub-projects albeit not not published alongside the releases. See open issue
DocURL
cert-manager security audit by Ada Logics, 2024https://ostif.org/wp-content/uploads/2024/03/AdaLogics-2023-cert-manager-audit-report.pdf
Vulnerability Reporting Processhttps://github.com/cert-manager/community/blob/main/SECURITY.md
Graduation Issuehttps://github.com/cncf/toc/issues/1306

Overview

cert-manager is a X.509 certificate controller for Kubernetes and OpenShift. It plugs into certificate authorities such as Let’s Encrypt and HashiCorp Vault and automatically renews certificates before they expire.

This document focuses primarily on cert-manager itself. The wider cert-manager project includes a variety of subprojects which can help with various tasks relating to managing certificates in Kubernetes, but those are out of scope for this document.

Background

cert-manager is the de-facto standard Kubernetes operator for obtaining and renewing X.509 certificates.

cert-manager comes with its own Kubernetes custom resource and is widely extensible by the means of “external issuers” that communicate to cert-manager with the intermediate of the cert-manager custom resources and “external DNS-01 webhooks” that cert-manager uses when validating ACME certificates.

cert-manager stores private key material in Kubernetes Secret resources.

Actors

The following table describes each actor.

ActorDescription
cert-manager controllerThe controller contains the logic for the built-in issuers such as the ACME issuer as well as logic for ensuring certificate issuance or reissuance is triggered. It connects to the Kubernetes API server and reads and writes Secret resources as well as the cert-manager custom resources such as the Certificate resource.
cert-manager webhookThe webhook is an HTTPS server internal to the Kubernetes cluster that lets the Kubernetes API server validate cert-manager's custom resources when they're created or edited by users.
cert-manager cainjectorThe cainjector aims to secure other Kubernetes webhooks. It is used by projects such as Cluster API. It connects to the Kubernetes API server to read and writes custom resource definitions as well as secret resources.
cert-manager acmesolverThe acmesolver is a separate pod created by the cert-manager controller when an ACME certificate is being issued. It accepts traffic from the internet in order to validate HTTP-01 ACME challenges.

Actions

cert-manager-adalogics.drawio

ActionActorDescription
Create CertificateRequestActors with RBAC permission - usually cert-managerA CertificateRequest's creation precedes certificate issuance. CertificateRequests (CRs) are not usually private - they're signed by a private key and contain no secrets. They're sent to issuers to obtain a signed certificate. Permission to create CRs is powerful since it may allow actors to issue certificates using any configured issuer in the cluster.
Issue via Certificate Resourcecert-manager controllerThe cert-manager controller acts on Certificate resources, which are a more user-friendly way of creating certificates. When issuance is required, cert-manager will (if required) create a temporary private key (in a Secret resource) to use for signing the CertificateRequest resource and which will be used with the certificate after issuance. Once signed, the certificate is added to a Secret resource along with the private key, and the temporary Secret is removed.
Generate Private Keycert-manager controllerThe controller is often required to generate a private key for a certificate / certificate signing request. These keys are generated using the Go standard cryptography library using cryptographically secure random number generators.
Validate Custom Resourcecert-manager webhookThe webhook validates resources as they're created or updated. It speaks to the Kubernetes API server to do this, and has a privileged position to be able to deny resources from being created if they're invalid.
Validate HTTP-01 Challengecert-manager acmesolverThe acmesolver pod is started in response to a need to validate HTTP-01 ACME challenges. It listens on a socket for a request for a specific path, in line with the requirements of the ACME spec. The spec states that this server must listen for HTTP (not HTTPS) requests. The pod is terminated after the challenge completes.

Goals

cert-manager primarily uses Kubernetes RBAC to control access to resources, meaning that cluster administrators ultimately have control over who can issue certificates and how certificates are issued.

As a corollary to this, cert-manager seeks to only allow users with valid RBAC to be able to perform certificate related tasks.

The tool also relies heavily on Kubernetes Secrets, and the security of issued Certificates is reliant on the assumption that in-cluster Secret resources are secure.

The nature of cert-manager’s implementation means that its underlying security relies heavily on the security of the underlying cluster. If a cluster has an exposed or vulnerable etcd component, that could mean the disclosure of key material handled by cert-manager since key material is stored in Secrets which will in turn be stored in etcd.

Non-goals

cert-manager doesn’t seek to handle anything outside of its scope of issuing and renewing certificates.

For example, while an X.509 certificate is required for TLS, it is not sufficient to configure a secure TLS server: settings like the TLS version and cipher suites are relevant for that. cert-manager makes no effort to help with that process. It only seeks to provide a valid and secure certificate.

cert-manager also doesn’t provide any guarantees around TLS trust. Trusting certificates is an important foundation of TLS, and cert-manager has a sister project called trust-manager to help with this. The core cert-manager project doesn’t seek to address this.

Self-assessment Use

This self-assessment is created by the cert-manager team to perform an internal analysis of the project’s security. It is not intended to provide a security audit of cert-manager or function as an independent assessment or attestation of cert-manager’s security health.

This document serves to provide cert-manager users with:

  • an initial understanding of the project’s security posture
  • where to find existing security documentation
  • cert-manager’s plans for security, and
  • a general overview of cert-manager security practices

This document provides the CNCF TAG-Security with an initial understanding of cert-manager, and the self assessment is required as part of the graduation process for cert-manager within the CNCF. This document also paves the way for a joint assessment document with the CNCF / TAG-Security in the future.

Security functions and features

Critical Security Components

  • cert-manager controller: the controller is a pod that has access to secret resources across all namespaces.

Security Relevant Security Components

  • cert-manager cainjector: the cainjector is a pod that has access to secret resources for a given namespace
  • cert-manager acmesolver: HTTP server which listens publicly as required by the ACME specification
  • cert-manager webhook: validates newly created and newly changed resources with the permission to deny a resource from being created.

Project Compliance

The cert-manager project does not comply with any specific security standard. Some general suggestions for compliance with some hardening solutions are available on the cert-manager website .

Secure Development Practices

Development Pipeline

  • Regular Trivy scans are defined in the cert-manager repo and are run regularly (for example, on the controller )
  • Regular govulncheck scans are configured for the cert-manager repo . Govulncheck is a much more effective security scanner for Go, and we address anything it reports with a great deal of urgency.
  • golangci-lint’s gosec integration is run regularly and is run against all PRs which are candidates for merging to enforce good practices
  • We sign all cert-manager containers and the cert-manager helm chart during the build process, which is itself entirely defined in open source auditable code.

Communication Channels

Ecosystem

The cert-manager project is widely used across the ecosystem: most Kubernetes controller projects, such as Cluster API, use it to provide certificates for their webhooks.

It also has integrations with various notable cloud native projects including Istio, Linkerd, SPIFFE, SPIRE and more.

Security Issue Resolution

Responsible Disclosures Process

The cert-manager project has a well defined responsible disclosure process, which is documented in a SECURITY.md file in every repo in the project.

Those files link back to a central file, which is located in the community repo .

The project has a dedicated security disclosure Google group which all maintainers have access to, and maintainers are encouraged to sign up for that list with both work and personal emails so that they can address valid reports quicker.

Our documentation for what we’ll accept specifically calls out that we won’t accept plain security scanner reports, because usually these are low effort and filled with false positives and because we run scanners ourselves, too.

We haven’t yet had a serious disclosure through this mechanism. So far, we’ve not had to appoint anyone to take the lead on a response, because all responses have been picked up quickly by a member of the maintainer team.

Vulnerability Response Process

Our process defines that we’ll respond to valid reports within 3 working days. In practice we tend to be much faster than this, although again most reports are false positives.

Appendix

Known Issues Over Time

There haven’t been any CVEs or notable security reports made about cert-manager. The most interesting result is from the ADALogics security audit linked at the top of this document, in which a maliciously crafted certificate could exhaust the memory of cert-manager using a particular invalid custom Subject. This was only present in a feature which was alpha-level at the time and so had to be enabled specifically at install time.

Our govulncheck and trivy scanners regularly find issues of various importance, and we patch them quickly depending on the severity and exploitability of the issues. cert-manager has been affected by security bugs in Go itself, and in several dependencies.

Another example comes from the security audit, which found that some servers were missing timeouts or limits, which have now been added. Mostly, these servers are only accessible in-cluster and exploiting these issues would have required a presence in-cluster.

CII Best Practices

cert-manager passes the OpenSSF best practices badge:

Case Studies

We don’t have any particular case studies, but we have several tutorials on the website which point to how it can be used:

  1. Use cert-manager to obtain let’s-encrypt certificates for an nginx ingress (see Nginx Ingress tutorial )
  2. Use cert-manager to issue certificates for service-mesh communication (see istio-csr )
  3. Use cert-manager to integrate a Kubernetes cluster in a company-wide PKI setup (see Venafi tutorial )

In addition, cert-manager’s popularity means that there is a lot of content on the wider internet about it. Searching on Google or YouTube will provide a lot of tutorials and examples.

This section is about discussing the differences between cert-manager and other similar projects.

Direct alternatives:

These direct alternatives can be used to replace cert-manager for a subset of the use-cases that we support. These projects are smaller in scope, this means that they might be a better solution for a very specific use-case. However, due to their specificity, they are less popular and receive updates and bugfixes less frequently and have a smaller community.

Indirect alternatives:

These alternatives are alternative ways to provision certificates in a Kubernetes cluster. They are mainly based on secret-provisioning, meaning that they need to be used in combination with another system to make sure all certificates are renewed in time.