CSR (Certificate Signing Request) on Digi Hunch

Creating X.509 TLS certificate for workload on Kubernetes

Sun, 29 Aug 2021 23:19:00 -0400

In deployment automation, I often had to create self-signed X.509 certificate for testing TLS traffic into Kubernetes. Sometimes self-signed, sometimes signed by a CA. This post summarized the approaches I’ve taken.

Create self-signed certificate with OpenSSL

Traditionally, this is done in three OpenSSL commands:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority'
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.orthweb.com'
openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

I have an older post to cover the basics of cryptography in TLS certificate and PKI. In the three commands above, the first produces a private key and self-signed certificate for a CA. The second creates a private key and a CSR for the web site. The third one uses the CA’s signing private key to sign the CSR from the website. The output is the certificate for the website.

Workloads running in Kubernetes typically consume certificates stored in Kubernetes Secret. The cons of this approach is that it usually requires an extra step to import the certificate files into Kubernetes Secret. For example:

kubectl create -n orthweb secret generic orthweb-cred --from-file=tls.key=server.key --from-file=tls.crt=server.crt --from-file=ca.crt=ca.crt

Note, people use the term self-signed certificate loosely. It sometimes means literally a certificate that is self-signed, like the one generated above. Sometimes, I had to self-sign a CA, then use the CA to sign one certificate for the server and one for the client. Because the application being tested requires that the client and server’s certificates both under the same CA. This would involve a few more commands, for example:

# Self-sign a CA
openssl req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=issuer.digihunch.com/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

# Generate a CSR for server
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=server.digihunch.com/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com -keyout /tmp/server.key -out /tmp/server.csr

# Use the self-signed CA to issue a certificate to the server
openssl x509 -req -sha256 -days 3650 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt -extfile <(echo subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com)

# Generate a CSR for clietn
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=client.digihunch.com/emailAddress=client@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr

# Use the self-signed CA to issue a certificate to the client
openssl x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

In the example above, it is important to note that even though the server’s CSR contains subject alternative name (SAN), I still have to specify the SAN again when signing the certificate for the server.

Similar to OpenSSL there are other toolkits such as CFSSL that supports specifying configuration files. However, the steps in Shell command are generally not always easy to automate.

Create self-signed certificate with Helm

Moving to the context of workload deployment in Kubernetes, running openSSL command isn’t always a viable option. For example, generating a certificate in the middle of deployment using a Helm Chart.

In Helm, template functions is for this purpose. In my Korthweb project I used genSignedCert to create self-signed certificate and then store the key, certificate and CA certificate as Kubernetes Secret:

{{- $dbtlscert := genSignedCert .Values.dbtls.certCommonName nil (list .Values.dbtls.certCommonName) 365 $ca }}
apiVersion: v1
kind: Secret
metadata:
 name: {{ .Values.dbtls.certCommonName | quote }}
 namespace: {{ $.Release.Namespace | quote }}
type: kubernetes.io/tls
data:
 tls.crt: {{ $dbtlscert.Cert | b64enc | quote }}
 tls.key: {{ $dbtlscert.Key | b64enc | quote }}
 ca.crt: {{ $ca.Cert | b64enc | quote }}
{{- end }}

The cons of this approach is that the syntax is not straightforward. As indicated in Helm documentation: Helm Chart templates are written in the Go template language, with the addition of 50 or so add-on template functions from the Sprig library and a few other specialized functions. While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates. Many resources on Go templates may be helpful as you learn about templating.

Create self-signed certificate with Cert-Manager

The Cert Manager project is very popular to produce X.509 certificates directly in Kubernetes secret. We can install cert manager using Helm:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true
kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager.io

Alternatively, FluxCD’s documentation on Kustomization dependency uses Cert Manager as an example. It is a good way of installing cert-manager if you have GitOps pattern.

Creating self-signed certificate for website is fairly simple. It starts with bootstrapping a CA issuer. Take the manifest below as an example. When creating the first certificate, make sure to specify isCA=true, so it stores the signing private key along with its own certificate in the ca-secret. Then use the newly created CA as issuer to create the X.509 certificate for the website.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: my-ca
 namespace: orthweb
spec:
 isCA: true
 commonName: my-ca
 secretName: ca-secret
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: selfsigned-issuer
 kind: ClusterIssuer
 group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: my-ca-issuer
 namespace: orthweb
spec:
 ca:
 secretName: ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: orthweb-cert
 namespace: orthweb
spec:
 commonName: orthweb.com
 secretName: orthweb-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.orthweb.com
 - dcm.orthweb.com
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: my-ca-issuer
 kind: Issuer
 group: cert-manager.io

The site certificate is directly stored in Kubernetes Secret as specified in the secretName field. To fetch the certificate text, we need to decode the secret entry, for example:

kubectl -n orthweb get secret orthweb-secret -o jsonpath='{.data.ca\.crt}' | base64 -d

Note that the example above uses ECDSA algorithm with size 256 for private key and certificate. It requires that the TLS client to support ECDSA algorithm as well. For more supportability, you can use RSA algorithm (2048 or 4096 size).

In addition to creating self-signed certificate, Cert Manager supports a number of other issuer types. For example, the support of ACME issuer type enables integration with Let’s Encrypt.

Cert Manager can secure Kubernetes Ingress resources with a sub-component called ingress-shim. It is configured via annotation on the Ingress resource.

Cert Manager

Create CA-signed certificate manually

For a certificate signed by a CA, there are may paid options, from manual, to self-help, to automated. The classic manual way is using OpenSSL, generating key, CSR. The CA takes CSR to sign a X.509 certificate returned to the website administration.

Many CA websites charges for a fee and makes it easy. For example, this site currently uses certificate from SSLs.com. Apart from the fee-for-cert option, there is a website named “SSL for free“, a CA with free option for 90-day single-domain, non-wildcard certificate and we can request it simply on their website, with proof of domain ownership.

The other popular free option is Let’s Encrypt, which also employs ACME protocol. The protocol requires ACME challenges to be satisfied in order to proof domain ownership. There are a few types of challenges:

HTTP-01 challenge
DNS-01 challenge
TLS-SNI-01 challenge
TLS-ALPN-01 challenge

I have used the HTTP-01 and DNS-01 challenges. The DNS-01 challenge requires adding TXT records to DNS configuration. The HTTP-01 challenge requires adding a DNS A-record to resolve to the server, then two URIs with pre-defined value.

When I first set up this site I used certbot (the client program for letsencrypt) to create certificate every 90 days from the wordpress server, following this guide, including solving DNS-01 challenges.

Create CA-signed certificate automatically with cert manager and letsencrypt

With Kubernetes, cert-manager has the ability to integrate with let’s encrypt for full automation. Here is a good blog post on this. Domain verification is still required but it can be done automatically. We first need to register an A record that resolves host name to the Ingress IP to enable this automation. The domain ownership validation may use the ACME protocol. This should also work on private networks with private DNS and ACME protocol using a private boulder server.

Take domain name demo1.digihunch.com for example, if ingress exposes a public IP address which the domain name resolves to, then we can configure certificate with the following manifest:

kind: IngressClass
metadata:
 name: istio
spec:
 controller: istio.io/ingress-controller
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt
spec:
 acme:
 privateKeySecretRef:
 name: letsencrypt
 server: https://acme-staging-v02.api.letsencrypt.org/directory
 solvers:
 - http01:
 ingress:
 class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: demo
spec:
 dnsNames:
 - demo1.digihunch.com
 issuerRef:
 kind: ClusterIssuer
 name: letsencrypt
 secretName: demo-tls

This example uses Istio as ingress controller but the method works regardless of the controller technology behind Ingress. In the ClusterIssuer object, we’re telling it to use the staging server from letsencrypt. We also specify http01 as challenge type, and that the ingress type is istio. In the Certificate object, we provided dnsName and specified ClusterIssuser. We also tell it to store the credentials to a secret named demo-tls.

When we apply the resources above, the ClusterIssuer connects to letsencrypt server via ACME protocol. Since the DNS name already resolves to the Public IP that the ingress is hosting, the ClusterIssuer configures the required Ingress, Services and Pods accordingly so the token to satisfy the challenge is presented at the designated URI. Instead of a staging server, we can also use production ACME server for production deployment. Note that the production ACME endpoint has a stricter rate limit.

When the ACME validation is in progress, it is important to ensure that port 80 is open and there is no other mechanism (such as routing rule, authorization requirement, mandatory redirect to 443) that blocks access from letsencrypt server.

Bottom line

Cert Manager is deployed in Kubernetes, supporting a variety of issuer types. As a Kubernetes-native tool, it is a no-brainer for Kubernetes workload for X.509 certificate. Compared with using template function in Helm, it is not dependent on template function and the syntax is consistent (YAML). Compared with OpenSSL or other binary tools, it is easy to integrate with the platform.

Public Key Infrastructure (PKI) – Introduction

Thu, 08 Apr 2021 22:07:00 -0400

A public-key infrastructure (PKI) is a set of roles, policies, hardware, software and procedures needed to create, manage, distribute, use, store and revoke digital certificates and manage public-key encryption. The algorithms are based on Publick-key cryptography. The format of the digital certificate is defined in X.509 standard.

Certificate Authority – CA digitally signs and publishes the public key to user. Website requesting certificates start with a key pair. It then converts public key into CSR (certificate signing request), including the identity. Once the identity of requestor is validated, CA will sign the public key of requestor, using its own private key. The output of this is the certificate.

Registration Authority – CA may delegate some roles to registration authority (RA). RA is responsible for accepting requests for certificates and authenticating the entity making the request. However, RAs do not have the signing authority of a CA. Note that Microsoft may have referred to a subordinate CA as an RA, which is incorrect according to X.509 PKI standards.

Validation types

Domain Validation: domain ownerships is usually verified via DNS record.
Organization Validation: the organization name and address are verified and put into the certificate.
Extended Validation: verifies existence and location of the legal entity, as well as domain ownership. EV cannot be issued as a wildcard certificate.

Private Certificate Authorities

You can create private CA and use it to sign certificates. Your user need to manually install and trust your private CA so that all certificates issued from the CA will inherit that trust. For revocation, you will also need to maintain an HTTP server for the certificate revocation list, or an OCSP responder.

Certificate Revocation List (CRL)

SSL certificates include information on how to access a certificate revocation list. Client will download and check this list to make sure the certificate has not been revoked. This mechanism has largely been replaced by OCSP responders.

Online Certificate Status Protocol (OCSP)

The OCSP protocol is a replacement for CRLs, with the benefit of being more real-time and requiring less bandwidth. The general operation is similar: clients are to query to OCSP responder to check if a certificate has been revoked.

Commercial vs non-profit CA

	Commercial (e.g. SSLs.com)	Non-profit (e.g. Let’s Encrypt)
Validation	DV, OV and EV	DV only
Wildcard	Supported	Supported (using DNS-01 challenge via ACME v2)
Cost	Not Free	Free
Expiration	1-3 years	90 days

ACME protocol

Traditionally, there are several command-line utilities such as openssl, cfssl, or keytool (Java) to manage certificate related tasks. The process are mostly manual. The Internet Security Research Group (ISRG) developed the ACME (Automated Certificate management Environment) protocol.

In this protocol, there is a certificate management agent (client) on the given web server. The agent generates a key pair and shares it with the CA at the outset of the validation process. Once validation is finished and the agent is verified as the proven owner of the key pair. It can use its key to digitally sign the CSRs it generates and sends to the CA via HTTPS requests. The CA uses the CSR, along with its associated public key, to issue the certificate and send it back to the agent. The agent downloads and installs it, then notifies the designated contact. The agent can be automated to check in with the CA at given intervals to rotate certificates and keys.

Let’s encrypt adopts ACME protocol by using Boulder on the server side, and the most commonly used client is certbot. Smallstep also introduced ACME support in step CA in 2019.

Lets Encrypt

I have used let’s encrypt several times because it is free and easy to manage with certbot, which can be installed using brew on Mac. Here’s how I quickly generate certificate manually:

DOMAIN=orthwebdemo.digihunch.com
echo $DOMAIN
sudo certbot -d $DOMAIN --manual --preferred-challenges dns certonly
# get ready to change txt record
ls /etc/letsencrypt/live/orthwebdemo.digihunch.com/

The --manual switch starts interactive prompts, which includes configuring TXT record and wait for the update.

Open source implementations

Here is a list of open-source implementations of PKI management:

OpenSSL: classic tool for PKI management. The Mac/BSD implementation and GNU implementation are slightly different.
Keytool: Java’s Key and Certificate Management Tool that supports formats used in Java
Cfssl: introduced by CloudFlare to simplify the PKI management process. On Ubuntu, the apt package name is golang-cfssl
Hashicorp Vault: CA, secret management and encryption.
Boulder: implemented in Go based on ACME protocol. Let’s Encrypt uses Boulder on the server side.
EJBCA: a full-featured, enterprise-grade implementation in Java, managed by Swedish company PrimeKey Solutions AB.

Managed CA as service

AWS Certificate Manager: [Update] as of Sep 2022, the managed CA capability was spun off as a new service called AWS Private Certificate Authority, to distinguish from the certificate management capability.
EJBCA Enterprise, as Azure Market place
Google Cloud Certificate Authority Service API