digital certificate on Digi Hunch

Creating X.509 TLS certificate for workload on Kubernetes

Sun, 29 Aug 2021 23:19:00 -0400

In deployment automation, I often had to create self-signed X.509 certificate for testing TLS traffic into Kubernetes. Sometimes self-signed, sometimes signed by a CA. This post summarized the approaches I’ve taken.

Create self-signed certificate with OpenSSL

Traditionally, this is done in three OpenSSL commands:

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Health Certificate Authority'
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=*.orthweb.com'
openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

I have an older post to cover the basics of cryptography in TLS certificate and PKI. In the three commands above, the first produces a private key and self-signed certificate for a CA. The second creates a private key and a CSR for the web site. The third one uses the CA’s signing private key to sign the CSR from the website. The output is the certificate for the website.

Workloads running in Kubernetes typically consume certificates stored in Kubernetes Secret. The cons of this approach is that it usually requires an extra step to import the certificate files into Kubernetes Secret. For example:

kubectl create -n orthweb secret generic orthweb-cred --from-file=tls.key=server.key --from-file=tls.crt=server.crt --from-file=ca.crt=ca.crt

Note, people use the term self-signed certificate loosely. It sometimes means literally a certificate that is self-signed, like the one generated above. Sometimes, I had to self-sign a CA, then use the CA to sign one certificate for the server and one for the client. Because the application being tested requires that the client and server’s certificates both under the same CA. This would involve a few more commands, for example:

# Self-sign a CA
openssl req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=issuer.digihunch.com/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

# Generate a CSR for server
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=server.digihunch.com/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com -keyout /tmp/server.key -out /tmp/server.csr

# Use the self-signed CA to issue a certificate to the server
openssl x509 -req -sha256 -days 3650 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt -extfile <(echo subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com)

# Generate a CSR for clietn
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=client.digihunch.com/emailAddress=client@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr

# Use the self-signed CA to issue a certificate to the client
openssl x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

In the example above, it is important to note that even though the server’s CSR contains subject alternative name (SAN), I still have to specify the SAN again when signing the certificate for the server.

Similar to OpenSSL there are other toolkits such as CFSSL that supports specifying configuration files. However, the steps in Shell command are generally not always easy to automate.

Create self-signed certificate with Helm

Moving to the context of workload deployment in Kubernetes, running openSSL command isn’t always a viable option. For example, generating a certificate in the middle of deployment using a Helm Chart.

In Helm, template functions is for this purpose. In my Korthweb project I used genSignedCert to create self-signed certificate and then store the key, certificate and CA certificate as Kubernetes Secret:

{{- $dbtlscert := genSignedCert .Values.dbtls.certCommonName nil (list .Values.dbtls.certCommonName) 365 $ca }}
apiVersion: v1
kind: Secret
metadata:
 name: {{ .Values.dbtls.certCommonName | quote }}
 namespace: {{ $.Release.Namespace | quote }}
type: kubernetes.io/tls
data:
 tls.crt: {{ $dbtlscert.Cert | b64enc | quote }}
 tls.key: {{ $dbtlscert.Key | b64enc | quote }}
 ca.crt: {{ $ca.Cert | b64enc | quote }}
{{- end }}

The cons of this approach is that the syntax is not straightforward. As indicated in Helm documentation: Helm Chart templates are written in the Go template language, with the addition of 50 or so add-on template functions from the Sprig library and a few other specialized functions. While we talk about the “Helm template language” as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates. Many resources on Go templates may be helpful as you learn about templating.

Create self-signed certificate with Cert-Manager

The Cert Manager project is very popular to produce X.509 certificates directly in Kubernetes secret. We can install cert manager using Helm:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true
kubectl get pods -n cert-manager
kubectl get crd | grep cert-manager.io

Alternatively, FluxCD’s documentation on Kustomization dependency uses Cert Manager as an example. It is a good way of installing cert-manager if you have GitOps pattern.

Creating self-signed certificate for website is fairly simple. It starts with bootstrapping a CA issuer. Take the manifest below as an example. When creating the first certificate, make sure to specify isCA=true, so it stores the signing private key along with its own certificate in the ca-secret. Then use the newly created CA as issuer to create the X.509 certificate for the website.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: selfsigned-issuer
spec:
 selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: my-ca
 namespace: orthweb
spec:
 isCA: true
 commonName: my-ca
 secretName: ca-secret
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: selfsigned-issuer
 kind: ClusterIssuer
 group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: my-ca-issuer
 namespace: orthweb
spec:
 ca:
 secretName: ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: orthweb-cert
 namespace: orthweb
spec:
 commonName: orthweb.com
 secretName: orthweb-secret
 duration: 2160h
 renewBefore: 72h
 subject:
 organizations:
 - digihunch
 dnsNames:
 - web.orthweb.com
 - dcm.orthweb.com
 privateKey:
 algorithm: ECDSA
 size: 256
 issuerRef:
 name: my-ca-issuer
 kind: Issuer
 group: cert-manager.io

The site certificate is directly stored in Kubernetes Secret as specified in the secretName field. To fetch the certificate text, we need to decode the secret entry, for example:

kubectl -n orthweb get secret orthweb-secret -o jsonpath='{.data.ca\.crt}' | base64 -d

Note that the example above uses ECDSA algorithm with size 256 for private key and certificate. It requires that the TLS client to support ECDSA algorithm as well. For more supportability, you can use RSA algorithm (2048 or 4096 size).

In addition to creating self-signed certificate, Cert Manager supports a number of other issuer types. For example, the support of ACME issuer type enables integration with Let’s Encrypt.

Cert Manager can secure Kubernetes Ingress resources with a sub-component called ingress-shim. It is configured via annotation on the Ingress resource.

Cert Manager

Create CA-signed certificate manually

For a certificate signed by a CA, there are may paid options, from manual, to self-help, to automated. The classic manual way is using OpenSSL, generating key, CSR. The CA takes CSR to sign a X.509 certificate returned to the website administration.

Many CA websites charges for a fee and makes it easy. For example, this site currently uses certificate from SSLs.com. Apart from the fee-for-cert option, there is a website named “SSL for free“, a CA with free option for 90-day single-domain, non-wildcard certificate and we can request it simply on their website, with proof of domain ownership.

The other popular free option is Let’s Encrypt, which also employs ACME protocol. The protocol requires ACME challenges to be satisfied in order to proof domain ownership. There are a few types of challenges:

HTTP-01 challenge
DNS-01 challenge
TLS-SNI-01 challenge
TLS-ALPN-01 challenge

I have used the HTTP-01 and DNS-01 challenges. The DNS-01 challenge requires adding TXT records to DNS configuration. The HTTP-01 challenge requires adding a DNS A-record to resolve to the server, then two URIs with pre-defined value.

When I first set up this site I used certbot (the client program for letsencrypt) to create certificate every 90 days from the wordpress server, following this guide, including solving DNS-01 challenges.

Create CA-signed certificate automatically with cert manager and letsencrypt

With Kubernetes, cert-manager has the ability to integrate with let’s encrypt for full automation. Here is a good blog post on this. Domain verification is still required but it can be done automatically. We first need to register an A record that resolves host name to the Ingress IP to enable this automation. The domain ownership validation may use the ACME protocol. This should also work on private networks with private DNS and ACME protocol using a private boulder server.

Take domain name demo1.digihunch.com for example, if ingress exposes a public IP address which the domain name resolves to, then we can configure certificate with the following manifest:

kind: IngressClass
metadata:
 name: istio
spec:
 controller: istio.io/ingress-controller
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt
spec:
 acme:
 privateKeySecretRef:
 name: letsencrypt
 server: https://acme-staging-v02.api.letsencrypt.org/directory
 solvers:
 - http01:
 ingress:
 class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: demo
spec:
 dnsNames:
 - demo1.digihunch.com
 issuerRef:
 kind: ClusterIssuer
 name: letsencrypt
 secretName: demo-tls

This example uses Istio as ingress controller but the method works regardless of the controller technology behind Ingress. In the ClusterIssuer object, we’re telling it to use the staging server from letsencrypt. We also specify http01 as challenge type, and that the ingress type is istio. In the Certificate object, we provided dnsName and specified ClusterIssuser. We also tell it to store the credentials to a secret named demo-tls.

When we apply the resources above, the ClusterIssuer connects to letsencrypt server via ACME protocol. Since the DNS name already resolves to the Public IP that the ingress is hosting, the ClusterIssuer configures the required Ingress, Services and Pods accordingly so the token to satisfy the challenge is presented at the designated URI. Instead of a staging server, we can also use production ACME server for production deployment. Note that the production ACME endpoint has a stricter rate limit.

When the ACME validation is in progress, it is important to ensure that port 80 is open and there is no other mechanism (such as routing rule, authorization requirement, mandatory redirect to 443) that blocks access from letsencrypt server.

Bottom line

Cert Manager is deployed in Kubernetes, supporting a variety of issuer types. As a Kubernetes-native tool, it is a no-brainer for Kubernetes workload for X.509 certificate. Compared with using template function in Helm, it is not dependent on template function and the syntax is consistent (YAML). Compared with OpenSSL or other binary tools, it is easy to integrate with the platform.

Cryptography Basics 1 of 2

Wed, 10 Jul 2019 20:07:18 -0400

I have been dabbling with OpenSSL commands to achieve what I needed during IT implementation, but I decided to spent some time to overcome the conceptual hurdles around cryptography. In this domain, following other people’s instructions through the project does not produce much learning value when too many concepts cloud around. Let’s take the bull by the horn.

This article is purely conceptual. There are already lots of step-by-step guideline about acquiring a website certificate. The intention is to elucidate the core concepts on IT cryptography, and then connect the dots to form the big picture in cryptography.

First, let’s distinguish three basic concepts: encoding, hashing and encryption. They are in essence all mathematical functions, but one does not need to understand the underlying algorithm in order to understand what they are.

Encoding – Transform data into a format so it is readable by external system. Encoding is about interoperability. It is not about security whatsoever. Example: ASCII, BASE64, UNICODE.

Hashing – Mathematic algorithms to generate digest of content. A digest is usually fixed-size, non-reversible and deterministic. You cannot restore content from its digest (non-reversible, one-way calculation). Two different contents results guarantees different digests (deterministic). Digest as a result of hashing is mostly about data integrity. For example, in file download you can calculate MD5 hash and compare the result against the digest given by the source. Another common use, is to hash all password in database. In that sense, hashing has to do with security. Popular algorithms are MD5 and SHA-256.

Encryption – Mathematic algorithms that only succeeds if correct parameter (key) is provided. The function is deterministic and reversible (two-way). The parameter (key) used for calculation (encryption) and reverse calculation (decryption) can either be the same or different.

Cryptography involves all three concepts, but it mainly addresses issues around encryption. We break down encryption into two categories, symmetric key encryption (aka private key encryption) and asymmetric key encryption (aka public key encryption).

Symmetric Key Encryption: using the same key (shared secret) for encryption and decryption. Both parties need to keep it secret. Popular algorithm is AES.

Asymmetric Key Encryption: involves a pair of public key and private key. Public key is given out to external systems. Private key is kept secret. Popular algorithm is RSA. You can run some experiments on this page, and I’ve made some additional notes (all based on RSA algorithm):

Public Key and Private Key are NOT interchangeable. The size of an RSA private key is usually much larger than its public key;
You can encrypt with either key, and decrypt with the other, so long as you specify the key type at the time of encryption or decryption;
You can generate public key from private key (ssh-keygen -y); but not the other way round;
Using a wrong key to encrypt or decrypt leads to failure, instead of wrong result.

Obviously symmetric key encryption is the original form of encryption and requires both parties to keep the key secret, permanently. This is not realistic in real life between organizations. This challenge leads to the adoption of asymmetric key where the public key can be published, for external party to encrypt outgoing messages, whereas the private key is kept secret within the owner, only to decrypt incoming messages.

Knowing this distinction, we can introduce two concepts that are built on top of asymmetric key encryption.

Digital Signature: if an entity signs a document digitally. The digital signature is the digest of the document (hash of the content) encrypted with the signer’s private key.

Digital Certificate: contains owner’s public key and digital signature of issuer (digest of owner’s public key encrypted by issuer’s private key). Client (e.g. browser with CA’s public key preloaded) should not trust the owner’s public key until it compares its digest against decryption result of digital signature (using CA’s public key).

The key difference here is that a digital certificate involves third party. Digital signature by itself cannot address impersonation, which is addressed by digital certificate issued by third party (certified authority). This requires digital certificate must follow some standard, and the most prevalent one is:

X.509: a standard format for digital certificates. It contains:

a public key
an identity (a hostname, or an organization, or an individual)
a signature (either signed by CA or self-signed)

Since the advent of certificate, there are tons of global organizations that need to manage keys and certificates. When we create a secure connection now, we only need to get a certificate from an intermediate CA, thanks to existing certificate chain. It involves lots of work for large organizations to maintain keys and certificates, which requires:

Public Key Infrastructure (PKI): the IT infrastructure to create, manage, distribute, use, store and revoke digital certificates and public keys.

One of the important technique to manage keys and certificates is the encoding. Keys and certificates are usually wrapped with different encoding formats based on what need to be done on them. Here is a summary of encoding formats

Format	Encoding	what is stored	Possible Suffix
PEM	DER file (binary content) encoded in Base64. Certificate files typically include clear text statement “BEGIN CERTIFICATE” and “END CERTIFICATE”	single certificate, certificate chains or private keys	.pem, .crt, .cer, .key
DER	Binary format in early days; it’s complex and don’t use unless with a specific purpose.	single certificate, certificate chains, or private keys	.der, .cer
PKCS#7	Base64 encoded ASCII file, typically include clear text statement “BEGIN PKCS7” and “END PKCS7”	only certificates or certificate chains; no private keys	.p7b .p7s
PKCS#8	Similar to PEM as base64 encoded format but for storing private key only, can be password protected	Private key	.key
PKCS#12	a binary format, heavily used by Microsoft products	certificate, certificate chains or private keys; public private key pair	.pfx .p12
OpenSSH	used by OpenSSH to store public keys (as specified in RFC4253)		.pub

Note: above are just encoding formats. Just by file extension you cannot tell whether the file is a key or a X.509 certificate. When you are configuring certificates, you may come across the following file extensions as well:

.key this extension can indicate any kind of key, but usually it is a private key (used along with .crt file)

.csr certificate signing request, including a public key and an identity required by CA. CA needs this file to issue a certificate. CSR could be encoded in Base-64 or DER

.cer or .crt a certificate, usually in X.509 v3 (public key + identity + signature), the encoding could be PEM or DER.

.jks -> java key store file type. It can be either a key store (private key along with certificate) or trust store (certificate) for Java application. Refer to the section for Java applications.

In the next article, we will examine some use case involving the concepts introduced in this post.