korthweb on Digi Hunch

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

Helm – Configuration Management for Kubernetes Resources

Mon, 26 Jul 2021 19:28:22 -0400

Developer ships application in Docker container, so it can eventually hosted in Kubernetes cluster. However, there are still some installation steps, before the application can operate online in production. In this post, we use the container image of Orthanc application as a starting point. We first build services in Kubernetes to go through these steps. Then, to automate the steps, we build a helm chart. The code is kept in Korthweb project, in which the manual directory has the files requirement for manual deployment, and the helm directory is the helm chart.

Manual Deployment

The manual deployment steps include different kinds of activities, such as creating X.509 certificates, apply config map, create Kubernetes deployment using the YAML declarations, and use helm to install dependency. The steps need to take place in a particular sequence. Some step requires pulling information from secrets created in the previous step. This is why the deployment is not portable.

In order to automate the steps, one might think of wrapper script, which is very limited. A configuration management tool is needed in this scenario. Two common options are Kustomize, and Helm. Kustomize is a native tool which can be run by kubectl. It is also driven by declarative statement in YAML, which is simple to grasp. However, in lack of a templating mechanism, Kustomize may require wordy statements. Helm, on the other hand, comes with a templating mechanism which greatly increase reusability, making it more suitable for complex steps required in installation.

Helm Repo and Chart

Helm is known as package manager for applications running on Kubernetes. Helm defines an application as a collection of related Kubernetes resources, and it manages application deployment through a templated approach. An installation workbook is called a chart. Charts are kept in repositories. There are some well-known repositories, such as Bitnami, Helm stable. You need to add a repostory before using the Helm Charts in it. To add a repo, run:

helm repo add bitnami https://charts.bitnami.com/bitnami

You can host your own repo (public or private) as well. To search for charts across repositories, the best place is artifact hub, which indexes charts from a lot of public repositories. To search for charts from the repositories added, run:

helm search repo postgres

Template is the soul of Helm chart. A Helm chart consists of a directory of files following specific pattern so Helm can understand how to deploy the application. For example, the chart name is the name of the working directory. Under the directory, the values.yaml and chart.yaml defines variables and constants, both serving as template inputs. The template directory is the most important part of the directory where the installation logics are defined. Helm runs the entire directory hierarchy (except for paths specified in .helmignore file) through a Go template rendering engine. The template result spec out the detailed steps.

A great example of using Helm chart to simplify installation is the wordpress chart by Bitnami. You can install all the required components in a single command:

helm install my-release bitnami/wordpress

The helm chart in Korthweb project is also an evolving helm chart I created for installing Orthanc application.

Helm V3 (released in late 2019) includes an important architectural change – the removal of tiller. This means Helm can operate on the client-side – a significant simplification. Helm graduated from CNCF project in 2020. There are also a few changes in V3, as outlined here, including the consolidation of requirements.yaml into Charts.yaml.

Template and Function

As discussed, templating is the key towards reusability and flexibility in configuration management. We’ve worked with Jinja2 template engine in Ansible and Python. Here in Helm, we use Go templates with some enhancement. The syntax is mostly based on Go template, which is somewhat similar to Jinja2. Helm also added all functions from the Sprig library, making it more powerful and flexible than Jinja2. Helm chart developer should be very familiar with these functions, as well as the best practices. For example, the cryptographic and security functions in Sprig library gives us the ability to create self-signed X509 certificates during installation.

Since template introduces another layer of abstraction, to help troubleshooting we should be able to preview rendered template with the template command:

helm template orthanc | less

The command above simply renders template without attempting to execute the chart. To go one step further, you can dry-run the installation with:

helm install orthweb ./orthanc --debug --dry-run | less

Although Jinja2 (using {% … %} to express template control) and Go (using {{ … }} to express template control) have different syntaxes, one aspect that is similar between them, is chomping whitespace with minus sign (-). This is pretty common in templating language. The documentation of both Jinja2 and Helm have a section on whitespace control. Not paying attention to this nuance may cause pesky errors.

Dependency

The Orthanc application relies on Postgres database, which itself is deployed by a separate helm chart. This can be specified in Chart.yaml (Helm V3), like this:

dependencies:
 - condition: postgresql-ha.enabled
 name: postgresql-ha
 repository: https://charts.bitnami.com/bitnami
 version: 7.8.x

The values of variables of the dependency chart can be specified in values.yaml of the root chart. They can also be imperatively specified as a parameter of helm install command.

The section above also requires the dependency chart to be downloaded into the charts sub-directory. This can be done with:

helm dependency update

Then you will notice a file with tgz extension in the charts sub-directory. Note that when you change the version of the dependency package in Chart.yaml, then you will need to run the command again. Alternatively, this command can be automatically executed before helm install if you specify the switch –dependency-update with helm install.

The main chart (e.g. wordpress) is referred to as parent chart, and the charts it depends on are referred to as sub-chart (e.g. mariadb, memcached). When it comes to managing property values, values from parent chart can override those from sub-chart, as explained here. On the other hand, values from sub-chart can override those from parent chart in two formats: export format (keyword exports) and child-parent format (keyword import-values). This is something to be careful and we can use the aforementioned template command to display the rendered values.

Hooks

Helm does a great job in figuring out the dependency relationship between kubernetes objects defined in the chart, and create them in order. So typically you do not need hooks for objects in the chart. However, in certain circumstances, such as cleaning up after uninstallation, we may need hooks. Here is a list of available hooks.

It is worth-noting that hook is not tied to an action. Instead it is tied to a kubernetes resource. The resource could be a job, a config map, etc. The resource is tied to a hook simply by resource annotation.

Moving to GUI

Helm is a command-line tool. For a team with varying levels of familiarity with command-line, GUI-based tool is a better option. For that, some enterprises adopt Rancher, a comprehensive Kubernetes cluster management platform. Rancher manages many aspects of Kubernetes cluster through web portal. One aspect is the support of helm chart. Rancher can be install on a cluster of its own. For demo, it can also be installed on docker desktop, a single-node Kubernetes cluster by Docker. In both cases, Nginx ingress controller needs to be configured.