pacs on Digi Hunch

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

How imaging devices talk to each other (in DICOM)

Sun, 15 Nov 2020 18:40:00 -0400

Overview

In the previous post I briefly touched on DICOM as the crucial standard in medical imaging for both data exchanging and data storage. It is important to understand that DICOM is such a massive standard that, beyond data exchanging and storage, has expanded into many different areas around imaging, that no device (or information system) can ever implement every single aspect of the standard. A device or information system complies to (and implements) a subset of the DICOM standard. The manufacturer must provide a document (DICOM conformance statement) to spec out which parts of the standard are implemented. Care providers (e.g. hospitals) are supposed to review these specs as part of the procurement process to ensure interoperability with existing information system.

Medical Imaging Informatics

When it comes to diagnosis, historical examinations provide baseline reference for radiologist. They are sometimes even more revealing than current imaging data acquired from a patient. Compared to the current exams, historical ones are referred to as priors. Priors are useful only if they are relevant to the current exams in terms of body part, modality, and exam procedure (the particular problem being studied). The effort to find out and pre-load relevant priors so they are ready to display along with current exams, is called “prefetch”. With huge demand in exchanging imaging data, “prefetch” has developed into its own sub-market in imaging informatics industry.

Trust me, this is a difficult undertaking (and why I was in the industry). First, patient identities in each healthcare organization are usually different for lack of universal medical record number; patient’s name can change (marriage, divorce, or just for fun); or it can be unavailable, if patient is simply not in a condition to provide identify (e.g. trauma). Second, even if you get “who’s who” right, you’d have to dig into all his history for useful information from several different systems. The definition of relevant prior can be different depending on the specific medical specialty. Then, the old data are typically stored in a slow part of the storage from their source system, yet the patient might be bleeding and dying on the table, waiting for prior retrieval like pulling teeth. Last, but not least, the priors being retrieve might be from a modality of previous generation from 1990s; good luck with current display application. Sorry that sounds a lot but in real life, there are even more challenges. In General, there are four categories of applications that need to support DICOM:

Acquisition devices: Modalities need to store newly acquired studies persistently;
Routing applications: usually by the name of some routers, gateways or bridges that receive imaging studies, decorate the metadata (because a lot of legacy devices can’t do it), and send to one or multiple defined destination;
Archives (e.g. PACS, VNA): They usually use dedicated database (metadata) and storage systems (pixel data). They are the repository of imaging data and must provide full support of transfer capability;
Peripheral applications that uses DICOM data, such as 3D post-processing or DICOM testing (grassroot dicom, dcmtk)

No matter what the devices are, they must follow certain protocols in order to communicate with each other.

Data structure and encoding

DICOM PS3.5 defines data types in VR (value representation). Each VR has its own purpose, allowed characters, and length limit. For example:

VR	Definition	Allowed Characters	Length Limit
SH Short String	A string of characters		16 maximum
LO Long String	A string of characters		64 maximum
AE Application Entity	A string of characters that identifies a DICOM application running on a compliant device		16 maximum
CS Code String	A string to represent code		16 maximum
PN Person Name	Person’s name, with caret (^) as delimiter		64 maximum
UI Unique Identifier	An ID that uniquely identify an item, such as 1.2.840.100008.1.1	0-9 and period (.)	64 maximum
DA Date	A string to represent date YYYYMMDD	0-9	8
US Unsigned Short	Unsigned binary integer, 16 bits long		2
SQ Sequence of other items	Sequence of other items
UN Unknown	A string of bytes where the encoding of contents is unknown

Value Representatives

DICOM metadata is a dataset comprised of a set of data elements. Each element includes tag, (optional) VR, length of value, and the actual value, as shown below:

DICOM data elements

VR is optional because it can be implicitly determined based on DICOM data dictionary defined in PS3.6. The most common tags are:

Level	Tag	Meaning	VR
Patient	0010,0010	Patient’s Name	PN
Patient	0010,0020	Patient ID	LO
Patient	0010,0021	Issuer of Patient ID	LO
Patient	0010,0024	Issuer of Patient ID Qualifier Sequence	SQ
Patient	0010,0030	Patient’s Birth Date	DA
Study	0008,0020	Study Date	DA
Study	0008,0050	Accession Number	SH
Study	0008,0061	Modalities In Study	CS
Study	0008,1030	Study Description	LO
Study	0020,000D	Study Instance UID	UI
Study	0020,0010	Study ID	SH
Series	0008,103E	Series Description	LO
Series	0008,0015	Body Part Examined	CS
Series	0008,0060	Modality	CS
Series	0020,0011	Series Number	IS
Series	0020,000E	Series Instance UID	UI
Series	0020,0060	Laterality	CS
SOP inst	0008,0016	SOP Class UID	UI
SOP inst	0008,0018	SOP Instance UID	UI
SOP inst	0012,0010	Transfer Syntax UID	UI
SOP inst	0020,0013	Instance Number	IS

Common DICOM tags

When VR is not explicitly spelled out, the data encoding is known as implicit VR. The opposite is explicit VR, where each data element spells out the VR type. When storing numeric value, such as tags or numeric values for tags, the predominant format stores lower byte before higher bytes, known as little endian. Rarely seen in DICOM is big endian, the opposite order of storing numeric values.

The tag include a group number (e.g. 0020) and element number (e.g. 0013). If group number is odd, it is a private tag not defined in PS3.6

The actual length of tag value shall always be even number. Odd-sized value should add an additional character (e.g. trailing space), known as even-length padding to meet this requirement.

Above is the basic rules for DICOM data structure and encoding. For more information about encoding, including transfer syntax, refer to this previous post.

Transactions (DIMSEs and SOP classes)

DICOM has its own information model of real world. It requires some clinical knowledge to come to full understanding. For technical people, we just need to understand the patient-study-series-image hierarchy.

One patient may have multiple studies
Each study may include one or more image series
Each series has one or more images

Images are also referred to as SOP instance, a general terms that include not only images, but also reports, and other types of objects.

A business transaction in DICOM is termed DIMSE, for example:

Name	Group	Type	Description
C-STORE	DIMSE-C	Operation	A stores an image to B
C-MOVE	DIMSE-C	Operation	A tells B to store an image to C
C-FIND	DIMSE-C	Operation	Query for patient, study, series, images
C-ECHO	DIMSE-C	Operation	DICOM level “ping”
N-EVENT-REPORT	DIMSE-N	Notification	Report an event
N-ACTION	DIMSE-N	Operation

DIMSE

Some DIMSEs involves multiple SOP classes based on the type of image being processed. For example, SOP Class UID (10.2.840.10008.5.1.4.1.1.1) represents storage for CR image. A c-store transaction includes:

Requestor (C-STORE SCU) sends a request (C-STORE-RQ), to store specified SOP class in certain transfer syntax
The request is followed by the actual data (PDU)
Once completed, the Response (C-STORE SCP) respond with C-STORE-RP

Handshake (Association)

In the previous C-STORE example, proper syntax that are mutually supported must be used in order for the requestor and receiver to process the transaction. Both parties learn each other’s supported transfer syntax through an upfront handshake process known as DICOM association.

In DICOM association, the initiating party presents a list of supported pairs of SOP class and transfer syntax. Each pair is called a presentation context, and the responding party must respond to each presentation context in the proposed list, with either yes or no.

This negotiation is similar to the cipher negotiation in TLS handshake.

For more detailed information, please refer to Pianykh’s book “DICOM, a practical introduction and survival guide“.