imaging on Digi Hunch

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

DICOM data encoding

Sat, 30 Jun 2018 20:42:36 -0400

DICOM is a standard for medical imaging exchanges, originally in radiology, but later expanded into other departments where mass imaging data are acquired, such as cardiology. One part of the DICOM standard defines how to lay out the data without providing any official code implementation. It is up to each vendor to implement their application and declare what parts of DICOM standard they are compliant to in their individual conformance statement. Therefore it is common that different vendors have different perspectives of whether each other’s implementation is compliant.

Many arguments of this topic revolves around the encoding of DICOM data (or in strict term, Information Object Definition, IOD). In this article I try to clarify DICOM IOD encoding, best practices from vendor support. A more comprehensive coverage of this topic is chapter 5 of Oleg Pianykh’s book, which discussed the basics such as implicit vs explicit VR, big vs little endian.

The very purpose of standard is to determine a common protocol in which two application communicate with each other. A strong standard leaves no room for ambiguity in implementation and unfortunately, many healthcare IT standards are weak standards. When two devices from two vendors fail to communicate properly, the healthcare provider (device buyer) should take the lead of moderation, because they suffer the most pain from proprietary implementation and they benefit the most from good interoperability. In reality however, healthcare organizations with insufficient technical competency in their information technology team, usually leave it in Vendors’ hands to configure integration, with minimum supervision on standard conformance. This allows vendor to put in technologies that are just made to work, but not fully up to standard. This is not optimal. Remember: Proprietary technology = Vendor Lock-In

From vendor’s perspective, the implementation should just comply with DICOM standard. They should not accommodate to third party application that are incorrectly implemented. Vendor’s responsibility with customer is simply to proof that the data encoding is compliant with DICOM; or if otherwise is discovered, escalate to engineering with low level technical detail. It is a courtesy in the discretion of vendor’s support operation, to investigate and advise on the integrity of externally sourced DICOM data. In reality however, vendors are pressured to just make it work.

DICOM Objects

In DICOM encoding, an IOD, either in a C-Store or a part 10 file consist of hundreds of data elements. A data element (uniquely identified by a tag) can be either:

A single item;
A sequence (SQ) of multiple items;

Transfer Syntax

Below is the table that summarizes the metadata and pixel data encoding under different transfer syntax UID specified in (0020,0010). It is not meant to be a completed list.

transfer syntax UID	transfer syntax name	Metadata encoding	Pixel data encoding
1.2.840.10008.1.2	Implicit VR Endian: Default Transfer Syntax for DICOM	Implicit VR Little Endian	Implicit VR Little Endian
1.2.840.10008.1.2.1	Explicit VR Little Endian	Explicit VR Little Endian	Explicit VR Little Endian
1.2.840.10008.1.2.4.70	JPEG Lossless, Nonhierarchical, First- Order Prediction (Processes 14 [Selection Value 1]): Default Transfer Syntax for Lossless JPEG Image Compression	Explicit VR Little Endian	JPEG Lossless Compression
1.2.840.10008.1.2.4.80	JPEG-LS Lossless Image Compression	Explicit VR Little Endian	JPEG-LS Lossless Compression

Practically, Big Endian encoding is rarely used in DICOM. So is .99 so they are not covered. Little Endian simply refers to the reverse ordering of each pair of bytes. The rest of this article only discusses metadata encoding.

Encoding of Data Element of single item

Most DICOM parsers out in the market don’t have a problem with data elements of single items. However it is important to understand the encoding of single item before trying to understand sequence. Regardless of implicit or explicit VR, big or little endian, a single item is always encoded in the following sequence:

	Length	Data format	Example
Tag	Group number	2-byte	unsigned integer
Element number	2-byte	unsigned integer	0010,0010
VR (present only for explicit VR)	2-byte	2 ASCII characters	PN
Length of Value	2-byte	an even integer	0x000A
Value	determined by length of value	determined by VR	Smith^Joe

Note that in the example, length of value is 10 in decimal, and the value “Smith^Joe ” contains a trailing space to make up for 10 character length. It is required by DICOM that the length be even number of characters, which sometimes omitted and tolerated by different implementations. The corresponding DICOM’s guideline is here. On this page please understand Figure 7.1-1, Table 7.1-1 and 7.1-2 before reading on.

Encoding of data element with SQ type

When it comes to SQ (sequence), there’s much confusion about what are valid options for sequence encoding. There is also a good chance that a third party DICOM interpreter is incompletely implemented, and mistakenly complains correctly-encoded sequence as bad data. Symptoms include, but not limited to, A-ABORT an association, silence a TCP connection, complaining in their logs that the data is “corrupted”.

Here is the reference to DICOM standard as to the valid options for sequence encoding. The language is fairly abstract and I’m making some addition to elucidate it:

When determining the sequence encoding, DICOM needs to address two problems:
1. Define how to start and end a data item;
2. Define how to start and end the entire sequence;
You can explicitly specify the length of a data item, or leave it undefined; Similarly for the entire sequence, you can explicitly specify the length upfront, or leave it undefined. This leads to four possible combinations but one of them is invalid. The following table points to an example of each based on the tables in DICOM document:

	Sequence Length is explicit	Sequence Length is undefined
Data item length is explicit	Valid Format A Exemplified in Table 7.5-1, when sequence length is explicit, length of each individual data item must be explicit as well. – Sequence length is 0F00H, 3840 bytes – Data Item length is 04F8H, 1272 bytes – No delimiter (FFFE,E0DD or FFFE, E00D) is needed for sequence of data element – Sum of unit length equals total length: (1272 + 4 + 4 ) x 3 = 3840 Even though this particular example is implicit VR, the parser should know this is a sequence by the length calculation a. the length of data element (sequence) is 0F00H	Valid Format B Exemplified in Table 7.5-2, as well as the first Item in Table 7.5-3 Sequence length is undefined, marked by (FFFF,FFFF) as the length value of data element Data Item length is explicit defined, as follows: – 98A5 and B321 for the two items in Table 7.5-2 – 17B6 for the first item in Table 7.5-3 – FFFE,E000 marks the start of a data item
Data Item length is undefined	This is NOT a valid encoding option. It would be error prone, if the total length is explicitly defined but the unit length is not.	Valid Format C Exemplified in the second Item in Table 7.5-3 – Sequence length is undefined, marked by (FFFF,FFFF) as the length value of data element – Data ltem length is also undefined – FFFE,E00D followed by 00000000H marks the end of the data item – FFFE,E0DD followed by 00000000H marks the end of the sequence

As shown above, there are multiple valid options (A, B and C) to encode data item in a sequence and the entire sequence. If sequence length is undefined, explicit and undefined data item length can even co-exist within the same sequence. (Table 7.5-3)
if the length is left undefined at the beginning, you must clearly mark the end of the data item, or sequence using one of the special data elements.

Special Data Element used in SQ encoding:

FFFE,E000 (Data Item) – marks the start of each data item inside of SQ element; it shall be followed by a 4-byte field to indicate the length of the data item (either an explicit value or FFFFFFFFH to indicate undefined length)

FFFE,E00D (Item Delimitation) – marks the end of each data item only if the length of that data item is undefined; it shall follow the data item immediately and the length of itself shall be set to 00000000H

FFFE,E0DD (Sequence Delimitation) – marks the end of an entire sequence only if the length of that sequence is undefined; it shall follow the last item of the SQ element and the length of itself shall be set to 00000000H