tls on Digi Hunch

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

Service Proxy – from Nginx to Envoy

Wed, 09 Mar 2022 21:36:00 -0400

Update (Nov 20, 2022): 1. Envoy’s configuration schema can be hard to get used to. It is lacking examples because the documentation is mostly generated. Use its examples directory to find real-life configuration examples. 2. the Envoy implementation in the example project has been reverted in favour of Nginx.

Envoy proxy is the underlying technology for Istio, as well as a number of other service mesh products, such as AppMesh (AWS), Consul (Hashicorp) and OpenServiceMesh (Azure). Most of the capabilities of Isito is ultimately provided by Envoy proxy. Envoy has a page outlining its differences with similar technologies. I decided to take a look into Envoy by replacing Nginx with it.

Rate limiting and Circuit Breaker

In most SDLC, it is application developers that create backend APIs or server applications. Most developers specializes in application features, and cannot fathom all the nuances with the TCP/IP network stack. Nginx allows them to push network concerns (non-business features) to a dedicated proxy to handle the dynamics in network connection. Nginx can be configured as both a reverse proxy (handling incoming connection on behalf of the process) and a forward proxy (handling outgoing connection on behalf of the process). This is the prototype of sidecar pattern, an important idea behind service mesh. For example, when a sudden increase in connection to the server-side application, the server process could be either unresponsive (refer to “the queuing knee“, and Little’s Law), or OOM killed. When such interruptions are not automatically recovered, a downtime is caused. This traditionally requires some congestion control strategy for TCP/IP queue but two features provided by a network proxy can help circumvent this situation: rate limiting, and circuit breaking. Rate limiting keeps more requests above threshold from entering the queue. Circuit breaker releases downstream pressure by cutting out existing in-queue request. Nginx added both over the years but Circuit breaker still remains a premium feature exclusive to Nginx Plus. Envoy on the other hand has them free when it was launched.

Envoy also supports other advanced traffic management such as traffic shaping, and mirroring. It is on top of those features that Istio introduces its own abstraction such as virtual service, destination rules to its users. In that sense, we can think of Istio as a configurator (control plane) for Envoy proxy (data plane), similar to Ansible to Nginx proxy instances.

Dynamic Configuration via API

I used Nginx previously with traditional environment and loved its flexibility. As the system grows, I started to feel the pain of management overhead. With one of the production system, there were 25 + instances of Nginx each running on a VM and I managed configuration files with Ansible. Ansible pushes out configuration files and triggers a reload from each Nginx instance. In the cloud-native era where Pods are ephemeral, this kind of overhead would snowball to an unmanageable level. Envoy was designed for cloud-native applications, with all these kinds of problems in mind. Envoy has dynamic configuration. The majority of the configurations can be pulled from xDS API, or file system. Updating configuration drains connections gracefully without runtime having to reload the file. The idea of centrally managing Nginx instances with Ansible, also evolved into the concept of control plane.

TLS origination

In the Orthweb project, I used Nginx to proxy TLS and HTTP traffic, and performed TLS termination on both ports. This is know as TLS offloading. The traffic between the proxy and the upstream service takes place in the clear, even though they do not travel across different network interfaces in most cases. For a true end-to-end encryption, it is helpful to also encrypt the traffic between proxy and upstream server. This requires the capability of securing TCP traffic to upstream server. With Nginx, the ability to secure HTTP traffic to upstream server is offered in open-source. The ability to secure TCP traffic is available in Nginx Plus, or with self-compiled binary. In Envoy, both are available using the UpstreamTlsContext configuration.

More pros

In another post, I also discussed Nginx as a LDAP proxy to front services such as Kibana and Nifi. It requires a proxy service (ldap-auth in this case), to defer auth to third party. Envoy has this capability using a filter with extension for external authorization. Istio also exposes this capability, an enabler for the configuration proposed in my previous post. Envoy also uses WebAssembly for its extensibility.

Another useful feature is protocol detection. It can use filters to detect protocol (TLS or regular TCP) and route traffic to predefined destination.

Performance wise, this benchmark from 2018 ran a comparison among the popular options where Envoy leads by a margin.

Observability (logging, metrics and tracing) are well supported in Envoy. User can configure format of logs that takes effect immediately. There are many metrics that works with Prometheus and they are expandable using filters. On the tracing side, Envoy supports integration with jaeger, zipkin and datadog.

Basics of Envoy

The configuration of Envoy is more involving. There is an Envoy course by Tetrate, as well as two blog entries for envoy 101: Envoy as gateway proxy and File-based dynamic configuration. Another good way to get started is the Sandboxes projects, which covers a number of different areas of configuration. The admin port (by default at port 9901. Istio’s default is 15000) provides helpful information. If we need to turn on debug on some features, we can do so with curl:

curl -X POST http://localhost:9901/logging?client=debug

Stats are exposed at the same port:

curl -X GET http://localhost:9901/stats

When packets are received at a listener, the is first processed by listener filters. Then, depending on filter match, one or more network filter chains will further process the packet, including further actions, as illustrated below:

Envoy supports dynamic configuration, which uses a set of discovery services (xDS) APIs. Some of the important xDS APIs include:

LDS (Listener Discovery Service) – allows you to add listeners dynamically while Envoy is running
RDS (Route Discovery Service) – allows you to dynamically update routes for HTTP connection managers
CDS (Cluster Discovery Service) – allows you to update cluster definitions dynamically
EDS (Endpoint Discovery Service) – allows you to add or remove endpoints dynamically
Secret DS

The relation can be illustrated in this diagram below:

This post from Tetrate has more examples.

Nginx to Envoy

For the advantages of Envoy, I decided to migrate from Nginx to Envoy on my Orthweb project. Using Envoy as service proxy is not where Envoy is mostly used (as sidecar), but it is how Envoy was originally used at Lyft to replace ELB in 2015.

The original Nginx configuration was referenced in this old blog post. The Envoy setup also covers both TCP (DICOM) and HTTP (HTTPS) traffic. For HTTP traffic, it also encrypts the traffic to upstream. Below is what it looks like:

admin:
 address:
 socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
 listeners:
 - name: https_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 443
 filter_chains:
 - filters:
 - name: envoy.filters.network.http_connection_manager
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
 codec_type: AUTO
 stat_prefix: ingress_http
 route_config:
 name: local_route
 virtual_hosts:
 - name: app
 domains:
 - "*"
 routes:
 - match:
 prefix: "/"
 route:
 cluster: service-https
 http_filters:
 - name: envoy.filters.http.router
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 - name: dicomtls_listener
 address:
 socket_address:
 address: 0.0.0.0
 port_value: 11112
 filter_chains:
 - filters:
 - name: envoy.filters.network.tcp_proxy
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
 stat_prefix: downstream_cx_total
 cluster: service-dicomtls
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
 common_tls_context:
 tls_certificates:
 - certificate_chain: {"filename": "/etc/ssl/certs/site.pem"}
 private_key: {"filename": "/etc/ssl/certs/site.pem"}
 validation_context:
 allow_expired_certificate: true
 trusted_ca: {"filename": "/etc/ssl/certs/site.pem"}
 require_client_certificate: false
 clusters:
 - name: service-https
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-https
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 8042
 transport_socket:
 name: envoy.transport_sockets.tls
 typed_config:
 "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
 - name: service-dicomtls
 type: STRICT_DNS
 lb_policy: ROUND_ROBIN
 load_assignment:
 cluster_name: service-dicomtls
 endpoints:
 - lb_endpoints:
 - endpoint:
 address:
 socket_address:
 address: orthanc-backend
 port_value: 4242
layered_runtime:
 layers:
 - name: static_layer_0
 static_layer:
 envoy:
 resource_limits:
 listener:
 https_listener:
 connection_limit: 1000
 overload:
 global_downstream_max_connections: 5000

From Nginx to Envoy, to achieve nearly the same functionalities, it takes 100 lines of configuration instead of less than 30. The configuration also appears more abstract, which is one of the cons of Envoy to consider before the migration.

Update from Jan 2025 – To align with what most uses in Orthanc community, the project switched back to using Nginx. The envoy configuration is kept here.

From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms

Wed, 29 Dec 2021 22:50:16 -0400

Update: also read my other article here on the different generations of ingress technologies.

In my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.

The word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article.

Background

In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node’s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node’s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.

Kubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.

Requirement

Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:

The ability to proxy TCP traffic over an arbitrary TCP port
The ability to terminate TLS/SSL encryption for TCP traffic

The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I’m not confident.

I also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:

Product	Requirement #1	Requirement #2
F5 driven Nginx Ingress Controller	Supported	There have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.
HA Proxy Ingress controller	Supported	There’s no mention of this in documentation.
Kong’s Kubernetes Controller (with TCPIngress CRD)	Supported	Documentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.
Traefik Lab’s IngressRoute (with IngressRouteTCP CRD)	Supported	Documentation seems to suggest that it is supported by IngressRoute.

My general impression is that Req #2 isn’t very popular so the providers either don’t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP’s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.

Introduction to Gateway

In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:

This sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to “Route” resource type.

This conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.

The diagram above stolen from Gateway API website illustrate the management model for each resource.

Kubernetes Gateway Implementations

The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.

Because of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification.

For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API’s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong’s Kubernetes Ingress Controller follows the same path.

The Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.

Until the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.

Istio Gateway implementation

As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.

First, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I’ve compared three major service mesh technologies as below:

Linkderd	Istio	Consul
– lightweight – uses linkerd2-proxy – the original service mesh – no Gateway implementation (deal breaker in my use case)	– feature rich – uses Envoy proxy – complex but getting better	– initially a service discovery and distributed key-value store – lacks observability features but getting better – uses Envoy proxy

It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in.

Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.

Istio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).

In addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:

Here is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called “Ingress”, nor a resource type named “Ingress Gateway”, although they may be loosely used to refer to “Gateway resources configured to manage ingress traffic”.

Istio Gateway Installation

There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm.

With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:

install CRDs using the base chart
install istiod using the istiod chart. Here is an example of values provided to Helm installer.
install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition.

This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.

Secure web application deployment

Sun, 16 May 2021 15:19:41 -0400

In Nov 2020, I created OrthWeb project, a deployment of Orthanc’s server. Orthanc is a DICOM viewer and repo shipped in Docker container. In the deployment project, I use Terraform to provision infrastructure, including a managed PostgreSQL instance, an EC2 instance for docker runtime, and the init script to bring up the web service. I whipped up the project for a demo, and skipped some security configurations. For example, the password was stored in clear text in Terraform configuration. The web certificate is stored in the repository. I recently had some time to fix that. My effort leads up to the conclusion that this requires a better platform (i.e. managed Kubernetes cluster). So I wanted to note down how I got there.

Secret store

In AWS, both parameter store and secret manager can act as secret store. Secrets manager comes at higher cost but some additional features, such as built-in password generator, secret rotation, and cross-account access. We use Secret Manager but we generate password within Terraform because we need to specify password during database provisioning. Secret store requires certain special characters to be eliminated. Terraform can specify the special characters allowed. For EC2 instance to pull from secret manager, the following entities are needed:

A secret store
A VPC endpoint to expose secret store to subnet via private route.
The VPC endpoint needs its own security group
The instance profile of the EC2 instance must contain an IAM role to get secret value
The security group of EC2 instance needs to allow traffic to secret store
The script from EC2 instance uses VPC endpoint

This is a common pattern for interaction between computing object and VPC endpoint. The details are in compute.tf, network.tf, secgrp.tf and secret.tf. The secret name needs to be partially randomized to avoid naming conflict with deactivated secrets.

Passing Secret to container

Here is an example CLI command to pull secret:

$ aws secretsmanager get-secret-value --secret-id DatabaseCreds51c1db4172ae9c54 --query SecretString --output text --endpoint-url https://vpce-0897b168cf1c60df2-khx32o7f.secretsmanager.us-east-1.vpce.amazonaws.com | jq -r .password

The connection is made via private network route (whether the instance is in public or private subnet). Traffic is encrypted in TLS. Once in the operating system, the secret is available as standard output and can be stored to file, or saved in environment variable.

My first attempted approach is docker’s secret store and config so that I do not have to store secret in plain text on the file system. I eventually give up this approach due to several hiccups. First, secret and config are part of Docker swarm service. So it requires initializing docker swarm before I could port in the secret, with the following command:

$ docker swarm init
$ echo mdbuser123 | docker secret create db_un -
$ echo m1p@ssw0rd | docker secret create db_pw -
$ echo 10.2.32.41 | docker config create db_ep -

The content of the secret and config are presented as files to the container file system at different locations, as can be verified this way:

$ docker service create --name="redis" --secret=db_un --secret=db_pw --config=db_ep redis:alpine
$ docker container ls
$ docker exec -it c8ed2a278ca8 sh
# cat /db_ep
# cat /run/secrets/db_un
# cat /run/secrets/db_pw

This is a great way to pass secret and config to container applications. However, since the values are stored as content of file, the main application must be able to load file content as its own configuration value. In my specific scenario, the application expects explicit value in its configuration file, or environment variable.

On the other hand, Docker document states that docker secrets do not set environment variables directly. this was a conscous decision, because env var can unintentionally be leaked between containers. In other word I could present secrets as files but the application cannot use it. There is potentially a workaround here which is great function wise but an additional layer of complexity.

Moreover, I later discovered that this isn’t even a viable approach if I use docker compose. This is because I must declare those entries from secret store or config store as external, and external secrets are not even available to containers created by docker-compose.

With reluctance, I store the config and secret keys and values to a file, and use the env_file section in docker compose to import them as environment variables. The application can pick up environment variables as configuration values.

X509 Certificate

We use a self-signed X509 certificate, along with the private key. The creation is straightforward. However, when I tested on Mac, the browser does not load the page for this reason. Since macOS 10.15, the certificate requires several extensions: ExtendedKeyUsage, Subject alternative names and DNS name. The native openssl from the operating system is outdated (v 1.0.2) and I had to install openssl11 package and create it as follows:

openssl11 req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/private.key -out /tmp/certificate.crt -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=digihunch.com/emailAddress=info@3.237.97.93 -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:digihunch.com

The Mac uses libreSSL backed openSSL utility and can achieve the same with slightly different command line argument.

Next Step

The limitation with passing secret concerns me and I’m looking to move to managed Kubernetes platform where secrets can be ported to environment variable of Pods. We can also consider ECS in AWS which allows to inject sensitive data from secret manager to container.

Automatic deployment of Orthanc on AWS

Sun, 08 Nov 2020 00:54:06 -0400

[Update] I changed reverse proxy from Nginx to Envoy. Here‘s the detail.

[Update] Some security improvement was introduced in may 2021. Here‘s detail.

[Update] Here’s the link to the orthweb repository.

In this project we introduce a medical imaging web service based on Orthanc, an open-source project of DICOM server, and a pipeline to deploy such server automatically and consistently. We deploy Orthanc on AWS automatically. This little project involves a number of technical deets in DevOps, to deliver a web application prototype with an automated deployment pipeline.

A brief on imaging

In medical imaging, scanning devices are the data collectors. It consists of various categories of scanners, such as Computed Tomography (CT), and Ultrasound (US). They are collectively referred to as modality, but vary significantly in terms of image generation and hardware manufacturing. The challenges to exchange data between these heterogeneous scanning devices and centralized computers came around as early as the 1980s, which brought about ACR-NEMA standard in 1985, under the initiative between American College Radiology (ACR) and National Electrical Manufacturers Association (NEMA). The standard lately evolved into DICOM (Digital Imaging Communication in Medicine), a comprehensive set of standard in the ISO framework that governs modern imaging data storage and exchange across several disciplines (radiology, cardiology, pathology, etc) that operate around images in medicine.

In addition to defining a file format to store imaging data, DICOM also includes an upper layer protocol that dictates how two compliant devices (referred as application entity, each identified by AE title) can negotiate a common syntax to transfer objects (e.g. an image, a report or a discovery). Upper layer refers to layer 5-7 in OSI model, or application layer in TCP/IP model.

Imaging server

Once scanner acquires images from patient, they stores the exams to imaging server for persistent storage. The functionalities of such server expands overtime since 1990s and hence go by different names in different eras, such as PACS (Picture Archive and Communication Systems), VNA (Vendor Neutral Archive) and EI (enterprise imaging) archive. Regardless of naming, they can be generally seen as a highly specialized variation of enterprise content management system. They are usually hosted with a centralized database to index clinical information at patient, exam and image levels. The other key component is the persistent storage devices, usually in the form of a NAS.

Orthanc is an open-source initiative for such imaging servers. It provides a DICOM endpoint, allowing scanning devices to store medical images. It also provides a web viewer allowing users to see the images stored. It is released for many platforms, including Docker images.

Infrastructure as code

We use Amazon Web Service (AWS) for infrastructure as service, and Terraform as the tool to provision resources off AWS, in a reliable and consistent mechanism, known as Infrastructure-as-Code. Terraform is an alternative to CloudFormation, AWS’s proprietary infrastructure-as-code technology. Terraform is developed by Hashicorp as an open-source project, and therefore is vendor neutral. It supports multiple public cloud vendor through different providers. Each provider accesses the vendor specific SDK. For example, the AWS provider integrates with AWS SDK. As a result, the code used in one vendor cannot just be applied to a different vendor without a major overhaul. Terraform’s current version is 0.13 as of Oct 2020, and has gone through some syntax changes since version 0.11. Terraform also produces files for state management locally in the working directory.

When executing, Terraform combines all files in the working directory to assess variables, and create required resources. It is compatible with the most of AWS resources. For example, you can specify user data with templates when creating EC2 instances. You can also create managed service instance as long as it is supported by the provider.

Architecture

Orthanc web server stores data in sqlite by default, but also has a plugin to support PostgreSQL, an open-source relational database. AWS has managed service (RDS) based on PostgreSQL. In this project, we create an RDS instance that span across two availability zones for minimum high availability. Orthanc also supports storing imaging data including pixels in PostgreSQL, which obviates the need for a dedicated file storage system.

We deploy the application in Docker’s containers for compatibility and portability. The Orthanc server is shipped in Docker images, available in Docker hub registry. The docker environment is configured as part of EC2 instance bootstrapping, including installing packages with YUM, initializing and customizing environment variables. The docker-compose file, and the auxiliary configuration files are provided in the repo. The bootstrapping script installs git and pulls required files from this GitHub repo.

This demo project does not include load balancing, DNS management, or container orchestration.

Security

Orthanc’s web browser natively supports HTTPS. However, the DICOM port does not support TLS natively, as their development has made clear in the FAQ. This leaves a severe security vulnerability because all patient data (protected health information in HIPPA context) would be sent across the Internet in the clear, visible to every network interface along the route. To address this issue we brought in Nginx as a reverse proxy to work at TCP layer to terminate encrypted traffic for Orthanc’s DICOM end point. DICOM upper layer works on top of TCP layer.

In Nginx literature, this use case is referred to as SSL Termination for TCP Upstream Servers. Note that Nginx is providing layer 4 capability in this use case so the certificate and key configuration should not be placed under http section of the configuration file. This layer 4 capability in fact enables security configurations of all protocol that operates in upper layers and can be used in a broad range of situations. It is also noteworthy that Nginx can re-encrypt the traffic on the way out to upstream, for even tighter security control measure as outlined in this use case.

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
 worker_connections 1024;
}

stream {
 upstream dicom_backend {
 server orthanc-backend:4242;
 }

 server {
 listen 11112 ssl;
 proxy_pass dicom_backend;

 ssl_certificate conf.d/site.pem;
 ssl_certificate_key conf.d/site.pem;
 ssl_protocols SSLv3 TLSv1 TLSv1.2 TLSv1.3;
 ssl_ciphers HIGH:!aNULL:!MD5:ECDH+AESGCM;
 ssl_session_cache shared:SSL:20m;
 ssl_session_timeout 4h;
 ssl_handshake_timeout 30s;
 }
}

It is also helpful to use Nginx to terminate HTTPS traffic, using a pair of certificate and key. When testing with self-signed certificate I realized that Chrome browser has specific requirement on self-signed certificate, or it won’t load the page. So the certificate has to be created as instructed here.

For better security, it is advisable that the RDS instance is provisioned in private subnet, with its data encrypted both in-transit and at-rest. Docker service should also manage sensitive information as secrets.

Summary

The deliverable is stored in this Github repo. The docker part of it can be executed on MacBook with PostgreSQL. The entire hardware stack represented by terraform code, can be executed against AWS to create required resources. Checkout README for further instruction. To emulate a modality, one will need a TLS supported DICOM application entity, Horos is a great project on MacOS to serve this purpose, both as DICOM-compliant sender and a viewer. Alternatively, consider some command-line based DICOM toolkit such as dcmtk, or grassroot dicom.

SASL Authentication Mechanisms

Thu, 19 Mar 2020 22:53:00 -0400

Introduction

Authentication is used in many protocols (such as LDAP binding) and it usually involves sending password. Given the nature of authentication protocol, its traffic encryption is usually mandatory. Simple Authentication and Security Layer (SASL) is introduced to ensure the security during authentication. It is not a single protocol, but rather a framework for authentication and data security involving many protocols. The intent is to decouple authentication mechanisms from application protocols, thus allowing any authentication mechanism (under SASL) to be used in any application protocol (that supports SASL). Application protocols that support SASL typically can also be built on Transport Layer Security (TLS), whose latest versions (1.2 and 1.3) are considered more secure.

None (ANONYMOUS)

The server basically does not authenticate the client. The client connects to the server anonymously. Under SASL framework, this may also be referred to as ANONYMOUS mechanism.

Simple (PLAIN)

In simple authentication method the password is sent to server in the clear. This is subject to eavesdropping and is not secure. It is still surprisingly widespread in legacy configurations probably due to the simplicity of configuration. This option should not be available in cloud environment. Under SASL framework, this may also be referred to as PLAIN mechanism.

CRAM-MD5 and DIGEST-MD5

CRAM-MD5: described in RFC 2195, using HMAC-MD5 algorithm. In this challenge-response scheme based mechanism, the client’s password is protected during authentication, but the application session (e.g. LDAP) traffic is not encrypted. It includes random data from the server and is slightly better than Simple authentication. However, this authentication method is not recommended either.

DIGEST-MD5: described in RFC 2831. This is very similar to CRAM-MD5 but is is somewhat stronger because it includes random data from both the client and server. In addition, it also provides a provision to ensure connection integrity and confidentiality (a data security layer).

GSSAPI

Generic Security Service Application Program Interface (GSS-API) is an API specification for programs to access security services. GSS-API by itself does not provide any security. Instead, security-service vendors provide GSSAPI implementations – usually in the form of libraries installed with their security software. These libraries present a GSSAPI-compatible interface to application developers who can write their application to use only the vendor-independent GSSAPI.

Under SASL, the dominant GSSAPI mechanism implementation in use is Kerberos version 5. GSSAPI allows Kerberos implementations to be API compatible. In many contexts, GSSAPI simply implies Kerberos.

NTLM

NT LAN Manager (NTLM) is a challenge-response based Microsoft security protocols. It is implemented in a Security Support Provider (SSP), which combines the older LAN Manager authentication protocol, NTLMv1, NTLMv2 and NTLM2 Session protocols in a single package. Group policy manages whether these protocols are used or can be used. NTLM passwords are considered weak because they can be brute-forced very easily with modern hardware. It might still be enabled in server configuration as a backup mechanism to Kerberos.

TLS (EXTERNAL)

CRAM-MD5, DIGEST-MD5, GSSAPI, and NTLM are more commonly referred to as SASL mechanisms (in loose terms). These mechanisms allow for a secure password exchange without requiring TLS by trying to address the authentication traffic encryption problem at application layer. Using TLS this can also be address at transport layer. TLS can be used in combination with any of the mechanisms above but usually TLS/Simple mechanism is sufficient. In many occasions the mechanisms under SASL can be replaced by simple authentication encrypted with TLS. Under the SASL framework, this may also be referred to as EXTERNAL mechanism so TLS (in strict terms) is also considered a SASL mechanism.

Summary

This article outlined several authentication protocols under SASL. Since SASL is the framework that intends to govern all authentication protocols, the use case of these protocol can be widespread. For example, in LDAP you can find all of them. Here is a comparison across them:

	Password on wire	Session
Simple	Clear	No Encryption
SASL/CRAM-MD5	Encrypted	No Encryption
SASL/DIGEST-MD5	Encrypted	No Encryption
SASL/GSSAPI	Kerberos	Encryption
TLS:SIMPLE	Encrypted	Encrypted

Common SASL implementation includes Cyrus SASL and GNU SASL. There are also some API implementations that supports some of SASL mechanisms, such as SSPI.

Lightweight Directory Access Protocol (LDAP)

Mon, 02 Mar 2020 21:11:00 -0400

Introduction

Originally LDAP only refers to the connectivity protocol to the directory server. This term is being used loosely today and it also refers to the actual directory service that supports and complies with LDAP. LDAP v3 is the current version developed in RFC 2251.

A directory is information about some set of entities such as people, organization, or stones. An example of directory would be /etc/passwd file in Linux. A directory server is simply an application with the main purpose of maintaining directories. Typically, the read traffic is high whereas write traffic is low. LDAP is a general-purpose directory server. It can store information about people, or cars, or rocks. You just need to define what a person’s entry looks like as well as what a rock’s entry looks like. The general architecture of LDAP provides the capability nedded for managing large amount of diverse directory entries.

An LDAP entry consists of DN (distinguished name) and attributes. An attribute may have one or more attribute names and they are defined in attribute definitions. Attribute names are not case-sensitive. An attribute may have one or more values if multiple values are allowed for that attribute. Attribute values may be case-sensitive depending on the definition.

A special attributed named objectclass attribute provides information about what type of record it is, and what attributes canbe given to the record. For example, the organization name (o) is required for any entry with an organization object class. While a record may have multiple object classes, one of these object classes must be the structural object class for the record. A structural object class determines what type of object the record is.

In addition to regular attributes, the directory server may also attach special operational attributes to an entry. Operational attributes are used by the directory server itself to store information about entries. Such attributes are not designed for use by end users, and are usually not returned during LDAP searches.

An LDAP schema defines types of records in a directory and how those records might relate to each other.

Information in an LDAP directory is organized into one or more hierarchies where, at the top of the hierarchy, there is a base entry, and other entries are organized in tree-like structures beneath the base entry. Each node on the hierarchy is an entry, with a DN and more than one attributes. This hierarchically organized collection of entries is called a directory information tree (DIT). In DIT, LDAP directories stores data in hierarchical relationships. The root entry sits at the top and subordinate entry is beneath that, which in turn may have its own subordinate entries. Each of these records has its own DN, and its own attributes. The DN of each entry is composed of two parts: the relative DN (RDN) and the full DN of the superior entry.

LDAP is nothing other than a special sort of database that organizes data into tree structures, like a file system hierarchy. This view is more easily seen by comparing an LDAP directory to a relational database system (RDB), where SQL is the protocol and RDBMS is the service. LDAP refers to both the protocol and the service.

OpenLDAP

A common LDAP implementation is openldap. OpenLDAP suite can be broken up into four components:

Servers: slapd (stand-alone LDAP Daemon) provides LDAP services.
Clients: ldapsearch is used to manipulate LDAP data
Utilities: support LDAP servers
Libraries: provide programming interfaces to LDAP

Installing OpenLDAP requires libldap-2.3-0, slapd, ldap-utiles packages. It is configured in /etc/ldap/. An HDB (hierarchical database) needs to be specified in the configuration.

To test as a client, the first thing that must happen is the client must authenticate to the server (via simple bind or SASL Bind). LDAP server verifies the identity, permission as well as password provided by the client.

LDAPTLS_REQCERT=never ldapsearch -x -o ldif-wrap=256 -H ldaps://ldap.digihunch:636/ -b "OU=Admin,OU=Service Department,DC=digihunch,DC=com" -D "gh\ldap-bind-user" -w 'S@f35+P@55w0rd' "(objectclass=user)" -s sub -d 9

The command above first sets client environment variable LDAPTLS_REQCERT to never, in case the client is being asked to provide certificate. Then the ldapsearch command performs the bind.

To search the directory, the client needs to provide the followings:

Base DN: where in the directory to start from
Scope: how deep in the tree to look
Attributes: what information to be retrieved per result
Filter: what to look for

Below is an example of ldapsearch (-b for Base DN, -s for Scope, -S for attributes, stdin for filter):

ldapsearch -x -o ldif-wrap=256 -H ldaps://ldap.digihunch:636/ -b "OU=Admin,OU=Service Department,DC=digihunch,DC=com" -D "gh\ldap-bind-user" -w 'S@f35+P@55w0rd' "(memberof=CN=Security-Admin,OU=Admin,OU=Service Department,DC=digihunch,DC=com)" -s sub -S name

Users with appropriate permissions may also other directory operations using ldapadd, ldapmodify, ldapdelete, ldapcompare, ldapmodrdn, ldappasswd, ldapwhoami, etc

Apart from those in Openldap toolkits, there are many other tools such as Apache Directory Studio that allows you to perform similar functionality with a user interface.

LDAP security

Historically LDAP servers listens to port 389 through which traffic is sent in clear text. This is a bad security practice known as “insecure bind”. To secure LDAP traffic, two prevalent approaches are Secure LDAP and StartTLS.

Secure LDAP was the original attempt to secure LDAP traffic as an addition to LDAP v2. It is also known as LDAPS, LDAP over TLS/SSL or LDAP channel binding (“channel binding” just refers to the establishment of encrypted channel following TLS handshake. It provides a facility to tie an authentication exchange to security services provided at a lower layer. Defined in RFC 5056). Secure LDAP operates on port 636 on the server side and TLS handshake must be established for traffic encryption. Client application usually need to import the certificate of LDAP server. As part of TLS 1.2 protocol, the server may also request client certificate during ServerHello message. The presence of CertificateRequest means the server either demands client certificate, or tries to get client certificate (i.e. TLSVerifyClient is set to demand or try, which is only visible on the server). If client cert is only attempted, the LDAP client may choose to ignore it. If client cert is demanded, then a two-way TLS authentication is required and thus the client must proof its identity to the server. This Secure LDAP configuration requires the server to listen to both 389 and 636 ports on the same server to support both secure and legacy applications, which is unnecessary. Secure LDAP therefore is not the preferred approach.

The standardized way of implementing SSL/TLS in LDAP v.3 is to use the StartTLS method. This method should be implemented whenever possible. If an AD server supports StartTLS, the client can start with a STARTTLS command to the server so that the server begins the TLS encryption process. In the binding phase, TLS handshake follows a LDAP_START_TLS_OID command through port 389.

Here’s the summary of the three LDAP configuration mode:

	Legacy	SecureLDAP (aka LDAPS, LDAP over TLS/SSL)	StartTLS
Listening port	389	636	389
Traffic Encrypted	No	Yes	Yes
Standard	Yes but this should always be avoided since it is insecure	Introduced in the time of LDAP v2, but the option is deprecated (although still supported) by RedHat	Introduced in LDAP v3. This may be left as the only valid option.

Note that one of the recent changes that drives may customer away from the legacy mode is the requirement for LDAP channel binding on Windows servers, with a target date of March 2020. Our current strategy at CS is to direct customer towards Secure LDAP as we do not support StartTLS yet and we know we do support LDAPS. Although Secure LDAP itself is somewhat legacy this would not hold long. According to this Wikipedia page:

The use of LDAP over SSL was common in LDAP Version 2 (LDAPv2) but it was never standardized in any formal specification. This usage has been deprecated along with LDAPv2, which was officially retired in 2003.

The trade off between StartTLS and TLS/SSL exists not only in LDAP protocol, but also in many other protocols such as SMTP (port 2525, 25, 587). StartTLS is also called Opportunistic TLS. The standard is in the relevant RFC documents.

Networking Basics 3 of 3 – common network protocols and technologies

Fri, 20 Dec 2019 10:28:00 -0400

The 5 layer TCP/IP model (or its more rigorously defined alternative OSI model) leads to a whole world of network protocols. Understanding these new protocols requires one to map it out agains the network layers (e.g. at Layer 4 whether it is TCP or UDP, etc) .

VPN is a whole family of technologies with many flavours of implementation. The previous posting covered some basics of the idea, as well as the two common forms (remote access VPN and site-to-site VPN). The VPN implementation protocols vary a lot.

PPTP(Point-to-Point Tunnelling Protocol) is outdated and less secure
IPSec (Internet Protocol Security)
L2TP (Layer 2 Tunnelling Protocol) replacement of PPTP, more secure, more overhead and slightly slower.
OpenVPN – very secure, and reliable and supported by communities all over the world.
TLS/SSL and SSH connections may be considered VPN as well.

Phone service protocols:

VoIP (voice over IP, operating at network layer) – allows one to make and receive phone calls over the network. Communication on the IP network is perceived as less reliable in contrast to the circuit-switched public telephone network because it does not provide a network-based mechanism to ensure that data packets are not lost, and are delivered in sequential order. It is a best-effort network without fundamental Quality of Service (QoS) guarantees. Voice, and all other data, travels in packets over IP networks with fixed maximum capacity. This system may be more prone to data loss in the presence of congestion[a] than traditional circuit switched systems; a circuit switched system of insufficient capacity will refuse new connections while carrying the remainder without impairment, while the quality of real-time data such as telephone conversations on packet-switched networks degrades dramatically. Therefore, VoIP implementations may face problems with latency, packet loss, and jitter.
SIP (session initiation protocol, operating at application layer) – a VOIP signaling protocol responsible for the creation and tearing down of media connections. So it supports all types of media.

ALG (application layer gateway, aka proxy server) – a software component that manages specific application protocols such as SIP and FTP. An ALG acts as an intermediary between the Internet and an application server that can understand the application protocol. ALG proxies connection to destination on behalf of client. This adds

Application Layer Gateway

DDNS (dynamic domain name service) – a router service that assigns your device a fixed domain name even though you are using dynamic IP.

NAT – another family of technologies, usually implemented in the following

Port preservation: source port chosen by a client is the same port used by the router
Port forwarding: NAT application redirects a communication request from one address and port number combination to another while the packets are traversing a network gateway, such as a router or firewall.
Port triggering: a dynamic form of the port forwarding model. Generally, port triggering is used when the user needs to use port forwarding to reach multiple local computers. Port are close when they aren’t in use (more secure) protocol used is UPnP

DMZ – a physical or logical subnet that contains external facing service to untrusted network (e.g. Internet). The purpose is to add an additional layer of security so an external network can assess what is exposed in DMZ while the rest of network remains firewalled.

VPN passthrough – a feature that allows any device connected to the router to establish outbound VPN connections. Most modern router already have this built in.

WAN optimization – a collection of techniques for increasing data transfer efficiencies across wide-area networks.

Deduplication
Compression
Latency optimization
Caching/proxy
Protocol spoofing
Traffic shaping

Network performance tuning

Performance tuning in network covers a variety of skills. It is important to understand in which layer the problem occurs. Anything above layer 4 is more likely to be an application issue. For ethernet performance tuning, I found this page and this page to be helpful in my practices.

Networking basics 2 of 3 – Layer 4 and common network configurations

Sat, 07 Dec 2019 23:19:00 -0400

Transport Layer

Transport Layer handles multiplexing & de-multiplexing through ports. Port is more or less a virtual concept. Source port is usually ephemeral. Two dominant protocols are TCP and UDP.

TCP relies on acknowledgement. TCP control flags are SYN, ACK, FIN, URG, PSH, RST, ECE, CWR. TCP connection is established by 3-way handshake and torn down by 4-way termination.

TCP handshake and termination

Socket – the instantiation of an end-point in a potential TCP connection. A socket can be in one of the following states:

LISTEN: a TCP socket is ready and listening for incoming connections;
SYN_SENT: a SYNC request has been sent but connection hasn’t been established yet;
SYN_RECEIVED: a socket previously in a LISTEN state has received a SYNC request and sent a SYN/ACK back;
ESTABLISHED: connection is up;
FIN_WAIT: FIN sent, ACK hasn’t been received yet;
CLOSE_WAIT: connection has been closed at the TCP layer but the application that opened the socket hasn’t release the hold on the socket yet;
CLOSED: connection fully terminated;

TCP packet format

TCP is a connection-oriented protocol

Transport layer is responsible for re-sending data if data is lost
Sequence # is important because packet may arrive out of sync but receiver reassemble them in order
There is a lot of overhead (acknowledgement, establish connection first, tear down connection afterwards)

On the other hand, UDP is connectionless. A good example is video streaming, where it is okay to lose a few packet along the way, in exchange of bandwidth saving.

Firewall may operate at different layers but it is most commonly used at transport layer, to block traffic based on port.

Application Layer

There is no dominant protocol at this layer. IIS, Nginx and Apache are examples of applications operating at this layer.

Basic network configurations

Standard modern network configuration involves: IP address, subnet mask, gateway and DNS server.

DNS – global and highly distributed network service that resolves domain name into IP address. There are many steps in DNS resolution. DNS service listens on port 53. Two famous free public DNS servers are 8.8.8.8 and 8.4.4.4. DNS servers have five categories:

Caching name servers: store known domain name lookups in cache. TTL today can be a few hours, much shorter than what it used to be in early days;
Recursive name servers: perform full DNS resolution request;
Root name servers;
TLD name servers;
Authoritative name servers;

DNS resolution steps

DNS uses UDP protocol and it can generate a lot of traffic (TCP is impractical. If implemented in TCP, it would have required 44 packet for a DNS query, which is very expensive considering DNS query is just a precursor of the real traffic)

Anycast DNS – any one of a number of DNS servers can respond to DNS queries, and typically the one that is geographically closest will provide the response. This reduces latency, improves uptime for the DNS resolving service and provides protection against DNS flood DDoS attacks.

DNS record types:

A record: domain name to IP address. DNS service round robin across multiple A records
AAAA (quad A) record: domain name to IPv6 address
CNAME: redirect traffic from one domain to another (e.g. test.com to www.test.com so you can minimize IP references)
MX record
SRV record
TXT record: originally for human consumption, freeform text for configuration purpose.

A FQDN (fully qualified domain name) can have up to 127 domains, but only three in most cases. (i.e. subdomain.domain.topleveldomain)

DNS zones – allow for easier control over multiple levels of a domain. DNS zones are configured in zone files.

Domains vs zones – Domains are broken into zones for which individual DNS servers are responsible. A domain represents the entire set of names/machines that are contained under an organizational domain name. For example, all domain names ending with “.com” are part of the “com” domain. A “zone” is a domain less any sub-domains delegated to other DNS servers. A DNS server could be responsible (authoritative) for all records under the “xyz.com” domain, but by defining NS-records for “abc.xyz.com”, this part of the domain is delegated to other DNS servers – and possibly a different company/entity. A zone contains exactly one SOA-record describing the general properties of the zone, and any number of other DNS records. Entire zones can transferred from a primary DNS server to secondary DNS servers through Zone Transfers. A domain administrator would be responsible for creating zones, and delegating responsibility for these zones to an administrator and DNS server.

Reverse DNS lookup – query for FQDN by IP. This is commonly used by email servers where anti-spam mechanism on the receiver needs to validate that sender’s IP is associated with a domain as claimed. This is also used in logging application to convert IP into human-readable domains in the log data. Reverse DNS lookups query DNS server for a PTR (pointer reserve record). If the server does not have a PTR record, it cannot resolve a reverse lookup.

DHCP

DHCP operates at application layer and helps you to configure IP automatically with a lease, through automatic allocation, or fixed allocation based on MAC, etc. DHCP process involves address allocation, renewal, and release. Address allocation takes four steps:

Client sends a broadcast to discover DHCP server;
DHCP server broadcast a DHCP offer;
Client requests IP address from the DHCP server;
Server acknowledged the DHCP request;

Here is an illustration of DHCP address allocation.

DHCP can also be used to set NTP address.

NAT

Network Address Translation (NAT), can be implemented in many different ways in different OS. Essentially, it is a technology that allows a gateway, usually a router or firewall, to rewrite the source IP of an outgoing IP datagram while retaining the original IP in order to rewrite it into the response. Two categories of NAT are:

Basic NAT: provides a one-to-one translation of IP addresses, aka one-to-one NAT. Basic NATs can be used to interconnect two IP networks that have incompatible addressing.
One-to-many NAT: maps multiple private hosts to one publicly exposed IP address, aka IP masquerading. This can be a security measure so that no external host can establish to your computer without knowing your actual IP. Source port conflict can be managed in two ways:
- port preservation: When making outgoing connection, NAT preserves the ephemeral port number used by internal client that initiates the connection; if two clients happen to use the same ephemeral port, then NAT picks a random port to initiate outgoing TCP connection;
- port forwarding (port mapping): Forward traffic to certain destination based on the port of incoming request that NAT receives.

Proxy

Proxy refers to a concept rather than a specific implementation. It exists on almost every layer in the network model, and act on behalf of a client in order to access other service. For example,

Web proxy used to be used to cache web traffic data in slow Internet but it is not necessary any more because 1) there is not much speed benefit; 2) website today is much more dynamic.
Reverse proxy is a popular architecture of web server, such as Nginx, to act as a front end of web servers, as well as point of decryption so web servers are free to just serve the content.

VPN

VPN – allows for extension of a private or local network to host that might not be on that local private network by using encrypted tunnel. There are many flavours of implementation for many purposes. It is a general concept rather than a specific protocol (just like NAT). VPN client provisions the computer with a virtual interface with an IP that matches the address space of the private network, and establish a VPN tunnel to it. Most VPNs work by using the payload section of transport layer to carry an encrypted payload that actually contains an entire second set of packets: the network, the transport and the application layers of a packet intended to traverse a network. Basically, this payload is carried to the VPN’s endpoint where all the other layers are stripped away and discarded. Then, the payload is unencrypted, leaving the VPN server with the top three layers of a new packet. This gets encapsulated with the proper datalink layer information and sent across the network. This process is completed in the inverse in the opposite direction. VPN usually requires strict authentication procedures and encryption. VPN can also be used to establish site-to-site connection (aka point-to-point VPN) where individual user doesn’t have to establish connections on their own. Both sites needs specialized hardware to achieve this. Site-to-site VPN is a good alternative to WAN when two site don’t need to transfer large amount of data for very fast speed.

WAN

Wide Area Network – act like a single network, but span across multiple physical locations, it requires that you contract the link across the internet with ISP. ISP handles data link from one site to another.

Wireless Network

Wireless protocol (802.11 family) defines how Wifi operates at physical and data link layer. Wifi networks operates on 2.4GHz and 5GHz frequency bands. Wireless frame is fairly different from Ethernet frame due to the nature of wireless transmission.

Wireless frame

Wireless access point is a device that bridges the wireless and wired portions of a network. A single wired network might have many wireless access points to cover a large area.

Wireless network can be configured in a few main ways:

Ad-hoc network – nodes all directly speak to each other. No supporting infrastructure is needed but not most common. It can be powerful tool during disasters.
Wireless LAN (WLAN) – one or more access points act as abridge between wireless and wired network. This is the most common type in business world where the wired LAN provides link to the Internet.
Mesh networks – a hybrid of the two above

Wireless Security

Wireless transmission is across the air so encryption is more important. The number of bit in the encryption key corresponds to how secure the encryption is.

WEP (encryption technology) provides low level of privacy (40-bit encryption) and it is not preferred today;
WPA provides 128-bit key encryption;
WPA2 provides 256-bit key encryption and is most common today.
MAC filtering also help security in wireless

Basic networking troubleshooting

ICMP ping to test general quality of connection;
traceroute discovers the path between two nodes and give you the information along the way;
netcat checks port and host address (telnet is retiring);
nslookup: very powerful in interactive mode for resolution tools;

IPv6

IPv4 address running out in November 2019 is a major crisis. IPv6 becomes more critical to implement for many organizations. IPv6 address is very long with numbers. Numbers represents a natural way that computer thinks and operates, but not for human. So IPv6 comes with two rules to shorten the address:

You may remove any leading 0;
Any number of consecutive groups can be replaced with two colons;

IPv6 header looks like below:

IPv6 header

Coexistence – the optimal approach for existing networks is to focus not on transition but on coexistence. Coexistence may live a long period with these phases: 1) Turn on IPv6 routing in their existing IPv4 networks and start using it; 2) Contract IPv6 service with their upstream, peer, and downstream neighbours; 3) Use the IPv6 protocol in addition to IPv4 in their applications and services both on server equipment and on their clients; 4) turn off IPv4 at some point when it is no longer a business requirement.

Cryptography basics 2 of 2

Sun, 08 Sep 2019 21:21:41 -0400

My previous post outlines several core concepts around cryptography, such as asymmetric key encryption, digital certificate, the encoding formats and relevant file extensions. In this article, we continue to explore cryptography use cases, where these concepts are connected and put into application.

The most important use case is TLS handshake. I cannot stress enough how paramount this scenario is. This use case and its variation can be found in almost every situation where connection needs to be secured.

TLS handshake: the process in which client and server establish secure connection. During the handshake, two parties agree on TLS version, decide on cipher suite, authenticate the identity of each other (although client identity authentication is less common), and generate session key for symmetric encryption after the handshake. Details steps are very important. Here is a fairly thorough reference, and here is a great diagram:

TLS handshake

Secure browser connection: The most widespread use of TLS handshake is to secure browser connection with HTTPS:

Browser initiates connection to the server;
Server sends browser its certificate (public key + digital signature signed by CA);
Browser has preloaded public key of CA and uses it to decrypt digital signature and get the digest of public key;
Browser calculate digest of received public key and compares it against the digest from the previous step;
If the result is the same, public key is trusted, a green lock is displayed; otherwise, a warning is displayed;
proceed to the rest of the steps in TLS handshake.

Secure connection in Java: Java applications manages keys and certificates through two classes: java.security.KeyStore and java.security.TrustStore. Suppose a Java application client initiates TLS connection to server. The server application will present its certificates from server’s Key Store. The client will use certificates stored in client’s Trust Store to verify the identity of the server. Once validated, the client then presents certificates stored in client’s Key Store back to the server for validation.

A KeyStore keeps keys and certificates for your own application. Typically, you store a KeyPair in a KeyStore file.
A TrustStore keeps the certificates of external systems that your application trusts.

Below is a great diagram:

Secure Java application in TLS

JKS file is Java’s version of PKCS#12 (private key + certificate, password protected). Entries in a JKS file must have an “alias” that is unique. The JKS file type can be used for both Key Store and Trust Store. When it’s used in Key Store, it contains a certificate and private key for the Java application. When it’s used in Trust Store, it only contains certificate from external trusted applications. Note that JKS is the default keystore format until Java 8. Since Java 9 the default keystore format is PKCS12.

Finally, we can introduce some tools:

Command line Tools

openssl is a versatile tool for cryptography and keytool is a similar tool for Java applications.

keytool is for store keys/certificates in Java Key Store or Trust Store.

Conclusion

I do not include any openssl or keytool command in this article in order to remain theoretical. However, once through the conceptual hurdles, one should become fairly comfortable picking up the tools and understand why each command is needed to achieve its purpose.