testing on Digi Hunch

DICOM testing over TLS

Sat, 18 Feb 2023 01:05:00 -0400

I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario.

In a nutshell, I changed from dcm4che to dcmtk binary builds. I’ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section.

Background

To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well.

I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123!

To turn that test into a C-Store test, simply add a DCM file as input:

./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM

The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify –tls12 as the version, and with –tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store.

This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).

Moreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package.

The test case, data and the tool

We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here.

Before mocking with dcmtk‘s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.

For a full-blown testing with TLS, we should have two levels of CA as below:

The chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:

Our DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.

In addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are:

Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key
Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA’s private key
Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA’s private key

Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:

-d (shorthand for –debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for –verbose) instead.
+tla (shorthand for –anonymous-tls): enable anonymous TLS (without client certificate)
+tls (shorthand for –enable-tls): enable full TLS (with client certificate), followed by client key and certificate files
-rc (shorthand for –require-peer-cert): –require-peer-cert, require peer (server) certificate
+cf (shorhand for –add-cert-file): –add-cert-file, add server certificate so client can trust it.

For C-ECHO, the testing commands with and without client certificate look like:

$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112
$ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112

For C-STORE, the testing commands with and without client certificate look like:

$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm
$ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm

As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.

Orthweb Test

The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:

IssuerComName=issuer.orthweb.digihunch.com
ClientComName=dcmclient.orthweb.digihunch.com
ServerComName=ca-central-1.compute.amazonaws.com

openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt

openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr
openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt

When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section.

Korthweb Test

Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:

curl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt

Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser.

On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:

openssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com < /dev/null
openssl s_client -showcerts -connect dicom.orthweb.com:11112 < /dev/null

The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.

When it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to “*” so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check.

The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well.

For example, I perform DICOM validation (Istio ingress) with the steps below:

# bhs: generate client key pair
openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr

# bhs: export intermediate CA credentials
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.key}' | base64 -d > bhs.int.ca.key
kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath='{.data.tls\.crt}' | base64 -d > bhs.int.ca.crt

# bhs: get intermediate CA to sign client cert 
openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt

# bhs: validate web request (without client certificate)
curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt

# bhs: validate DICOM c-echo request (with client certificate)
echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112

# bhs: validate DICOM c-store request (with client certificate)
storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm

On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:

echoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112
storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm

The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.

TLS profile

Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:

–profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile
–profile-bcp195 (+px): BCP 195 TLS Profile
–profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile
–profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired)
–profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA)

The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:

Implementation shall not negotiate TLS 1.0 or 1.1
Client and server shall prefer strict TLS configuration (as opposed to startTLS)
Ciphers that should be supported.
Recommend port 2762

These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6

3.6. Server Name Indication
TLS implementations MUST support the Server Name Indication (SNI)
extension defined in Section 3 of [RFC6066] for those higher-level
protocols that would benefit from it, including HTTPS. However, the
actual use of SNI in particular circumstances is a matter of local
policy.
Rationale: SNI supports deployment of multiple TLS-protected virtual
servers on a single address, and therefore enables fine-grained
security for these virtual servers, by allowing each one to have its
own certificate.

So technically, missing SNI is considered incompliant.

Summary

I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.

Istio Lab – Ingress and Egress

Tue, 16 Nov 2021 21:28:00 -0400

Istio is a popular open-source service mesh implementation using Envoy proxy. One of the benefit of using Istio is the ingress and egress it brings to native Kubernetes platform. This article is a hands-on guide to test Istio ingress and egress gateways on Minikube. It was tested on my MacBook. All the information in this post can be found in several different pages from Istio and Minikube documentation. The purpose of this article is a single cheat sheet for myself to quickly set up Istio ingress and egress testing.

Install Istio on Minikube

We start with configuring Minikube on MacBook (or change the command accordingly on other platform), and install metal LB load balancer. The steps with Istio also works with a regular Kubernetes platform.

minikube start --memory=12288 --cpus=6 --kubernetes-version=v1.20.2 --nodes 3 --container-runtime=containerd --driver=hyperkit
minikube addons enable metallb

Then we need to give load balancer an IP address range before setting up the load balancer. Minikube adds an interface named bridge100 to MacOS host environment. The IP address of the host can be found via the following command:

minikube ssh "ping host.minikube.internal -c 1"

For example, my host IP address is 192.168.64.1, so I dedicate the IP range 192.168.64.64 to 192.168.64.80 to the MetalLB load balancer. Then I need to provide this range when configuring the load balancer:

minikube addons configure metallb
kubectl -n metallb-system get po

After the configuration, make sure the MetalLB pod is up. Then we can install istio to the Kubernetes cluster. The steps are completed using istioctl following the instruction, I created a quick and dirty script to put the steps together:

#! /bin/bash
curl -L https://istio.io/downloadIstio | sh -
export PATH=$(realpath istio*/bin):$PATH
ln -sfn $(ls -d -- istio-*) istio
if istioctl x precheck; then
 echo ready to install istio and lable namespace for istio-injection
 istioctl install --set profile=demo -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY
 kubectl label namespace default istio-injection=enabled
else
 echo failed precheck
 exit 1
fi

The script above download the istio zip file, unzips it, installs Istio (in demo mode and set outboundTrafficPolicy to REGISTRY_ONLY), labels default namespace for proxy injection. To use istioctl without spelling out full path, we can run the following command again separately outside of script:

export PATH=$(realpath istio/bin):$PATH

To validate, check istioctl version command.

Install Sample application

For simplicity, the rest of this article uses httpbin to test Istio’s ingress and egress. For completeness, this section gives the steps to install bookinfo application as it is used in other Istio testing as well. It also help understand how to ingress to an application with Istio using an IP provided by the load balancer. Skip this section if just need to test with the simple httpbin application. The Bookinfo application comes within the istio 1.12 directory. In the following steps from the istio directory, we install some observability add-ons as well as the bookinfo application. Then we test access to the website.

kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/kiali.yaml
kubectl apply -f samples/addons/jaeger.yaml
kubectl apply -f samples/addons/grafana.yaml

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS productpage:9080/productpage | grep -o "<title>.*</title>"

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
curl -s "http://${GATEWAY_URL}/productpage" | grep -o "<title>.*</title>"

On my laptop, the value of $GATEWAY_URL is 192.168.64.64:80. Therefore I can browse to site http://192.168.64.64:80/productpage with local browser. We can also check kiali dashboard:

istioctl dashboard kiali

Then we are good to examine ingress and egress.

Prep Work

The following custom resources are related to traffic management in istio:

Virtual Service (vs)
Destination Rule (dr)
Service Entry (se)
Gateway (gw, Istio’s ingress or egress)

We will expand on the role of each in a separate article. To execute command from within a Pod, we also use sleep application which is essentially a Pod with a sleeper thread for the purpose allowing user to run troubleshooting command. The sleep application is provided in istio directory:

kubectl apply -f samples/sleep/sleep.yaml
kubectl get po -l app=sleep

Find out the Pod name to execute command. Then we use httpbin website to emulate an external service, and use the sleep Pod (suppose the name is sleep-557747455f-q99cz) to access the service:

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/headers

As we previously set outboundTrafficPolicy.mode=REGISTRY_ONLY, this step is expected to output nothing. If this is not the case, it means the cluster may have ALLOW_ALL as outbount traffic policy, and you will need to follow this guide to set it to REGISTRY_ONLY. With REGISTRY_ONLY, in order to allow access, we need to register a service entry, by applying the document below:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: httpbin
spec:
 hosts:
 - httpbin.org
 ports:
 - number: 80
 name: http
 protocol: HTTP
 resolution: DNS

Now we should be able to access httpbin service as it is registered:

kubectl get se httpbin -o yaml

Using the exec command above again, we should see output from the curl command.

Ingress Testing

We deploy an httpbin service in order to test ingress. The YAML declaration is prepared by Istio installation package:

kubectl apply -f samples/httpbin/httpbin.yaml

Then we deploy the ingress:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: httpbin-gateway
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - "httpbin.digihunch.com"

The selector above indicates that the gateway deployed to a Pod labeled with istio: ingressgateway. Then we create a corresponding virtual service for this gateway:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: httpbin
spec:
 hosts:
 - "httpbin.digihunch.com"
 gateways:
 - httpbin-gateway
 http:
 - match:
 - uri:
 prefix: /status
 - uri:
 prefix: /delay
 route:
 - destination:
 port:
 number: 8000
 host: httpbin

Note that the hosts setup must match the configuration in gateway. The virtual service defines two services at URIs /status and /delay to expose, and we can confirm the creation of virtual service:

kubectl get vs httpbin -o yaml

We can then test with curl command from MacOS host to verify that the ingress is working. We need to use the IP address of the service (192.168.64.64 with port 80 as previously indicated):

curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/status/200
curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/delay/2

We use -HHost to emulate this request going to httpbin.digihunch.com (otherwise we’d have to fake DNS entry in /etc/hosts). Both command should give their return without error. The second command will give return after a delay of 2 seconds.

This validates the success of ingress setup.

Egress Testing

To visit an external service, there are four options:

Just access external service from Pods, when you have global.outboundTrafficPolicy.mode=ALLOW_ANY
Use ServiceEntry (as illustrated above in Prep Work)
Bypass envoy proxy (not recommended, because there is no role of service mesh)
Configure an Egress Gateway

In the Prep Work, we’ve seen how to use Service Entry. However, there is no control of how Pods access the external service. We are going to configure an Egress Gateway to manage the outgoing traffic flow. This allows you to manage route, and apply observability features to the outgoing traffic, something the Service Entry alone will not provide. The egress gateway have two typical use cases as explained on Istio website.

First, we want to confirm that the egress pod exists, with a quick command:

kubectl get po -n istio-system -l app=istio-egressgateway && kubectl get se

It should exist as we installed istio as demo version. We then also need a service entry, as we have already configured in Prep Work. At this point, the egress gateway resource has not been created.

Now, in a separate terminal, we monitor the log of egress pod

kubectl logs -f istio-egressgateway-7d5d69dcfd-kvwxv -n istio-system

With the -f (follow) session on, if we visit external service as we did during Prep Work, there is no new entry pop up at the tail of the log. This is because the external access is still not managed by egress. Now we need to add egress, with the following resources applied:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: istio-egressgateway
spec:
 selector:
 istio: egressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - httpbin.org

Then we setup the virtual service with related rules:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: vs-for-egressgateway
spec:
 hosts:
 - "httpbin.org"
 gateways:
 - istio-egressgateway
 - mesh
 http:
 - match:
 - gateways:
 - mesh
 port: 80
 route:
 - destination:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subset: httpbin
 port:
 number: 80
 weight: 100
 - match:
 - gateways:
 - istio-egressgateway
 port: 80
 route:
 - destination:
 host: httpbin.org
 port:
 number: 80
 weight: 100

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: dr-for-egressgateway
spec:
 host: istio-egressgateway.istio-system.svc.cluster.local
 subsets:
 - name: httpbin

With all applied, we can repeat the access command using curl, and watch for the log tail.

kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/ip

The log should output a line saying curl command access, which indicates that this outbound access is managed by istio egress gateway. The log line looks like below and in the example the IP address 10.244.2.4 is the Pod IP address.

[2021-11-22T01:22:47.547Z] "GET /ip HTTP/2" 200 - via_upstream - "-" 0 45 818 816 "10.244.2.9" "curl/7.80.0-DEV" "9ec0a193-77f0-90a5-b132-21c3259307da" "httpbin.org" "3.216.167.140:80" outbound|80||httpbin.org 10.244.2.4:42372 10.244.2.4:8080 10.244.2.9:52710 - -

Looking at the virtual service declaration, you might noticed multiple routes in the virtual service. The first route (mesh) tells sidecars to route request to egress gateway. The second route (istio-egressgateway) tells egress gateway to route traffic to external service as destination.

In real life, there’s more to consider. For example, we usually want the host to allow on egress to be a wild card host such as *.api.google.com. While the hosts field in virtual service can be wild card, the route destination in virtual service has to be a list of fully qualified domains. The other situation is the destination website may issue an HTTP redirect to a different host, which also needs to be registered as a destination. These details can add up to administrative effort.

Egress example

An other real life scenario is the use of authorization policy. Authorization policy rules use SPIFEE identity of a Pod. If a Pod initiates egress connection to destination host in TLS, then the Pod’s SPIFFEE identity and its identity for external TLS connection are in conflict.

With Azure AKS

The steps above also works on other cloud platform such as AKS. This is an AKS cluster that I tested the steps. The Terraform code provisions a private cluster, as well as a Bastion Host. From the Bastion host, we can perform the same steps as above.

When Istio gets installed, it will created a Service object called istio-ingressgateway in the istio-system namespace. The service is of load balancer type and by default will create a public load balancer in Azure AKS.

The test steps will also work for private load balancers. To tell Azure to create private load balancer on a specific subnet, the Service object needs to include annotation. We have to customize Istio installation in order to do that, using the following overlay file (azoverlay.yaml):

apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
metadata:
 namespace: istio-operator
 name: custom-istiocontrolplane
spec:
 profile: demo
 values:
 gateways:
 istio-ingressgateway:
 serviceAnnotations:
 service.beta.kubernetes.io/azure-load-balancer-internal: "true"
 service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "suitable-porpoise-lb-subnet"

This is inspired by this post. Then the command to install Istio becomes:

istioctl install -f azoverlay.yaml -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

This concludes the hand-on guide for Istio ingress and egress labs. I will need to write another post explaining the related concepts.