[{"content":"To automate and secure machine learning operations, one needs to know about programming with SageMaker SDK. While algorithms are data scientists\u0026#8217; wheelhouse, infrastructure engineers like myself need to understand the skeleton of the code in order to orchestrate the activities, optimize performances, and secure the communications.\nIt was overwhelming to begin with. Programming in machine learning heavily depends on specific frameworks, SDKs, and is highly experimental. The experimental nature entails the followings in the notebook workload:\ndata engineers may want to load data once, and play with different data processing logics if the program fails halfway through, the data scientist can rerun from the failed step, or even execute the cells out of order some lines of code (for training or inference), may demand significantly more resource than other lines; These makes machine learning workload more challenging and require more flexibility at the infrastructure layer. However, most of the tutorials target data scientist aspirant and lack the perspective of infrastructure operations.\nIn this post I\u0026#8217;ll go over what I\u0026#8217;ve learned about training and inference workload. I will not focus on specific algorithms. Instead, I\u0026#8217;ll use the most easy-to-understand algorithm, solving a typical problem. I\u0026#8217;ll also use a simple framework (SK Learn) but try to reveal what is common amongst other supported frameworks.\nFrameworks and Data Amongst many programming frameworks, I need one that is generic, beginner friendly and compatible with SageMaker AI. The candidates are:\nScikit-learn: good for classical machine learning activities; TensorFlow and PyTorch: industry standards supporting a wide range of tasks; from simple models to advanced deep learning applications; Hugging Face, PyTorch and TensorFlow can handle NLPs; XGBoost excels at handling tabular data The use case statements above might be over-simplifications but at least Scikit-learn is a solid choice for beginners. It\u0026#8217;s also good to keep Pytorch and TensorFlow in mind. As to the machine learning problem, I want to go easy with a simple classification problem with Logistic Regression. Despite the word \u0026#8220;regression\u0026#8221; in the name, logistic regression is an algorithm to solve classification problems (whereas linear regression solves a regression problem).\nI use the simplistic iris flower data set to solve the classification problem: identifying the species (setosa, versicolor, or virginica) based on the length and width of sepal and petal. The dataset is very clean and does not reflect real-world messiness, making it a good for testing machine learn concepts, although not for training in production. The input is four numbers. The prediction output is a number (0,1, or 2) representing the species identified.\nTraining and Inference Machine Learning concerns with indeterministic results. We first train the models with input data along with known results. Once trained, we can store the model as artifact with version control. Then we serve the model so we can feed it with unseen data and use the output as predictions. This machine learning flow has a lot in common with SDLC. Training resembles building of an application, the output artifact is the trained model, which we store as flat file or tarballs. In a model lifecycle, we track versions of trained models, quality control the models, decomission them and so forth. To perform inference, we serve the model in different ways, such that consuming application can use it for predictions.\nTo get started, let\u0026#8217;s look at the following simple snippet, where we load data, train a simple model and use it for inference.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 import numpy as np import pandas as pd from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score # Step 1: Load the Iris dataset data = load_iris() X = data.data # Features (sepal length, sepal width, etc.) y = data.target # Target (species labels) # Step 2: Split the dataset into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) # Step 3: Create and train a logistic regression model model = LogisticRegression(max_iter=200) model.fit(X_train, y_train) # Step 4: Predict using the test set y_pred = model.predict(X_test) # Step 5: Evaluate the model accuracy = accuracy_score(y_test, y_pred) print(f\u0026#34;Model accuracy: {accuracy * 100:.2f}%\u0026#34;) Note that the snippet does not use any SageMaker SDK libraries yet. The example is simple enough to run anywhere. In real life however, the invocations of .fit() method and sometimes .predict() method can be resource-demanding. In such experiments, even a simple change to the hyper-parameter may dramatically increase the resource requirement. This experimental nature calls for a mechanism to allow execution of certain lines of code, in a different runtime environment. The SageMaker SDK provides two mechanisms: a @remote decorator and a RemoteExecutor class. Take the decorator as an example, it employs Python\u0026#8217;s decorator to implement a wrapper of the function in the code so that the function can execute remotely on a different machine. The code looks like this:\nimport numpy as np import pandas as pd from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score # Step 1: Load the Iris dataset data = load_iris() X = data.data # Features (sepal length, sepal width, etc.) y = data.target # Target (species labels) # Step 2: Split the dataset into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) # Step 3: Create and train a logistic regression model: put the action in a function and mark it to execute remotely from sagemaker.remote_function import remote @remote(instance_type=\u0026#34;ml.m5.large\u0026#34;) def train(X_train,y_train): model = LogisticRegression(max_iter=200) model.fit(X_train, y_train) return model model=train(X_train, y_train) # Step 4: Predict using the test set: put the action in a function and mark it to execute remotely @remote(instance_type=\u0026#34;ml.m5.large\u0026#34;) def predict(model): return model.predict(X_test) y_pred = predict(model) # Step 6: Evaluate the model accuracy = accuracy_score(y_test, y_pred) print(f\u0026#34;Model accuracy: {accuracy * 100:.2f}%\u0026#34;) Note that this snippet requires a SageMaker SDK library (sagemaker.remote_function) and must run on a Jupyter Lab notebook. That comes with the flexibility to run any function on any type of instance allowed in the SageMaker domain. When a function runs remotely from Jupyter workspace notebook, SageMaker has to spend time provisioning and bootstrapping an EC2 instance, which usually takes a noticeable delay and the stdout gives a summary of billable seconds. SageMaker also capture these remote executions as training job uniformly.\nSageMaker SDK Abstractions We can summarize the activity pattern from the two example scripts as below:\n","date":"2025-04-20T07:20:00-04:00","image":"/wp-content/uploads/2025/04/feature-sagemaker-sdk.webp","permalink":"/2025/04/training-and-inference-in-sagemaker-ai/","title":"Training and Inference in SageMaker AI"},{"content":"Machine learning workflows are highly experimental. To smooth out the processes, Amazon SageMaker AI packages many features as managed services. As an infrastructure specialist, I want to remain compliant. At a basic level, compliant architecture means multi-account structure and hub-and-spoke VPC topology in a landing zone. While the multi-account operating model for SageMaker is well documented, these ML managed services obscure the network configuration. I\u0026#8217;m always looking for low-level insights on where the computing activity is happening and how the application traffic flows in and out of our VPCs. I don\u0026#8217;t always get straight answers and I often have to experiment them out. This post is a review of the networking aspects of SageMaker AI I recently learned.\nNaming Shenanigans Unfortunately, I have to start with how AWS has renamed these services, so the terms remain clear throughout the post.\nIn Dec 2024, AWS renamed SageMaker to SageMaker AI. The name of SageMaker going forward represents the overarching AWS service for machine learning, data, analytics and generative AI. Here is a video for clarification. I\u0026#8217;m not a fan of how they repurpose the names. As of date there are still a lot of content referencing SageMaker AI as SageMaker, whose meaning has changed.\nIf that\u0026#8217;s not enough, here\u0026#8217;s another one. The SageMaker Studio launched originally in 2019 for model development. In 2023 that became SageMaker Studio classic, in favour of the newly launched studio, taking the name of SageMaker studio. In Feb 2025, AWS deprecated SageMaker Studio classic. You can only create SageMaker Studio in SageMaker AI. At the SageMaker level, AWS launched SageMaker Unified Studio, the all-encompassing development environment for data analytics, generative AI, and so on. In this post though, we talk about many features under SageMaker AI and SageMaker Studio. While the service UIs are picturesque, we remain focused on two questions: how these services interact with resources on our VPCs, and how they connect to the Internet.\nWorkload Categories We divide machine learning workload into three categories, based on network connectivity pattern: notebooks, model hosting, and pipeline jobs.\nNotebooks are where data scientists carry out experiments by running experimental scripts on performing hardwares (depending on the tasks), usually within IDE application as Jupyter Labs, Code Editors. Data scientist users may perform any machine learning related activities such as model evaluation, etc. It is possible that one part of a notebook only requires consumer grade CPU and another part of the notebook program requires a performant GPU. It all depends on the nature of the program code.\nModel is the key artifact in Machine Learning workflows. Models themselves are files stored in S3 buckets. The machine learning engineers performs two most common activities. They train the model, and feed the model with unseen data for new output (inference). In simple workflows, data scientists may build, train a model and run inference all from within the same notebook. As the experiment concludes and the team wants to operationalize the inference, it makes senses the run inference in a client-server architecture. This calls for a inference endpoint acting as the server, backed by the trained model, operating on a single or an autoscaling group of instances. The client application feeds the endpoint with unseen data, often using REST API calls, and expects inference results.\nPipeline steps like training do not operate on a server. They are similar to Notebook workloads. The difference is that pipeline steps are headless executions. The steps are non-interactive without engaging the Studio GUI. Many other types of activities in machine learning are in similar pattern, such as model evaluation, model optimization or any general processing such as a Python script. I consider them similar to training activities. Since we orchestrate these headless activities with pipelines (e.g. SageMaker pipeline), and each step may execute on some specialized instance depending on the computing requirements. Collectively, I call these activities the pipeline jobs.\nLet\u0026#8217;s look at these workloads through the networking lens.\nStudio Notebooks The most common Studio app is some kind of notebooks, such as Jupyper Lab. However, this category can generally include all kinds of SageMaker Studio apps, e.g. Canvas, Code Editor. I use the term Studio app and Studio notebook interchangeably but the APIs mostly refer to these as apps, such as AppNetworkAccessType.\nIn the app, a user may create one or more spaces each specifying the backing instance type. The configuration that influences the instances\u0026#8217; networking setup is in SageMaker AI domain\u0026#8217;s Network Setting. There are two parts of this AppNetworkAccessType setting:\nNetwork Mode (also called AppNetworkAccessType in AWS SDK): PublicInternetOnly (default): only EFS traffic goes through the specified VPC and subnets. Other studio traffic (e.g. API calls) goes through the Internet Gateway of the VPC that the studio manages internally VpcOnly: all studio traffic goes through the specified VPC and subnets. This delegates the responsibility of connectivity to endpoints to the VPC\u0026#8217;s owner. VPC and Subnet: to place EFS mount points on. Also route other studio traffic in VpcOnly mode. There are one diagrams on the documentation for each network mode (PublicInternetOnly on the left; VPCOnly on the right):\nThe diagrams (as of March 2025) are not accurate because the PublicInternetOnly mode also has a domain managed ENI per app space. The VpcOnly mode is when the ML do not like the idea that a Notebook instance can bypass centrally managed Internet path. The team must ensure the endpoints are reachable, either via Internet, or via routable VPC endpoint (e.g. Gateway Endpoint for S3 and Interface endpoint for the rest). In a hub-and-spoke setup it might be another dedicated VPC that provides the interface endpoints centrally.\nThe domain settings include a few configurations on the underlying instance. For example, SecurityGroupIds specifies the security groups associated with the ENIs. DockerSettings enables Docker daemon on the instance, allowing users to test container workload in local mode. Note that in the more constraint VPC-only mode, Docker pull and push operations outside of Amazon Elastic Container Registry aren\u0026#8217;t supported. To pull or push from ECRs users also need to white-list account IDs of the private ECRs in the VpcOnlyTrustedAccounts setting.\nIn comparison with the other two types of workload, the network traffic for studio notebooks are the easiest to control because they are all configured at the SageMaker AI domain level. Once users with user profiles under a domain creates a notebook, the domain or user profile determines the network mode, and subnet values and security groups. Users themselves cannot change these settings.\nInference Endpoint The machine learning realm has a few established programming frameworks to host a model file behind an inference endpoint. For example, TensorFlow, PyTorch, Scikit-learn, and even Fast API. Amazon SageMaker AI supports many such frameworks and makes it straightforward. Managing the frameworks requires complex dependency management, a typical use case of containers. Apart from choosing a proper container image, user also selects instance types. These machine learning special purpose instances are pricier than their commodity counterpart. In low-traffic workflows, users may provision inference endpoint on-demand and use it in a controlled time-window, or just use serverless inference endpoint if the model supports it. When creating inference endpoint, the CreateModel API is used. Under the VpcConfig attribute, two parameters are at play: Network Isolation and VPC-Subnet configuration. I summarize them as below based on the documentation:\nNetwork IsolationVPC-Subnet ConfigurationDescriptionDisabledNot specifiedSageMaker AI containers are able to access external service and resources on the public Internet; but not able to access resources inside your VPC SpecifiedSageMaker AI containers communicate with resources inside your VPC through an ENI (Elastic Network Interface). Users are responsible for managing network access to your VPC and Internet.EnabledNot specifiedSageMaker AI container cannot communicate with resources inside your VPC or on the public InternetSpecifiedThe download and upload operations are routed through your VPC, but the inference (and training) containers themselves continue to be isolated from the network, and do not have access to any resource within your VPC or on the internet. The Network Isolation option governs the container connectivity option. If we do not expect the inference activity to make outgoing network calls (except for downloading artifact and packages), then we should enable network isolation. On the other hand, if the inference container needs resources on VPC or on the Internet, disable network isolation. Either way, we specify the VPC so that we manage the routing through VPC. In a compliant networking setup where Internet access must be centralized, the VPC-Subnet configuration must always be configured. The network isolation value depends on the nature of inference workload. However, what seems to be missing in the Studio UI is the activity to enforce that VPC-Subnet configuration is always specified.\nAs a workaround, we can use the SageMaker Domain\u0026#8217;s IAM role to contain such attempt at API level. Below is an example of deny policy:\n{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Action\u0026#34;: [ \u0026#34;sagemaker:CreateModel\u0026#34; ], \u0026#34;Condition\u0026#34;: { \u0026#34;BoolIfExists\u0026#34;: { \u0026#34;sagemaker:VpcSubnets\u0026#34;: \u0026#34;false\u0026#34; } }, \u0026#34;Effect\u0026#34;: \u0026#34;Deny\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34;, \u0026#34;Sid\u0026#34;: \u0026#34;DenyModelcreationIfNotOnVPC\u0026#34; }, { \u0026#34;Action\u0026#34;: [ \u0026#34;sagemaker:CreateModel\u0026#34; ], \u0026#34;Condition\u0026#34;: { \u0026#34;ForAnyValue:StringNotEquals\u0026#34;: { \u0026#34;sagemaker:VpcSubnets\u0026#34;: [ \u0026#34;subnet-999999999999999\u0026#34;, \u0026#34;subnet-111111111111111\u0026#34; ] } }, \u0026#34;Effect\u0026#34;: \u0026#34;Deny\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34;, \u0026#34;Sid\u0026#34;: \u0026#34;DenyModelcreationIfAnySpecifiedSubnetIsNotIntended\u0026#34; } ] } We can use this policy in conjunction with the AmazonSageMakerFullAccess managed policy. The request to create a mode gets denied, either if VpcSubnets are not specified, or they are but not from the preset list of subnet IDs. Once the user selects subnets, corresponding ENIs will get created in the subnets too and user needs to specify security groups for the ENIs.\nEnforcing in IAM policy requires that the user who creates endpoint either on SageMaker Studio UI or SageMaker SDK must know the exact subnet IDs as well as appropriate security groups. This requires access to the VPC and can turn into an operation pain point if the users are not well versed with networking. Ideally subnet configuration should also be enforceable at the domain level. Pipeline Jobs Most machine learning jobs do not need to function behind an endpoint (i.e. server-side), for example, training, labeling job, model optimization, hyper parameter tuning, etc. In operation, we often use a pipeline to orchestrate these short-lived, non-interactive, headless jobs. Therefore, I simply refer to them as pipeline jobs. They sometimes rely on special purpose instance types. In most cases, they need access to either the Internet or other resources available via customer VPC, a connectivity pattern similar to that of interface endpoints.\nTo create such resources, SageMaker AI domain user either operate on SageMaker Studio, or program with SageMaker SDK. To make it easy to specify network isolation and subnet configurations, the SDK even has a class for NetworkConfig that can pass to many types of processors (steps).\nfrom sagemaker.network import NetworkConfig security_group_ids = [\u0026#39;sg-#\u0026#39;] subnets = [\u0026#39;subnet-#\u0026#39;] enable_network_isolation = True network_config = NetworkConfig( security_group_ids=security_group_ids, subnets=subnets, enable_network_isolation=enable_network_isolation ) script_processor = ScriptProcessor( image_uri=\u0026#39;my-script-processor-image\u0026#39;, command=[\u0026#39;python3\u0026#39;, \u0026#39;script.py\u0026#39;], instance_type=\u0026#39;ml.m5.large\u0026#39;, role=role, network_config=network_config ) The network_config parameter exists as an argument in the creation method of many other resources via SDK. However, the SageMaker UI domain does not have a mechanism to enforce it one way or another. Leaving this option open to users is not what every organization wants either. We could exercise control as much as we can with condition keys such as sagemaker:VpcSubnets in the deny policy for SageMaker IAM role as the example above shows. The Service Authorization Reference document lists out in which SageMaker SDK calls the sagemaker:VpcSubnets condition key (or equivalent) exists. A proactive IAM policy to safeguard all the applicable SDK calls would be helpful as a workaround to the missing enforceability at domain level for SageMaker AI.\nNote that on the SageMaker Studio\u0026#8217;s Pipeline tool there is a Network configuration seemingly for the pipeline. However, the CreatePipeline SDK call does not have an argument about network configuration. The PipelineDefinition argument requires a JSON format input to define the pipeline configuration and the network configuration is defined per step in the definition. Another perspective to look at this issue is how we can give a pipeline step (or in general any job runtime) the flexibility to connect to Internet, and in the mean time remain in control of its network connectivity. It depends on the intended security posture but we mainly look at these two questions on the requirement:\ncan the job runtime access the VPC? can the job runtime access the Internet on its own path? Depending on the answer, we can configure a Job in three ways:\nConfigurationDescriptionaNeither #1 or #2 are allowedMost secure but might be overly restrictive because the job runtime may need to download artifacts. This requires enabling Network Isolation and specify VPC-Subnet configuration.bEither #1 or #2 is allowed, but not both at the same time.Exclusively allowing #1 is more secure because the VPC can manage access to Internet for the job runtime. Exclusively allowing #2 breaks the central Internet access pattern and should not be allowed if central Internet access is a compliance requirement.cBoth #1 and #2 are allowed at the same timeThis configuration should not be made possible due to exfiltration risk. This classification of network configuration for ML pipeline job, is quite similar to DevOps pipeline job agent (think of Azure DevOps agent or Terraform agents). The user may use service provider\u0026#8217;s agent which come with its own Internet access, or choose to self-host the agent to allow access to VPC but the VPC\u0026#8217;s owner is then responsible for managing Internet routing through the custom VPC. SageMaker AI makes c impossible, which is good. SageMaker administrator needs to evaluate the requirement between a and b and determine how to enforce it with IAM policy.\nSummary As an infrastructure security specialist, I investigated networking options in SageMaker AI. When any user is performing any task in SageMaker AI, I am concerned with two questions:\nHow does the instance behind the endpoint connects to the internet and to our VPCs; How do we enforce the connectivity pattern (enforceability); For the studio notebooks workload, the control is through SageMaker domain. For Inference endpoint and pipeline jobs, the control is reactive. We use IAM policies and role to deny attempts to create resources with bad network configuration. The limitation is that the SageMaker users have to know what network configuration options are available.\nAs a result, the infrastructure security specialist must clearly define the required network configuration, and communicate it out to SageMaker AI users.\nPrevious PostThe Leanest Web and Email Hosting Next PostTraining and Inference in SageMaker AI ","date":"2025-04-01T00:53:00-04:00","image":"/wp-content/uploads/2025/04/feature-sagemaker-networking.webp","permalink":"/2025/04/sagemaker-ai-and-networking/","title":"SageMaker AI and Networking"},{"content":"This site has been quiet for a while. During this time, I migrated the hosting platform again, and refactored email solution. This post, is another note about how I finally came to the most cost-effective web and email solution for a small business, with a solid security posture, and at nearly no cost.\nFor background, I have had this website for more than five years now, and revamped it a couple times. I have been with Amazon Lightsail at an already low cost. I didn\u0026#8217;t have a lot of emails. So I have been relying on the email forwarding feature from my old domain registrar. There have been a few challenges. The content has grown a lot. The speed gradually slows down. WordPress needs reinstall. The outgoing emails get flagged as unverified.\nWordPress Hosting Worried about too many plugins, I debated about changing content management platform. Unfortunately, it\u0026#8217;s not easy. There are some alternatives but none has all the features from all the plugins that I\u0026#8217;ve been using. Even if they do, it would be too much work: I\u0026#8217;d have to migrate the code snippets, highlights, etc, find similar themes and verify the new theme and new plugins work together. Despite of the criticism, WordPress today is still a predominant content management platform. So I stick to it.\nStaying with WordPress isn\u0026#8217;t maintenance-free. When PHP version has major update once in a while, I still had to re-install and migrate to a new server. Since I\u0026#8217;ll exercise migration anyways, why not also shop around for a new virtual private server. Surprisingly I learned that Google Cloud Platform has an always-free tier including a decent size virtual machine.\nAmazon Lightsail 7USD/moGCP Free tierClassmicroe2-microCPU2 vCPUs2 vCPUsMemory1 GB1 GiBDisk40GB SSD30 GB (standard persistent disk HDD)Data Transfer1TB1GB egress The vCPU in different platforms do not represent exactly the same compute capacity, but should be generally comparable. The egress data transfer usage was below 1GB on Lightsail so the new service is sufficient. By going with GCP\u0026#8217;s free tier, the only downgrade is the disk speed.\nThe GCP platform provides Bitnami package image and it\u0026#8217;s fairly simple to spin up a new VM. The setup is as simple as Amazon Lightsail and I don\u0026#8217;t have to deal much with networking. It doesn\u0026#8217;t work out to be completely free but the net cost is negligible. I also picked the Nginx-based WordPress image because I had wasted a lot of time dabbling with Apache which I never use elsewhere.\nWordPress Migration With the new server up in a few minutes the next step is migration. The All-in-One WP Migration and Backup plugin (by ServMask) is the Swiss army knife. There are some pitfalls though.\nFirst, the plugin has a size limitation in the free version. Luckily I can bypass it with changes to the PHP and Nginx configurations. For PHP configuration, change the line in /opt/bitnami/php/etc/php.ini, to upload_max_filesize = 512M. For the other change on Nginx as the reverse proxy, in /opt/bitnami/nginx/conf/nginx.conf, under http configuration, add:\nclient_max_body_size 512M; The second pitfall is trickier. I have to temporarily host the new site, either by an public IP, or by a different DNS name (such as new.digihunch.com), so I can log in and verify the functions. Once I log on to the new site, the new DNS name or IP will start to be written to the database. I\u0026#8217;m not sure which exact service does that. But once I put the new site on the production domain name (i.e. www.digihunch.com), the database still contains many entries referencing the temporary DNS name. This causes many insidious broken links. There is a plugin for this kind of issue: Better Search Replace by WP Engine. The plugin goes through all tables in the database and allows you to find and replace strings. It also supports dry-run mode and I managed to fix tens of thousands of bad references. In addition, I managed to change username using a plugin called Easy Username Updater.\nAnother important activity after migration, is to ensure the MFA by WordFence plugin continues to work, and re-scan the entire file system.\nCloudFlare Security With less egress data and an HDD, the VPS in GCP is slower. I need to speed it up with CDN. It is hard to resist the freebies by CloudFlare. I\u0026#8217;ve already been using it for security. Now it\u0026#8217;s time to dig deeper.\nCloudFlare greatly simplifies TLS certificates. It manages your certificate automatically. On the other side, it issues an origin certificate (and key, with long expiry) for communication with origin server. In full encryption mode, all I need to do is deploy origin certificate to Nginx. There is no manual renewal effort.\nNote this setup requires CloudFlare to manage DNS so I just transferred my domain to CloudFlare. Therefore I consolidate domain registrar, certificate vendor and CDN to one vendor. CloudFlare can manage DNS record in proxy mode, to hide the public IP of the origin server from clients. I can also control the security group of the origin server to only open port 443 to CloudFlare IPs.\nAnother annoying but necessary configuration is the redirect of zone apex and http requests. Specifically most websites needs the following redirects:\nIf request from client is http://www.digihunch.com, enforce https; If request is https://digihunch.com, redirect to https://www.digihunch.com; If request is http://digihunch.com, enforce https, then redirect following rule #2. I used to manage the these redirects in the reverse proxy (Apache). Now that I changed to Nginx, I offload these settings to CloudFlare, instead of configuring another reverse proxy again. For #1, I simply turned on \u0026#8220;Always use HTTPS\u0026#8221; and \u0026#8220;Automatic HTTPS Rewrites\u0026#8221; under edge certificates. For #2, I need a CNAME for zone apex to alias to www, along with a redirect rule. As a result, there is no need to ever open port 80 on the VPS, just to let the reverse proxy redirect URL.\nCloudFlare Cache Rules As to caching, I had to turn on a couple Cache rules to enable the caching. Cache hit was at about 50% and there are rooms to go higher if I enable reserve cache. Once cache is on, there are other considerations, such as expiring the cache, and by passing the cache during server deployment. CloudFlare allows you to purge cache by rules. There is also a Development Mode that temporarily allows you to bypass all cache for testing. With cache enabled, and the WordPress plugin activated, the page load time for recent posts appear to be faster than before.\nOne challenge is that the admin bar of WordPress went into the CDN and is served to visitors. To get rid of this behaviour I have two catch rules:\nA cache-everything rule at order 1, to make everything eligible for cache; A bypass rule at order 2 to bypass cache conditionally. The condition expression reads: (http.cookie wildcard \u0026#34;wp-.*\u0026#34;) or (http.cookie wildcard \u0026#34;wordpress_logged_in_*\u0026#34;) or (http.cookie wildcard \u0026#34;wordpress.*\u0026#34;) To make use of the cache space. It is important to have small image sizes. Last year I added many AI generated feature images, most of which are 2MB in PNG format. I have to convert many PNG images into webp format, which only takes a fraction of the space and is widely accepted by most browsers today.\nSometimes there are other annoyances when it comes to speeding up page loading. For example, I noticed that with integration between Google Tag Manager and Microsoft Clarity, the home page issues a call to https://www.clarity.ms/tag/tag-id-xyz. This call blocks the rendering of a big chunk of home page by a minute! I didn\u0026#8217;t notice it because my Brave browser\u0026#8217;s ad-block removes that call. Nonetheless, the issue has been there for any visitor without built-in ad-block in their browser. The lessons learnt is that you always test with standard browsers, or at least in Brave browser with shields down.\nSMTP service I have been using email forwarding for incoming emails, and Gmail\u0026#8217;s feature to send email from a different address or alias, for outgoing emails. This feature works but oftentimes the email gets marked as unverified on the recipient side. That gives it a good chance to appear as spam.\nI need similar email forwarding mechanism and sending service. The free tier of CloudFlare goes a long way. Receiving is simply about creating Email routing rules, with a few MX and TXT records on DNS created and managed by CloudFlare. These records ensure CloudFlare routing rules captures all emails coming to the domain.\nFor outgoing emails, I have to get rid of the alias-based mechanism by Gmail. I need a proper SMTP server but CloudFlare doesn\u0026#8217;t offer one. Resend seems like a popular choice with a good amount of free tier usage. However I\u0026#8217;m hesitant to introduce another single-purpose platform so I just resort to Amazon SES for a full-feature, low-cost SMTP service. I do have to request production access, stating the server is for transactional email. This allows the SMTP service to arbitrary recipient address.\nThe other benefit of having my own SMTP server is to enable email integration in WordPress, which enables other important features such as web form, WordFense security alert and admin password reset by email.\nSending Reputation To keep outing email from being marked as spam, it is important to understand what mechanisms are at play (SPF, DKIM, DMARC) to maintain sending reputation. At minimum, we configure SPF, DKIM and DMARC. SPF (Sender Policy Framework) identifies which mail servers are allowed to send mail on behalf of your custom MAIL FROM domain through a DNS TXT record that is used by DNS. The receiving mail server checks the SPF record of the sender\u0026#8217;s domain to see if the email came from an authorized server. For example, when I use the SMTP server by Amazon SES, and customize the MAIL FROM field to mail.digihunch.com, I need an MX and a TXT record for mail.digihunch.com to tell receivers that it authorized amazonses.com to send email on its behalf.\nDKIM (DomainKeys Identified Mail) is an email authentication method used to verify that an email message was sent by an authorized mail server and that the message content hasn’t been altered in transit. When an email is sent, the sending mail server generates a unique digital signature (based on the content of the email) using a private key. This signature is added to the email header. DNS Lookup: The receiving mail server looks up the sender\u0026#8217;s domain in DNS to find the corresponding public key for DKIM authentication. Amazon SES configuration requires three TXT records to store the sender\u0026#8217;s key information.\nWith SPF and DKIM configurations, the Amazon SES setup complies with DMRAC authentication protocol, making it less likely to be flagged as spam. The email comes off as mailed by amazonses.com and signed by digihunch.com.\nIn addition, there is Brand Indicators for Message Identification (BIMI) protocols that enables logo for your email on the recipient\u0026#8217;s inbox. I managed to set up BIMI with SES, without Verified Mark Certificate (VMC), a protocol that requires evidence of ownership of the logo.\nAlthough SPF, DKIM and DMARC cannot guarantee the email is not marked as junk, this is the best thing you can do. There are other techniques with diminishing return on effort.\nSummary This post summarizes my journey to land on the leanest web and email hosting solutions for small business. After five years of improvement, the tech stack that brings me low cost, solid security posture and lightening-fast speed, consists of: virtual machines from Google Cloud Platform (or Lightsail from AWS), web security and CDN by CloudFlare with Email supported by Gmail and Amazon SES (or Resend).\nInitially with GCP, I tried to stick to the free tier but I did notice the standard persistent disk struggling, especially when I run system scan from WordFence. To overcome that I upgraded the disk to balanced disk (SSD backed) out of the free tier. There might be even cheaper alternatives but the marginal value isn\u0026#8217;t worth the effort. Also, I configured the caching in CloudFlare for read, adjusted the home page, and schedule IO-intensive activities such as bi-weekly snapshot to quiet hours. The combination gave the site a 100 score in the CloudFlare speed testing.\nPrevious PostFirewall Deployment Patterns Next PostSageMaker AI and Networking ","date":"2025-02-27T22:14:11-05:00","image":"/wp-content/uploads/2025/04/feature-web-email-host.webp","permalink":"/2025/02/the-most-cost-effective-web-and-email-hosting/","title":"The Leanest Web and Email Hosting"},{"content":"The Hub-and-Spoke topology is the most common topic in the discussion for building cloud infrastructure design. This topology appeared in both AWS and Azure design papers and had been around as a very important option in physical networking design. The AWS whitepaper Building a Scalable and Secure Multi-VPC AWS Network Infrastructure has thorough discussion on the topology. This topology often feature a Transit Gateway as the hub. In addition to workload VPCs, the network topology often includes some special-purpose VPCs, such as interface endpoints VPC, or shared tooling VPCs. One of the special-purpose VPC is the inspection VPC. It is a key design area to suit the need of inspection and traffic management for the business and the design may vary a lot depending on the available inspection tools such as a Firewall appliance. Inspection Requirements The most common situation with an enterprise is connecting with on-prem networking. Options including Direct Connect, site-to-site IPsec or SD-WAN overlay. The business decides whether and at what level they would like to inspect the traffic between on-prem and their VPCs. Here is an example.\nConnectivityInspection RequirementBetween Workload VPCs (East-West)No inspectionBetween a workload VPC and a special-purpose VPCNormal InspectionIngress Traffic from Internet to Workload VPCDeep Packet InspectionEgress Traffic from Workload VPC to InternetDeep Packet InspectionBetween Workload VPC and on-prem networking over Direct ConnectNormal Inspection\u0026#8230;\u0026#8230; With a normal inspection, the firewall appliance only checks the information in the packet\u0026#8217;s header, such as the source and destination IP addresses, port number, etc. With deep packet inspection, the appliance examins a larger range of metadata as well as the data in each packet. DPI provides a more effective mechanism to perform network packet filtering and find otherwise hidden threats. It is however an expensive operations from a performance standpoint. Ultimately the business makes the call but it is important to identify ALL connectivity scenarios in this phase and explicitly document the decision and rationales. They can choose from an NGFW product or the Network Firewall service from AWS, depending on capability required.\nInspection Architecture At minimum, inspection is required for ingress and egress traffic to and from workload VPC. The design must account for both routing and inspection. Many would use the same VPC for ingress/egress traffic and for inspection. It is also possible to separate these two purposes into two different dedicated VPCs: an inspection VPC that hosts firewall services or appliances, and an ingress/egress VPC that directs traffic from and to the Internet but we must route the traffic to the inspection appliance. If all traffic to be inspected has to be routed through the Transit Gateway both ways, the cost would be high. In 2020 AWS introduced Gateway Load Balancer (GWLB) to address this use case. The recommended pattern using GWLB allows you to place firewall appliance and a GWLB in one VPC, and place the GWLB endpoint (GWLBE) in a different VPC. The connectivity between GWLBE and GWLB is backed by HyperPlane, a technology that also enables other endpoint service such as PrivateLink. The connectivity between GWLB and the appliance take place with Geneve encapsulation. This pattern places any appliance behind an endpoint, so long as the appliance supports Geneve protocol. The GWLB technology enables a number of inspection patterns based on distributed ingress paths, as summarized in this document. Distributed ingress/egress means each workload VPC can have their own Internet Gateway and NAT gateways. They must configure their route table so as to send the traffic via GWLBEs to inspection appliances. In general, I recommend this pattern over the centralized ingress/egress patterns where only the inspection VPC can take ingress traffic from Internet Gateway. The Network architectures for ingress traffic inspection presentation from 2021 ReInvent covered this topic as well, especially about the scaling benefit of distributed ingress.\nFirewall deployment patterns The firewall deployment pattern available differ between vendors and the requirements. Since the GWLB pattern places appliances behind the GWLB, the appliances rely on Geneve traffic that GWLB forward over. Some vendors may argue that this pattern keeps the NGFW product from performing other tasks that do not support Geneve traffic. One example is Network Address Translation. The native NAT gateway services is very expensive (consider fck-nat as an alternative for NAT). Many clients want to use the NAT feature of the NGFW product. The architecture therefore has to be adjusted in favour of centralized egress. Review this post about one-arm mode and two-arm mode.\nIf we have to go with central ingress/egress anyways, there are still numerous options. Take FortiGate for example, while the GWLB pattern of deployment is supported, other available options include:\nTraditional pattern with multiple interfaces across different subnets in the inspection VPC (L3 mode) Integration with Transit Gateway using Transit Gateway Connect Attachment Integration with Transit Gateway using Transit Gateway VPN Attachment I regard the first option as traditional because it does not directly integrate with Transit Gateway and it is very similar to how we deploy them in a physical networking environment. Fortigate refers to it as L3 (NAT/route) mode. In this mode the Firewall appliance can also influence network routing. The second and the third options are similar except for different types of Transit Gateway attachments are used. The reason to directly integrate with Transit Gateway is so that the Transit Gateway can route the traffic for inspection therefore no need for a Gateway Load Balancer, and thus no dependency on the firewall features supporting Geneve. The second option builds a GRE (Generic Routing Encapsulation) tunnel over a Transit Gateway Connect attachment as the transport tunnel, and uses BGP to exchange routes between the Transit Gateway and the appliance. It treats the firewall instances as SD-WAN appliance and has performance benefit. The third option uses VPN attachment with the main benefit of encryption if it is part of compliance requirement.\nRules The rule configuration for Firewall is critical to the operation of the entire multi-VPC network configuration. Unfortunately, there is no standard with the rule syntax across majore NGFW vendors, leading to challenges for customers to swap vendors. Most flavours of rules have common elements such as\nAction (Pass, drop, or alert) Source and Destination Protocol and Port A very common open-source firewall rule syntax is the Suricata-compatible format. One important adopter is the AWS Network Firewall, which supports both stateful and stateless rule groups. With stateful rule group, there are two options for how the Suricata engine evaluates rules. With \u0026#8220;Action Order\u0026#8221; option, Suricata engine evaluates the rules in the order of: pass, drop, reject and alert. You can use the priority attribute to influence evaluation; With strict order, the rules are evaluated in the order of the rule definition; It is important to be aware of the rule evaluation order since it impacts the firewall behaviour deeply.\nSummary In the networking infrastructure design, ingress and egress routing are the most critical one-way door decision. This decision must account for both routing and inspection. While there are many options, we usually start with capturing the key requirements. In this post we reviewed how to approach the requirement, a key technology Gateway Load Balancer and some firewall deployment patterns with FortiGate as an example. The approach is similar for other NGFW vendors, such as Palo Alto, Check Point or Cisco Secure Firewall. Previous PostCloud Certifications for Learning? Next PostThe Leanest Web and Email Hosting ","date":"2024-11-16T16:24:15-04:00","image":"/wp-content/uploads/2025/04/feature-fw-deploy.webp","permalink":"/2024/11/firewall-deployment-patterns/","title":"Firewall Deployment Patterns"},{"content":"Certification count is a measurement of capability of a consulting practice. At individual level it sometimes serves to navigate the learning journey, especially for engineers aspiring to advance to an adjacent technical area, for example, a network engineer moving to cloud engineer, or a data engineer transitioning to machine learning engineer. Personally I don\u0026#8217;t find certifications the most efficient way to learn. However, it is the most measurable way. The market has a plethora of choices for certifications but the quality is not very consistent. Through years of studying for different certificates, I have learned to treat certificates as necessary evil and be extremely cautious before investing time to it. After I sat another exam yesterday, I decided to write down my views of certifications and alternative ways of studying a topic in cloud technologies.\nDosage of Marketing A few years ago, I sat the Azure exams on AI fundamentals (AI-900) and AI engineer associate (AI-102). Out of the two I found the fundamentals one better, despite of lower level. That is because AI-900 focuses on the core concepts that are broadly useful. In contrast, the AI-102 exam, at least at that time, felt like a marketing exam all about what managed AI services Azure has for what kinds of problems. I’m not a fan of marketing exams but admittedly all these exams by cloud service providers contains some dose of marketing. You just can\u0026#8217;t completely avoid marketing from these exams. It’s just a matter of how much. My first principle: be wary of the marketing scheme behind the exam.\u0026nbsp;\nWhy is this important? We choose an exam because we want to dedicate precious time to learn. Unfortunately, the time we, aspiring IT engineers starting to study a new topic, is the time we are the most vulnerable to marketing schemes. Not only are we eagerly open to devouring the course contents, we’re also laser-focused and repeating a lot. This tunnel vision while cramming the exam allows biases to sneak in and reinforce in favour of the product names in the exam.\nIf you’re about to build a container platform and your first flash of thought is Elastic Container Service, you should congratulate Amazon on the marketing win. If your decision process relies too much on these mental shortcuts that the exam study has covertly built, you are developing a tendency of not being analytical to engineering problems. This makes you a worse engineer than you could otherwise have been.\u0026nbsp;\nA wise learner has be cautious about the dosage of marketing in these exams. One indicator of high dose of marketing, is the portion of questions focusing managed and unique services in the particular cloud service provider. The certifications by non-profit organizations such as CompTIA and Linux Foundations tend to have low dose of marketing in the exams.\nStability of\u0026nbsp;Topics The time studying an exam is an investment. We want the certification to remain stable over the course of several years. New certification exams should only launch under rigorous review of the necessity, maintainability and the direction should not change drastically. Unfortunately, this is not always the case. Let me pick on AWS specialty exam this time. If you have been in the certification game for long, you might remember the following specialty certifications:\nBig Data specialty — launched in Oct 2016, and deprecated in Apr 2020 in favour of Data Analytics specialty and Database specialty Data Analytics specialty — launched in Apr 2020, retired in Apr 2024 Database specialty — launched in Mar 2020, retired in Apr 2024 SAP on AWS specialty — launched in Apr 2022, retired in Apr 2024 Alexa Skill Builder — launched in May 2019, retired in Mar 2021 These changes are a main reason I am very hesitant to go for specialty certifications with AWS. Some even had a lifespan of less than two years. In my opinion, learning is worth it but spending time on those short-lived certificates is a completely waste of time. In Lindy effect, the longer a non-perishable item has been around, the longer it\u0026#8217;s likely to persist into the future. Technologies such as TCP/IP, Unix, Object Oriented Programming, have been around for decades and yet still profoundly impact a production system today. Only lasting knowledge is worth more time. As of spring 2024, AWS introduced two certificate level exams: Data Engineer Associate and Machine Learning Engineer Associate. So it seems that some specialty certifications are being consolidated into the associate-level. All previous associate-level exams had been around for more than a decade. Given that record of stability, I decided to venture the AWS Certified Machine Learning Engineer Associate certification.\nBreadth and\u0026nbsp;Depth The next important factor to consider is the breadth and depth of certification exams.\u0026nbsp;\nSome certifications are targeting different audiences. For example, I personally regard the AWS Certified Cloud Practitioner as targeting non-technical folks and would not carry much weight for an experienced cloud engineer. On the other hand, I chose AWS Certified Machine Learning Engineer Associate over the Machine Learning Specialty because the latter targets data scientists and is a lot more demanding on theories. Even if I grind it out there wouldn’t be opportunity in my day-to-day responsibility to digest and reinforce what I have jammed in my head.\u0026nbsp;\nI’m dubious on certifications for a single tool but there seems to be more and more of them these days. The only exception is if the tool plays such a crucial and fundamental role in the IT systems, and it is complex but there is no alternatives. For example, Red Hat as a classic distribution of Linux operating system, Kubernetes as a complex container orchestration service. However, certifications on single tools such as Prometheus, Istio, Cilium, and Terraform, are just too narrow to worth the effort. In my opinion these certifications provide very limited return on investment. We just need to learn these tools as we go with documentations, or go through the essential workshops.\nUpkeep Renewal requirement is another consideration, especially if the organization requires your active certification status. Having a streamlined renewal process is a big plus because once certified our main duty is on the actual work. Kudos to the Azure certifications that only require a shortened free on-line exam for renewal. It is quite a hassle to have to retake the exam again. The validity of the certification is also subject to arbitrary changes. For example in 2024 Linux Foundation changed the validity of CKA exam from 3 years to 2 years.\nCertification inflation is real. There might have been a time when they measure one\u0026#8217;s competency in the technical field. But today very few of them can differentiate one\u0026#8217;s technical competency, thanks to Goodhart\u0026#8217;s law. In addition to the certification inflation, another threat to the value is the question leaking. The longer the exam has been administered, the more likely the questions are leaked all over the place. This gives an unfair advantage for users of question dumps. And because the scores are scaled results, this systematically penalize honesty, and ultimately impairs the value of the certification.\nMore ways of learning Last year I had a number of certifications all up for renewal, I gave up many of them that are time-consuming. They did help with learning but only up to certain extent. Unfortunately, so long as I work in consulting I still have to live with it. As a result, I learned to be very picky about which ones to pursue to get the most value of my time. At the end of the post, I must reiterate that I don\u0026#8217;t personally find certifications are best way to learn. In the preparation, I always tend to focus on acing the exam itself. Over the years, I\u0026#8217;ve also used a few other ways in my learning, and can summarize them as such:\ncertification exam: most measurable but least effective in my experience. online courses: highly dependent on the quality of the individual course. Low-quality ones are just workshops. The best ones walk you through a classic project to fill the knowledge gap. workshops: my preferred way going into a new topic. Vendor managed ones may over-emphasize on tools and under-emphasize on concepts. side projects: most effective but requires a real-life problem and can be time consuming quality white paper: an effective way of learning going into new topic in a systematic way read a book: as long as the book content is up to date, this is the most systematic approach blogging: effective but time-consuming talk: to someone using the technology hands-on, free of fluff I hope every cloud engineers can also choose certifications wisely and maximize their leanring.\nPrevious PostDebating between count and for_each in Terraform Next PostFirewall Deployment Patterns ","date":"2024-10-09T16:03:29-04:00","image":"/wp-content/uploads/2025/04/feature-cloud-learning.webp","permalink":"/2024/10/choosing-cloud-certifications-wisely/","title":"Cloud Certifications for Learning?"},{"content":"In Terraform, we often have to create an array of resources of the same type but similar attribute values. For code reusability, manageability and for DRY principle, it\u0026#8217;s better to use loop. Terraform HCL supports loop via the use of meta-argument. Currently, there are two options to drive a loop: count and for_each .\nProblem with count loop The book Terraform Up and Running (Chapter 5 Terraform Tips and Tricks) regards count as Terraform\u0026#8217;s oldest, simplest and most limited iteration construct. One of the big limitations is the shifting of index if the length of resource array changes. The point comes with a good example:\nvariable \u0026#34;user_names\u0026#34; { description = \u0026#34;Create IAM users with these names\u0026#34; type = list(string) default = [\u0026#34;neo\u0026#34;, \u0026#34;trinity\u0026#34;, \u0026#34;morpheus\u0026#34;] } # The Example from the book Terraform Up and Running resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { count = length(var.user_names) name = var.user_names[count.index] } As you execute Terraform apply, three IAM users will be created, with the plan looking like:\n# aws_iam_user.example[0] will be created + resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { + name = \u0026#34;neo\u0026#34; (...) } # aws_iam_user.example[1] will be created + resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { + name = \u0026#34;trinity\u0026#34; (...) } # aws_iam_user.example[2] will be created + resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { + name = \u0026#34;morpheus\u0026#34; (...) } Then if you remove \u0026#8220;trinity\u0026#8221; from the variable user_names, and run terraform plan, the plan would look like:\nTerraform will perform the following actions: # aws_iam_user.example[1] will be updated in-place ~ resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { id = \u0026#34;trinity\u0026#34; ~ name = \u0026#34;trinity\u0026#34; -\u0026gt; \u0026#34;morpheus\u0026#34; } # aws_iam_user.example[2] will be destroyed - resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { - id = \u0026#34;morpheus\u0026#34; -\u0026gt; null - name = \u0026#34;morpheus\u0026#34; -\u0026gt; null } Plan: 0 to add, 1 to change, 1 to destroy. In this plan, instead of deleting the second user, it renames the second user and deletes the third user. While the plan matches the code logic, it is often an unwanted result, considering the resource could be one that many other resources depends on, such as a subnet.\nThis is a good example of the problem with count. Terraform identifies each resource in the generated list of resource by position(index) . When the length changes, the index shifts. If you remove an item from the middle of the list, Terraform will delete every resource after the deleted item, then re-create all the resources that come after the deleted one. As a consequence, you may loose availability or even worse, lose data.\nEmbrace for_each loop If we modify the example above to use for_each, the code looks like:\nvariable \u0026#34;user_names\u0026#34; { description = \u0026#34;Create IAM users with these names\u0026#34; type = list(string) default = [\u0026#34;neo\u0026#34;, \u0026#34;trinity\u0026#34;, \u0026#34;morpheus\u0026#34;] } # The Example from the book Terraform Up and Running resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { for_each = toset(var.user_names) name = each.value } This results in the creation of three IAM users. If you remove the \u0026#8220;trinity\u0026#8221; user from the middle of the input collection and apply, the plan looks like this:\nTerraform will perform the following actions: # aws_iam_user.example[\u0026#34;trinity\u0026#34;] will be destroyed - resource \u0026#34;aws_iam_user\u0026#34; \u0026#34;example\u0026#34; { - arn = \u0026#34;arn:aws:iam::123456789012:user/trinity\u0026#34; -\u0026gt; null - name = \u0026#34;trinity\u0026#34; -\u0026gt; null } Plan: 0 to add, 0 to change, 1 to destroy. The plan suggests that Terraform will delete the very resource that was taken out from the middle of the input collection and no existing resources in the array are impacted.\nNote that in the code snippet above, we use function toset() to convert the input list to a set (ordered and de-duped list of string). This is because we can only loop over a set or map when creating an array of resource. If the array of resource being created have another attribute whose value needs to be individualized, we can loop over a map and store the individualized attribute values as key-value pairs.\nA few pages down, the book discusses an important limitation for both count and for_each. The length of the resource array that you are creating with count or for_each meta-argument must not be computed from other resources. Terraform must be able to compute count and for_each during the plan phase, before any resources are created or modified. The length of the resource array can be from hardcoded values, data sources, or even a list of other resources to create in the same file, so long as the length can be determined during the plan, instead of not being computed from other resource outputs.\nA real-life example with classic pattern The book then touches on another advantage of for_each: the ability to create multiple inline blocks within a resource. The guide from Hashicorp documentation also has a section on when to use for_each Instead of count, with a similar example. The section merely mentions when to use count in the opening sentence: If your instances are almost identical,\u0026nbsp;count\u0026nbsp;is appropriate.\nThat makes for_each sound like a no-brainer, after reading all the literatures about this topic. In my experience with a specific use case at the beginning, count feels more efficient. The example from the book is too simplistic. To better compare the two options, I need a realistic example. Let\u0026#8217;s consider this use case where, after creating a VPC, I need to create the followings:\none NAT gateway for each availability zone (each NAT Gateway maps to one subnet and one allocation ID) one public subnet for each availability zone one public IP allocation in each availability zone We can summarize the relationships between resources in the following diagram:\naws_subnetaws_subnetaws_eipaws_eipaws_nat_gatewayaws_nat_gatewaysubnet_idsubnet_idallocation_idallocation_idText is not SVG \u0026#8211; cannot display\nI deliberately pick this example because they are self-contained. So are all code snippets in this post. The example also demonstrate a classic relation between resources that we can find everywhere in infrastructure automation. Here\u0026#8217;s another example off the bat:\ncreate an array of aws_subnet, each has a subnet_id attribute; create an array of aws_route_table, each has a reout_table_id attribute; now, create an array of aws_route_table_association, each referencing one aws_subnet (by subnet_id) and one aws_route (by route_table_id); If we address the NAT gateway example, we\u0026#8217;re good with many other resources that shares the same relation pattern. In the next section, we\u0026#8217;ll first implement the NAT gateway example, using count loop.\nImplementation using count The use case exemplifies the pattern where we have multiple types of resources related to each other. We need a loop in each type of resources, resulting in multiple arrays of different resource types. Moreover, the elements in the array for aws_nat_gateway has 1-to-1 mappings with both the array for aws_subnet, and the array for aws_eip.\nWith count, I created Terraform code with everything in a single main.tf file for the convenience of illustration, like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 provider \u0026#34;aws\u0026#34; {} variable \u0026#34;vpc_cidr_block\u0026#34; { type = string default = \u0026#34;147.206.0.0/16\u0026#34; } variable \u0026#34;public_subnets_cidr_list\u0026#34; { type = list(any) default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;] #default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;, \u0026#34;147.206.8.0/22\u0026#34;] } data \u0026#34;aws_availability_zones\u0026#34; \u0026#34;this\u0026#34; {} resource \u0026#34;aws_vpc\u0026#34; \u0026#34;base_vpc\u0026#34; { cidr_block = var.vpc_cidr_block } resource \u0026#34;aws_internet_gateway\u0026#34; \u0026#34;internet_gw\u0026#34; { vpc_id = aws_vpc.base_vpc.id } resource \u0026#34;aws_subnet\u0026#34; \u0026#34;public_subnets\u0026#34; { count = length(var.public_subnets_cidr_list) vpc_id = aws_vpc.base_vpc.id cidr_block = var.public_subnets_cidr_list[count.index] map_public_ip_on_launch = true availability_zone = data.aws_availability_zones.this.names[count.index] } resource \u0026#34;aws_eip\u0026#34; \u0026#34;nat_eips\u0026#34; { count = length(var.public_subnets_cidr_list) } resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { count = length(var.public_subnets_cidr_list) subnet_id = aws_subnet.public_subnets[count.index].id allocation_id = aws_eip.nat_eips[count.index].id depends_on = [aws_internet_gateway.internet_gw] } The intent is to create subnet, public IP and NAT gateway for two availability zones. I also want to add one more AZ in the future and have the code to handle the addition gracefully. To add the new AZ, I uncomment line 10 and comment out line 9. The plan after this code change looks like this:\nTerraform will perform the following actions: # aws_eip.nat_eips[2] will be created + resource \u0026#34;aws_eip\u0026#34; \u0026#34;nat_eips\u0026#34; { (...) } # aws_subnet.public_subnets[2] will be created + resource \u0026#34;aws_subnet\u0026#34; \u0026#34;public_subnets\u0026#34; { + availability_zone = \u0026#34;us-east-1c\u0026#34; + cidr_block = \u0026#34;147.206.8.0/22\u0026#34; + id = (known after apply) (...) } # aws_nat_gateway.nat_gws[2] will be created + resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { + allocation_id = (known after apply) + subnet_id = (known after apply) (...) } Plan: 3 to add, 0 to change, 0 to destroy. The plan creates a set of resources required for the new availability zone without touching any existing resource, which is expected.\nWhy do I only focus on the use case of adding a new subnet in new AZ, and not deleting or modifying CIDR on an existing subnet? That\u0026#8217;s because we rarely do that with production. We rarely remove the use of an availability zone. Nor do we modify the CIDRs on an existing subnet. In fact, AWS SDK does not even have an API to change CIDRs on a subnet or a VPC. In our infrastructure operation, we make such decisions upfront so they are immutable once provisioned. We simply don\u0026#8217;t need to consider all the possible CRUD actions on a resource.\nSo, the count loop does just the job. Now, what about for_each?\nImplementation with for_each: first attempt Since for_each takes a set or map, I have to make some adjustment. My first attempt looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 provider \u0026#34;aws\u0026#34; {} variable \u0026#34;vpc_cidr_block\u0026#34; { type = string default = \u0026#34;147.206.0.0/16\u0026#34; } variable \u0026#34;public_subnets_cidr_list\u0026#34; { type = list(any) default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;] # 2 AZ #default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;, \u0026#34;147.206.8.0/22\u0026#34;] # 3 AZ } data \u0026#34;aws_availability_zones\u0026#34; \u0026#34;this\u0026#34; {} resource \u0026#34;aws_vpc\u0026#34; \u0026#34;base_vpc\u0026#34; { cidr_block = var.vpc_cidr_block } resource \u0026#34;aws_internet_gateway\u0026#34; \u0026#34;internet_gw\u0026#34; { vpc_id = aws_vpc.base_vpc.id } locals { subnet_config = [ for i in range(length(var.public_subnets_cidr_list)) : { cidr = var.public_subnets_cidr_list[i] az = data.aws_availability_zones.this.names[i] } ] } resource \u0026#34;aws_subnet\u0026#34; \u0026#34;public_subnets\u0026#34; { for_each = { for idx, rec in local.subnet_config : idx =\u0026gt; rec } vpc_id = aws_vpc.base_vpc.id cidr_block = each.value.cidr map_public_ip_on_launch = true availability_zone = each.value.az tags = { Name = \u0026#34;PUBLIC-SUBNET\u0026#34; } } resource \u0026#34;aws_eip\u0026#34; \u0026#34;nat_eips\u0026#34; { for_each = toset(var.public_subnets_cidr_list) tags = { Name = \u0026#34;NATEIP\u0026#34; } } data \u0026#34;aws_subnets\u0026#34; \u0026#34;public_subnets\u0026#34; { filter { name = \u0026#34;tag:Name\u0026#34; values = [\u0026#34;PUBLIC-SUBNET\u0026#34;] } depends_on = [aws_subnet.public_subnets] } data \u0026#34;aws_eips\u0026#34; \u0026#34;nat_eips\u0026#34; { filter { name = \u0026#34;tag:Name\u0026#34; values = [\u0026#34;NATEIP\u0026#34;] } depends_on = [aws_eip.nat_eips] } locals { nat_gw_config = [ for i in range(length(var.public_subnets_cidr_list)) : { subnet_id = data.aws_subnets.public_subnets.ids[i] alloc_id = data.aws_eips.nat_eips.allocation_ids[i] } ] } resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { for_each = { for idx, rec in local.nat_gw_config : idx =\u0026gt; rec } subnet_id = each.value.subnet_id allocation_id = each.value.alloc_id depends_on = [aws_internet_gateway.internet_gw] } Note that I have to create a couple of data resources (nat_eips and public_subnets) and local variables (subnet_config and nat_gw_config) in order build the required map data structures and feed them to the for_each parameters.\nAfter Terraform apply, let\u0026#8217;s edit public_subnets_cidr_list with the additional subnet CIDR for the 3rd AZ. The plan looks like this:\nTerraform will perform the following actions: # data.aws_eips.nat_eips will be read during apply # (depends on a resource or a module with changes pending) \u0026lt;= data \u0026#34;aws_eips\u0026#34; \u0026#34;nat_eips\u0026#34; { + allocation_ids = (known after apply) (...) } # data.aws_subnets.public_subnets will be read during apply # (depends on a resource or a module with changes pending) \u0026lt;= data \u0026#34;aws_subnets\u0026#34; \u0026#34;public_subnets\u0026#34; { + id = (known after apply) + ids = (known after apply) (...) } # aws_eip.nat_eips[\u0026#34;147.206.8.0/22\u0026#34;] will be created + resource \u0026#34;aws_eip\u0026#34; \u0026#34;nat_eips\u0026#34; { + allocation_id = (known after apply) (...) } # aws_nat_gateway.nat_gws[\u0026#34;0\u0026#34;] must be replaced -/+ resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { ~ allocation_id = \u0026#34;eipalloc-0ffe32519d20b9b7f\u0026#34; # forces replacement -\u0026gt; (known after apply) # forces replacement ~ subnet_id = \u0026#34;subnet-0b4b202056c75bb0a\u0026#34; # forces replacement -\u0026gt; (known after apply) # forces replacement (...) # (1 unchanged attribute hidden) } # aws_nat_gateway.nat_gws[\u0026#34;1\u0026#34;] must be replaced -/+ resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { ~ allocation_id = \u0026#34;eipalloc-0ff1d32fb271b5cf4\u0026#34; # forces replacement -\u0026gt; (known after apply) # forces replacement ~ subnet_id = \u0026#34;subnet-09b6486fbe44e9795\u0026#34; # forces replacement -\u0026gt; (known after apply) # forces replacement (...) # (1 unchanged attribute hidden) } # aws_nat_gateway.nat_gws[\u0026#34;2\u0026#34;] will be created + resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { + allocation_id = (known after apply) + subnet_id = (known after apply) (...) } # aws_subnet.public_subnets[\u0026#34;2\u0026#34;] will be created + resource \u0026#34;aws_subnet\u0026#34; \u0026#34;public_subnets\u0026#34; { + availability_zone = \u0026#34;us-east-1c\u0026#34; + cidr_block = \u0026#34;147.206.8.0/22\u0026#34; (...) } Plan: 5 to add, 0 to change, 2 to destroy. Wait a second, I expect the template to create a new subnet, a new elastic IP and a new NAT gateway in that new AZ. But why does it plan to delete the two existing NAT gateways and recreate two? This doesn\u0026#8217;t make for_each an appealing option at all.\nApart from the interruptive plan, there are also other problems. First, since the aws_subnet resources requires cidr_block and availability_zone values, I have to build a map (subnet_config) for its resource array to consume. Similarly, I have to build a second map (nat_gw_config) to create resource array for aws_nat_gateway, which requires subnet_id and allocation_id. This map takes more work to build. Because of the 1-to-1 relationship between subnet_id and alloc_id, I have to fetch the values from two data sources (line 47-61), use a common index (line 63-70). Can I neat it up and combine two maps into one? Not really. Because the second map (nat_gw_config) uses a data source depending on the subnets, which depends on the first map (subnet_config). Trying to combine the maps causes circular dependency!\nAlso, the additions of data sources makes the code less readable. As Marcel L pointed out in his post, two cons with for_each are: complexity and requiring a map (to store multiple attribute values). Now we seem to have one more: it may cause unintended deletions\nIs for_each a bad idea? Let\u0026#8217;s find out why for_each could destroy two existing NAT gateways.\nNotice that I built the map nat_gw_config by looping through the list of variable public_subnets_cidr_list. After apply, we appended it one more string at the end, without changing the existing order. However, the devil lies in the order of the string lists returned from the data sources. By printing this map, we found that the originally value before the AZ addition is:\nindexalloc_idsubnet_id0eipalloc-0ffe32519d20b9b7fsubnet-0b4b202056c75bb0a1eipalloc-0ff1d32fb271b5cf4subnet-09b6486fbe44e9795 Based on this, NAT Gateway with index 0 is created with eipalloc-***b7f and subnet-***b0a. NAT Gateway with index 1 is created with eipalloc-***cf4 and subnet-***795. After we add the third AZ, and apply the run, the new map, with a new alloc_id and a new subnet_id looks like this:\nindexalloc_idsubnet_id0eipalloc-0b70460721596e33f (new)subnet-0b4b202056c75bb0a1eipalloc-0ffe32519d20b9b7fsubnet-0335071ced2dc9922 (new)2eipalloc-0ff1d32fb271b5cf4subnet-09b6486fbe44e9795 There are two factors at play. When the data sources return the ids (data.aws_subnets.public_subnets.ids and data.aws_eips.nat_eips.allocation_ids), the return is sorted. It doesn\u0026#8217;t matter whether the order alphabetical or the opposite. Because in any given order, the randomly generated new ID, can fall anywhere in the list. In this particular result, the new alloc_id falls at the beginning, and the new subnet_id falls in the middle. As a result, NAT Gateway with index 0 and 1 are both changed. Therefore they have to be destroyed and replaced.\nAll these come from having to build a map. The values of each object in the map come from two different data sources. The values are not predetermined and contain a random part. When we add more AZ, the entire map get shuffled, leading to deletion of existing resources. Yikes.\nImplementation with for_each: second attempt The first draft of this post drew some ideas on Reddit. One redditor pointed out that the snippet above with for_each isn\u0026#8217;t the optimal way. With some tricks to we can manage the map so that it maintain relative order if we have to add new AZ. The strategy is:\nAvoid using data sources to retrieve attribute values Use a unique key to identify objects in the map; Directly look up from the resource by the unique key We\u0026#8217;re able to do #2 and #3 because when a resource has the for_each argument set, the resource itself becomes a map of objects. We can then locate that resource by the key. We can determine what that key is so long as it uniquely identifies the resource. Below is the revised code snippet with for_each:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 provider \u0026#34;aws\u0026#34; {} variable \u0026#34;vpc_cidr_block\u0026#34; { type = string default = \u0026#34;147.206.0.0/16\u0026#34; } variable \u0026#34;public_subnets_cidr_list\u0026#34; { type = list(any) default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;] # 2 AZ #default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;, \u0026#34;147.206.8.0/22\u0026#34;] # 3 AZ } data \u0026#34;aws_availability_zones\u0026#34; \u0026#34;this\u0026#34; {} resource \u0026#34;aws_vpc\u0026#34; \u0026#34;base_vpc\u0026#34; { cidr_block = var.vpc_cidr_block } resource \u0026#34;aws_internet_gateway\u0026#34; \u0026#34;internet_gw\u0026#34; { vpc_id = aws_vpc.base_vpc.id } locals { subnet_config = { for cidr in var.public_subnets_cidr_list : md5(cidr) =\u0026gt; { cidr = cidr az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)] } } } resource \u0026#34;aws_subnet\u0026#34; \u0026#34;public_subnets\u0026#34; { for_each = local.subnet_config vpc_id = aws_vpc.base_vpc.id cidr_block = each.value.cidr map_public_ip_on_launch = true availability_zone = each.value.az } resource \u0026#34;aws_eip\u0026#34; \u0026#34;nat_eips\u0026#34; { for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) =\u0026gt; cidr } } locals { nat_gw_config = { for cidr in var.public_subnets_cidr_list : md5(cidr) =\u0026gt; { subnet_id = aws_subnet.public_subnets[md5(cidr)].id alloc_id = aws_eip.nat_eips[md5(cidr)].allocation_id } } } resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { for_each = local.nat_gw_config subnet_id = each.value.subnet_id allocation_id = each.value.alloc_id depends_on = [aws_internet_gateway.internet_gw] } In this example, I use the MD5 hash of CIDR as the unique identifier key to ensure we have a consistent mapping between allocation id and subnet id. When a new AZ is created, the new allocation-subnet id pair will have its own new key. The unique key can be any identifier (even the CIDR itself) as long as it is unique and we do not change the selection of unique key after the first apply.\nOne more shot with for_each The code snippet above got rid of data sources, but still have to leverage two local values (subnet_config and nat_gw_config) as helpers. Are they absolutely necessary? Not really. The Terraform documentation has a page about References to Values, where it states:\nIf the resource has the\u0026nbsp;count\u0026nbsp;argument set, the reference\u0026#8217;s value is a\u0026nbsp;list\u0026nbsp;of objects representing its instances. If the resource has the\u0026nbsp;for_each\u0026nbsp;argument set, the reference\u0026#8217;s value is a\u0026nbsp;map\u0026nbsp;of objects representing its instances. In other words, using for_each with a map as input, we\u0026#8217;re also creating a map as output, which is the resource array itself. The key is the same as the input map. Therefore, we can reuse the key. I know that sounds too abstract. Here\u0026#8217;s the code refined:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 provider \u0026#34;aws\u0026#34; {} variable \u0026#34;vpc_cidr_block\u0026#34; { type = string default = \u0026#34;147.206.0.0/16\u0026#34; } variable \u0026#34;public_subnets_cidr_list\u0026#34; { type = list(any) #default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;] # 2 AZ default = [\u0026#34;147.206.0.0/22\u0026#34;, \u0026#34;147.206.4.0/22\u0026#34;, \u0026#34;147.206.8.0/22\u0026#34;] # 3 AZ } data \u0026#34;aws_availability_zones\u0026#34; \u0026#34;this\u0026#34; {} resource \u0026#34;aws_vpc\u0026#34; \u0026#34;base_vpc\u0026#34; { cidr_block = var.vpc_cidr_block } resource \u0026#34;aws_internet_gateway\u0026#34; \u0026#34;internet_gw\u0026#34; { vpc_id = aws_vpc.base_vpc.id } resource \u0026#34;aws_subnet\u0026#34; \u0026#34;public_subnets\u0026#34; { for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) =\u0026gt; { cidr = cidr az = data.aws_availability_zones.this.names[index(var.public_subnets_cidr_list, cidr)] } } vpc_id = aws_vpc.base_vpc.id cidr_block = each.value.cidr map_public_ip_on_launch = true availability_zone = each.value.az } resource \u0026#34;aws_eip\u0026#34; \u0026#34;nat_eips\u0026#34; { for_each = { for cidr in var.public_subnets_cidr_list : md5(cidr) =\u0026gt; cidr } } resource \u0026#34;aws_nat_gateway\u0026#34; \u0026#34;nat_gws\u0026#34; { for_each = aws_subnet.public_subnets subnet_id = aws_subnet.public_subnets[each.key].id allocation_id = aws_eip.nat_eips[each.key].allocation_id depends_on = [aws_internet_gateway.internet_gw] } Voila. I use md5 of the CIDR as the key again, first to create both aws_subnet and aws_eip. I also followed the example of chaining for_each between resource types. This way, when creating aws_nat_gateway, I can reference an instance in each resource array by the same key. Chaining for_each is very handy. But admittedly, it takes several iterations for me to get there. The code is neater, but not as straightforward to read due to the list/map comprehension.\nConclusion I came across a team where the code review guideline favours for_each strongly. I see where that comes from after reading the book. But I don\u0026#8217;t find count to be evil. That triggered my initiative to dive deep into this topic. In this post we brought up a classic pattern of relationship between resources, and examined several ways to implement them using count and for_each. Using count can be straightforward but carries the risk of index shifting if additional element is added in the middle of the resource array. On the other hand, for_each is more powerful, but it requires some crafting with the Python-style list/map comprehension.\nMy recommendation is, start with a holistic look at the types of resources to create with loop, and how they are related with each other. Go with count if index shifting isn\u0026#8217;t a risk. For example, when you need to create one instance of a resource conditionally. Otherwise, use for_each loop if the team is comfortable with the list/map comprehension. In some cases where we need to conditionally create several instances of the same resource, we can use a technique such as:\nfor_each = variable.disabled ? {} : data.any_resource.map for_each = variable.disabled ? toset([]) : data.any_resource.list In fact, the recommendation from AWS Terraform best practice is highly in favour of for_each.\nPrevious PostTest Open ID Connect Flows Locally Next PostCloud Certifications for Learning? ","date":"2024-08-27T22:36:39-04:00","image":"/wp-content/uploads/2025/04/feature-tf-cnt-foreach.webp","permalink":"/2024/08/debating-between-count-and-for_each-in-terraform/","title":"Debating between count and for_each in Terraform"},{"content":"Earlier this year, I had to integrate an application with an identity provider. Both claim to be compliant with Open ID Connect. But when they don\u0026#8217;t get along, I must find out where it breaks to determine which party isn\u0026#8217;t compliant. Therefore, I had to really get to the transaction-level details. As a result, I was eager to find out a way to test Open ID Connect Flows locally. Considering that I discussed OIDC in several past articles and use cases (authenticating to kube-api server, Kubernetes workload, ROSA), I feel it is important to be able to test Open ID Connect flows locally.\nArchitecture What makes local testing difficult is that there are several parties involved and they act in different roles. Take Authorization Code Flow as an example, the main actors are:\nUser Agent: the browser; Authorization Server: an identity store, also referred to as identity provider; Resource server: an HTTP server that returns protected resource; OIDC client application: the component that communicates with identity store on behalf of the resource server; Apart from the bowser (obviously running on my laptop), we can group the the other actors in different patterns. Here I illustrate some options below:\nServer PatternExample #4Server Pattern\u0026#8230;Server PatternExample #2Server Pattern\u0026#8230;Server PatternExample #1Server Pattern\u0026#8230;Client\u0026nbsp;Pattern Ex.#3Client\u0026nbsp;Pattern Ex.#3Client PatternExample #2Client Pattern\u0026#8230;Client PatternExample #1Client Pattern\u0026#8230;Advanced Reverse Proxy(e.g. Nginx Plus, Traefik Enterprise)Advanced Reverse Proxy\u0026#8230;App (with built-in\u0026nbsp;OIDC support)App (with built-in\u0026nbsp;OIDC suppor\u0026#8230;Identity Provider withOIDC support\u0026nbsp;(e.g. Active Directory)Identity Provider wit\u0026#8230;OIDC Client App PatternsOIDC Client App PatternsOIDC Authorization Server Patterns(including Identity Provider)OIDC Authorization Server Patterns\u0026#8230;HTTPResourceServerHTTP\u0026#8230;oauth2-proxyoauth2-proxySelf-hosted OIDC-capableIdentity Providers(e.g. KeyCloak as IdP)Self-hosted OIDC-capable\u0026#8230;App (without\u0026nbsp;OIDC capability)App (without\u0026nbsp;OIDC capability)OIDClibraryOIDC\u0026#8230;reverse\u0026nbsp;proxycapabilityreverse\u0026nbsp;proxy\u0026#8230;oidc clientcapabilityoidc client\u0026#8230;Commercial OIDC-capableIdentity Providers(e.g. MS Entra-ID, Google)Commercial OIDC-capableIde\u0026#8230;(optional)reverse proxy(optional)\u0026#8230;HTTPResourceServerHTTP\u0026#8230;(optional)reverse proxy(optional)\u0026#8230;App (without\u0026nbsp;OIDC capability)App (without\u0026nbsp;OIDC capability)HTTPResourceServerHTTP\u0026#8230;Any OIDC-capableIdentity Broker\u0026nbsp;e.g. DexAny OIDC-capableIden\u0026#8230;Other Protocol(e.g. LDAP, SAML)Other Protocol\u0026#8230;StandardOIDC FlowStandard\u0026#8230;Server PatternExample #3Server Pattern\u0026#8230;Identity Provider withOIDC support\u0026nbsp;(e.g. Active Directory)Identity Provider wit\u0026#8230;KeyCloak asIdentity BrokerKeyCloak as\u0026#8230;Other Protocol(e.g. LDAP, SAML)Other Protocol\u0026#8230;Text is not SVG \u0026#8211; cannot display\nIn essence, we need an OIDC client app and an authorization server. And I find the most simplistic test architecture to be pattern #2 (without reverse proxy) on the client side, and pattern #1 on the server side. In the next section let\u0026#8217;s discuss why I prefer this test architecture.\nChoice of Tools The principle of this lab is that we can focus on the OIDC flows itself and simplify every other aspects as much as we can.\nThe authorization server needs to access my client app. As a result, if I use a public authorization server, such as Azure or Google, I\u0026#8217;d have to host my client app on a public IP with domain name as well. This creates churns. To see how much hassle this can involve, review my article Istio External Authentication lab. Instead, I need a tool to host the authorization server on my laptop. And yes, it\u0026#8217;s KeyCloak. It is a well-renewed open-source project for identity and access management. It is also the upstream project of RedHat SSO. Another reason is it operates on PostgreSQL database which is a common relational database technology with release in Docker images.\nOn the client side, we can have our web service with built-in OIDC capability, which usually requires development work in a language with OIDC library. Alternatively, if the client app lacks such capability, we would address this capability in a different component on the client side. It can either be a standalone, purpose-built proxy, or a generic reverse proxy with OIDC capability. For this, I examined a few options. Nginx and Traefik have OIDC support for a fee in their Enterprise product. Apache has a module mod_auth_oidc for free but it requires building the plug-in on my own for Mac platform. Eventually I landed on the purpose-built option, using the oauth2-proxy open source project. This project can act as both actor #3 (with a minimal HTTP server) and actor #4 (OIDC client). So it also saves me from hosting a separate web server. Also, I have used it in the past in the Istio External Authentication lab.\nIn reality, each message in the OIDC flows must be TLS encrypted. In local testing though we don\u0026#8217;t really care. Similarly we don\u0026#8217;t necessary need a reverse proxy if it plays no role in the OIDC flow. Both Keycloak (with PostgreSQL) and oauth2-proxy are released in Docker images. As a result, we are ready to roll with only three tools: the browser on local host, Keycloak and oauth2-proxy in the Docker daemon, which provides great portability.\nTo get started, let\u0026#8217;s have two fictitious domains, web.digihunch.com, and keycloak.digihunch.com. In the Docker compose manifest, I name the services based on their hostname so that they can reference each other from within the container network namespace. To access the service from the host, I force the DNS resolution to localhost in /etc/hosts on my Mac, and make sure to declare the same host port in the port mapping (4180 for dummy web service; and 8080 for Keycloak).\nConfiguration I created a Docker compose file as below, to set up my test:\nservices: web.digihunch.com: container_name: oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:latest command: - --http-address - 0.0.0.0:4180 environment: OAUTH2_PROXY_COOKIE_SECRET: NYZaClZinINKwxNGzEDeFGh64W6tmq1eB6uHQPa4S5o OAUTH2_PROXY_CLIENT_ID: dh-user-client OAUTH2_PROXY_CLIENT_SECRET: pmkwBjkVesrj7fw1MY7h5s9e3cmAKXgc OAUTH2_PROXY_PROVIDER: oidc OAUTH2_PROXY_OIDC_ISSUER_URL: http://keycloak.digihunch.com:8080/realms/digihunch-users OAUTH2_PROXY_PASS_ACCESS_TOKEN: true OAUTH2_PROXY_EMAIL_DOMAINS: \u0026#39;*\u0026#39; OAUTH2_PROXY_REDIRECT_URL: http://web.digihunch.com:4180/oauth2/callback OAUTH2_PROXY_PROVIDER_DISPLAY_NAME: DHCKC OAUTH2_PROXY_COOKIE_CSRF_EXPIRE: \u0026#39;5m\u0026#39; OAUTH2_PROXY_COOKIE_CSRF_PER_REQUEST: true OAUTH2_PROXY_COOKIE_SECURE: false # Needed for HTTP connection #OAUTH2_PROXY_UPSTREAMS: file:///var/www/static/#/home/ # serve page at /home path OAUTH2_PROXY_UPSTREAMS: static://202 volumes: - ./config/oauth2-proxy.cfg:/etc/oauth2-proxy.cfg # - ./config/www:/var/www/static/ ports: - 4180:4180 networks: - oidc_network restart: unless-stopped depends_on: - keycloak.digihunch.com postgres-db: image: postgres container_name: postgresdb restart: always shm_size: 128mb ports: - 5432:5432 networks: - oidc_network volumes: - ./data/pgdata:/var/lib/postgresql/data environment: - POSTGRES_USER=master - POSTGRES_PASSWORD=masterpass - POSTGRES_DB=keycloak keycloak.digihunch.com: image: quay.io/keycloak/keycloak command: start environment: # Based on Hostname:v2 https://www.keycloak.org/docs/25.0.0/upgrading/#migrating-to-25-0-0 KC_HOSTNAME: http://keycloak.digihunch.com:8080 #KC_HOSTNAME_ADMIN: For simplicity, no separate management URL or port KC_HOSTNAME_BACKCHANNEL_DYNAMIC: true KC_HTTP_ENABLED: true ## Otherwise HTTPS is the enforced by default. KC_HEALTH_ENABLED: true KEYCLOAK_ADMIN: admin KEYCLOAK_ADMIN_PASSWORD: kcadminpass KC_DB: postgres KC_DB_URL: jdbc:postgresql://postgres-db/keycloak KC_DB_USERNAME: master KC_DB_PASSWORD: masterpass ports: - 8080:8080 networks: - oidc_network restart: always depends_on: - postgres-db networks: oidc_network: enable_ipv6: false driver: bridge For OAuth2-Proxy, the official image for oauth2-proxy is distroless. So if you have to troubleshoot its container file system, you need to access it via an assistant container:\ndocker run --rm -it --name debugger --privileged --pid container:oauth2-proxy --network container:oauth2-proxy busybox sh # to see target container\u0026#39;s file system, go to: ls -l /proc/1/root/ Alternatively, we can use the one that Bitnami releases but be vary of some nuances.\nIn the Keycloak part, I use environment variables and had to watch out for the recent changes on hostname v2. We should first start up the Keycloak service. From http://keycloak.digihunch.com:8080, we can login using the credential specified in the environment variables, then we can create a client app: Create a new realm (dropdown-\u0026gt; Create realm) with name digihunch-users Switch to this realm from the dropdown, create a couple users (e.g. dhadmin@www.digihunch.com), and set password. Create a group (e.g. myadmin) and join the user to the group Under the same realm, create a client, with OpenID Connect as type, client ID being dh-user-client. turn on client authentication (without Direct access grants) Save the client for now and grab the client secret. Note that the OIDC discovery document (http://keycloak.digihunch.com:8080/realms/digihunch-users/.well-known/openid-configuration) should come online. Update the docker compose file: The value for OAUTH2_PROXY_CLIENT_SECRET is from the client secret; The value for OAUTH2_PROXY_OIDC_ISSUER_URL should be http://keycloak.digihunch.com:8080/realms/digihunch-users; The value for OAUTH2_PROXY_CLIENT_ID is dh-user-client; Restart all services including the web. Log on to keycloak and go back to the client configuration in the realm. Under settings. Put in valid redirect URIs as http://web.digihunch.com:4180/oauth2/callback and save the client. Now, let\u0026#8217;s start an private browser session, and browse to http://web.digihunch.com:4180/. The browser should redirect you to keycloak\u0026#8217;s login page. Once log in is successful, it should redirect you to the static response with 202 code as the manifest configured.\nSummary In this post, I laid out the steps to test the login for OIDC authorization code flow locally as a starting point. For a bullet-proof solution, I recommend taking a look at KeyCloak administration guide. For example, we typically disable the master realm for security. There is a similar test setup on Otka blog but I simplified all the aspects that I regard as distractors. There are many other flows that can be tested. However, some of the testing still requires client pattern #1 if we need to initiates an activity from the client application.\nPrevious PostIAM Roles for any workload Next PostDebating between count and for_each in Terraform ","date":"2024-08-05T13:24:34-04:00","image":"/wp-content/uploads/2025/04/feature-local-oidc-test.webp","permalink":"/2024/08/test-open-id-connect-flows-locally/","title":"Test Open ID Connect Flows Locally"},{"content":"Background A few month back a client of mine wanted to use GitLab pipeline to deploy infrastructure on AWS with Terraform. The key question is how to authenticate the Terraform process running in the pipeline to AWS with temporary credential. Having worked it out on GitHub, my proposal at time was to add OIDC provider to represent the GitLab runner. After a few months, they told me that they are self-hosting their GitLab instance. The idea above was based on exposing an identity provider document on the public Internet, which the client is unable to do. Now, I have an idea: IAM Roles Anywhere.\nIAM Role Introduction Many 101 tutorials asks beginners to create standalone IAM users (or group) with IAM policies directly attached. For programatic access they also include creating a pair of access key and secret access key and pass them along to an external application. The keys are long term credentials, and worse, never expires. The leakage of these long-term credentials had been such a headache that AWS strongly discourage the use of long term credentials. You can feel the discouragement when trying to create an access key through the web console, or by the banners on top of the documentation page about how to do so. The recommendation is use temporary security credentials. In the context of AWS that means IAM roles. The users must assume an IAM role by issuing an API call, and the Security Token Service (STS) grants temporary credential in response.\nCLICLIRequest: AssumeRole*Request: AssumeRole*Response:\u0026#8211; AccessKeyId\u0026#8211; SecretAccessKey\u0026#8211; SessionTokenResponse:\u0026#8230;Text is not SVG \u0026#8211; cannot display\nThis diagram has several variations. For example, the request can be AssumeRole, AssumeRoleWithSAML and AssumeRoleWithWebIdentity, depending on whether and how the user info is federated with external identity store. The returned response, a triplet of three values, makes the temporary credential that we should use in any secure environment. They must be renewed before expiry. This model works not only for human identity (e.g. SAML integration, OIDC integration, cross-account access) but also for workload identity (e.g. EC2 instance profile, Lambda execution role, ECS task role, etc). Another good example is IAM Role for Service Account (IRSA), where a web identity represents a Kuberentes Service Account to gain role credential using the AssumeRoleWithWebIdentity API. In this post however, I\u0026#8217;d like to explore more about the IAM role for EC2 instance profile.\nThe IMDS service For EC2 instance, we all know that we can associate an IAM role as the instance profile and grant the process using AWS SDK running on the instance with permissions associated with the IAM role. At a lower level, this relies on the IMDS (Instance Metadata Service) running on the instance.\nIf an EC2 instance\u0026#8217;s profile points to an IAM role, a process running on the instance using AWS SDK will also need to get the triplet from the STS. It is summarized in this diagram:\nIMDS v2169.254.169.254IMDS v2\u0026#8230;AWS SDKAWS SDKRequest: AssumeRoleRequest: AssumeRoleResponse:\u0026#8211; AccessKeyId\u0026#8211; SecretAccessKey\u0026#8211; SessionTokenResponse:\u0026#8230;ApplicationProcessApplication\u0026#8230;EC2 Instance\nEC2 I\u0026#8230;Text is not SVG \u0026#8211; cannot display\nThe IMDS is a service available on one of the link-local IP address (169.254.169.254) on the EC2 instance. Requests made to this IP address are not routed elsewhere. The Instance Metadata Service (IMDS) is a means for the cloud service provider\u0026#8217;s virtualization layer to share information with the processes on the operating system of a virtual machine. It responds with information related to the instance itself, such as the subnets, IAM role, instance ID, AMI ID, security group. The instance metadata also includes user data script for cloud init process to consume, and most relevantly, the role credential for the instance. This also requires that the IMDS service to have connectivity to the STS endpoint, either via interface endpoint or over the Internet. The AssumeRole calls are logged in CloudTrail.\nAll major cloud vendors (AWS, Azure and GCP) uses the IMDS mechanism, and this mechanism obviously draws the attention of bad actors. I find some good articles on this here and here.\nIAM Role for EC2 Workload For EC2 instances at AWS, the initial IMDS v1 was introduced in 2012 and allows a GET method to fetch instance metadata. The IMDS v1 is subject to attacks such as SSRF (Server-side request forgery). In 2019 AWS introduced IMDS v2 which tackles those vulnerabilities. As of date, the recommendation is to use IMDSv2. Here is an example of how to fetch instance metadata, including the credential:\n# Grab a token TOKEN=`curl -X PUT \u0026#34;http://169.254.169.254/latest/api/token\u0026#34; -H \u0026#34;X-aws-ec2-metadata-token-ttl-seconds: 21600\u0026#34;` # Get top-level instance metadata information curl http://169.254.169.254/latest/meta-data/profile -H \u0026#34;X-aws-ec2-metadata-token: $TOKEN\u0026#34; # Get the name of the role curl -H \u0026#34;X-aws-ec2-metadata-token: $TOKEN\u0026#34; http://169.254.169.254/latest/meta-data/iam/security-credentials # Get the credential for the role session curl -H \u0026#34;X-aws-ec2-metadata-token: $TOKEN\u0026#34; http://169.254.169.254/latest/meta-data/iam/security-credentials/InstanceProfileRoleName These commands emulate how the SDK library fetches the credentials to assume the instance profile role. You can also find similar commands on the documentation. However, there isn\u0026#8217;t much details about how the instance metadata service interacts with the STS service, except a general statement:\nThese security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials. When creating a new EC2 instance, make sure that the instance metadata option has http_endpoint enabled, to enable the IMDS service. Also set http_tokens to required, which would run IMDSv2 exclusively. With that setup, the application does not have to mana to use an SDK version that supports IMDSv2. Another metadata option is http_put_response_hop_limit, with default of 1. This limits the number of hops in the metadata request. If the process runs from a Docker container with bridge networking mode, set it to 2 or the process cannot even secure a token.\nIMDS v2169.254.169.254IMDS v2\u0026#8230;AWS SDKAWS SDKRequest: AssumeRoleRequest: AssumeRoleResponse:\u0026#8211; AccessKeyId\u0026#8211; SecretAccessKey\u0026#8211; SessionTokenResponse:\u0026#8230;ApplicationProcessApplication\u0026#8230;EC2 Instance\nEC2 I\u0026#8230;BridgeNetworkBridge\u0026#8230;Docker DaemonDocker DaemonText is not SVG \u0026#8211; cannot display\nThe diagram above illustrates this scenario with two hops.\nIAM Role Anywhere Concept As this point, we know that the process of workload assuming an IAM role, is essentially using SDK to gain role credentials from instance metadata. In addition to using SDK and instance metadata, AWS also supports using X.509 certificate to gain role credentials. As a result, workload no longer needs AWS SDK, and it doesn\u0026#8217;t rely on instance metadata from an EC2 instance. This mechanism is known as IAM Role Anywhere, and it greatly expands the use cases for IAM Role. To make this work, we first have to provide a certificate authority (CA) to AWS as a trust anchor. It can be any X.509 CA including AWS Private CA. The IAM Roles Anywhere will allow any end-entity endorsed by this trust anchor, to assume an IAM role as specified. We also need to create a profile, in which we can add IAM policies directly, or link to an IAM roles with a trust policy for service principal rolesanywhere.amazonaws.com. To gain role credential, the requestor must provide both the private key, and its end-entity certificate. The certificate proofs the endorsement of the CA as Role\u0026#8217;s trust anchor specifies. The private key proofs the requestor\u0026#8217;s identity. The requestor uses the aws_signing_helper utility to request role credentials. The utility is compatible with the credential_process feature in AWS config, which passes the returned role credentials to the AWS config profile for AWS CLI or SDK running on external virtual machine.\nIAM Role Anywhere Lab Let\u0026#8217;s tweak the three commands from this old post of mine to create the test materials: a self-signed CA and a certificate signed by the CA:\nopenssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj \u0026#39;/CN=Health Certificate Authority\u0026#39; -addext basicConstraints=critical,CA:TRUE,pathlen:1 -addext keyUsage=keyCertSign cat \u0026gt; ext.cnf \u0026lt;\u0026lt;EOF [v3_leaf] keyUsage = digitalSignature basicConstraints=CA:false EOF openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj \u0026#39;/CN=*.digihunch.com\u0026#39; openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt -extfile ext.cnf -extensions v3_leaf I tweak them to add the X.509 extensions to meet the requirement for signature validation. We need the following files from the output.\nca.crt -\u0026gt; the certificate of the CA. We provide this file as the trust anchor server.crt -\u0026gt; the certificate of the server, we need it as the end-entity certificate server.key -\u0026gt; we need to present this file to proof identity of the requestor First, we go to AWS console and create a new trust anchor. Copy the content of ca.crt as the certificate. Then we can create a profile with an IAM role, with the trust policy looking like this example. Then we can request the role credential with one command:\naws_signing_helper credential-process --certificate server.crt --private-key server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN Moreover, if the workload supports AWS SDK or can use CLI but not an EC2 instance, we can bake this in the AWS profile on the external machine:\n[profile myprofile] output = json credential_process = aws_signing_helper credential-process --certificate /path/server.crt --private-key /path/server.key --trust-anchor-arn $TRUST_ANCHOR_ARN --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN We can even configure this in any pipeline as code to allow deployment from a non-AWS pipeline. Summary In summary, apart from native AWS services, an IAM role can trust the following types of principals:\nNative IAM identity such as an IAM user or an IAM group Authenticated identity from SAML identity provider that IAM is configured to trust Authenticated identity from OIDC identity provider that IAM is configured to trust Validated identity endorsed by a Certificate Authority that IAM designate as a trust anchor The first type is rarely used because few organizations uses AWS IAM as their identity store. Most organizations have their identity store with federation capability via SAML. On the other hand, a lot of modern applications adopts identity stores with OIDC compliance. Now with IAM Role Anywhere, any entity with X.509 identity can also assume an IAM role. It works with any CI/CD pipeline, whether it is self-hosted. Also, it is now more important to keep the keys safe. Previous PostManaging EC2 instances across accounts with Ansible Next PostTest Open ID Connect Flows Locally ","date":"2024-07-14T23:53:49-04:00","image":"/wp-content/uploads/2025/04/feature-iam-role-anywhere.webp","permalink":"/2024/07/iam-roles-for-any-workload/","title":"IAM Roles for any workload"},{"content":"I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is consistency in the VM management, over time, and across platforms (AWS, on-prem, Azure, etc). Even for AWS shops, in the last few years many enterprises have adopted AWS landing zone with the multiple AWS account prescriptive pattern. However AWS Systems Manager still lacks integration with AWS Organization (except for a few non-core capabilities). This creates the demand of managing EC2 instances across AWS accounts. In this post, we propose a secure method to manage a fleet of EC2 instances from multiple AWS accounts, using Systems Manager . It also enables connectivity from an Ansible control node.\nPrerequisites This proposal ties a few CLI tools together, including AWS CLI, SSH, Ansible, etc. It also requires the cloud engineer to understand how they work. I\u0026#8217;ll start with the choice of the tools.\nAbove, I discussed the benefit of Ansible. Since Ansible operates on SSH, we\u0026#8217;ll still have to use SSH tools. Even though SSM agent provides a way to connect to EC2 instance without requiring an RSA key pair, we still need SSH since it is a well-established industry standard (RFC4253) and the foundation of Ansible. These two technologies are not mutually exclusive. In fact, the SSM agent provides a secure enhancement to the operation with SSH. Traditionally, on each EC2 instance we\u0026#8217;d have to run SSHD services which opens TCP port 22 (or alternative TCP port as configured). For authentication we favour key pair over password but the open port is still an attack surface vulnerable to brute force and DDoS attacks. For EC2 instances on private networks there is no reachability to the instance\u0026#8217;s SSH port unless the bastion is also in a connected network. As I cover in a post, the SSM Session Manager comes in handy. The SSM agent operates from the instance and communicate outbound to AWS backend. Since the SSM agent runs under a privileged user on the OS, you can perform OS-level commands through SSM. Further, AWS developed a Session Manager plugin with AWS CLI, allowing AWS CLI as a proxy command when making an SSH connection. Therefore SSM enables SSH connection without requiring port 22 to be open. In addition we\u0026#8217;ll need to use RSA key pair as required for SSH, which is also an improvement to the security posture. That explains the dependent tools. On the Ansible control node, apart from Ansible itself, we need the latest version of AWS CLI with the Session Manager plugin, we need to configure AWS CLI properly to connect to EC2 instances across multiple AWS accounts. Configure AWS CLI This section discusses how to configure AWS CLI. I have a couple of handy aliases for productivity but they are not essential. For example, I often need to check the IAM identity making the call, and I often need to list out all profiles configured. So I added the following two entries in the ~/.aws/cli/alias file:\n[toplevel] whoami = sts get-caller-identity --no-cli-pager --output yaml profile = configure list-profiles With that I have an alias to check IAM identity and available profiles. Then we can start configuring the profiles for CLI (in the file ~/.aws/config). Since we\u0026#8217;ll be working with multiple AWS accounts, we have to manage multiple CLI profiles, which implies that:\nWe better use the --profile switch to explicitly specify profile being used, instead of relying on the AWS_PROFILE environment variable; As a security best practice, we should not configure profiles with long-term IAM credential in the config file; We must ensure the CLI doesn\u0026#8217;t prompt for log-in every time we switch profile To satisfy #3 there are many ways but we\u0026#8217;ll discuss two: using cross-account IAM role, and using AWS SSO.\nBonus point if you enable auto-complete for AWS CLI.\nConfigure AWS CLI Profiles With cross-account IAM role, the idea, is that the client start with one IAM identity, and use that IAM identity to assume roles on several other accounts. The configuration looks like this:\n[profile jump_account] credential_process = /opt/bin/awscreds-custom --username helen [profile target_account_1] role_arn = arn:aws:iam::123456789011:role/OrganizationAccountAccessRole source_profile = jump_account [profile target_account_2] role_arn = arn:aws:iam::123456789012:role/OrganizationAccountAccessRole source_profile = jump_account In this example, you start with an validated identity in the jump account, then assume a privileged IAM role named OrganizationAccountAccessRole on the target accounts. Typically such IAM roles are pre-configured (e.g. in an multi-account landing zone) with appropriate trust policy to allow principals from the jump account. Once you\u0026#8217;re validated as the IAM identity in the jump account, then you can use profiles for target accounts without being prompted for credentials again.\nIf your have configured IAM Identity Center for the multi-account environment, consider an alternative approach using sso login. The configuration usually looks like this:\n[sso-session sso] sso_start_url = https://myorg.awsapps.com/start/ sso_region = us-east-1 sso_registration_scopes = sso:account:access [profile target_account_1] sso_session = sso sso_account_id = 123456789011 sso_role_name = AWSAdministratorAccess [profile target_account_2] sso_session = sso sso_account_id = 123456789012 sso_role_name = AWSAdministratorAccess This is often used by human users with SSO credential. In this example, to authenticate the sso session, start with command \u0026#8220;aws sso login\u0026#8221; . Then you can use all profiles by specifying --profile switch without having to log in again.\nConfigure SSH to EC2 via SSM The EC2 instance must connect to Systems Manager endpoint, before one can SSH to the instance using the plugin. Once connected, you should find the instance in Fleet Manager. For this to happen, there are a few conditions. First, the instance must be able to reach the endpoint, either via public Internet, or via VPC interface endpoints if in a private subnet. Second, the instance profile must contain an IAM role with appropriate permissions. We can use AWS managed policy AmazonSSMManagedInstanceCore in the role. In addition, if we record the SSM session to an S3 bucket with encryption the instance profile must have permission to use the encryption key.\nSSM agent uses the IAM role. The agent runs as a service on Linux or Windows machines. Many AWS managed AMIs come with SSM agent pre-installed. If that is not the case, you\u0026#8217;d install the agent in your own AMI, or in user data which requires downloading the installer. With these configuration you\u0026#8217;d be able to connect to the instance via SSM. We can use AWS CLI SSM command, or AWS web console to start an SSH session. To use SSH CLI utility, we install the session manager plugin along with AWS CLI on the SSH client machine. We also specify a public key for EC2 instance and run ssh command with the private key. The SSH configuration needs a configuration such as:\nhost i-* mi-* ProxyCommand sh -c \u0026#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters \u0026#39;portNumber=%p\u0026#39;\u0026#34; User ec2-user IdentityFile ~/.ssh/id_rsa With this entry, you may directly SSH by instance ID (usually starting with i-* and mi-*), and the specified Proxy Command with SSM session document AWS-StartSSHSession will be invoked. Configure Ansible Inventory We can SSH to an instance (without port 22 open, on top of SSM) using the method above. Similarly, we can also configure Ansible to connect to the instance, without port 22. The inventory configuration looks like this:\nmytest: hosts: instance1: ansible_host: i-00aabbffcc7755221 ansible_user: ubuntu ansible_ssh_common_args: -o ProxyCommand=\u0026#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1\u0026#34; instance2: ansible_host: i-eedd88ff66aa22442 ansible_user: ubuntu ansible_ssh_common_args: -o StrictHostKeyChecking=no -o ProxyCommand=\u0026#34;sh -c \\\u0026#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters \u0026#39;portNumber=%p\u0026#39; --profile target_account_2 \\\u0026#34;\u0026#34; Note I used two patterns for ansible_ssh_common_args that are similar. Both works. Each entry references its own profile. This is necessary because Ansible does not have the knowledge which instance belongs to which profile\u0026#8217;s account.\nAnother way to get Ansible connect to instances is to use the connection plugin community.aws.aws_ssm, by specifying ansible_connection: aws_ssm (e.g. as host variable) and other required variables (e.g. profile). This method does not require SSH channel but it requires an S3 bucket, and hence IAM permission on the controller node.\nAnsible supports dynamic inventory, in three ways: inventory source file (with existing plugin), custom inventory plugin, and inventory script (in Python). Take source file as an example for EC2, add the followings as the content of aws_ec2.yaml:\nplugin: amazon.aws.aws_ec2 # Attach the default AWS profile aws_profile: target_account_1 compose: ansible_host: instance_id ansible_user: \u0026#34;\u0026#39;ubuntu\u0026#39;\u0026#34; ansible_ssh_common_args: \u0026#34;\u0026#39;-o ProxyCommand=\\\u0026#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --profile target_account_1 \\\u0026#34;\u0026#39;\u0026#34; Then we can display the rendered inventory list, and Ansible-ping the instances. ansible-inventory -i aws_ec2.yaml --list -y | less ansible all -i aws_ec2.yaml -m ping For more flexibility, for the composed variables, we could use jinja2 expression to generate the value. In both ways, we produce an inventory source per profile using the built-in inventory plugin aws_ec2. For greater flexibility, such as consolidating instances from all accounts into a single inventory, consider writing your own inventory script, or even own inventory plugin.\nConclusion In this post we propose a way to manage instances across AWS accounts. Two main challenges are establishing the communication channel (SSH on top of SSM) and generating inventory data in Ansible. Some AWS services can generate inventory data, such as resource data sync in Systems Manager, or using AWS Config Aggregator. It is unfortunate that neither way produces the inventory data in a format that is directly compatible with Ansible inventory. Therefore, you might have to create a custom Ansible dynamic inventory script (in Python) that reads from the inventory data from AWS Config Aggregator (which supports AWS Organization). The side benefit of this script is that it is usually faster than the built-in aws_ec2 inventory plugin.\nPrevious PostPublic Key Infrastructure 3 of 3 – PKI Implementation Next PostIAM Roles for any workload ","date":"2024-05-27T14:00:22-04:00","image":"/wp-content/uploads/2025/04/feature-ec2-ssm.webp","permalink":"/2024/05/managing-ec2-instances-across-aws-accounts-ssm/","title":"Managing EC2 instances across accounts with Ansible"},{"content":"After the last two post, now we can focus on PKI implementation. The use case is software testing, where we need to create and recycle a lot of short-lived certificates. Typically, we don\u0026#8217;t have to create public certificates because testing workload is internal. Also, hosting a public CA is much more involving. In this post we go over some options to host private CAs.\nRoot CA I\u0026#8217;m assuming we are creating a self-signed root CA for the organization. Large organizations usually keep their root CA offline and store the key material in highly protected configuration such as HSM. That is not something we can easily emulate. Nor are we concerned with the details of protecting root CA. In order to enable configuration of intermediate CAs, all we need for root CA, is to create a self-signed certificate along with the key. We can do this with OpenSSL following the first section of this post. Now we\u0026#8217;ll use open-source Step CA. In my opinion, this tool is handier. For example, we can create a root CA this with a single command:\nstep ca init This will start a prompt with a few questions. Then it will create a root CA and an intermediate CA. The keys and certificates are stored in ~/.step/ directory. Note that by default, the root CA certificate has a path length of 1. To make the path length more than 1, we\u0026#8217;d have to customize the creation. We can specified the desired path length in a certificate template and create a self-signed certificate with the template:\ncat \u0026lt;\u0026lt; EOF \u0026gt; root.tpl { \u0026#34;subject\u0026#34;: {{ toJson .Subject }}, \u0026#34;issuer\u0026#34;: {{ toJson .Subject }}, \u0026#34;keyUsage\u0026#34;: [\u0026#34;certSign\u0026#34;, \u0026#34;crlSign\u0026#34;], \u0026#34;basicConstraints\u0026#34;: { \u0026#34;isCA\u0026#34;: true, \u0026#34;maxPathLen\u0026#34;: 2 } } EOF step certificate create --kty=RSA --size 4096 --template root.tpl \u0026#34;DigiHunch Root CA\u0026#34; root_ca.crt root_ca.key The command outputs the certificate and key of our test root CA. With that, next, we\u0026#8217;ll create subordinate CAs with a few different tools.\nAWS Private CA as intermediate CA I\u0026#8217;ll start with AWS Private CA. It was a spin-off service from AWS Certificate Manager and is fairly simple. However, we need to understand its capacity limit. AWS Private CA documentation has a page on RFC Compliance, which lists what in RFC 5280 are supported and what are not. It performs very basic CA functions. It does not perform domain validation, hence no ACME support. We can create root CA as well but here we\u0026#8217;ll create an intermediate CA:\nIn AWS Console, create a Private CA. Indicate you want to create a subordinate CA. You may specify CRL distribution and OCSP endpoint. However, you\u0026#8217;re responsible for the providing CRL and/or hosting OCSP service; In the Private CA, we need to create CA Certificate. We need to provide CSR to our root CA to sign this CA\u0026#8217;s certificate. The root CA can be either another AWS Private CA or external CA. In this case we choose external CA and export the CSR to a file (e.g. dh.csr). We can use Step CA to sign the request. Note that we have to give reasonable value for expiry date. Because ACM create certificate with 13 month validity period by default, the expiry date must be at least 13 months from the current date. We also need to set path length. In this case, I put 0 so the Private CA can only issue end-entity certificate. The command looks like this: step certificate sign pca.csr \\ /Users/digihunch/.step/certs/root_ca.crt \\ /Users/digihunch/.step/secrets/root_ca_key \\ --profile intermediate-ca \\ --not-after=2027-01-24T07:20:50.52Z \\ --path-len=0 In the AWS console, we paste the generate certificate content in the as certificate body, and the root CA\u0026#8217;s certificate as certificate chain. If the Private CA serves as parent CA of one more layer of CA, set the path length to 1. This requires the path length to be at least 2. Once the CA certificate is installed, we have completed creating a private CA. We can reference this CA from ACM (AWS certificate manager) when creating a private certificate. Since there is no validation support, ACM will allow you to claim any domain. Currently it does not support ACME-based certificate automation. However, you can use AWS CLI or any AWS based automation tool to create your private certificate.\nStep CA As a cheaper alternative, we can host private CA using step CA on virtual machines. One of the benefits is ACME support (http-01 challenge). In the init command in this post, we already create the intermediate CA certificate and configuration. In this part of the lab, we\u0026#8217;ll have two servers: the CA server (pki.digihunch.internal) and the web server (web.digihunch.internal). Make sure the DNS resolution works for both servers. The architecture looks like this:\nCA Server\n(ACME Server)CA Server\u0026#8230;Step CAStep\u0026#8230;pki.digihunch.internalpki.digihunch.internalWeb Server\n(ACME Client)Web Server\u0026#8230;Step CLI Step\u0026#8230;web.digihunch.internalweb.digihunch.internalephemeralephemeralTCP 443TCP 80/challenge/response/challenge/responsestandalone\nmodestandalone\u0026#8230;/acme/order//acme/order/Text is not SVG \u0026#8211; cannot display\nThe CA server runs the Step CA process acting as ACME server. We deploy the CA in standalone mode (instead of linked or hosted deployment), meaning it\u0026#8217;s not connected to any cloud services. We can host the service on port 443 so make sure the process has port binding permission within the operating system, and the firewall (security group) allows traffic via port 443. The Web Server runs Step CLI acting as ACME client. On the web server, we run Step CLI in standalone mode (instead of webroot). During the challenge-response phase, the CLI will get the required random number from ACME server, and host it the as the response on port 80. Make sure that Step CLI process has port binding permission in the OS, and the firewall (security group) allows traffic via port 80.\nMake sure the DNS resolution works for both servers. On the CA server, we fetch the fingerprint (in preparation for setting up Step CLI on web server). Then we add an ACME provisioner, and host the CA server with a single command:\nstep certificate fingerprint $(step path)/certs/root_ca.crt step ca provisioner add myacme --type ACME step-ca $(step path)/config/ca.json The CA server is listening on port 443 (by default). Now we can test ACME process with any ACME compatible client. Let\u0026#8217;s use step CLI on the web server:\nstep ca bootstrap --ca-url https://pki.digihunch.internal:443 \\ --fingerprint \u0026lt;fingerprintvalue\u0026gt; step ca certificate web.digihunch.internal acme.crt acme.key \\ --acme https://pki.digihunch.internal/acme/myacme/directory The bootstrapping step is to establish trust on the CA server. Then the provisioning process should complete automatically:\nHere the step CLI command uses standalone mode by default so it is important to ensure reachability (port 80, DNS name) from the CA server when step CLI hosts the response. You can also go with webroot mode, where Step CLI generate the file to the web root directory so the response becomes available. This is helpful when you already run another web hosting process such as Nginx on the web server. This blog post covers the usage of other ACME-compatible tools with Step CA as private CA server.\nAnother well-known tool is HashiCorp Vault. There is already a good tutorial here and all the OpenSSL command in the tutorial can be replaced with Step commands. Cert Manager on Kubernetes The Cert Manager project on Kubernetes makes PKI work simple. In my discussion about self-signed certificate on Kubernetes, I covered how to use Cert Manager to create self-signed certificate. However, a corporate with multiple clusters may need to chain each cluster-wide issuing CA to an intermediate CA outside of the cluster. Let\u0026#8217;s look at this architecture as an example of a full PKI implementation:\nData CenterData CenterRoot CA\nOfflineRoot CA\u0026#8230;Root CA\nCertificateRoot CA\u0026#8230;self-signself-signIntermediate CA\non-premInte\u0026#8230;Intermediate CA\non-premInte\u0026#8230;AWS CloudAWS CloudsignsignOps AccountOps AccountIntermediate CA\nCertificateIntermedi\u0026#8230;AWS Private CA\nIntermediate CAAWS Pri\u0026#8230;Resource Access ManagerResou\u0026#8230;Workload AccountWorkload AccountElastic Kubernetes\nService ClusterElastic\u0026#8230;nsnamespace\nworkload1namesp\u0026#8230;nsnamespace\ncert-managernamesp\u0026#8230;ClusterIssuerWorkload1IssuerWorkload1\nCertificateWorkload\u0026#8230;podWorkload1Workload1CA CertificateCA Certif\u0026#8230;signsignnsnamespace\nworkload2namesp\u0026#8230;Workload2IssuerWorkload2\nCertificateWorkload\u0026#8230;podWorkload2Workload2CA CertificateCA Certif\u0026#8230;signsignnsnamespace\ningressnamesp\u0026#8230;IngressIssuerIngress\nCertificateIngress\u0026#8230;podIngress\nPodIngress\u0026#8230;CA CertificateCA Certif\u0026#8230;signsignsignsignsignsignsignsignsaService\nAccountService\u0026#8230;Elastic Kubernetes\nService ClusterElastic\u0026#8230;nsnamespace\nworkload1namesp\u0026#8230;nsnamespace\ncert-managernamesp\u0026#8230;ClusterIssuerWorkload1IssuerWorkload1\nCertificateWorkload\u0026#8230;podWorkload1Workload1CA CertificateCA Certif\u0026#8230;signsignnsnamespace\nworkload2namesp\u0026#8230;Workload2IssuerWorkload2\nCertificateWorkload\u0026#8230;podWorkload2Workload2CA CertificateCA Certif\u0026#8230;signsignnsnamespace\ningressnamesp\u0026#8230;IngressIssuerIngress\nCertificateIngress\u0026#8230;podIngress\nPodIngress\u0026#8230;CA CertificateCA Certif\u0026#8230;signsignsignsignsignsignsignsignsaService\nAccountService\u0026#8230;Extend Corporate PKI to Cloud for Kubernetes workloadExtend Corporate PKI to Cloud f\u0026#8230;Text is not SVG \u0026#8211; cannot display\nIn this architecture, we extend corporate on-premise root CA to the cloud. The Ops account hosts the AWS Private CA, and shares it out to workload account(s) using Resource Access Manager. In each EKS cluster, the Private CA serves as the cluster-level issuer, which can issue CA certificates across Kubernetes namespaces. In each namespace, there is a Cert Manager issuer responsible for issuing certificates within the namespace. Cert Manager supports many issuers and we\u0026#8217;re using the AWS Private CA issuer. To demonstrate the gist of this architecture, in our lab we\u0026#8217;ll create one Kubernetes cluster in the same AWS account as the Private CA, and create a Cert Manager certificate in one namespace (e.g. ingress). To get started, create a private CA with the instruction in this post and ensure that the private CA has path length of 1. Then create an EKS cluster. You may use my CloudKube project to provision this cluster in Terraform or any other ways. We use the IRSA model to grant a service account access to the Private CA. First, we create an IAM policy and reference the ARN of the private CA:\n{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;awspcaissuer\u0026#34;, \u0026#34;Action\u0026#34;: [\u0026#34;acm-pca:DescribeCertificateAuthority\u0026#34;, \u0026#34;acm-pca:GetCertificate\u0026#34;, \u0026#34;acm-pca:IssueCertificate\u0026#34;], \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:acm-pca:\u0026lt;region\u0026gt;:\u0026lt;account_id\u0026gt;:certificate-authority/\u0026lt;resource_id\u0026gt;\u0026#34; } ] } Name this policy \u0026#8216;PCA-Access\u0026#8217;. It is the minimum required permission. Now let\u0026#8217;s create a service account, along with an IAM role that uses this policy:\neksctl utils associate-iam-oidc-provider \\ --region $AWS_REGION \\ --cluster $CLUSTER_NAME \\ --approve eksctl create iamserviceaccount \\ --cluster=$CLUSTER_NAME \\ --namespace=cert-manager \\ --name=aws-pca-sa \\ --role-name EKSCertManagerPrivateCARole-$CLUSTER_NAME \\ --attach-policy-arn=arn:aws:iam::123456789012:policy/PCA-Access \\ --approve In order to run the command successfully, you need the IAM permission to create IAM role, as well as API access to the cluster. Now we can install both Cert Manager and the Private CA Issuer.\nhelm repo add awspca https://cert-manager.github.io/aws-privateca-issuer helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --version v1.13.3 \\ --set installCRDs=true \\ --create-namespace helm install aws-ca awspca/aws-privateca-issuer \\ --namespace cert-manager \\ --version v1.2.7 \\ --set serviceAccount.create=false \\ --set serviceAccount.name=aws-pca-sa Note that when installing Private CA issuer with helm, specify the service account that we created earlier (aws-pca-sa). We install both to the cert-manager namespace, where we\u0026#8217;ll create a ClusterIssuer. Now we can declare the following manifest:\napiVersion: awspca.cert-manager.io/v1beta1 kind: AWSPCAClusterIssuer metadata: name: pca-cluster-issuer-rsa namespace: cert-manager spec: arn: arn:aws:acm-pca:ca-central-1:383500642091:certificate-authority/7f2d7b38-2508-4492-81f0-b5b85427c99c region: ca-central-1 --- apiVersion: v1 kind: Namespace metadata: name: ingress --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: ingress-ca-cert namespace: ingress spec: isCA: true commonName: ingress-ca secretName: ingress-ca-secret privateKey: algorithm: RSA size: 2048 issuerRef: name: pca-cluster-issuer-rsa kind: AWSPCAClusterIssuer group: awspca.cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: ingress-ca-issuer namespace: ingress spec: ca: secretName: ingress-ca-secret --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: ingress-cert namespace: ingress spec: commonName: web.digihunch.com secretName: web-digihunch-secret duration: 2160h renewBefore: 72h subject: organizations: - digihunch dnsNames: - web.digihunch.com privateKey: algorithm: RSA size: 2048 issuerRef: name: ingress-ca-issuer kind: Issuer group: cert-manager.io In this manifest, we create a CA certificate, along with a CA in the ingress namespace. With that, we create an end-entity certificate. As a result, we should find the certificates ready:\nkubectl -n ingress get certificate NAME READY SECRET AGE ingress-ca-cert True ingress-ca-secret 24s ingress-cert True web-digihunch-secret 24s The certificate can be reference by the workload in the namespace. Because an Issuer can only issuer Certificate in the same namespace, we need a issuer in the namespace where the certificate will live. The other benefit that Cert Manager brings, is the support of ACME challenges, which is an enhancement to AWS Private CA.\nSummary We discussed several approaches in PKI implementation. PKI is a fundamental requirement in a software testing environment today. Having an internal public key infrastructure enables many other use cases too. For example, the team may use SSH user certificate. You can also host a CA with Step. PKI allows an enterprise to configure SSL inspection on their Next Generation Firewall. The IAM Role Anywhere feature on AWS also operates on an organization\u0026#8217;s own PKI.\nPrevious PostPublic Key Infrastructure 2 of 3 – Certificate Automation Next PostManaging EC2 instances across accounts with Ansible ","date":"2024-03-30T00:13:00-04:00","image":"/wp-content/uploads/2025/04/feature-pki-3.webp","permalink":"/2024/03/public-key-infrastructure-3-of-3-use-cases/","title":"Public Key Infrastructure 3 of 3 – PKI Implementation"},{"content":"Following the last post on PKI, we\u0026#8217;ll discuss automation of certificate issuance. Two key activities to automate are: validation of the requestor and issuance of the certificate.\nValidation Validation isn\u0026#8217;t always required. For private CAs, the trust boundary does not go beyond the internal engineering team, there is little incentive to perform any validation. AWS Private CA is based on this idea. The requestor can claim to be any identity. The private CA, when issuing the certificate, does not perform any validation. Neither is there a need to convince any entity outside of the trust boundary of the validity of the certificate. Validation is optional. For public facing certificate however, validation is a must because we\u0026#8217;re convincing every browser in the world of the validity of the certificate requestor. Common validation levels include:\nDomain Validation (DV): the certificate requestor must demonstrate the right to administratively manage the affected DNS domain. Organization Validation (OV): in addition to the DV criterion, the issuer verifies the actual existence of the requestor\u0026#8217;s organization as a legal entity. Extended Validation (EV): the certificate requestor must persuade the certificate provider of its legal identity, including manual verification checks y a human. Unlike DV and OV certificates, only a subset of CAs can issue EV certificates. For both OV and EV, a certificate provider publishes its vetting criteria through its certificate policy. They require human validation of any registrants. At corporate level, EV certificates are required for sensitive public-facing workloads (e.g. banking, financial, health information). For non-sensitive public-facing workloads, DV certificates may be sufficient. For non-public facing workloads, such as software testing, they may go with DV certificates or no validation at all, depending on the specific use case. Since I set up PKI for the latter, I\u0026#8217;ll focus on DV. DV is the most basic level and can be fully automated. DigiCert, a well-known trusted third party, has a detailed page on the differences among DV, OV and EV.\nCertificate Automation As to automation, there are some common certificate automation protocols:\nACME (Automated Certificate Management Environment): commonly used in web server automation. SCEP (Simple Certificate Enrollment Protocol): commonly used in enterprise environments for managing certificates in the network devices such as routers, switches and IP phones. EST (Enrolment over Secure Transport): a more secure alternative to SCEP suitable for various use cases beyond network devices. CMP (Certificate Management Protocol): more comprehensive protocol with a wide range of functionalities for complex certificate management scenarios. SCEP is common in network industry. EST and CMP target very specific scenarios. We\u0026#8217;ll examine ACME as it\u0026#8217;s most relevant to the use case of web service. The biggest advocate of ACME is Let\u0026#8217;s Encrypt, a non-profit CA run by ISRG that provisions X.509 certificates at no charge. Let\u0026#8217;s Encrypt is the world\u0026#8217;s largest CA, aiming to secure all websites with HTTPS. ACME only issues DV certificates, since they can be fully automated.\nThe ACME Protocol The ACME protocol automates validation and issuance. The certificate requestor will have to use an ACME-capable client. The certificate provider (CA) needs to act as ACME server. At a high level, the flow looks like this:\nI came across this good diagram on the ACME flow from a post from small step. It has all the transactions in detail. As it shows, the delivery (issuance) of certificate material is based on HTTP POST method. The domain validation process is based on a challenge-response model. The ACME specification makes this an extension point, with the following most comment challenge types:\nHTTP-01 (HTTP Challenge): the domain in question needs to host a random number at a random URL under /.well-known/acme-challenge on port 80. The CA will fire an HTTP GET request to that URL. This is easy to configure because we usually have full control on the web server. There must be network connectivity between the web server and the CA to allow HTTP traffic. DNS-01 (DNS Challenge): the requestor provisions a TXT record with random value. The ACME server does not need to connect to the web server. It only needs to perform a DNS lookup to confirm the challenge. However, the certificate requestor needs the privilege to modify DNS record. TLS-ALPN-01 (TLS ALPN Challenge): ALPN is the protocol during TLS negotiation. The client presents a self-signed TLS certificate containing the challenge response as a special X.509 certificate extension. This challenge type is useful when a security policy requires the CA to reach the client via a TLS connection. DEVICE-ATTEST-01 (Device Attestation Challenge): This is for Apple Managed Device Attestation (ADA) and other secure zero-touch provisioning (SZTP) applications as part of your device management (MDM) strategy. Certificates identify specific hardware devices, via permanent device IDs. These are typically client certificates that can be used for device authentication. At the implementation level, Let\u0026#8217;s encrypt drives its public CA with Boulder. It supports two challenge types. When hosting a private CA, you can use Boulder too. Some feel Boulder is complicated and you can consider the following alternatives:\nLabCA: based on Boulder and supports hosting in docker. Step CA (open source): a simple CA solution Cert Manager: very popular choice on Kubernetes Hashicorp Vault: a secret management solution including certificate management capability with ACME support. You may combine different solutions for all level of CAs. For example, use Step CA for internal root CA, and Cert Manager for intermediate CAs for Kubernetes workloads. On the client side, Let\u0026#8217;s Encrypt recommends Certbot. However, there are many choices. Step CLI (by Step CA), acme.sh, etc. Let\u0026#8217;s Encrypt compiled a list here. If we\u0026#8217;re not seeking automation with ACME in our process, and just want to manually sign certificates, we can use generic tools (e.g. openSSL, cfssl, easyRSA, etc). They act both as client (gingnerate CSR) and server (signing CSR) using different command switches.\nRenewal and Revocation Lifecycle management involves renewal and revocations. Renewal is essentially re-issue certificates closer to expiration date. In software testing, we often use short-lived certificates, to ensure that our test scenario covers automated certificate renewal as well. It is the responsibility of requestor to initiate the renewal, and distribute the renewed certificates. With Let\u0026#8217;s Encrypt, the renewal process will challenge the requestor again for validation purpose. However, in some cases, the certificate provider may choose not to perform validation on every renewal. For example, short-lived certificate gets renewed every week, while validation is performed every year. During the renewal process, the private key of the website does not change. Note the difference between renewal and rekey. If the website\u0026#8217;s private key is compromised, then instead of renewal, we should re-issue a private key and request a new certificate.\nRevocation is a challenging process. To declare that a certificate should no longer be trusted, there are currently two ways: CRL and OCSP but both have drawbacks. CRLs are lists of all the certificates that a CA has issued but revoked. This list can grow very large. It is not feasible for the application (e.g. Browser) to download the giant list for each CA regularly and check for every website that matches the CA. OCSP provides a query-based method. The application can query the revocation status against the OCSP endpoint. It however brings its own challenges. The OCSP server is subject to downtime. The network connectivity between application and OCSP server causes latency. Many applications simply treats query timeout as not revoked. To reduce the load, application may cache OCSP responses, leading to potentially out-dated status. Worse, a malicious CA can track website of the application user.\nLet\u0026#8217;s encrypt has a page on these challenges, and it proposes a new browser-summarized CRLs. It was still a recent effort so we\u0026#8217;ll see how that plays out.\nSummary Following the first post on the PKI concepts, we discussed the automation of certificate issuance in this post. In the next one, let\u0026#8217;s go over some labs. Previous PostPublic Key Infrastructure 1 of 3 – Basics Next PostPublic Key Infrastructure 3 of 3 – PKI Implementation ","date":"2024-03-08T00:14:00-04:00","image":"/wp-content/uploads/2025/04/feature-pki-2-1.webp","permalink":"/2024/03/public-key-infrastructure-2-of-3-certificate-automation/","title":"Public Key Infrastructure 2 of 3 – Certificate Automation"},{"content":"In 2021, I wrote an intro to Public Key Infrastructure (PKI). Now that I have to host my own certificate authority, I decide to dive a little deeper into PKI in this series of posts. In software testing scenario, we need to issue (and recycle) a lot of certificates, and manage their lifecycle events such as (renewal, revocation). As a result, the corporate should establish its own private key infrastructure (PKI). This effort includes hosing their own Certificate Authority.\nConcepts IT professions often use the acronyms PKI and CA interchangeably in the context of implementation. However, strictly speaking, PKI is the entire framework (strategy, policy, etc) around managing certificate at corporate level, and CA is the specific entity that issues certificates. When it comes to architecting PKI, a key design decision is the hierarchy of CAs. Microsoft has a good page explaining the common options. In a single-tier CA hierarchy, the root CA is also issuing CA. The root CA as the anchor of trust of the PKI also issues certificates to the end entities. Obviously, this does not scale. It is only for sandbox testing, and not suitable for any environment that needs to scale. It also carries significant risk because the private key at the anchor of trust has to remain online.\nWith a two-tier CA hierarchy, the root CA only issues certificates to its subordinate CAs. Therefore the root CA can go offline. As a result, the subordinate CAs issue certificates to end entities. Since the root CA can remain offline (except for issuing certificates for new CAs), the chance compromise is reduced significantly. This option also scales better with multiple issuing CAs. If one of them gets compromised, then only the end entities of the compromised CA are impacted. The root CA still needs to be online from time to time. Apart from granting new subordinate CA, the team may also need root CA online to sign CRLs, which is a management overhead.\nRoot CA \u0026amp;\nIssuing CARoot CA\u0026#8230;Root CA\n(Offline)Root CA\u0026#8230;Issuing CAIssuing\u0026#8230;Issuing CAIssuing\u0026#8230;Issuing CAIssuing\u0026#8230;Root CA\n(Offline)Root CA\u0026#8230;Issuing CAIssuing\u0026#8230;Issuing CAIssuing\u0026#8230;Issuing CAIssuing\u0026#8230;Intermediate CA\n(Offline)Intermed\u0026#8230;Intermediate CA\n(Offline)Intermed\u0026#8230;Issuing CAIssuing\u0026#8230;Issuing CAIssuing\u0026#8230;Single-Tier\nCA ModelSingle-Tier\u0026#8230;Two-Tier CA ModelTwo-Tier\u0026#8230;Three-Tier CA ModelThree-Tier\u0026#8230;Text is not SVG \u0026#8211; cannot display\nThe two-tier hierarchy introduces another problem. The root CA also needs to restrict the certificates that its subordinate CAs can issue. For example, one issuing CA can only issue certificates in the *.dev.digihunch.com domain, and another CA can only issue *.ops.digihunch.com. No subordinate CA shall issue \u0026#8220;rogue\u0026#8221; certificate beyond their authorized scope. In order to enforce issue boundaries on issuing CAs, RFC 5280 defines multiple ways to express constraints, such as basic constraints (including path length), name constraints, policy constraints, and EKU.\nIn a three-tier CA hierarchy, the top-level is still a root CA that stays offline. One level below, is a layer of CAs that also stay offline and we refer to them as intermediate CAs. Going down one more level down, there are the issuing CAs for end-entity certificates. Oftentimes, we use the intermediate CAs as policy CAs where we introduce restrictions to the subordinates. In this model, the root CA can remain offline nearly all the time because we can issue CRLs at the intermediate level. This is more flexible, but also more management overhead.\nImplementation Options The two-tier hierarchy is good in most scenarios but the three-tier hierarchy is also common for large organizations. The next consideration is implementation strategy. Take two-tier model for example. We can think about these options:\nImplement a self-managed PKI, with an internal root CA. The internal subordinate CAs are the issuing CAs. They are chained to the internal root CA. Implement a self-managed PKI, with its certificate purchased from a commercial CA. The internal subordinate CAs are the issuing CAs. They are chained to the external root CA, as the TPP (trusted third party). The external CA may or may not be a public root CA. Purchase certificates from a commercial CA that are chained to a public root CA Option 3 essentially delegates the PKI to a commercial provider. This is usually not a favourable option due to the hefty charge and minimal control. In option 2, when the third party is a public root CA, you can have all your certificates with public trust, although this is a pricey option too. Even if the third-party isn\u0026#8217;t a public root CA, there is still a benefit of delegating the management of Root CA to commercial provider.\nOption 1 is for use cases where we need a lot of certificates quick, for example, in agile development iterations. The certificates do not need trust beyond the organization, and the risk of root CA being compromised is manageable. In this option all CAs are private CAs. Option 2 on the other hand, is flexible in terms of trust boundary. You can just rely on the commercial provider as a TTP. If the TPP supports public root CA, you can issue certificates for public facing workload. In other words, you can choose either a private CA or a public CA. When we work with a private CA, we have to import the certificate to the trust stores of the organization.\nCACACASCASRA\nregion-2RA\u0026#8230;RA\nregion-1RA\u0026#8230;RA\nregion-3RA\u0026#8230;Certificate Authority Service:\n\u0026#8211; Holds the CA keys and certs\u0026#8211; signs CSRs from RAs\n\u0026#8211; trusts the RAs implicitlyCertificate Authority Service:\u0026#8230;Registration Authority:\n\u0026#8211; Authenticates requests\u0026#8211; Relays CSRs and Certs\u0026#8211; Connect to CASRegistration Authority:\u0026#8230;local\nclientslocal\u0026#8230;local\nclientslocal\u0026#8230;local\nclientslocal\u0026#8230;Text is not SVG \u0026#8211; cannot display\nSome PKI topologies splits a CA into two sub-components: the Certificate Authority Service (CAS, or simply CA) and the Registration Authority (RA), as the diagram above shows. With many locations, each location has an RA that communicates with a with a central CAS. The RAs receive requests from local clients and are responsible for authenticating these requests and pass the validated ones along to the CAS. The validation can take place in ACME protocol. The CAS implicitly trusts RAs and will sign the validated requests from RAs before sending them via API calls. The RA then relays the certificates back to the local requestors. This topology is more scalable with the CAS focusing only on signing and the RAs on validation and passing the requests along. Large organizations may have their central CAS hosted on-prem, and remote RAs in the CSP regions.\nComponents for X.509 Certificate X.509 is the standard for digital certificate. The X.509 standard has the following most important fields:\nSubject: The name of the subject (e.g. a user, service, device), commonly represented as X.500 formate distinguished name (DN). For website, the value can be CN=digihunch.com Serial Number: A unique identifier for each certificate that a CA issues. Issuer: DN of the CA. For a self-signed root CA, the issuer is the subject. They are different otherwise, such as in subordinate CA certificates and end entity certificates. These fields were introduced in version 1. In addition, X.509 version 3 certificates introduces extensions that provide additional functionality and features to the certificate. Each extension comes in two flavours: critical and non-critical. A certificate-using system MUST reject the certificate if it encounters a critical extension it does not recognize or a critical extension that contains information that it cannot process. A non-critical extension MAY be ignored if it is not recognized, but MUST be processed if it is recognized.\nX.509 CertificateVersionVersionCertificate Serial NumberCertificate Serial NumberCertificate Algorithm Identifier for Certificate Issuer\u0026#8217;s SignatureCertificate Algorithm Identifier for Certificate Issuer\u0026#8217;s\u0026#8230;IssuerIssuerValidity PeriodValidity PeriodSubjectSubjectSubject Public-Key Information (Algorithm Identifier and Value)Subject Public-Key Information (Algorithm Identifier and\u0026#8230;Issuer Unique IdentifierIssuer Unique IdentifierSubject Unique IdentifierSubject Unique IdentifierExtensionsExtensionsCertificate Authority\u0026#8217;s Digital SignatureCertificate Authority\u0026#8217;s Digital SignatureExtension Fields \u0026#8230;Extension Fields \u0026#8230;OptionalOptionalOptionalOptionalSubject Alternative Name (SAN)Subject Alternative Name (SAN)Key UsageKey UsageBasic ConstraintsBasic ConstraintsName ConstraintsName ConstraintsCRL distribution Points (CDP)CRL distribution Points (CDP)Authority Information Access (AIA)Authority Information Access (A\u0026#8230;Subject Key Identification (SKI)Subject Key Identification (SKI)Authority Key Identification (AKI)Authority Key Identification (A\u0026#8230;Extended Key Usage (EKU)Extended Key Usage (EKU)Certificate PoliciesCertificate PoliciesCommon ExtensionsCommon ExtensionsText is not SVG \u0026#8211; cannot display\nHere are some common extensions that a lot of implementations use:\nSubject Alternative Name (SAN): only used in end entity certificates, not in CA certificates. The format of SAN is flexible and it does not have to be X.500 DN. For website certificate, we often place alternative DNS names here. Key Usage: The intended scope of usage for a private key is specified through the Key Usage and Extended Key Usage (EKU) extensions in the associated certificate. Example: \u0026#8220;Certificate Sign, CRL Sign\u0026#8221; Basic Constraints: Used to distinguish between end-entity cert and CA cert. You should also specify path length. The value can be: \u0026#8220;CA:TRUE, pathlen:1\u0026#8221;. If the path length constraint is 0, the CA may have one more level of subordinate CA. But these subordinate CAs must have path length of 0 on their own certs, and cannot extend one more level as their own subordinates. Those subordinate CAs can only issue end-entity certificates. If the path length value is none, then there is no restriction on the levels of subordinate CAs. Name Constraints: for CA certs only, defined in RFC5280, to limit the scope to certain names on the certificate that the CA issues. Client must verify that a certificate is allowed to be signed by CA. CDP (CRL distribution points): URL(s) where the application or service can retrieve the certificate revocation list (CRL). AIA (Authority Information Access): URL(s) where the application or service can retrieve the revocation list for CA\u0026#8217;s certificate SKI (Subject Key Identifier): the SHA-1 hash of the subject\u0026#8217;s public key AKI (Authority Key Identifier): the SHA-1 hash of the issuer\u0026#8217;s public key EKU (Extended Key Usage, also Enhanced Key Usage): an object identifier (OID) for each application or service a certificate can be used for. It needs to align with Key Usage Certificate Policy: reference to the certification practice statement (CPS) of the issuer. During exchange, any relying party can access the assurance level associated with the certificate, and decide on the level of trust to put in the certificate. Policy Constraints: for path validation, it can be used to prohibit policy mapping or to require that each certificate in a path contain an acceptable policy identifier. Policy Mappings: in CA certificates to restrict the certificates that the CA can issue. This isn\u0026#8217;t a complete list. Plus, X.509 V3 also supports custom extensions. Here is a sample certificate. In real life, the way each implementation uses these extensions may vary slightly, so interoperability issues between PKIs exist. When designing your own PKI, it is important to examine the usage of extensions to comply with the standard. For example, Microsoft has the following recommendations to restrict certificates: For subordinate CA certificates, the Basic Constraints extension should be present and marked as critical The cA field should be set to TRUE The pathLenConstraint field should be set to the minimum value required to enable the business scenario (i.e. 0 if that CA will issue certificates only to End Entities) The EKU extension should be present and contain the minimum set of EKU object identifiers (OIDs) to enable the business scenario. Furthermore, the anyExtendedKeyUsage OID (2.5.29.37.0) should not be specified. Summary This post discussed the basic concepts in public key infrastructure. In the next post, I\u0026#8217;ll cover the automation of certificate issuance.\nPrevious PostWorkload Identity on Kubernetes 2 of 2 – EKS Next PostPublic Key Infrastructure 2 of 3 – Certificate Automation ","date":"2024-02-20T00:05:00-04:00","image":"/wp-content/uploads/2025/04/feature-pki-2.webp","permalink":"/2024/02/public-key-infrastructure-1-of-3-basics/","title":"Public Key Infrastructure 1 of 3 – Basics"},{"content":"I discussed in my previous post on workload identity and dived into how it works in AKS (Azure Kubernetes Service). In this post I will continue the topic with AWS as the example. From the perspective of CSP, we consider any running process on the cloud resource as workload. Therefore, I\u0026#8217;ll start with control plan and node identities. From the perspective of a Kubernetes platform, the term workload mostly refers to applications running in Pods. So later in this article I\u0026#8217;ll distinguish two mechanisms for Pod Identity: IRSA and EKS Pod Identity. EKS Control Plane and Node Identity AWS directly associate an IAM role with EKS control plane and an IAM role with each node group. We don\u0026#8217;t need an extra step of assigning a \u0026#8220;managed identity\u0026#8221; (as in Azure) to a cluster or to a node group ( and then bind a role to the identity). You can find this pattern from Terraform code. Each aws_eks_node_group resource has a node_role_arn attribute to link to its IAM role, and a cluster_name attribute to link to the cluster. Each aws_eks_cluster resource has a role_arn attribute for cluster\u0026#8217;s permission. The cluster\u0026#8217;s IAM role is usually bound to managed policies like AmazonEKSVPCResourceController and AmazonEKSClusterPolicy. The IAM role that is assigned to the node group is the exact IAM role of the instance profile of each node. The kubelet process on the nodes are the main users of this role and the permission should not be broader than what it needs to do. This role usually have a few managed policies such as AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonSSMManagedInstanceCore and AmazonEC2ContainerRegistryReadOnly. The node role applies to self-managed node and managed node. When using Fargate to provide computing capacity, each Fargate profile will use its own IAM role, to connect to the cluster and pull container images. This IAM role is known as Pod Execution Role. For a private cluster, the place to run the command would be a bastion host with connectivity to the cluster\u0026#8217;s API endpoint. Refer to this post about the connectivity to private cluster.\nIAM Role for Service Account (IRSA) When AWS launched EKS in 2018, Kiam was a popular open-source project to grant Pods access to AWS resources. In 2019, AWS introduced the official mechanism, IRSA (IAM Role for Service Account). IRSA ties a Kubernetes identity (in the form of Service Account) to an IAM role in AWS. IAM allows creation of web identity based on OIDC. EKS can act as an OIDC issuer. This requires a few points of configurations, via the cluster API and via cloud the endpoint. The eksctl utility makes it simple with two commands:\n$ eksctl utils associate-iam-oidc-provider \\ --cluster $CLUSTER_NAME \\ --approve $ eksctl create iamserviceaccount \\ --cluster=$CLUSTER_NAME \\ --namespace=kube-system \\ --name=aws-load-balancer-controller \\ --role-name AmazonEKSLoadBalancerControllerRole \\ --attach-policy-arn=arn:aws:iam::112233445566:policy/AWSLoadBalancerControllerIAMPolicy \\ --approve The first command creates an OIDC web identity integrated with the EKS cluster, and the second creates a Service Account in Kubernetes and links it to the identity, and assign an IAM policy. These two commands must run under certain conditions. The AWS CLI identity for first command requires the the permission to add OIDC provider. The second needs the permission to create an IAM role. In addition, it requires kube API access to the cluster. So the command needs to run from an environment that can access both the cluster\u0026#8217;s API and AWS API. The IAM identity provider is somewhat similar to a managed identity with OIDC federated credential in Azure. However, unlike managed identity, here in AWS we cannot create the OIDC identity until after the cluster creation. In other words, the OIDC web identity\u0026#8217;s lifecycle does not decouple with the cluster lifecycle. We have to create a new web identity every time we create a new EKS cluster. In large organizations, the permission to create a new web identity is highly restricted.\nEKS Pod Identity There are a few other limitations with IRSA. As this blog post suggests:\nFurther, cluster administrators have to update the IAM role trust policy each time the role is used in a new cluster during scenarios like blue-green upgrades or failover testing. Additionally, as customers grow their EKS cluster footprint, due to the per cluster OIDC provider requirement in IRSA, customers run into the per account OIDC provider limit. Similarly, as they scale the number of clusters or Kubernetes namespaces in which an IAM role is used, they run into IAM trust policy size limit, which makes them duplicate the IAM roles to overcome the trust policy size limit.\nAWS brings the new mechanism \u0026#8220;EKS Pod Identity\u0026#8221; at reInvent 2023. In this mechanism, user can hook up an IAM role directly to a Kubernetes service account, without having to resort to a web identity and OIDC integration. Users just need to create a Pod Identity Association, using the CreatePodIdentityAssociation API, with the following parameters:\nCluster name Namespace ARN of the IAM role serviceAccount Both AWS CLI and ekscli already support the CreatePodIdentityAssociation API. Before creating a Pod Identity Association, we need to install the add-on \u0026#8220;Amazon EKS Pod Identity Agent\u0026#8221;, and ensure that the node roles have the permission. That is because the agent needs to use AssumeRoleForPodIdentity API. We also need an IAM role, with the trust policy principal being \u0026#8220;pods.eks.amazonaws.com\u0026#8221; and our own choice of resource tags as condition. Note that another implicit prerequisite is that the programming running in the Pod use a newer version of AWS SDK to access cloud resource. This blog post has good details, including a diagram and a walk-through. Comparison Both EKS Pod Identity and IRSA are here to stay. I\u0026#8217;m afraid this is going to create confusions. I put the following table for their comparision:\nIRSAEKS Pod IdentityPros\u0026#8211; in use since 2019\n\u0026#8211; support EKS, EKS-A, ROSA\n\u0026#8211; support all EKS versions\u0026#8211; support role session tags\n\u0026#8211; no dependency on OIDC identity provider\n\u0026#8211; create an IAM role once for all clusters. the role can be created before cluster\n\u0026#8211; cross account access through resource policies and chained AssumeRole operationCons\u0026#8211; Cannot create OIDC identity provider, until the cluster is ready\n\u0026#8211; One OIDC provider per cluster, with the risk of hitting quota\n\u0026#8211; Trust policy sprawl as more clusters are created\u0026#8211; the program has to use newer version of SDK. \u0026#8211; ony support EKS\n\u0026#8211; Pod Identity Agent (DaemonSet) can\u0026#8217;t run on FargateCompairing IRSA with EKS Pod Identity The blog post also contains a long table for their comparison. In the near future, I will have to check the SDK version of a workload in order to assess whether EKS Pod Identity will function. This is a restriction because it depends upon software builder disclosing the SDK version used. The EKS cluster also needs to host daemonSet on a node agent. On the other hand, go with IRSA if portability between EKS and EKS-A and ROSA is of concern, because the IAM service principal pods.eks.amazonaws.com is dedicated to EKS. The blog post also gives the migration step as follows:\nEnsure EKS cluster is above 1.24, and install the add-on for EKS pod identity agent. Ensure the SDK running in pod meets the version requirement. Update the IAM role\u0026#8217;s trust policy with the new principal \u0026#8220;pods.eks.amazonaws.com\u0026#8221; So the EKS Pod Identity mechanism still requires an IAM role. It does not required an OIDC identity. The service account connects to IAM role via an agent on the node. Summary A good design concerns not only functionality, but also streamlined configuration experience. EKS Pod Identity is a great improvement over IRSA heading the right direction. It just came out two months ago so still too early to adopt, especially without knowing the workload details. For now I tend to use pod identity as a backup mechanism when IRSA isn\u0026#8217;t available for some reason. However, I recommend starting to introduce the Pod Identity mechanism for all new EKS clusters and new workloads.\nPrevious PostWorkload Identity on Kubernetes 1 of 2 – AKS Next PostPublic Key Infrastructure 1 of 3 – Basics ","date":"2024-01-08T00:05:00-04:00","image":"/wp-content/uploads/2025/04/feature-workload-identity-eks.webp","permalink":"/2024/01/workload-identity-on-kubernetes-2-of-2-eks-and-rosa-on-aws/","title":"Workload Identity on Kubernetes 2 of 2 – EKS"},{"content":"As applications are moved to the cloud, the application workload hosted on virtual machines need to interact with cloud resources. For this, we need an IAM solution with two mechanisms:\na (non-human) identity in the cloud service platform (CSP), to represent the application; a way to grant permission to this identity, so it can manage resources CSPs such as Azure and AWS have their own implementations of the two mechanism. In Azure, we have Entra workload identity (including service principal and managed identity) for the first mechanism, and Azure roles for the second. On AWS, they are the identity pool capability of Amazon Cognito and IAM role. Next, what about the workload on managed Kubernetes service? Essentially, we will need to more mechanisms:\na native Kubernetes identity to represent the workload (Pod); a way to map the Kubernetes identity to the identity in CSP Kubernetes Service Account is designed for the first item. The second mechanism is for the CSP to address. In this post, let\u0026#8217;s examine this in Azure. Specifically, how does Azure manage workload identity with Azure Kubernetes Service (AKS). Node Identity and Cluster Identity in AKS Let\u0026#8217;s define what exactly is a workload identity. In Azure we think of it as one type of non-human identity. In our context, workload identity in the broader sense contains:\nthe identity that represents the control plane (or the whole cluster) the identity that represents the node (or kubelet process) the identity that represents the application in a Pod (workload identity in the narrow sense); So it is important not to confuse these identities. In this section, I\u0026#8217;ll focus on 1 and 2 since they are part of workload identity in the broad sense. In the rest of the sections, I\u0026#8217;ll discuss 3, and use the narrow sense of workload identity. When we create an AKS cluster, we create both a cluster control plane and a node pool. Both the control plane and the nodes need to provision cloud resources using Cloud API from Azure. For example, if we use Terraform\u0026#8217;s AzureRM provider to create an azurerm_kubernetes_cluster resource, then we specify the cluster\u0026#8217;s identity using service_principal or identity block. We specify the nodes\u0026#8217; identity using the kubelet_identity block, because kubelet is the process that runs on each node. Even though a cluster builder might be tempted to assign the same identity to both Control plane and kubelet, the security best practice is to keep them separated. It is also the responsibility of the cluster builder to distinguish activities by the control plane and by kubelet process on each node, and attache an Azure Role with minimum privilege to each of the identities.\nThese two types of identities (control plane and kubelet) are relatively straightforward. In order to use them, we don\u0026#8217;t have to play with Kubernetes objects. In the next section, we\u0026#8217;ll continue to discuss the identities that represents each Pod in Azure. We now refer to them as workload identities, but the first available technology was pod managed identity.\nPod Managed Identity in AKS When I first worked on Azure Kubernetes, Pod managed identity was in preview and was the recommendation. However, Microsoft renamed it (to Microsoft Entra pod-managed identities) and then deprecated it after a couple years of preview. As of Oct 2022, the recommended mechanism becomes Microsoft Entra Workload ID. For simplicity, we refer to the deprecated mechanism as \u0026#8220;Pod Identity\u0026#8221;. We discuss pod identity only for the purpose of understanding why it is no longer recommended and what is missing in it. For new workload deployment, we should always use workload identity. For Pod Identity to work, a feature flag EnablePodIdentityPreview must turn on. Pod Managed Identity operates on a Kubernetes controller called MIC (Managed Identity Controller) and a DaemonSet called NMI (Node Managed identity). You start with an Azure managed identity with appropriate roles. Once you installed Pod Identity, there will be two CRDs AzureIdentity and AzureIdentityBinding. To grant Azure permissions to a Pod, you create a CR for each CRD. The AzureIdentity CR connects to your Azure managed identity. You also create an AzureIdentityBindign CR. When declaring a Pod, you link to AzureIdentityBinding by using the label aadpodidbinding. There are two problems with pod managed identity. First, there is a vulnerability when it works with kubenet as network plugin. This vulnerability requires an additional mitigation step. Second, it does not make use of Kubernete\u0026#8217;s Service Account. Let\u0026#8217;s discuss in the next section why it\u0026#8217;s favourable to use Kubernetes\u0026#8217; ServiceAccount. Kubernetes Service Account In Kubernetes RBAC model, Service Account can bind to Roles to gain access to other Kubernetes resources. The most common use case is allowing the running application in a Pod to access other Kubernetes resources. When it comes to letting an application in Pod access cloud resources in the CSP, it makes sense to use Service Account, for a a consistent pattern.\nA service account must carry a token to function. Each namespace has a default service account with the token mounted automatically. Each Pod created in a namespace uses the default service account of the namespace, unless otherwise specified. However, many security organizations do not considered this default behaviour as the best practice. For example, CIS Kubernetes benchmark 1.8 has these two recommendations:\nEnsure that the default service accounts are not actively used (5.1.5) Ensure that Service Account Tokens are only mounted where necessary (5.1.6) In other words, we should create non-default service account with automountServiceAccountToken set to false. Then when declaring a Pod, we explicitly specify the service account and where to grab the token for the service account. One way to pass ServiceAccount token is through volume projection. To allow a Pod to access Azure resources, we use the combination of Kubernetes Service Account and Microsoft Entra workload identity. Workload Identity for AKS Microsoft introduced Entra Workload Identities in late 2022 to address IAM issues around machine identities. It comes with some modern features such as conditional access (e.g. location-based access, anomaly sign-in detection, etc). A workload identity can be:\napplication: an abstract entity as the global representation of your application for use across all tenants; service principal: the local representation of a global application object in a specific tenants; managed identity: a special type of service principal that eliminates the need for developers to manage credentials In our use case for AKS workload, we also make use of Azure\u0026#8217;s Managed Identity. This part is the same as the pod identity mechanism. However, here we create a federated identity credential for managed identity. The OIDC federated identity credential is issued by the AKS cluster. Within the AKS, the service account references the identity by client_id. Here is the documentation for the whole process.\nOne of the improvements in Entra workload identity for AKS, is the use of service account, which obviates the use of CRDs. Another improvement is the use of federated identity, whose lifecycle is tied to the cluster. This pattern is not only neater, but also standard. We map a service account to a managed identity with federated credential. Summary On managed Kubernetes services, we need an integration mechanism to grant Kubernetes workload access to cloud resources. We discussed what\u0026#8217;s needed in this integration mechanism and looked at Azure Kubernetes as an example. In the next post, we\u0026#8217;ll discuss how this issue is addressed in Elastic Kubernetes Service on the AWS side.\nPrevious PostWordPress Security Basics Next PostWorkload Identity on Kubernetes 2 of 2 – EKS ","date":"2023-12-23T16:43:00-04:00","image":"/wp-content/uploads/2025/04/feature-workload-identity-aks.webp","permalink":"/2023/12/workload-identity-on-kubernetes-1-of-2-aks/","title":"Workload Identity on Kubernetes 1 of 2 – AKS"},{"content":"Background In 2019, I moved this site to WordPress hosted on an Amazon Lightsail instance. There were few visits at that time so I lived with the single-server architecture. The website traffic has since been in steady growth but I have been too busy to catch up with the WordPress security setup. In July 2023, a malware impacted this site as well as the web traffic. It took me several months to fix a few related issues but the traffic still has not fully recovered. This post is about the lessons.\nThe Incident I first noticed the issue when I clicked on links to my web page from Google result and got redirected to some spam site. It did not happen 100% of time, but it is annoying enough. In the mean time, from Google search analytics I noticed traffic volume going up with a lot of traffic going to URLs that I did not recognize or create. Somehow these URLs have a lot of clicks and impression counts. These are signs of artificial traffic.\nObviously the site was hacked. The first thing to determine is whether the server access was compromised. From the audit log (/var/log/auth.log and auth.log.gz) I can see a lot of brute force attempts to connect but fortunately none was successful. That also prompt me to change the default SSH port and use ECDSA key pair. Since the OS access is safe, the hack happens at the WordPress level. I suspected the sideloaded plugins from a few days ago. So I immediately removed all sideloaded plugins. The attack is called malicious redirect. The plugin puts creepy pages in WordPress directory without my awareness and direct user traffic via my website. To clean up the damage, I looked into my WordPress directories at /opt/bitnami/wordpress and found many suspicious signs:\nThere are directories with weird names, such as rexall-vitalmin, or q4lee3, etc In each of those directories there was an index.php file and .htaccess file; Those directories also have other files which look like red herrings; All those files have the same date time (from July 6); Other directories to look at are /bitnami/wordpress/wp-content/plugins, where I noticed two directories (named gokyfozaxy and q199n071) that are not accounted for; and /bitnami/wordpress/wp-content/themes/, which contains unknown directories.\nClean up and hardening In one of the .htaccess file I noticed segments of mojibake (garbled texts). I first tried to manually remove those files, but the problems stayed. Because the malicious redirect did not happen consistently on every single click, I sometime had false impression that the problem went away. However, the challenges with manual cleaning are: 1. there are too many bad files (.htaccess and php files containing mojibake segments); 2. some existing files are impacted with mojibake segments too. I found a free plugin called WordFence to scan the file directory for malicious chagnes, and delete the bad files or bad segments. I also tried a paid scanner (Malcare) which found an bad file in /bitnami/wordpress/wp-content/themes/. However, it also blocked my site so I removed Malcare right away. Using the combination of WordFence and Malcare appears to have cleared up the offending files. After restarting apache, the bad URLs are no longer redirecting to spam sites.\nThis time, I decided to harden the WordPress system given the evidence of brute force attack at different point of entries. At OS level, I mentioned the changes to SSH daemon configuration. At WordPress level, I used WordFence to perform several levels of scans for problems and and improved posture such as admin user\u0026#8217;s MFA. I also noticed a few unrecognized wordpress users and used wordpress CLI to delete those and other unused users.\nThe wordpress.org website has some general guidance on what to do when a site is hacked, and a general guidance on hardening WordPress.\nBack Links Another clean up work I had to do is dealing with back links. Back links are URLs from other sites that references this site. There are several situations:\nIf it\u0026#8217;s a made-up URL, then it returns 404. In my case, these are URLs that stopped working once I cleaned up my server from the incident. However, the sources are still using these bad URLs. They are bad back-links; If it\u0026#8217;s a legit URL, look at if it\u0026#8217;s hot linking, such as another site directly access an image from my site. These are bad back-links; If it\u0026#8217;s legit URLs, and the referrer site has a good domain authority score. These are likely to be good back links Generally, it is painful to deal with bad back links because I\u0026#8217;m not in control. I used a few free backlink checker tools (e.g. Links report on Google Search Console, SEOMATOR, SEMRush free) and found a lot of spammy sites that I had to request Google to disavow. Otherwise, they may negatively impact the search performance.\nRepercussions In the next few months, my pages are no longer a stop for their redirect. However, web request for those invalid URLs keep coming. The bad pages are still in Google\u0026#8217;s cache. There are a lot of page request with 404 return code, and we consider this an HTTP flood. The problem now is that the amount of 404 return code is impacting how my site ranks in search engine. To make it worse, the amount of these requests with invalid URL increase since August. To fix this, there are two measures. First, in Google search console, I have to tell Google to remove those URLs from its cache. I have identified a number of prefix patterns, and submitted a request for each URL pattern. It takes google a day to have them cleared. After that, the bad request will no longer come from Google users clicking on bad URL. In my case, the requests did not reduce significantly, suggesting that most of the requests come from bots. Therefore I had to figure out a way to prevent those bad request hitting my server, which is a typical web application firewall requirement. Looking for such a solution for my WordPress Security I landed on Cloudflare. Cloudflare is pretty user-friendly with an easy-to-understand reference architecture. When I started, Cloudflare can import my DNS records, and guided me to change my name servers so I delegate my DNS management it. When I first move to Cloudflare the website gives ERR_TOO_MANY_REDIRECTS. I ended up having to go to SSL/TLS and set encryption mode to Full (strict) to get rid of this error. I also have to re-configure email forwarding as a result of name server change.\nCloudFlare Even for a self-hosted single-server site, it is very beneficial to place an Application Firewall upfront for WordPress security. I find CloudFlare are very useful service that provides everything else you need to host the web site. For example, it contains a domain registry itself. It manages DNS and allows email forwarding. In addition, it helps generate TLS certificate etc. The free tier covers everything for a small website, with the Application Firewall as the core feature. Within the free tier I can have these features:\ndomain registrar and name servers (not for free but at a reasonable cost) SSL certificate (not for free but at a reasonable cost) Request event tracking redirect rule: zone apex to www, and /status to uptime status page return code 409 for obsolete URLs (using routes and workers) email routing and forwarding WAF rules (path, parameter, rate, etc) DDoS protection and Bot Fight mode hot-linking prevention (i.e. other sites references images on your site directly) I am still exploring features for CloudFlare. One stunning feature is routes and workers. Essentially you can serve a function in response to HTTP request at a specific route. This is particularly useful in scenarios where it is not straightforward to add web pages on the backend server. For example, I want requests with certain paths to return HTTP code 490 and do not want to mock with the WordPress server, we can make use of CloudFlare worker.\nLessons Learned For WordPress security, never use suspicious plugins. Keep an additional layer of defense in WordPress such as Wordfense. It helps block malicious traffic that went through the first layer. It also helps configure MFA for administrators. Previous PostAWS Systems Manager is an Omnipotent Hodgepodge Next PostWorkload Identity on Kubernetes 1 of 2 – AKS ","date":"2023-11-17T12:02:00-04:00","image":"/wp-content/uploads/2025/04/feature-wp-basic.webp","permalink":"/2023/11/wordpress-security/","title":"WordPress Security Basics"},{"content":"Introduction to Systems Manager AWS Systems Manager addresses a lot of SysOps requirements for configuration management, including server automation. In this domain, there is another AWS service called OpsWorks. However, with OpsWorks Stack, OpsWorks Chef and OpsWorks Puppet all coming EOL in 2024, the entire OpsWorks service is mostly deprecated. By partnering with leaders such as Chef and Puppet, OpsWork services represent the era when AWS needed to mirror the configuration management capability on premise, in an effort to convince customers migrating to the cloud. Today, AWS Systems Manager has evolved to fill a lot of gaps around configuration management for servers in the cloud. Although AWS Systems Manager sounds like a single service. It consists a collection of many seemingly disparate capabilities that serves similar requirements around configuration management. In fact, many of the Systems Manager capabilities are built on top of a couple of what I call core capabilities, such as Session Manager, RunCommand, Automation. This post will review these core capabilities and how Systems Manager employs them to expand with other capabilities.\nSSM Agent and Session Manager What enables all other capabilities is the SSM agent installed on the EC2 instances. The agent running as a systemctl task by ssm-user on EC2 instances. Most of AMIs come with this agent pre-installed. It stores the logs in /var/log/amazon/ssm/. This agent works with an instance profile with a role with the AmazonSSMManagedInstanceCore managed policy, in order to communicate with AWS Systems Manager (ssm.\u0026lt;region\u0026gt;.amazonaws.com) backend. Because of that, you also need to provide a network path to the backend endpoint, either via Internet, or interface endpoint. This communication also allows an IAM user to connect to an instance\u0026#8217;s shell. A common use case is for private instance that do not have Internet access but do have access to SSM backend endpoint. In a previous post I discussed using Session Manager to replace a bastion host to connect to EKS nodes. When launching an instance using an AMI with SSM pre-installed, the SSM agent should launch after all the config sets from Cloudformation Init are finished. As a result, the Cloudformation Init script is not able to communicate with SSM backend via the agent, unless you install and start SSM agent first on your own, in CloudFormation Init. To troubleshoot SSM, it is important to review its logs.\nThrough Systems Manager Hybrid Activation, the SSM agent can also work on virtual machines out of AWS and report back to with SSM backend. This gives on-prem servers the identities (instance tags, instance profiles) required for Systems Manager to manage them as if they were EC2 instances. As a result, extend Systems Manager capabilities to on-prem fleet (requiring advanced instances tier).\nTypes of SSM Documents There are several types of document that SSM uses, including:\nCommand Document Automation Document Package Document Session Document Policy Document Change Calendar Document The AWS documentation has a table on what they each are for. Here I\u0026#8217;ll focus on three types of documents: Command Document, Automation Document and Session Document.\nThe Command Document is for the RunCommand capability. It executes on EC2 instances usually performing tasks relating to the operating system or application. I think of a Command Document as an Ansible Playbook that consists of Ansible tasks. We can author Command document that runs configuration steps using plugins, such as aws:downloadContents, aws:runShellScript, etc. This feature directly competes with Ansible. To troubleshoot why a command fail on an instance, check the file ssm-document-worker.log in the ssm agent log directory. Each log entry should have a command ID as reference.\nThe Automation Document (aka runbooks) is for the Automation Capability. You can define sequence of actions for automation. There are many pre-defined actions such as executing AWS API calls (aws:executeAwsApi), run commands (aws:runCommand), or executing a Lambda function. Therefore a runbook requires an IAM role (Automation Role). The schema of action sequence (YAML or JSON) looks very similar to an Ansible playbook. The web console comes with an UI to visualize the action sequence but most of the time I\u0026#8217;d rather . Session Document is for Session Manager capability. AWS Systems Manager Session Manager uses Session documents to determine which type of session to start, such as a standard session, a port forwarding session, or a session to run an interactive command. In most cases, automation developers do not need to create their own Session document, because the pre-built ones are sufficient:\nAWS-PasswordReset AWS-StartInteractiveCommand AWS-StartPortForwardingSession AWS-StartPortForwardingSessionToSocket AWS-StartSSHSession In my experience, I use the AWS-StartSSHSession and AWS-StartPortForwardingSession documents most often. To establish SSH connection for forward port to connecting host for Remote Desktop session. To author your own document, reference the schema correctly and use the latest SSM agent. However, I would explore if any existing shared document in the library already covers what you need. For example, the command document AWS-JoinDirectoryServiceDomain help join a Windows server to a managed Active Directory domain. The command document AWS-RunPatchBaseline is used by Systems Manager Patch Manager capability to check and apply operating system patches. They include steps for Windows, MacOS and Linux instances. The automation runbook AWS-AttachIAMToInstance helps you add IAM role to an EC2 instance. RunCommand and Automation The Run Command capability run on top of SSM agent. You can specify one or more target instances. You also specify other other options such as command parameters, rate control and where the output goes. This capability allows an IAM user to run command directly on the OS of an instance (using an OS user ssm-agent) and centrally keep track of those command runs on the AWS side. The most common commands to run on the OS is packaged into Command Documents. There is even a Command Document that allows you to run a pre-built Ansible playbook. Another way this capability is extremely helpful, is that we can reduce the load of cloud init process. Traditionally, we put a log of logics in the user data script for the cloud init process to execute. The cloud-init mechanism comes from Linux OS and the execution of the user data script is not very transparent to troubleshoot. You have to check the cloud-init-output log from the OS. The use of the UserData script should be reserved for establishing communication with CloudFormation endpoint and SSM endpoint. From there, other automation tasks should be done using SSM capabilities (e.g. State Manager) for better manageabilities. Take an example of joining a newly provisioned Windows server to a domain. If we do this in user data script, we will have a few problems. First, we can only tell success/fail state from logs in the OS. Second, if an OS user inadvertently removed the instance from domain, there is no mechanism to capture that. If we use Systems Manager\u0026#8217;s RunCommand capability, along with State Manager association, the AWS management console will be able to tell whether domain joining is successful, and the association can detect when the instance is removed from domain, report this finding as out of compliance, and remediate the issue. We\u0026#8217;ll discuss State Manager in more detail in the next section.\nAs part of automation, we often have to invoke AWS API calls, which happens outside of any target VMs. The Automation capability of Systems Manager is for this scenario. You can orchestrate your API calls using Automation runbooks. These automation steps do not execute on any target EC2 instance, so they do not rely on SSM agent. However, it needs its own IAM role to perform API tasks. This capability saves you from having to run API calls by creating a new Shell environment to run AWS CLI, or from your own Lambda function using the boto3 SDK library. When we combine Automation and RunCommand capabilities, we can perform most of the automation orchestration steps. They are the core capabilities that further enable a variety of other Systems Manager capabilities.\nMaintenance Window and State Manager Maintenance window is a very straight forward capability to schedule RunCommand activity with a cron or rate expression. You can specify target by instance tags, define one or more tasks, and define a window of activity and at what point prior to the end of Windows should the agent stop performing more activities (cutoff). Each task can be a type of a RunCommand command, Step Function, Lambda function and automation runbooks.\nState Manager is a similar capabilities with a lot of feature overlap with Maintenance Window. State Manager operates on the concept of associations. An association connects target instances to command document or automation runbook to execute. Similar to Maintenance Window, you can specify a schedule expression, document parameters and instance tags. State Manager was brought in to combat configuration drift. The associated document should consist of idempotent scripts so that a State Manager association can repeatedly execute these documents to ensure compliance.\nMaintenance Window is more about scheduling one or more tasks. On the State Manager side however, association failure by default will be reported as out of compliance compliance. This is useful in scenarios such as keeping a Window instance in the domain, or keeping SSM agent up to date. You can choose either capability for many common setups but they have subtle differences. For example, for Patch management, you can use State Manager to detect missing patches and report compliance, and Maintenance Window to actually apply the missing patches. In fact, there is a document page on choosing between State Manager and Maintenance Windows to distinguish their best use cases.\nFleet Manager and Inventory Fleet Manager presents a centralized view for all instances for users to perform common administration tasks, such as exploring file systems and logs, admin users and groups, manage registry and events on Windows instances, check processes and performance metrics. It also gives shortcuts to patch nodes, run commands, start session, etc. I think of Fleet Manager as a minimalist configuration management UI. It is not as sophisticated as those from Ansible Tower or Puppet but it comes at no additional cost.\nA very useful feature of Fleet Manager is to run a web-based remote desktop to connect to Windows Instances. This saves the need for a bastion host as long as the instances have SSM connection. You will need the RSA private key to decrypt the Administrator password, which I would not recommend. If the Windows server is on a domain, you can enter your domain credential via Fleet Manager. If the users logged in via IAM identity center, Fleet Manager also has the login option for them via SSO using IAM Identity Center identity. When a user logs in this way, Fleet Manager uses RunCommand capability to execute AWSSSO-CreateSSOUser document against the server to create a local admin user.\nAnother aspect of configuration management is the inventory management. Unlike in Ansible, the term inventory in the context of Systems Manager refers to the metadata of instances, which includes installed applications, AWS components, network configurations, instance details, services, Windows registry and roles, etc. The full list of what is part of metadata is in the document and you can even define your own inventory item. To gather inventory data, we can makes use of a State Manager association to execute the AWS-GatherSoftwareInventory document. Once we set up the association, the agents will report inventory data back to Systems Manager. More importantly, we can create Resource Data Sync objects to write inventory data (along with compliance data) to S3 buckets, allowing downstream applications to consume. A common use case is to run Athena query against those bucket and produce QuickSight dashboard. Patch Manager and Compliance The Patch Manager also operates on State Manager associations. The automation runbook is AWSRunPatchBaseline, where you can just scan for missing patches or install them as well. The SSM document can run on all three platforms (Windows, Linux and MacOS) and determines which patches are missing relative to a the Patch Baseline. There should be at least one default Patch baseline. Each OS (e.g. Ubuntu, Debian, Amazon Linux, etc) classifies patches differently, and a patch baseline is a configuration that defines whether a patch is approved based on operating system and their classifications. The automation document also allows you to override the patch baseline. When executing the document to scan for patches, it records patch compliance information using the PutInventory API command. When using the document to install patches, you can run the document from a Maintenance Window and specify whether you need to reboot the target instance if required.\nThe compliance capability reports compliance status for instances. By default there are two types of compliance: association and patch. The association compliance detects whether a state manager association is failed on certain instances. The patch compliance, as just mentioned, checks whether patches are up to date relative to the specified patch baseline. You can also define custom compliance item (with put-compliance-items API) but the documentation isn\u0026#8217;t clear on what exactly it can achieve and where on the instance does it pull the compliance status. From the example in put-compliance-items, custom compliance type seems to check the installation of additional software package in the inventory.\nOther capabilities Amongst the other capabilities, the one I use the most often is parameter store, which is a way to store a variable for different services to consume. In the domain of change management, the change manager is a mini change management system. Organization can use it to manage their change process such as approvals. More importantly, you can fire automation runbook from change manager and tie it back to the change control item. Change calendar allows you to block changes during specific period. Both of them are organization level capabilities.\nWhen it comes to operations management, the Incident Manager capability allows you to create response plan for incidents. Response plan can execute runbook actions once an incident is logged. It also helps you notify the on-call incident response team. On the other hand, OpsCenter capability allows you to create OpsItem, which also includes a way to execute runbook. The OpsData can aggregate to Explorer, which is a centralized dashboard for operations data. The Explorer, OpsCenter and Incident Manager capabilities can operate at organization level. These capabilities around change management and operations management come nowhere close to full-fledged ITSM solutions such as ServiceNow or SMAX. However, they have the ability to trigger runbooks and natively integrate with other AWS services.\nThere is also a quick setup capability which uses pre-baked CloudFormation template to configure other services. For Patch manager the current recommendation is to use quick setup to configure patch policy.\nSummary Systems Manager has so many capabilities that I cannot cover everything in a single post. Here is a good walk-through. Some capabilities like session manager, fleet manager and state manager, are extremely helpful. However, in my opinion, there are two problems with grouping all these capabilities under Systems Manager. First, With too many different capabilities, this service lacks focus, which makes it difficult to learn. Second, some capabilities have overlap with other capabilities, or another AWS services, which also makes it confusing. I try to sort out how these capabilities enable each other in the diagram below: This diagram may not be 100% accurate but it demonstrate the dependencies and can assist troubleshooting. For example, when compliance is missing data, check the execution history of run command. It also illustrates the key role of SSM agent as the underlying enabler of most of the other capabilities.\nOverall, Systems Manager is extremely powerful. You can try to replace your server management solutions (e.g. Ansible, Chef and Puppet) with Systems Manager configurations. With a good understanding of its capabilities, you can build your fleet automation in an efficient and scalable way. Previous PostOrchestrate Landing Zone with Landing Zone Accelerator on AWS Next PostWordPress Security Basics ","date":"2023-10-29T21:32:49-04:00","image":"/wp-content/uploads/2025/04/feature-ssm.webp","permalink":"/2023/10/the-systems-manager-hodgepodge/","title":"AWS Systems Manager is an Omnipotent Hodgepodge"},{"content":"Virtualization enables multi-tenancy, and containerization takes it further. Container allows for running many more service processes. Container introduces another layer of orchestration, calling for a platform of its own, which is capable of managing the lifecycle of thousands of containers. This makes it a bit more work, to releasing and operate containerized applications, due to the container platform layer, sitting between the application and the operating systems.\nKubernetes Cluster Kubernetes has emerged as the de-facto standard of container platform. Building a Kubernetes cluster with a cloud service provider requires configuring a number of disjointed services to work together. A functional and scalable cluster is the foundation of a robust container platform. Make sure the design of Kubernetes cluster is solid.\nKubernetes Storage Kubernetes was designed around the idea that Pods are ephemeral and so are their attached storage volumes. Now Kubernetes supports persistent storage but there are many nuances to consider before landing on a CSI-based storage solution.\nKubernetes Networking Containerization favours microservice architecture. Cluster design needs to decides on a CNI to enable Pod-to-pod communication. Further, a container platform needs to address application networking requirement using network policy, service mesh or similar technologies.\nCloud Native Workload CNCF promotes its own ecosystem for cloud native workload. Teams who move their applications to Kubernetes platform often have to reconsider the associated toolings. These teams need extensive investigation of available cloud native toolings. More on Container Platform Host legacy application in Docker 2 of 2 - My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months... Use Case I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process.\u0026hellip; Host legacy application in Docker 1 of 2 - This is my notes from containerizing a legacy application with Docker compose. We have to run multiple instances of our application because we're unable to secure additional VMs for this single-VM education environment. The application is target of containerization, because it requires mass reconfiguration (around TCP port) to run multiple\u0026hellip; Contact Digi Hunch for Professional Services.\n","date":"2023-10-22T14:48:54-04:00","image":"/wp-content/uploads/2025/04/menu-container-platform.webp","permalink":"/container-platform/","title":"Container Platform"},{"content":"Cloud is a delivery model of computing services over remote network. This model is enabled by virtualization technology and features a pay-as-you-go pricing plan for computing services. Public cloud providers are equipped with virtually unlimited capacity and are operating a broad suite of managed services. In the design of a cloud platform, we look at the five pillars in the well-architected framework: operational excellence, security, reliability, performance efficiency, and cost optimization. Through these lens, we mainly look at these areas:\nCloud landing zone A successful cloud platform enables application teams to focus on business requirement. The backbone of a cloud platform is a landing zone, which typically addresses security, networking and compliance requirement of the organization\u0026#8217;s IT footprint in the cloud. Both AWS and Azure have guidelines of multiple options to deploy landing zones.\nStorage design Enterprise applications often have specific requirements on IOPS and throughput. Selecting a storage service in a cloud platform, must also consider the high availability, disaster recovery and cost efficiency. Networking design Networking design has profound impact on the security posture and must be well thought out. It sets the foundation of high availability and fault tolerance. Also, how traffic flows in and out the system significantly affect the cost.\nInfrastructure as code There have been three categories of infrastructure as code, those based on markup language (ARM, CloudFormation), those based on general-purpose programming language (Pulumi, AWS CDK), and those based on Domain Specific Language (Terraform, Bicep). They have different levels of flexibility and different skill requirement. More on cloud platform IAM Roles for any workload - Background A few month back a client of mine wanted to use GitLab pipeline to deploy infrastructure on AWS with Terraform. The key question is how to authenticate the Terraform process running in the pipeline to AWS with temporary credential. Having worked it out on GitHub, my proposal at time\u0026hellip; AWS Systems Manager is an Omnipotent Hodgepodge - Introduction to Systems Manager AWS Systems Manager addresses a lot of SysOps requirements for configuration management, including server automation. In this domain, there is another AWS service called OpsWorks. However, with OpsWorks Stack, OpsWorks Chef and OpsWorks Puppet all coming EOL in 2024, the entire OpsWorks service is mostly deprecated.\u0026hellip; Istio External Authorization via OIDC - Istio service mesh allows application developers to offload non-core features to infrastructure layer. We explored authentication and authorization with Istio in a basic lab. In this post we continue to explore its capabilities with OIDC integration. This capability is made available thanks to the CUSTOM action in authorization policy, supported\u0026hellip; AKS Lessons Learned 2 of 2 - Even though Azure Kubernetes Service (AKS) is a managed service, building a cluster is not trivial. For help resources, I would start with the webinar \"Configure Your AKS cluster with Confidence\" from April 2021, which focuses on a set of working best practices (convention over configuration) but obviously not every\u0026hellip; AKS Lessons Learned 1 of 2 - In general, troubleshooting Kubernetes is tricky. That is because one has to get in and out of pods. I took two days to troubleshoot some networking issues with private AKS cluster. For the amount of of tricks I had to employ, I need to take some notes. The issue After\u0026hellip; Contact Digi Hunch for Professional Services.\n","date":"2023-10-22T14:48:43-04:00","image":"/wp-content/uploads/2025/04/menu-cloud-platform.webp","permalink":"/cloud-platform/","title":"Cloud Platform"},{"content":"Hunch Digital Services delivers professional IT services in platform engineering, which focuses on non-feature requirements, typically summarized as the five pillars including security, reliability, performance efficiency, operational excellence and cost optimization. Specifically, we specialize in areas such as cloud platform, container platform, automation and security. Our service categories include:\nSolution Design Create a design or blueprint that meets your specific business requirement. The purpose is to ensure that the chosen solution aligns with your organization\u0026#8217;s goals, objectives, and constraints, and that it is cost-effective, efficient, and sustainable.\nArchitecture Review Evaluate a proposed solution design against a set of architectural principles, standards, and best practices to ensure that the system meets its requirements and is scalable, maintainable, and secure. The purpose of architecture review is to ensure that the system design aligns with the organization\u0026#8217;s overall business strategy and objectives, and that it can be easily modified or extended to meet changing needs. System Design Review Evaluates the current status of an implemented solution. We examine various aspects of the current system, such as technical infrastructure, logs, operation overheads, and interviews with focused groups to gather feedbacks from system users and other stakeholders. We also make recommendations based on architectural principals and best practices.\nProof of Concept Evaluate the design by produce a prototype in order to understand the functionality, performance or interoperability. PoC activity helps stakeholders make informed decision about whether to proceed with the full implementation of the solution or not. Implementation Put recommendations and solution designs into action to ensure that the deployed technology aligns with the design and meets the specific needs of the client. Workshop and Demo A series of meetings to immerse the targeted group in a specific technical topic. Workshops familiarize the audiences with the technical topic through presentations, walk-throughs and labs. In addition to training, workshops also help generate insights and ideas, identify potential solutions or approaches, and develop a shared understanding of the complex problem at hand.\n","date":"2023-10-22T14:48:23-04:00","image":"/wp-content/uploads/2025/04/menu-services.webp","permalink":"/professional-services/","title":"Services"},{"content":"As a continuation to the last post, we explore the Landing Zone Accelerator on AWS (LZA) as an orchestration tool in this post. LZA borrows a lot from the ASEA, an accelerator project to deploy the security reference architecture (SRA). LZA is a multi-purpose project that consists of both the orchestration engine (the accelerator itself) and a few reference architectures (as configuration files).\nComparison with Control Tower First, let\u0026#8217;s sort out how LZA is related to Control Tower. Control Tower\u0026#8217;s main functionalities are available as an AWS service, with some customization capabilities available as a standalone solution on top of the service, as I discussed in the last post. Unlike Control Tower, LZA as a whole is a standalone solution. Luckily, the installation of the solution itself is highly automated.\nI see LZA both as an extension of Control Tower, and as a complement to Control Tower. It is an extension of Control Tower because LZA can co-exist with Control Tower. We can configure LZA to enable Control Tower and use its Account Factory to provision new accounts (alternatively but not recommended, we can opt out of Control Tower and manage account creation on our own). I also see LZA as a complement to Control Tower because it comes with full end to end automation scheme for networking infrastructure and most of the services involved. This is missing in Control Tower, which leaves it with users to provision networking infrastructure in the customization. Thanks to the infrastructure automation capability, even if you do not have a strong regulatory requirement, there are still good reason to go with LZA for its low-code automation capability. Below is a table that summarizes the differences:\nControl TowerLanding Zone Accelerator\u0026#8211; Multi-account management tool\n\u0026#8211; Governance layer\n\u0026#8211; Customization Framework to bring your own infrastructure automation\u0026#8211; can manage Control Tower \u0026#8211; low-code automation engine for infrastructure automation and service deployment based on CDK\n\u0026#8211; reference configurations based on common industry profiles and regulatory requirementsComparison between Control Tower and Landing Zone Accelerator As the name suggests, LZA is an accelerator so there is no expectation of its user knowing how to program infrastructure as code. However, it still expects its users to know YAML very well. Knowing how CloudFormation and CDK works can greatly help the users troubleshoot deployment. Reference architectures in LZA The input of LZA is configuration as code in YAML format. The LZA repository comes with a number of sample configurations to implement some industry-based best practices. The reference architectures currently include:\nGeneral best practices reference configuration: for clients other than the categories below; Government customers: US Gov Cloud (FedRAMP compliant, on aws-us-gov partition), US State and Local Government, China (on aws-cn partition), Canada Federal (CCCS compliant) TSE-SE (Highly Trusted Secure Enclave Sensitive Edition) on commercial partition for governments, national security, defence, and law enforcement customers reference architecture; Election: for election customers including elections agencies, committees and campaigns; Healthcare: for healthcare customers. However, the document does not mention HIPAA compliance or anything related to the HIPAA Reference Architecture; Finance and Taxation: for tax workload to secure Federal Tax Information (FTI) data; Education: for education industry customers. Many of these reference architecture shares a few common traits in the networking design. Take the CCCS reference as an example, the networking involves the followings:\nWorkload VPCs: consisting of a number VPCs for production and test environments; Shared services VPC: hosting common services such as pipelines, Active Directories, etc Endpoint VPCs: centrally hosting interface endpoints Perimeter VPCs: acting as ingress, egress and inspection VPCs. The Perimeter VPC hosts firewalls (either AWS Network Firewall or NGFW appliances behind Gateway Load Balancers). All the VPCs are centrally managed in an AWS network account, and are shared to other accounts using Resource Access Manager. The reference architecture document keeps the details of this architecture, which was derived from the security reference architecture (SRA).\nSpecial Purpose VPCs I consider the non-workload VPCs as special purpose VPCs. The shared services VPC is the most straight-forward. The Endpoint VPC is the most standardized. It is used to centrally host VPC interface endpoints for security and cost reasons. Unlike Gateway endpoint which is only available for S3 and DynamoDB, interface endpoint carries a standing charge and therefore should be consolidated. In addition, since interface endpoints are based on interfaces, we can centrally control the security group and interface policy. To integrate the endpoint VPC, not only do we need to create those interface endpoint. We also need to account for routing (using Transit Gateway route tables) and name resolution. For name resolution, we need to create a Route53 private hosted zone for each DNS name, such as ec2.us-east-1.amazonaws.com and associate them with each workload VPC. Note that the interface endpoints DNS name may not always follow the same format. See the exceptions in my old post. Also note that this would create a lot of associations (between Private Hosted Zone for each Interface endpoint and each workload VPC). For example, 20 workload VPC with 30 private hosted zones will create 600 associations. To overcome this, use Route53 profile (introduced in April 2024).\nAnother special purpose VPC is the perimeter VPC. This VPC vary greatly between customers because of different requirement and historical preferences. One of the key design areas is the placement of NGFW, which is discussed in this post.\nLZA Orchestration Engine The installation process may feel complex at the beginning because we have to first install the pipeline to that installs the pipeline. The initial setup consists the following steps:\nCloudFormation installs the installer. We start with a CloudFormation template to deploy the LZA installer itself. It deploys resources such as CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject). These resources are in the INSTALLER circle in the diagram below; The installer installs the accelerator core. In the LZA installer, the CodePipeline (AWSAccelerator-Installer) and CodeBuild project (AWSAccelerator-InstallerProject) drive the installation of the LZA. The input is the official LZA GitHub and we need a GitHub token for this step. The output is the actual LZA orchestration engine, including CodePipeline (AWSAccelerator-Pipeline) and CodeBuild (AWSAccelerator-BuildProject and AWSAccelerator-ToolkitProject). The user may specify their own GitHub repo as the configuration repo. Otherwise, a CodeCommit repo will be created. The LZA resources are shown in the CORE circle in the diagram below; The acceleration core configures the landing zone. The LZA orchestration engine deploys actual resources in the landing zone, with the CodeCommit repo (aws-accelerator-config) or the specified GitHub repo as input. If we enable Control Tower with LZA, we should first log in to management account and configure Landing Zone with Control Tower. we can also create (and register) the required OUs and accounts from Control Tower. Then we can deploy\u0026nbsp;Landing Zone Accelerator\u0026nbsp;with default configuration. After the initial setup, we will need to iterate over the aws-accelerator-config repo. We implement our landing zone design in YAML configuration following the schema documentation. Changes in the configuration repo will trigger the pipeline (aka LZA\u0026#8217;s orchestration engine) to redo step 3, whereas step 1 and step 2 are performed only once. The duration of step 3 is significantly longer than the first two steps. Pitfalls If LZA manages Control Tower, it expects existing OUs registered in Control Tower or it will report error. For account, LZA can create accounts listed in the manifest but not yet created. However, with the lengthy account vendor process for multiple account we run the risk of task time out in the pipeline.\nDuring the installation, some account may run into quota limit. For example, the Networking Account usually have more than five VPCs whereas the quota is 5 VPCs per region per account. We need to increase the quota on those accounts.\nThe full deployment usually creates some SCPs. However, if we ever need to re-deploy a configuration, some steps steps might be blocked by certain SCPs. Attempts to temporarily detach SCPs from OUs, or modify SCPs often get reverted. The cause is an EventBridgeRule in\u0026nbsp;us-east-1\u0026nbsp;region named\u0026nbsp;RevertScpChangesModifySc. The rule should be disabled temporarily to perform the troubleshooting activity. We can do this with the following steps:\nDisable the EventBridgeRule \u0026nbsp;RevertScpChangesModifySc , which is only present in us-east-1 region; Detach SCPs and note down what are detached, one OU at a time; Go to the failed CF stack in the region, delete the failed stacks (after turning off termination protection); Rerun the pipeline step from where it failed. This time it should go past the failure to the end, if SCP is the cause as per our assumption; Re-attach SCPs. Suppose Security and Infrastructure OUs share one group of SCPs, and Dev, Test, and Prod OUs share a different group of SCPs; Re-enable the EventBridgeRule; Even with the EventBridgeRule\u0026nbsp;RevertScpChangesModifySc disabled, when you re-run LZA deployment pipeline, the Accounts step will re-attach the SCPs using the AWSAccelerator-AccountsStack in the management account in us-east-1 region.\nIn general, how SCP works with organization structure is something to be very careful about, especially when the hierarchy consists of multiple layers of OUs. It is important to keep in mind, that deny statements in SCP take effect down the hierarchy, where as allow statements only affects the immediate child account of the OU where the SCP is attached to, as per the evaluation logic. As a result, an SCP with allow * statement (in the LZA-AWSFullAWSAccess managed policy) must be applied to Root, every OU at each level, and every account, for LZA to function. In addition, there are some hard limits for SCP. Each SCP has a size limit of 5120 characters, and each OU can attach a limit of 5 SCPs. Challenges Powered by CDK, LZA automates the creation of a lot of resources. The configuration files uses the deploymentTargets attribute to allow users to specify to which accounts or OUs the declared resources will be deployed to.\nSupporting many resources is a double-edge sword. Because the accelerator needs to go through every aspect of a landing zone, it is very slow to run. The accelerator pipeline may take as long as 40 minutes without any change to the configuration code. This is extremely slow if you just want to make some small changes in the configuration (e.g. update route table, add IAM role). Even though LZA supports many resources, it\u0026#8217;s not flexible with every resource. For example, today we can deploy IAM roles using RoleSet. However, in the trust policy of the IAM role you can only specify a two types of principals under the assumedBy attribute: account and service types. On the other hand a trust policy can support many other types of principals such as another IAM role.\nAZ Mapping Another important ability that LZA does not support is consistent AZ mapping across accounts. (Correction: this is now supported in LZA v1.5 as of Oct 2023). In some example LZA configurations, we deploy VPCs across multiple accounts using two or three availability zones referenced by their logical ID, such as us-east-1a and us-east-1b. However, AWS maps logical ID to physical ID and the mapping may be different in each AWS account. Using the same logical ID cannot guarantee the physical AZ are the same across account. As of LZA 1.5, the ability to reference physical ID in availabilityZone is supported in LZA configuration file.\nSummary I\u0026#8217;ve spent a lot of time on LZA recently. It is extremely powerful. LZA streamlined the landing zone deployment process with configuration as code. It also allows users to customize their landing zone towards their own architectural needs and compliance requirement. For example, you can declare arbitrary SSM parameters in each account. On the down side, the LZA deployment is time consuming through the pipelines. It tries to automate too many aspects of the infrastructure, which makes itself quite a complex project. Expect lots of changes in each new version. The idea of being a low-code solution is to make it simple for end users but it often sacrifices flexibility. For example, if you want to create an IAM role in each new account that references the Management account ID, it is not possible until such feature is implemented in the accelerator. When the accelerator pipeline fails, it still requires deep CloudFormation knowledge to troubleshoot. Previous PostOrchestrate Landing Zone with AWS Control Tower Next PostAWS Systems Manager is an Omnipotent Hodgepodge ","date":"2023-09-22T23:05:04-04:00","image":"/wp-content/uploads/2025/04/feature-aws-lza.webp","permalink":"/2023/09/orchestrate-landing-zone-with-landing-zone-accelerator-on-aws/","title":"Orchestrate Landing Zone with Landing Zone Accelerator on AWS"},{"content":"Following an introduction to AWS Landing Zone, I\u0026#8217;ll dive deeper into Control Tower as an orchestration tool in this post.\nMore on Landing Zone In data center operation, there are numerous tasks that other teams have to complete before the the deployment of an application. For example, the 42U server cabinet must be in place with dual powers. The cabinet comes with a network switch in the middle and each ethernet port must be provisioned. Once the server is connected to the central network, the NOC team assigns IP address, configure dynamic VLAN on the switches, and configures firewalls etc according to the connectivity requirement. The new server also needs to report to centralized monitoring solutions such as SolarWinds. In cloud operations, the scope and target of a landing zone is similar to those data center operations work, with networking being the core. The idea is that the landing zone ensures security, compliance and governance, so that applications (analogous to paratroopers) can focus on their primary responsibility. Although Landing Zone is a general concept for any cloud service provider (CSP), each CSP has some prescriptive guidances on setting up landing zone in their particular environment. For example, an AWS prescriptive landing zone typically covers the following apsects:\nMulti-account structure Identity and Access Management Governance (controls and guardrails) Networking Additional Security Services The multi-account best practice is an important aspect in AWS as the OU structure dictates how effective Service Control Policies (SCPs) can govern the entire footprint. A landing zone orchestration solution should also apply guardrails and controls based on the organization\u0026#8217;s security and compliance requirement. In addition, it is common expectation that landing zone orchestration solutions create required networking resources such that applications are ready to deploy securely. This post is about Control Tower.\nIntro to Control Tower Control Tower is a landing zone orchestration solution available as an AWS service. I have three impressions over Control Tower. First, it is good with managing multi-account structure. It integrates closely with AWS Organizations and requires client to have a dedicated log archive account and a dedicated security tooling account, which also serves as audit account. The second impression is Control Tower makes governance more straightforward. AWS seems to use the term control and guardrail interchangeably. Below is an illustration:\nAWS Control TowerAWS Co\u0026#8230;AWS Config\nRulesAWS Co\u0026#8230;AWS CloudFormation\nhooksAWS Cl\u0026#8230;AccountAccountOUOUAccountAccountAccountAccountService Control PolicyServic\u0026#8230;preventive controlpreventive controldetective controldetective controlproactive controlproactive controlText is not SVG \u0026#8211; cannot display\nThe detective controls only captures and reports violations. The preventive controls stops the violating API request. The proactive controls remediates the detected violations. Another way to look at the controls are the priority of the controls:\nMandatory controls: there are about 23 mandatory controls that are enforced on each account. As soon as a new workload account becomes part of Control Tower, these controls will come enforced. Optional controls including strongly recommended controls and Elective controls. There are many existing controls and AWS is still releasing new one (example). AWS Config service uses conformance pack to organize relevant controls. For example, there are sample conformance packs such as \u0026#8220;Operational Best Practices for \u0026#8220;NIST 800 53 rev4\u0026#8221; and \u0026#8220;Security Best Practices for EKS\u0026#8221;. In addition, you can even build your own conformance pack. Once you deploy a conformance pack, AWS config deploys additional rules against the current environment. The third impression is that Control Tower falls short with infrastructure automation. It provides a number of customization mechanism to leave infrastructure (mostly networking) automation with users. I will explain later.\nMandatory Accounts The function of Control Tower does not depend on a specific account structure that aligns with the multi-account best practice. However it does require three mandatory accounts to set up automatically when creating control tower. The account where Control Tower is configured is the management account. The log archive account is the owner of S3 buckets that hosts buckets for loggings. The audit account is a restricted account that\u0026#8217;s designed to give your security and compliance teams read and write access to all accounts in your landing zone. We also use this account as delegated administrator account for several security services at organization level. I\u0026#8217;ll elaborate in the next section.\nThe management account is directly under root OU and there is no SCPs applied to the management account. Also Control Tower does not turn on AWS Config recorder and configure delivery channel on the management account itself. The log archive account and audit account are under the Security OU (or otherwise named during the setup). At the end of Control Tower setup a set of mandatory controls will be applied at the OU level and thus effective to each account. Apart from the mandatory accounts, oftentimes there is an Infrastructure OU. Typically we place shared services account under this OU. The account is for common services such as central networking, managed Active Directory, DevOps pipeline etc. Another optional OU is workload OU, where we can create new workload accounts or enrol existing workload accounts.\nEach enrolled workload account have its CloudTrail configured with an organization-level trail, which sends log to the central bucket that log archive account owns. Each enrolled account also has AWS Config recorder configured with a delivery channel pointing to the configuration log bucket also owned by log archive account. Note that Control Tower does not turn on recorder on the management account itself.\nAccount Enrolment Most likely a customer is already in use of AWS Organization. They can delegate one account as management account, and use Control Tower console to create a landing zone for the intended regions. During the creation, they will have to create two additional account, one as log archive account, and the other as security tooling or audit account. This is in alignment with multi-account best practices. After the creation of landing zone, the two new account are automatically enrolled in Control Tower.\nGoing forward, the client should create new accounts using Control Tower\u0026#8217;s Account Factory to save a separate enrolment step. For existing accounts and OUs we\u0026#8217;ll need to enroll them into Control Tower. At the beginning, the Control Tower Landing Zone (CTLZ) only has the mandatory controls, which are the bare minimum governance. When we enroll existing OUs and accounts, these controls (mostly detective and preventive) will extend to the newly enrolled OUs or accounts. Therefore, it is important to not introduce numerous controls prior to having all accounts and OUs enrolled. It is important to have a proper OU hierarchy upfront because both detective and preventive controls (Config Rules and SCPs) are typically applied at OU level and effective to all the children of the the OU.\nTo enroll an existing account, the account must meet some prerequisites. An IAM role with cross-account trust must be manually created. Since each account can have only one AWS Config configuration recorder and delivery channel, if an existing account already has one of them, they must be manually deleted using CLI.\nAccount Factory defines what happens when we create a new account and it is where we can bake in the customizations. When it comes to infrastructure automation, Control Tower leaves pretty much everything up to the account factory customization.\nDelegated Administrator Many AWS services are available as organization wide service. For example, CloudTrail, GuardDuty and even AWS Organization. Most of these organization level services allows you to specify one account as delegated administrator for the entire organization. For example, for System Manager and Service Catalog, the best practice is to designate an operation account or shared services account as the delegated administrator. There are several security-related, organization-wide services, such as GuardDuty, Detective, Macie, Security Hub, Inspector, Audit Manager, and Firewall Manager. The best practices for these services is to designate the audit account as the delegated administrator. We think of the audit account as an aggregation point (or points for organizations that split the functionality across multiple accounts) for these AWS services. You can find this recommendation from the documentation of each service. However, currently Control Tower does not enforce this best practice. Even though Control Tower dashboard gives you visibility to security services such as GuardDuty and SecurityHub, you are still on your own to configure these services outside of Control Tower.\nWhether you had never used these services, or you had previously designated another account as administrator for these services, it is good time to align with the best practice for delegated administrator right after Control Tower setup. If you had previously designated other accounts, you need to take additional steps to revoke the old delegation and designate new delegated administrator account. These general steps can be performed from console or CLI and apply to all these services:\nFrom the old administrator account, remove all the members From the management account, revoke delegation From the management account, delegate new administrator account (audit account) From the new administrator account, invite all the members, and make sure to tell the service to automatically include new accounts going forward Note that when you invite an account, that account must not have created, invited or associated with other accounts (as a member). In other words, we should never let a non-delegated administrator account create, invite or associate other account as member.\nControl Tower Customization Control Tower by itself does not play a big role in networking automation. You can create a VPC when provisioning a new account in Control Tower. That\u0026#8217;s about what you can do. It does not address how the new VPC connects to existing networking space. It does not distinguish the existing VPC topology. To address this, Control Tower has a few customization options:\nAFC\nAccount Factory CustomizationAFT\nAccount Factory TerraformCfCT\nCustomizations for Control TowerSupported IaC languageAnything that Service Catalog Support (e.g. CloudFormation and Terraform HCL)Terraform HCLCloudFormationReadinessAFC is a native mechanism. You can specify the Service Catalog product during creation of an Account in Control Tower console.Users need to first bootstrap the solution, which creates Terraform pipelines and other components.Users need to first bootstrap the solution which creates a pipeline and other components.How it worksControl Tower launches the specified Service Catalog product (aka blueprint) during account creation.Quite complex. Explained in the section belowControl Tower posts a lifecycle events to Amazon EventBridge. A lambda function will process the event using pre-baked CloudFormation templates and Step Functions.Prebuilt customizationsNo. However, for certain products, there are some partner-maintained blueprintsYesNo AFC is fairly straightforward to use based on this post, as long as you know Service Catalog. It is a native capability of Control Tower. The other two ways for CT landing zone customization are not native capability and each require a separate bootstrapping process to deploy the pipeline-based solution.\nCustomizations for Control Tower In CfCT, we first bootstrap the solution from management account, using this CloudFormation template on GitHub. We usually need to version control the custom configuration so we should select \u0026#8220;AWS CodeCommit\u0026#8221; as the value for the CodePipelineSource parameter. The bootstrapping (execution of this CloudFormation template) takes about 5 minutes.\nThe solution is based on Control Tower\u0026#8217;s lifecycle event to trigger the pipeline with a source code repository (or S3 bucket). The source code repository (AWS CodeCommit) stores the custom configuration. A custom configuration consists of a manifest file in YAML format, made up with a number of resource sections. Each resource section references an artifact, either a CloudFormation template or a policy file to apply. The artifact can either be stored in the same repository, or in a remote S3 bucket. Account Factory Terraform Introduced in late 2021, the AFT solution is based on the terraform-aws-control_tower_account_factory repository. It is too complex as a solution in my opinion. At the beginning you need to have a designated OU and account for AFT and install the solution itself with Terraform. Note that, the solution does not address the state storage of the installation of the solution itself (chicken-or-egg). The installation also installed a few IAM roles required on the AFT management account. Having a landing zone management account and a AFT management account is not a neat setup.\nOnce installed, the solution consists of four repositories. Their purposes are as follows (copied from workshop instruction):\nAccount requests \u0026#8211; handles placing or updating account requests. See\u0026nbsp;example here. AFT account provisioning customizations \u0026#8211; manages non-Terraform customizations that are applied to all accounts. This stage runs before the global customizations stage. Examples available\u0026nbsp;here Global customizations \u0026#8211; Global customizations – manages Terraform-based customizations that are applied to all accounts created by and managed with AFT.\u0026nbsp;Examples available Account customizations \u0026#8211; Account customizations – manages Terraform-based customizations that are applied only to specific accounts created by and managed with AFT.\u0026nbsp;Examples available With this solution in place, when we create a new account via account request repo, quite a number of steps will happen after that, as the workshop instruction illustrated:\nWhen I first looked into how this work I\u0026#8217;m very concerned about the maintainability of this \u0026#8220;solution\u0026#8221;. The amount of services and pipelines involved in this solution makes it difficult to troubleshoot end to end. To improve that AWS even added an enhancement for request tracing. Nonetheless, I would not go with AFT just with how complex it looks like. For more details, check out the Control Tower workshop.\nSummary Control Tower helps you set up a Landing Zone without network infrastructure automation. It provides a couple of customization mechanisms, allowing you to bake in your own infrastructure automation. To build a landing zone, you can go with Control Tower in the following situations:\nyou have general regulatory requirement and need to enable governance capabilities your networking stack isn\u0026#8217;t centrally managed or isn\u0026#8217;t large enough to warrant infrastructure automation If you do need networking automation, then you will need the customization capability of Control Tower. In this case, your team should have good handle on infrastructure as code (CloudFormation or Terraform) Out of the customization options, AFC seems the least complex to me. The other two, especially AFT, is too complex. We use pipelines to trail and error with other stacks, and the pipeline solution itself must be simple.\nOn the other hand, if you environment has strong regulatory requirement, or you are seeking a prescriptive network architecture, or your team does not have the capacity with CloudFormation or Terraform Templates, consider Landing Zone Accelerator on AWS.\nPrevious PostAuthentication to kube-apiserver via OIDC Next PostOrchestrate Landing Zone with Landing Zone Accelerator on AWS ","date":"2023-08-19T17:25:00-04:00","image":"/wp-content/uploads/2025/04/feature-control-tower.webp","permalink":"/2023/08/control-tower-aws-landing-zone/","title":"Orchestrate Landing Zone with AWS Control Tower"},{"content":"Background There are many benefits of using OIDC to authenticate to kube-api server, especially with multiple clusters that need consistent log-in experience. With the last post on how OIDC Authorization Code Flow works, now I will discuss options with authentication for kubectl to connect to kube API server.\nTo start, let\u0026#8217;s look at the anatomy of kubeconfig file. The full schema is in the documentation. Looking at my kubeconfig file, there are three sections:\nclusters: each entry specifies a cluster\u0026#8217;s name, server address and certificate authority data (in base64 encoding or a file location). users: each entry specifies a username. Some users are identified with client key and certificate. Some specify a command to provide client authentication. Refer to the authentication strategies. contexts: each entry links a user to a cluster Therefore, the key to use OIDC integration, is to use command to provide client authentication. Vanilla Kubernetes The documentation on authenticating has a diagram on how to use OpenID Connect tokens. The diagram does not give details on how access_token and id_token were obtained. So it could be any OIDC flow (Authorization Code Flow, Implicit Flow, etc) as we have discussed.\nAlthough the instruction does not mandate which OIDC flow to use, we should use Authorization Code Flow in this architecture. The API server needs to trust the OIDC issuer, and the document covers how to configure API server.\nIn the diagram, step 2 and step 3 are required by kubectl itself does not perform these activities. All kubectl does is carry the JWT token in the Authorization Bearer. Vanilla Kubernetes does not provide a solution for OIDC integration. It only provides some instructions and we still need some helper scripts to glue all these instruction steps together.\nThere are many open-source project for this purpose. For example, Jetstack has kube-oidc-proxy and Int128 developed kubelogin. Other projects such as k8s-auth-client, k8s-oidc-helper, and gangway are no longer being updated. The kubelogin project remains influential. It has a clear diagram too:\nFrom the diagram we can see kubelogin proposes authorization code flow. Also, one design concern with Kubernetes control plane is the placement of endpoint. From this diagram we can see that even if the cluster endpoint is on private subnet, OIDC integration should still work. The control plane (specifically kube-apiserver) initiates outbound connection to OIDC Provider. There is no inbound connection to it from the OIDC provider.\nIn the kubelogin setup, the redirect URI is set to localhost:8080 because it stands up a server on the same host where browser is running. The browser can always resolve localhost. For a full configuration steps, Okta has this blog post on how to use kubelogin as helper, and Okta as Authorization Server to authenticate kubectl via OIDC. Step-by-step with kubelogin We\u0026#8217;ll go through an example with int128/kubelogin because it works with any Kubernetes flavour, including managed Kubernetes services. It is also fairly simple. The instruction covers a few types of Authorization Servers (Google Identity Platform, KeyCloak, Dex with GitHub, Okta and Ping Identity). I\u0026#8217;ll take KinD cluster as an example and use Azure AD as Authorization Server.\nFirst, we\u0026#8217;ll register an App in Azure Portal. Go to \u0026#8220;App Registrations\u0026#8221; and \u0026#8220;New registration\u0026#8221;. Give it a name \u0026#8220;kubeoidc\u0026#8221; and set Redirect URI to \u0026#8220;Web\u0026#8221; with URL \u0026#8220;localhost:8000\u0026#8221;. Click on Register.\nThe next page shows the details for this app. The Application (client) ID is important for next steps. Click on \u0026#8220;Add a certificate or secret\u0026#8221;, then \u0026#8220;New client secret\u0026#8221;, put in expiry and description. The secret value is generated and displayed on the next page, which is important for our next step. We also need to find out the issuer URL. From the App page above, click on \u0026#8220;Endpoints\u0026#8221; and find out the URL from field \u0026#8220;OpenID Connect metadata document\u0026#8221;. My metadata document URL looks like: https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0/.well-known/openid-configuration The OIDC Issuer URL is the part before .well-known. In this case, it is:\nhttps://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0 Now we have collected what we need for the next few steps: ClientID, ClientSecret and OIDC Issuer URL. We can then create the KinD cluster, and reference ClientID and IssuerURL in the cluster configuration:\ncat \u0026lt;\u0026lt; EOF \u0026gt; kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane kubeadmConfigPatches: - | kind: ClusterConfiguration apiServer: extraArgs: oidc-issuer-url: https://login.microsoftonline.com/xx8x8xx8-7777-66yy-55b5-444aaaaa3322/v2.0 oidc-client-id: 99999e88-e777-6666-c5c5-c444444d3d22 - role: worker - role: worker - role: worker EOF kind create cluster --config=kind-config.yaml The command will create cluster and configure kubeconfig file with a user named kind-kind as admin, which isn\u0026#8217;t what we need. Now we use kubelogin helper command:\nkubectl oidc-login setup \\ --oidc-issuer-url=ISSUER_URL \\ --oidc-client-id=YOUR_CLIENT_ID \\ --oidc-client-secret=YOUR_CLIENT_SECRET This command will act as the client in the OIDC flow and prompts you to log in to Azure in a Browser. Once logged on, it gives you the next few commands to run. Since we\u0026#8217;ve already created a cluster with the Issuer URL and Client ID, we can skip creating cluster, and run the steps to:\nbind the Azure user to a ClusterRole set up the kubeconfig locally with a user oidc, which needs to execute the oidc-login command Lastly, we can test the oidc user with kubectl --user=oidc get nodes. We can also set the context to use oidc user by default. Voila.\nFrom this example, we learned how to configure OIDC integration for any Kubernetes distros. The steps that we need to take are:\nOn the cluster side, we customize kube-api-server flag with OIDC provider info. Specifically, we provided Issuer URL and client ID in this example. In the OIDC Flow diagram, this step establish a trust from the Resource Server (K8s cluster) to the OIDC provider. On the kubectl side, kubectl itself cannot fulfill all the duties of a client app in the OIDC Flow. It needs a helper script and we\u0026#8217;ve made friend with int128/kubelogin. On the Identity Store side, we expect it to be an OIDC-compliant Authorization Server. Otherwise, we consider using Dex as a broker in between. When it comes to managed service, many allows us to customize the OIDC related flags for kube-api-server. Let\u0026#8217;s look at how some managed services get this to work.\nAzure Kubernetes Service I find the kubectl authentication with AKS highly opinionated in its documentation. The recommendation is using Azure Active Directory as identity store but I don\u0026#8217;t find it work with other OIDC providers.\nTo work with Azure Active Directory, you would configure the cluster and specify Azure role-based access control with the Group UUID in Azure AD. On the client side, you will need to install Azure\u0026#8217;s kubelogin utility. Do not confuse it with int128\u0026#8217;s kubelogin, which is for any cluster. This kubelogin is just for Azure. Once installed, you can use az-cli command to update your kubeconfig file, which call this utility from kubeconfig. To examine details about client configuration and AAD integration, check out the Terraform template in the azure directory of my cloudkube project.\nThe document only covers Azure AD integration and I tried to find if there\u0026#8217;s a way to integrate with third-party OIDC providers. Unfortunately I have no luck. ChatGPT points me to a page about enabling OIDC provider but it is in the context of workload identity and it does not allow you to customize the issuer. So it\u0026#8217;s completely irrelevant. Because you cannot customize OIDC issuer, etc, it simply won\u0026#8217;t work with any third-party OIDC provider. Sure enough, most of Azure\u0026#8217;s client use Azure AD anyways.\nAzure makes it streamlined to configure OIDC integration of AKS with Azure AD, its own identity store. To my disappointment, it is currently not possible to integrate with third-party OIDC provider for authentication at cluster endpoint. ROSA RedHat OpenShift on AWS (ROSA) is a. However it reflects how OpenShift configures third-party identity provider. Let\u0026#8217;s first create a cluster off custom VPC and private endpoint. This is covered in my previous post about ROSA. After the cluster creation, we\u0026#8217;re at the point where we can run oc command against cluster endpoint from Bastion host because it is a private cluster. However, being a private cluster is irrelevant to how we configure OIDC integration. We\u0026#8217;ll use Azure AD again as OIDC provider. So we need to register an app the same way as I did above in the kubelogin example. We need to have Client ID, Secret and OIDC issuer URL. For RedirectURI, go to your OpenShift console, and under Cluster, click on the cluster name → access control → identity providers → select OpenID. Note the page clearly states that this is Authorization Code Flow, and the OAuth Callback URL is provided. Use it to regiter App in Azure, and fill in the page with Client ID, Secret and OIDC issuer URL. Also fill in other fields accordingly and click on Add. Now you should have it configured!\nIdentity Providers for OpenShift cluster\nThe next step is trying to login. From Bastion host, run oc login and it will give me an URL to use. The URL contains the cluster endpoint, which resolvable from the Bastion Host itself. However I need a Browser session here, so I have to run Bastion host as SOCKS5 proxy and tell Chrome on my MacBook to use it:\n~ open /Applications/Google\\ Chrome.app --args --proxy-server=\u0026#34;socks5://localhost:1080\u0026#34; The browser session redirects to Azure AD for log in. Once completed the webpage will display a token that I can use with oc login command. Run this command with token from Bastion, I\u0026#8217;m logged in:\noc login --token=sha256~3ZT5JGWELOcBzfohftEm9D2UwoOVFvATASuZk3_uxps --server=https://api.dhc.62q3.p1.openshiftapps.com:6443 oc whoami oc get no At this point, if I run oc whoami, I get the user name. However, this user cannot do anything. This is because it is not associated with a role yet. You grant more permission to this user: go back to OpenShift console, Clusters → ClusterName → Access Control → Cluster Roles and Access → Add user. Here you can map the user name to a role (let\u0026#8217;s say ClusterAdmin). Then this user will have its priviledge:\nThe whole Flow works with private cluster, because the redirect URI is resolvable from the Bastion host. If you chose to expose cluster endpoint publicly (not recommended), you can perform the above steps directly from your MacBook or Laptop. So the ROSA experience has been smooth. Unlike kubectl, the oc utility can act as the Client App in Authorization Code Flow. The other part of the configuration such as client secret and issuer URL are made in OpenShift console. Good job!\nElastic Kuberentes Service EKS allows you to specify OIDC issuers from console or CLI to set up third-party OIDC configuration. There is a blog post from Okta on this, which works for private clusters. In the instruction, the author first manually created kubeconfig file with int128 kubelogin, and then bind ClusterRole with the user.\nThe blog post is very detailed. Instead of repeating it, I would like to discuss two SSO models available in AWS. I summarize them as below:\nEKS ClusterEKS ClusterRoleBinding\nClusterRoleBindingRoleBinding\u0026#8230;OIDC compatible\nIdentity ProviderOIDC compatible\u0026#8230;EKS ClusterEKS ClusterSAML compliant\nIdentity ProviderSAML compliant\u0026#8230;IAM Role via\nPermissionSetIAM Role via\u0026#8230;AWS IAM\nIdentity CenterAWS IAM\u0026#8230;SAMLSAMLConfigMap\naws-authConfigMap\u0026#8230;groupgroupgroupgroupEKS SSO Model 1 \u0026#8211; IAM Identity CenterEKS SSO Model 1 \u0026#8211; IAM Identity CenterEKS SSO Model 2 \u0026#8211; direct OIDCEKS SSO Model 2 \u0026#8211; direct OIDCOIDCOIDCRoleBinding\nClusterRoleBindingRoleBinding\u0026#8230;Role\nClusterRoleRole\u0026#8230;Role\nClusterRoleRole\u0026#8230;OIDC claim:\nuser=john\ngroup=adminOIDC claim:\u0026#8230;AssumeRole\nWithWebIdentityAssumeRole\u0026#8230;Pipeline\nUserPipeline\u0026#8230;Text is not SVG \u0026#8211; cannot display\nBoth are SSO models for EKS. Model 1 (IAM Identity Center) is home grown as AWS using IAM Identity Center (formerly AWS SSO). Users start with an IAM principal (AWS construct) and use the aws-auth config map to tie them to Kubernetes groups. This AWS blog post and this support article are based on the IAM Identity Center model. On the other hand, Model 2 (Direct OIDC) is the vanilla Kubernetes approach. It takes group claim from OIDC identity token. The Okta blog post is based on this model.\nThe IAM identity center model works with SAML compliant identity providers, oftentimes Active Directory, although there seems to be a plan to support OIDC-compliant identity provider as well in the future. Even if it was supported today, I\u0026#8217;d still prefer model 2 for working with any OIDC compliant identity provider because it\u0026#8217;s simple. Why not leverage K8s\u0026#8217; native capability? For identity providers that do not support OIDC natively, or does not issue group claim (e.g. Google workspace), as we discussed, we can also consider alternatives such as Dex as identity broker. However, this model comes handy when a pipeline user with IAM role needs to authenticate into EKS.\nOne should always go for the direct OIDC model, if the upstream identity provider supports OIDC. The provider itself can even be AWS Cognito User pool. Cognito User Pool itself supports federated identity but again, I would directly connect EKS cluster to the OIDC compatible identity provider, instead of going through Cognito User Pool. As a result, the only use case where Cognito user pool is absolutely necessary, is when we need the Cognito user pool itself as the identity provider, as we have in CloudKube\u0026#8216;s eks directory. Summary Configuring OIDC provider for cluster endpoint can be confusing and we need to understand how OIDC flows work. I dived into OIDC in the previous post and in this post, I explained how to get it to work with vanilla Kubernetes. I summarized the three requirements and looked at the OIDC provider option with some managed services. Then I went through OIDC options for AKS, ROSA and EKS.\nPrevious PostOAuth 2.0 and OIDC 2 of 2 Next PostOrchestrate Landing Zone with AWS Control Tower ","date":"2023-07-28T09:20:00-04:00","image":"/wp-content/uploads/2025/04/feature-kubectl-oidc.webp","permalink":"/2023/07/authenticate-kube-apiserver-via-oidc/","title":"Authentication to kube-apiserver via OIDC"},{"content":"I wrote a brief on this topic a while back. Now I need to configure OIDC in a few occasions I decide to dive deeper into the flows this time. As I stated in the last post, Nate Barbettini\u0026#8217;s presentation from 2017 was awesome and I viewed it again. Slides are available here. Another great reference is this post from DeepNetwork Developer\u0026#8217;s blog.\nBack Channel and Front Channel To understand why there are several different flows, it is important to understand the difference between back channel and front end channel.\nIn web development, traditional architecture involves a frontend (e.g. Browser, or any client-side app) and backend server. The web frontend is written in HTML, CSS, JavaScript, etc. There are also web frameworks such as Django, Angular, to save developers time. Backend (server-side) is responsible for storing and organizing data to ensure frontend can function. There might be multiple backend servers, such as session cache, data store, API server, etc. In a nutshell, server-to-server communication is back channel, and browser-to-server communication is front channel. From security perspective, we regard front-channel as less secure, because we have less control of the location of the front-end and browser is easy to tamper with.\nBack Channel and Front Channel (source https://www.okta.com/blog/2019/04/oauth-when-things-go-wrong/) For traditional web applications with client-server architecture, when they communicate with third-party servers, they can initiate the communication from their backends, creating a back channel for better security posture. Single-page applications (SPAs) are applications without their own backends. When building an SPA, the front-end developer deals with frontend frameworks such as React, Angular or Next.js. When SPAs have to communicate with third-party APIs, they have to create a front channel. Also the API must support Cross-Origin Resource Sharing (CORS) for the browser to allow the cross-domain communication.\nIn the web development world, there is also Native App (aka Mobile App). Native App to Server communication is also considered back channel as we consider the client-side (Mobile App) secure. Nate\u0026#8217;s talk makes the following recommendation for the flow (grant type) and I rephrase it as below:\nArchitectureRecommendationTraditional Web Application (client-server architecture)There are both front and back channels. Use authorization code flowSingle Page Application (SPA, e.g. JavaScript) with API backendFront channel only. Use implicit flow Native mobile AppUse authorization code flow with PKCEMicroservices and APIs (Machine-to-Machine)Back-channel only. Use Client credentials flow The message, is that we should use Authorization Code Flow and use back channel, so long as the required component (backend) exists in the architecture. Note that the terminology for these types of applications may differ. For example, when you try to create an OIDC client for Amazon Cognito user pool, here\u0026#8217;s how it categorizes client capability (app type):\nPublic client: A native, browser, or mobil-device app. Cognito API requests are made from user systems that are not trusted with a client secret. Confidential client: A server-side application that can securely store a client secret. Cognito API requests are made from a central server. Get used to different terms describing the same concept.\nOAuth 2.0 and OIDC flows The original problem that OAuth 2.0 (RFC6749) addresses is delegated authorization. In all OAuth flows, the authorization server issues an Access Token for the client to carry. The Access Token identifies the scope of resources that its carrier is authorized to access. However, third-party services do not always want to delegate authorization to the identity provider. They often just need identity information and want to perform authorization on their own. The OAuth 2.0 Access Token itself is all about permission and does not care about the identify of its carrier. It is not designed for authentication. OAuth 2.0 does not provide a standard way for Authorization Server to keep identity information of the principal. Many developers needs to address identity issue and they started to (mis)use the AccessToken to store identity information in custom fields, until OIDC came about.\nOAuth: Access Token only OIDC: Access + Identity Token OIDC is a thin layer (5%) on top of OAuth 2.0 and one important addition is the ID token. The resource server can, in addition to being asked to allow access, now can understand the identity of the principal requesting to access resources from the ID token. The OIDC layer also uses standard set of scopes and proposes a userinfo endpoint for client to get more details about user information. The authorization code flows in OIDC and OAuth2.0 are roughly the same except for the additions.\nIn terms of the flows supported, the OAuth flows are defined in RFC6749, including the following grants:\nAuthorization Code grant Implicit grant Resource Owner Password Credentials grant Client Credentials grant The classic grant type is Authorization Code. After verifying with user, the Authorization Server fires a call-back to the client to pass the authorization code. The client, then takes the authorization code, along with its client ID and client secret, to fire a request to Authorization server in exchange for Access Token. The implicit grant skips the Authorization Code step and the client gets the Access Token in a one-stop shop via callback over front channel, which is less secure. The other two grants are less often used.\nOn the OIDC side, the specification document discusses three flows:\nAuthorization Code Flow (specification 3.1) Implicit Flow (specification 3.2) Hybrid Flow (specification 3.3) The original OAuth2.0 flows should only be used in delegated authorization scenario. In most contexts, if we talk about ID token, and if our use case involves authentication, then we\u0026#8217;re talking about OIDC not just OAuth. Since Authorization Code Flow is the classic one, out of all these flows, in the next section we take a closer look at the Authorization Code Flow in OIDC.\nAuthorization Code Flow in OIDC We consider the Authorization Code Flow the baseline flow and others as variations of it owing to architectural limitations. When we mention OIDC we implicitly refers to the Authorization Code Flow unless the context suggests otherwise. Now let\u0026#8217;s zoom in on it:\nOIDC Authorization Code FlowOIDC Authorization Code FlowResource Owner\n(User)Resource Owner\u0026#8230;Client App\n(front \u0026amp; backend)Client App\u0026#8230;Authorization ServerAuthorization Se\u0026#8230;Resource ServerResource Server1. launch client app1. launch client app2. Token Request to /authorize2. Token Request to /authorize3. 302 redirect to prompt user to log in3. 302 redirect to prompt user to log in4. Authenticate and consent4. Authenticate and consent6. Redirect browser to the callback URI\n, with Authorization Code as a parameter 6. Redirect browser to the callback URI\u0026#8230;7. Request (HTTP) for Tokens\nat /token, using Authorization Code, ClientID and Client Secret7. Request (HTTP) for Tokens\u0026#8230;8.Validation8.Validation9. Response (HTTP) with IDToken and AccessToken9. Response (HTTP) with\u0026#8230;10. Issue API Request with Tokens10. Issue API Request with Tokens12. Receive API Response12. Receive API Response11. Validation11. Validationbackendback\u0026#8230;Authorization\nEndpointAuthorizat\u0026#8230;Token\nEndpointToken\u0026#8230;5.Validation5.ValidationText is not SVG \u0026#8211; cannot display\nHere is the narrative from my own words:\nThe user launches client application, which detects that user has not logged in, and redirect to log in page. The client app sends an HTTP request for authorization code to the /authorize endpoint of Authorization server. This request consists of the following fields: Response type: code, indicating it is requesting authorization code Scope: openid, standard for oidc RedirectURI: my.com/oidc-callback, Authorization Server will use this to call back with code. The authorization server redirects the user to a prompt for log-in The user completes authentication and consent The Authorization server validates user information within its identity provider The Authorization server fires an HTTP request call-back at the Redirect URI (on the backend), with Authorization Code. The client app issues an HTTP request for ID Token and Access Token to the /token endpoint of Authorization server. This request consists of: Authorization Code (received from previous step) Client ID Client Secret The Authorization server validates the information and process the request, and prepare the response with the following fields: ID Token: identifies the resource owner. Access Token: identifies what the client app can access Expiration (Optional) Refresh Token The client app receives the tokens above in the HTTP response from the /token endpoint With the Tokens, the client app issues API requests to the resource server The resource server independently validates the token The resource server send API response back the the client app. There are also a few points of configurations. First, the Resource Server needs to trust the Authorization Server. The Authorization Server uses its private key to sign the JWT tokens and the Resource Server needs the public key to validates it. Second, the authorization server needs to know about the client app. We usually configure the Authorization Server upfront, to generate the client ID and secret. The Client app will keep them as part of its configuration. It is important to note that, when I use the term client app (OIDC calls it client), the word \u0026#8220;client\u0026#8221; is relative to the Authorization server. The client app itself consists of both frontend (browser) and backend (aka relying party). In this flow, the authorization code is not exposed to browser.\nOther OIDC Flows Now we can discuss some flows for other architectures.\nThe Authorization Code flow has a close variation with the use of PKCE (Proof Key for Code Exchange). For native apps, postman and Okta recommend Authorization Code flow with PKCE. When client app first requests for authorization code, it also includes a challenge. After the callback, when it sends the authorization code back to authorization server in exchange for tokens, the request now adds a verifier. This way, even if the authorization code may not be securely saved, the authorization server can ensure it is the same client app that requests authorization code and that requests tokens.\nAn SPA or JavaScript app does not have a way to a. store Authorization code and b. listen on a call-back URI. As a result, it makes sense for the SPA to just fetch the Tokens directly. This make the implicit flow. The spec doc refers to it as simplified authorization code flow. The grant type is \u0026#8220;implicit\u0026#8221; because there is no intermediate credentials issued. In this flow, the Authorization server does not authenticate its client. The tokens may be exposed to resource owner or other applications with access to resource owner\u0026#8217;s user-agent. This flow improve the responsiveness but we should be wary of the security implications.\nOIDC Implicit FlowOIDC Implicit FlowResource Owner\n(User)Resource Owner\u0026#8230;SPA in Browser\n(frontend)SPA in Browser\u0026#8230;Authorization ServerAuthorization Se\u0026#8230;Resource ServerResource Server1. launch SPA1. launch SPA2. Token request to /authorize2. Token request to /authorize3. 302 redirect to prompt user to log in3. 302 redirect to prompt user to log in4. Authenticate and consent4. Authenticate and consent5.Validation5.Validation6. Response (HTTP) with IDToken and AccessToken6. Response (HTTP) with\u0026#8230;7. Issue API Request with Tokens7. Issue API Request with Tokens11. Receive API Response11. Receive API Response8. Validation8. ValidationText is not SVG \u0026#8211; cannot display\nAs stated above, for SPAs (or in-browser JavaScript), we have to choice but the Implicit Flow because the client app is front-end only and we do not consider it able to securely store credentials. Client apps that can securely store client credentials may benefit from Hybrid Flow. In the hybrid flow, when the authorization server fires callback, the callback includes a single-use authorization code, along with ID token, access token, or both, depending on the provided response_type. Then the client app sends it back to authorization server, along with client credentials, in exchange for a second ID token and access token. The specification has a good table that compares the three flows:\nApart from the three flows, there are also some \u0026#8220;unofficial\u0026#8221; OIDC flows, that are not discussed in the specification. For example, Auth0 adopted some original OAuth2.0 grants in conformance to OIDC, including Client Credentials Flow with OIDC and Resource Owner Password Flow with OIDC. The client credentials flow is for machine-to-machine identity and is not concerned with user identity.\nIdentify the Flow A challenge that I faced is to make sense of the Authorization Code Flow in real life. I realized that the components (Client App, Authorization Server and Resource Server) in Authorization Code Flow are conceptual. In real life we do not always find a counterpart that perfectly match their features. When we try to introduce OIDC for authentication, we often need to build our own solution, with additional tools.\nTake the client app for example. We would need one or several components in real life to perform the followings in order to qualify as a Client App in the sense of Authorization Code Flow, it needs to:\nknow the authorization endpoint and construct the HTTP request for Authorization code; stand up an HTTP service (relying party) to listen to call back, and parse the Authorization Code; securely store Authorization code, and have access to client ID and client secret; construct a request for tokens using client ID, secret and authorization Code received; parse the tokens from the response from Token endpoint to pass the tokens along When developers builds an application with OIDC integration capability, they\u0026#8217;d have to implement all these using the library of their programming language. In addition to application\u0026#8217;s own server, the OIDC module will need its own backend capable of doing all the activities above. The alternative option is to introduce a OIDC capable client proxy service.\nAs for the Resource Server, it needs to have a trust on the Authorization Server, so that it can cryptographically validate the tokens that the Authorization Server has issued using the well-known public key.\nOn the Authorization Server side, as we discussed. It needs to provision client ID and client secret that itself can later recognize when client app connects to it. It also needs to have both authorization endpoint and token endpoint. Often times, the authorization server contains identity store and we\u0026#8217;d like to call it the identity provider, but that is not always the case. A company may have a home grown identity store that does not support OIDC. In that case, to qualify as an OIDC Authorization server, they need a server proxy.\nOpen ID Connect Specifications Despite the different implementation by different vendors, we often need to resort to the official standard documentation. This Open ID connect page lists all the specification if you expand \u0026#8220;OpenID Connect specification\u0026#8221; under Final Specifications. The most commonly used ones are:\nOpenID Connect Core specification [OpenID.Core.Errata2], which covers the foundation and three login flows (Authorization Code, Implicit and Hybrid). This was developed early and the current version [OpenID.Core.Errata2] is from Dec 2023 but the two previous versions [OpenID.Core.Errata1] and [OpenID.Core.Final] had been around since 2014; Open ID Connect Session Management [OpenID.Session], another core document that stipulates how to manage sessions, finalized in Sept 2022; Open ID Connect Discovery 1.0 [OpenID.Discovery], which stipulates the hosting OIDC discovery document, finalized in Dec 2023; Open ID RP-Initiated Logout [OpenID.RPInitiated], one of the logout flow specification, drafted in 2020 and finalized in Sep 2022; Open ID Front-Channel Logout [OpenID.FrontChannel], one of the logout flow specification, drafted from March 2016 and finalized in Oct 2022; Open ID Back-Channel Logout [OpenID.BackChannel], one of the logout flow specification, drafted in 2016 and finalized in Sep 2022; Do read the specification when you\u0026#8217;re configuring integration. It is worth noting that apart from the Core specification which has been finalized for a decade, most of the other specifications did not finalize until late 2022. Therefore, it is important for integrators to validate the compliance state of the components in the implementation.\nOIDC Proxy As a result, with regard to OIDC, there are two categories of proxies: OIDC client proxy and OIDC server proxy. For example, in my previous post, I explained how to configure external authorization via OIDC in Istio. Looking at the diagram, it uses OAuth2 proxy to integrate with GCP as the authorization server. In this use case, GCP is natively OIDC capable, the the Hello Word App isn\u0026#8217;t. Therefore, the OAuth2-proxy that we introduced is an OIDC client proxy. For a corporate with Active Directory, the identity store only supports LDAP protocol. In order to quality the identity store as an OIDC Authorization Server, we would need a server-side proxy such as the LDAP connector in Dex, with the Active Directory as authentication source. The diagram of dex is a good summary of its role:\nIn some scenarios, we refer to this role as identity broker. Dex is an identity broker. Another important project to know is KeyCloak, which is sponsored by Red Hat and now a CNCF project. Although you can configure KeyCloak as an identity broker, it is much more than a broker. KeyCloadk is a full-fledged identity and access management solution on its own. It can act as the entire Authorization server. The diagram in this blog post summarizes its features well. It is for teams that wants to build their home grown identity store. Think of KeyCloak as a self-managed open-source alternative to IAM solutions such as Okta or Auth0.\nThe Amazon Cognito user pool plays a similar role. A user pool serves as an identity store to an app. The integration (no matter which flow and how Cognito calls them) is supposed to be OIDC compliant. However, as of date, the integration with Cognito user pool isn\u0026#8217;t. For example, the logout endpoint requires client_id parameter where as the RP initiated logout specification has it optional. On the other hand, it can federate its own identity pool with a third party via standard protocol including OIDC.\nIn any use case where we need to bring OIDC integration, we need to start with the flow recommendation for each architecture, then we examine the existing component against the flow diagram. From there, we can identify the missing pieces and determine where and how we should configure the proxy.\nSummary The OIDC topic confuses me big time every time I need to configure identity store. With this post, I was hoping to elaborate on the Authorization Code Flow for OIDC. See OpenID certification for a list of providers. Further I discussed the two categories of proxies in the OIDC picture. Hopefully, when the OIDC topic comes back again, I will be able to quickly match which is which, and identify the missing piece to build a solution.\nPrevious PostKubernetes Platform as a Service and Red Hat OpenShift Next PostAuthentication to kube-apiserver via OIDC ","date":"2023-07-13T21:24:01-04:00","image":"/wp-content/uploads/2025/04/feature-oidc-oauth-2.webp","permalink":"/2023/07/oauth-2-0-and-oidc-2-of-2/","title":"OAuth 2.0 and OIDC 2 of 2"},{"content":"The Three-layer model Kubernetes is so complex that it becomes a buzz word itself. I categorize the related work into three layers: a cluster layer, a platform layer and an application layer, by their purposes. The three layers are illustrated as below:\nKubernetes PlatformKubernetes PlatformKubernetes ClusterKubernetes ClusterApplicationApplicationAKS, EKS, self-built clusterAKS, EKS, self-built clusterROSA, AROROSA, AROOpenShift Container Platform\nSelf-managed platformOpenShift Container Platfo\u0026#8230;Text is not SVG \u0026#8211; cannot display\nLet\u0026#8217;s examine each layer in this model and where the Kubernetes Platform as a Service fits in.\nThe Kubernetes Cluster Layer At the bottom, the Kubernetes Cluster layer is the foundational layer. It focus on using self-hosted VMs or cloud resources to build a functional Kubernetes cluster and worker node groups. A functional cluster includes a highly available control plane, as well as scalable node groups that all communicate with the control plane. Cloud Service Providers like AWS and Azure provides managed Kubernetes service, which takes away the complexity (and flexibility as well) of managing control plane components such as etcd store and API server. The managed services also automatically provisions computing nodes and join them into the cluster. The cluster layer may also involve integration with of CNI and CSI, to ensure Pod-to-Pod communication and available storage classes. Professionals working at this layer are infrastructure experts who understand networking, storage, as well as how to manage cloud resources or VMs, infrastructure as code. On a daily basis, they deal with VPCs/V-Nets, subnets, EBS/Azure Disk, File storage, EC2/Azure VMs, etc. When the team is doing a bad job at this layer, you might see symptoms like unresponsive cluster API, orphaned worker nodes, or kubectl failing to connect to cluster endpoint.\nThe tenants (applications) of the Kubernetes platform does not directly interact with this layer. If you decide to switch CSP vendor, this layer requires 100% re-engineering because the managed Kubernetes service by each CSP is different.\nThe Kubernetes Platform Layer The Platform layer sits in the middle. When organization decides to adopt Kubernetes, they often underestimate the efforts required in this layer. This layer works on a functional cluster, without directly interacting with the underlying cloud resources. This layer involves any Kubernetes abstractions that do not creates tangible business value. Rather, this layer is an enabler. It allows the applications to deploy smoothly, evolve quickly, and more importantly, focus on the business.\nTeams working on this layer needs to be Kubernetes experts. On a daily basis, they play with common CNCF toolings, such as Prometheus, ArgoCD, Istio, Cilium, Tekton, Open Policy Agent, etc. They are comfortable with Operators, Helm Charts, Ingress, etc. Inside of the Kubernetes cluster, they also manage the foundational services such as Event streaming (e.g. Kafka), PostgreSQL database (e.g. PostgreSQL), software-defined storage (e.g. Ceph), service mesh (e.g. Istio), Authentication (e.g. Keykloak) , etc. These services act as the infrastructure layer to the business workload. If the team is doing a bad job, you would see data loss with database, observability service not populating data, ingress does not process request, etc. The tenants (application) share services in this layer. If you decide to switch CSP vendor. I estimate 80% of the work at this layer is portable, and 20% requires re-engineering. That is because each CSP offers different external resources, therefor the low level Kubernetes objects in this layer, such as storage classes, load balancers, supported CNIs are different. High level objects such as Kafka remains portable across platforms.\nThe Application Layer The next layer at the top is application layer. Workloads in this layer are directly linked to the business value. The applications are very diverse. Most of the time, the release team is the main player at this layer. If the organization develops its own application, the software development team also work at this layer. In terms of knowledge, the members of development team are experts in software engineering, and Software Development Life Cycle (SDLC), etc. On a daily basis, they deal with programming languages, product development, build and release. If they screw up their work, expect business errors, such as orders sent to wrong client, incorrect balance sheet, etc. This team has high visibility in the organization due to its direct link to business value.\nThis layer of work involves multiple tenants. Each tenant is isolated within their own namespace. When you switch CSP vendor, this layer should be readily portable with minimal effort.\nIt is also worth noting that, with solid platform and cluster layers, the team working at this layer do not write bespoke code for networking, observability, authentication and authorization, encryption and many other aspects not relevant to the core business. Once deployed, the application services are resilient, scale to demands, and cost efficient. This layer reaps the benefits of Kubernetes. Kubernetes Platform as a Service As the Kubernetes dust is still settling, a builder\u0026#8217;s title may not always reflect which layer she or he focuses on. Today it is pretty common for infrastructure engineers to expand their role into the platform layer, or likewise, a software engineer to drill down to the platform layer. The boundary between platform layer and cluster layer is clear. The cluster layer deals with underlying infrastructure, either in the cloud or on premise. They abstract away the complex infrastructure world from those working with the platform layer. The boundary between platform layer and application layer is a little tricky to articulate. The application layer focuses on implementing the business logics. The platform layer takes care of the functions that are not part of business logic but essential to the business application. Take an HTTP request for example, application developer should not have to write code to terminate TLS (not part of business logic). They should only write the code to process the HTTP request (business logic). TLS termination is delegated to an Ingress, to be configured by platform builders. The folks working at the Platform layer needs to interface with both sides. They provide Platform as a Service to the Application teams. However, their work appears mostly invisible in an organization. Their effort is oftentimes underestimated. There are several reasons for that. First, the platform layer does not directly create tangible business value. They are just someone else\u0026#8217;s enabler. Second, their building blocks involve a lot of abstractions by Kubernetes API. Third, the idea of platform engineering is newly emerged. There hasn\u0026#8217;t been a populous recognition of its value.\nRed Hat OpenShift Container Platform The platform team builds the platform with their choice of open-source tools. For clusters using OpenShift Kubernetes Engine, Red Hat introduces Open Shift container platform consisting of Red Hat\u0026#8217;s opinionated (but validated) choice of toolings, for example:\nOpenShift Service Mesh: Istio OpenShift Streams: Apache Kafka OpenShift GitOps: ArgoCD OpenShift Container Platform Pipelines: Tekton OpenShift Serverless: Knative OpenShift Data Foundation: Ceph Clients building their clusters with OpenShift Kubernetes Engine may build their own platform with the toolings in the OpenShift enterprise Kubernetes container platform. For more services, check out the documentation for OpenShift Container Platform. For customers with OpenShift Kubernetes Engine, their options to DIY platform are:\nEntry-Level: Red Hat OpenShift Kubernetes Engine: Enterprise Kubernetes distribution on RHEL CoreOS Mid-Level: Red Hat OpenShift Container Platform (RHOCP): Plus-Level: Red Hat OpenShift Platform Plus: RHOCP + advanced cluster management, security, data management essentials, enterprise container registry OpenShift runs the business model of Kubernetes PaaS.This is a unique business model that I do not find a matching competitor. Even if you choose to DIY your own platform, the Red Hat\u0026#8217;s choices are still a great reference. The OpenShift enterprise Kubernetes container platform maps perfectly to the platform layer of the three-layer model, aiming to simplify the work in the platform layer.\nManaged RedHat OpenShift At first, the OpenShift container platform started as a value add-on to the Kubernetes Engine. Now it\u0026#8217;s a separate product line in their business model. In the mean time, OpenShift partners with major CSPs, to develop the cloud service editions, including:\nRed Hat OpenShift on AWS (ROSA) Microsoft Azure Red Hat OpenShift (ARO) Red Hat OpenShift Dedicated \u0026#8211; on AWS and GCP Red Hat OpenShift on IBM Cloud These offerings are managed Kubernetes Platform as a Service in the cloud. Since RedHat is the only player in this model, we can refer to them as managed OpenShift services. In addition to an already-confusing world of Kubernetes platform portfolios, these offerings gives consumers even more options. On AWS for example, users have the following options:\nManaged Platform: OpenShift Dedicated, managed by Red Hat Managed Platform: Red Hat OpenShift Service on AWS (ROSA), managed by Red Hat and AWS Self-built cluster: OpenShift Container Platform This video discussed more details about these options, such as support model. It is also worth noting that these options tend to be much pricier than managed clusters such as EKS and AKS.\nSince a Managed RedHat Platform makes it easy to deploy, let\u0026#8217;s take ROSA as an example and create a cluster. To enable ROSA in AWS console, click on \u0026#8220;Getting Started\u0026#8221;. The next page ensures ROSA is enabled and checks other prerequisite such as meeting service quotas and creating ELB service-linked role, as show below:\nNow, with an AWS account (and ROSA enabled), a RedHat account, and the rosa-cli utility, we can create a cluster with just a few commands. As a note, be wary of the cost and do not forget to delete the cluster afterwards.\nCreate a ROSA cluster With the following set of commands, we can kick off cluster creation, using STS. We can bring our own VPC, so long as it meets certain prerequisites. I use the Terraform template in the vpc-base project, to create the underlying VPC. We\u0026#8217;ll need the followings from this template:\nThe CIDR range of the VPC: as input with a default The subnet Ids of the private subnet to place, printed in the output The subnets are private subnets, because we want to provision the cluster with private node and private endpoint. When we use rosa CLI, we provide the CIDR and subnet IDs.\n# start with AWS cli configured to the correct profile rosa login # with redhat account and past token rosa create account-roles --mode auto -y # this command creates the IAM roles ManagedOpenShift-*-Role, with RedHat account as trust entity rosa verify permissions # optional rosa verify quota # optional export ROSA_CLUSTER_NAME=\u0026#34;dhc\u0026#34; \\ OPENSHIFT_VERSION=4.13.4 \\ AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) \\ AWS_DEFAULT_REGION=us-east-1 rosa create cluster --sts --private \\ --cluster-name $ROSA_CLUSTER_NAME \\ --multi-az \\ --private-link \\ --region $AWS_DEFAULT_REGION \\ --version $OPENSHIFT_VERSION \\ --enable-autoscaling \\ --min-replicas 3 \\ --max-replicas 3 \\ --compute-machine-type m5.xlarge \\ --machine-cidr 147.206.0.0/16 \\ --subnet-ids subnet-052852a1fb4d7d2ad,subnet-06d8d40ae39d55c47,subnet-0f67ce08bc588012c The CLI will pick up the correct VPC by CIDR, and prompt you to confirm creation of private cluster. After the command kicks off, it will wait for OIDC provider creation, and role creation. Then it uses a Terraform template to create the related resources including VPC. Use this command to check status:\nrosa list clusters rosa describe cluster -c dhc When the second command displays the state of waiting (Waiting for OIDC configuration), we can create OIDC provider:\nrosa create operator-roles -c $ROSA_CLUSTER_NAME --mode auto --yes rosa create oidc-provider -c $ROSA_CLUSTER_NAME --mode auto --yes Throughout the process, we can monitor the install log (terraform output) with:\nrosa logs install -c dhc --watch In the log, you might see errors with terminals connecting to the terraform backend, which doesn’t necessarily indicate a cluster creation error. Always check the cluster state until it reports success. Kick the tires Eventually the describe cluster command will show ready state. We can now create an admin user:\nrosa create admin -c $ROSA_CLUSTER_NAME The command above prints an oc command (OpenShift CLI, equivalent to kubectl) with password to log in. Let\u0026#8217;s examine the cluster with oc. Because it is a private cluster, the endpoint is not available publicly. However, it is accessible from the Bastion host. Use the SSM Session Manager technique from my previous post to SSH to the Bastion Host, which should have oc installed. To install oc yourself, use HomeBrew on Mac. On Linux or Windows, log on to OpenShift console, go to Downloads on the left pannel and find it out under CLI tools.\nThe oc command may report insecure TLS on the login URL. Wait for a few minutes for the certificate to come off as safe. Once you run the oc command with password, it should return \u0026#8220;Login successful\u0026#8221; and then we can connect to the cluster:\n$ oc get node # or kubectl get node NAME STATUS ROLES AGE VERSION ip-147-206-135-41.ec2.internal Ready,SchedulingDisabled infra,worker 3m5s v1.26.5+7d22122 ip-147-206-155-141.ec2.internal Ready control-plane,master 25m v1.26.5+7d22122 ip-147-206-156-81.ec2.internal Ready worker 19m v1.26.5+7d22122 ip-147-206-164-21.ec2.internal Ready infra,worker 3m3s v1.26.5+7d22122 ip-147-206-179-90.ec2.internal Ready worker 19m v1.26.5+7d22122 ip-147-206-191-118.ec2.internal Ready,SchedulingDisabled control-plane,master 26m v1.26.5+7d22122 ip-147-206-192-232.ec2.internal Ready worker 19m v1.26.5+7d22122 ip-147-206-193-198.ec2.internal Ready infra,worker 3m20s v1.26.5+7d22122 ip-147-206-218-114.ec2.internal Ready control-plane,master 26m v1.26.5+7d22122 You can use oc the same way you\u0026#8217;d use kubectl. Both works through SOCK5 proxy. In the meantime, log in to the RedHat console with your Red Hat credential, you can see the cluster in Ready state as well:\nThe rosa create admin command creates a htpasswd type (username-password) of identity provider (IdP) with a user named cluster-admin and a preset password. In real life however, we often configure third party IdP with OIDC integration. I\u0026#8217;ll have to leave this to the next blog post.\nWe shall see the nodes as EC2 instances from AWS console as well. Note that there are three roles: control-plane, worker and infra. The infra nodes are for infrastructure services. These services (Ingress Controller, GitOps, Pipeliens) are the ones in the platform player as we discussed above. There are many customizations you can make in this installation process and I\u0026#8217;d have to defer to the ROSA documentation. To clean up, use the following ROSA command:\nrosa remove cluster -c dhc The output also gives you the command to delete operator roles and OIDC provider, for example:\nrosa delete operator-roles -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes rosa delete oidc-provider -c 23o4u3j98tqmlbtjo612opb7a4bbim5f --mode auto --yes Then we can destroy the VPCs using terraform.\nROSA with HCP Update Oct 2023:\nThe deployment above provisioned a few nodes for control plane, which add to the overall time to provision a cluster. In Aug 2023, there is a new option Hosted Control Plane (HCP) that came to allow users to provision a hosted control plane. This results in cost savings and shorter time to provision a cluster. Here is a table of comparison between the ROSA with HCP and ROSA classic.\nFinal words In this post, I discussed the three-layer model and pointed out that platform layer isn\u0026#8217;t as visible as the other two. I also experimented ROSA as a turn-key Kubernetes platform with its opinionated stack of services.\nSome misinformed organizations even skip the entire platform layer in their estimate of effort. They build a cluster, ran a hello-world service and assumes they can start putting applications on the Kubernetes cluster. There are also customers who purchased the entire Managed OpenShift platform but only use it as a cluster. Yikes!\nThe concept of Kubernetes platform, or generally platform engineering is still spreading. The consulting team that I worked in full-time last year re-branded itself as platform engineering. Marketings are pushing it. Builders are doing it. We\u0026#8217;ll keep an eye, on whether customers are buying it.\nPrevious PostConnect kubectl to private Kubernetes cluster in EKS and AKS Next PostOAuth 2.0 and OIDC 2 of 2 ","date":"2023-06-25T11:10:15-04:00","image":"/wp-content/uploads/2025/04/feature-rosa.webp","permalink":"/2023/06/kubernetes-paas-and-red-hat-openshift/","title":"Kubernetes Platform as a Service and Red Hat OpenShift"},{"content":"Managed Kubernetes services give user a cluster endpoint and a number of worker nodes, with the choice. For each access, users have the choice of making them publicly available, or keeping them on private networking. In my opinion, any deployment beyond personal hobbies, should use Kubernetes private cluster, with both cluster endpoint and worker nodes on private subnet. There is no reason to expose computing nodes or Kubernetes management traffic publicly. For worker nodes, it is fairly easy to put VMs on private network, but many companies still have the cluster endpoint exposed publicly. There are usually two reasons. First, their CI/CD agent is hosted somewhere else on the Internet (instead of on private network with private connectivity to Kubernetes cluster) and need to access Kubernetes cluster endpoint. Second, when the cluster needs to connect with third-party identity provider as OIDC provider, a two-way communication is needed. There is a classic pattern of using a public bastion host (jump box), with a bastion host on the public subnet, routable to the private endpoint of managed Kubernetes service. Clients then connect to the bastion host via port 22 on a public IP address. The authentication is based on SSH key pair, or worse, password. The port forwarding (aka SSH tunnelling) capability enables all the magics. Exposing a jump box in the public subnet with RSA key authentication is still not favourable. In this post, I\u0026#8217;ll examine some secure patterns to connect to private endpoint with improved security posture. AWS options There are two problems. First, how to establish connectivity to the Bastion host in a private subnet. Second, how to use the Bastion host to proxy traffic to the cluster endpoint also in private subnet. To the first problem, there are two potential solutions: SSM Session Manager, and EC2 Instance Connect (EIC) with EIC endpoint (EICE).\nSSM Session Manager was introduce in 2018. It runs an agent on the EC2, which initiates a connection to the SSM endpoint on the AWS side. This connection enables not only Session Manager, but also other Systems Managers (SSM) services such as Fleet Manager, Patch Manager and State Manager. The problem that session manager originally addresses is server management.\nAWS launched EC2 Instance Connect (EIC) in 2019, and EIC Endpoint (EICE) in 2023. EIC addresses the problem with managing SSH key pairs at scale. It dynamically generates an SSH key pair for server access, based on IAM permission. However, it still requires an instance to have its SSH port publicly accessible. With EICE, it is no longer a requirement. In the diagram, EICE is placed in a private subnet, allowing EICE service to reach private instances at their SSH port. Here is a comparison of the two:\nEC2 Instance Connect (EIC) with EIC EndpointSSM Session ManagerLocation of Bastion hostPrivate Subnet.Private SubnetNeed Ingress PortYes. Port 22 must open to the endpoint.No. SSM agent initiate outbound connection from the instanceTraffic PathAWS CLI → AWS EIC ES → EICE→EC2 InstAWS CLI → AWS SSM ES → SSM ← EC2 InstAuthenticationAWS IAM and ephemeral SSH key when using AWS CLI directly\nAWS IAM and long-term SSH key when using SSH proxy commandAWS IAM and long-term SSH key when using AWS CLI directly or SSH proxy commandWork with OpenSSHYesYesCostThere is no additional cost for using EIC.No additional cost, unless private SSM Endpoint. Let\u0026#8217;s take a look at each option.\nEC2 Instance Connect To use EIC, pick an AMI that has it pre-installed and ensure instance profile has correct policy, as the document states here. AWC CLI will make use of local OpenSSL client. So make sure there connection at port 22 is open. To make it work with EC2 instance on a private subnet, create an EC2 Instance Connect Endpoint on the VPC, and ensure that the security group of EC2 allows port 22 from the Endpoint. Run this command:\n$ aws ec2-instance-connect ssh --instance-id i-00ea30a6e02db33fe The command above simply generates a key pair internally, add the public key to the server side, and connect with SSH from the client side. The command takes you to an SSH session. Checking ps -ef | grep ssh on the client machine, you can see the full parameter of SSH, including the location of the ephemeral private key. However, if you use AWS CLI open-tunnel as proxy command to ssh, then you\u0026#8217;d still have to use the key pair used to create the EC2 instance. As suggested at the bottom of this blog post, the command is:\n$ ssh ec2-user@[INSTANCE] -i [SSH-KEY] -o ProxyCommand=\u0026#39;aws ec2-instance-connect open-tunnel --instance-id %h\u0026#39; This is a bummer, because with native SSH tool you do not get the primary benefit of EIC \u0026#8211; ephemeral key pair. SSM Session Manager Now let\u0026#8217;s look at SSM session manager. Similarly, it needs an agent installed and IAM role configured. You can connect to from web console but more importantly, from AWS CLI:\n$ aws ssm start-session --target i-0531b19bec8ad022d This command takes you to an SSH session with user ssm-user, without starting an OpenSSH client process locally. User do not have to manage key pair. There is also a document about using this command as proxy command, which uses an SSM document. I have one of the SSH config entry as:\nhost i-* mi-* ProxyCommand sh -c \u0026#34;aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters \u0026#39;portNumber=%p\u0026#39;\u0026#34; User ec2-user IdentityFile ~/.ssh/id_rsa This allows me to directly SSH to client using OpenSSL client (e.g. ssh i-0531b19bec8ad022d) by Instance ID. With this, I also need to specify my own OS user and matching private key.\nI know I will use the OpenSSH client a lot from pipelines because it is very powerful. In both options, I have to live with managing key pairs myself. With SSM session manager\u0026#8217;s proxy command, the instance does not need port 22 to open, which is a great advantage, in terms of security and operation. SSM Session Manager is a winner.\nSOCKS5 proxy for kubectl Either SSM Session Manager or EIC with EICE enables an SSH tunnel with key encryption between client (a local computer or a pipeline agent). On top of the SSH tunnel, we can build a SOCKS5 proxy. Kubernetes document has a good page on how to do this. I managed to get this to work with a few gotchas.\nPrivate SubnetBastion HostBastion HostPipeline AgentPipeline AgentSSH TunnelSSH TunnelSSHD\nProcessSSHD\u0026#8230;SOCKS5 Proxy\nby SSHSOCKS5\u0026#8230;kubectlkubectlaws cliaws cliEKS\nCluster\nEndpointEKS\u0026#8230;AWS Service Endpoint\nhttps://eks.us-west-2.amazonaws.comAWS Serv\u0026#8230;InternetInternetNote: kubectl calls aws-cli for authentication. So make sure that aws-cli uses the right profile and assumes the right role, if applicable.Note: kubectl calls aws-cli fo\u0026#8230;The SSH Tunnel is established on top of a proxy command using SSM session manager or EC2 Instance Connect with EIC EndpointThe SSH Tunnel is established on top of a\u0026#8230;Tell kubectl to use SOCKS5 proxy by the HTTPS_PROXY environment variable or by the proxy-url attribute in .kube/configTell kubectl to use SOCKS5 proxy by th\u0026#8230;Text is not SVG \u0026#8211; cannot display\nTo put this in practice, I first created a VPC stack with a bastion host using terraform template from my vpc-base project. The terraform output will give the next set of commands to run to create a private cluster, using a manifest rendered from the file private-cluster.yaml.tmpl:\n# cd aws_vpc # terraform init # terraform plan # terraform apply # ... run the given command ... # envsubst \u0026lt; private-cluster.yaml.tmpl | tee | eksctl create cluster -f - # Run this from a remote host without access to cluster endpoint. # Run terraform apply and terraform output contains the variables needed for the next steps # the command below may take 15 minutes to create a private cluster eksctl create cluster -f private-cluster.yaml aws eks update-kubeconfig --name private-cluster At this point, the kubeconfig file has been updated, but kubectl (from Internet or on-prem) is unable to connect to cluster endpoint (on private network). In order to BASTION_SECURITY_GROUP_ID=$(terraform output -raw bastion_sg_id) CLUSTER_SECURITY_GROUP_ID=$(aws eks describe-cluster --name private-cluster --query \u0026#34;cluster.resourcesVpcConfig.clusterSecurityGroupId\u0026#34; --output text) # In Cluster Endpoint\u0026#39;s security group, open up port 443 to Bastion host aws ec2 authorize-security-group-ingress --group-id $CLUSTER_SECURITY_GROUP_ID --source-group $BASTION_SECURITY_GROUP_ID --protocol tcp --port 443 # Test with connecting to Bastion host with ssh i-0750643179667a5b6, assuming .ssh/config file is configured as above. From the bastion host, you can test: # curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api # if you get an HTTP response, even an error code 403, the bastion host has TCP connectivity to cluster endpoint # then we can start an SSH session as a SOCKS5 proxy on the remote host ssh -D 1080 -q -N i-0750643179667a5b6 # add \u0026gt; /dev/null 2\u0026gt;\u0026amp;1 \u0026amp; to push it to background, or use ctrl+z after running the command # to validate that the SOCKS5 proxy is working, you can run the same curl command with a proxy parameter: # curl -k https://EC5405EE1846F19F9F61ED28FB12A6A9.sk1.us-west-2.eks.amazonaws.com/api --proxy socks5://localhost:1080 # you can instruct kubectl to use the SOCKS5 proxy with the following environment variable export HTTPS_PROXY=socks5://localhost:1080 kubectl get node # alternatively, add \u0026#34;proxy-url: socks5://localhost:1080\u0026#34; below server attribute in ~/.kube/config file. There are some pitfalls to watch for. On the remote host both ssh command and kubectl command implicitly uses AWS CLI. Therefore, make sure the profile and IAM role are correctly configured. For example, if SSM agent requires one IAM role, and kubectl is created with another IAM role, then make sure AWS CLI assumes the correct IAM role using environment variables, and use \u0026#8220;aws sts get-caller-identity\u0026#8221; to validate the IAM identity being used.\nWhat about AKS in Azure I touched on this in my Azure notes in 2021 and did a research again. Unfortunately, options are still fairly limited. The first option is to use a managed service called \u0026#8220;Azure Bastion\u0026#8221;, which requires public IP and a dedicated subnet with the exact name of AzureBastionSubnet, as well as some additional requirement. I\u0026#8217;m not impressed with these requirement because it is meant to be a managed service. The other option, is essentially to DIY a JumpBox. The idea is the same: put the jumpbox in a public subnet, which is routable to private subnets. When you need to connect to private VMs, get to the jumpbox first.\nApart from having to put the bastion VM on a public subnet, the pattern that we discussed above involving SOCKS5 proxy still works. Exposing a bastion host isn\u0026#8217;t ideal but it still reduces attack surface significantly, comparing to exposing the cluster endpoints of all Kubernetes API servers.\nSummary Many immature Kubernetes configurations exposes private endpoint publicly. Having cluster endpoint in private subnet greatly improves security posture. In my opinion, there are very few situations where cluster endpoint must exposed publicly. Having private endpoint should be mandatory for all Kubernetes cluster. In the next post, I also cover how to create a ROSA cluster with private endpoint.\nPrevious PostKubernetes with Multiple CPU Architectures 2 of 2 – Node and Workload Next PostKubernetes Platform as a Service and Red Hat OpenShift ","date":"2023-06-10T19:31:00-04:00","image":"/wp-content/uploads/2025/04/feature-kubectl-private-cluster.webp","permalink":"/2023/06/connect-kubectl-to-private-kubernetes-cluster-in-eks-and-aks/","title":"Connect kubectl to private Kubernetes cluster in EKS and AKS"},{"content":"The most common server CPU architectures today are amd64 (aka x86_64) and arm64. Although AMD developed the former first, Intel names it as x86_64 (or x64 for short). In terms of compatibility, they are the same. In general, arm64 architecture consumes less power and therefore mobile systems first favour it. Its power efficiency now drives a trend towards computing infrastructure. For example, Apple\u0026#8217;s MacBook moved to M1 processor in 2020. Since 2018, Amazon\u0026#8217;s Graviton processor has entered the third generation. In 2022, Azure also brought Ampere Altra processor, and GCP introduced ARM based VMs. Less power consumption ultimately leads to less computing cost.\nI can only see more workloads gradually move to servers with ARM architecture. With Kubernetes, we will most likely have a fleet of computing node consisting of hybrid CPU architectures. We can take a look at what the arm64 adoption entails for workloads on Kubernetes.\nARM64 architecture Graviton processor is on 64-bit Arm Neoverse cores, targeting for optimizing cloud-native workloads. Currently at AWS, the majority of arm64 instances use Graviton2 processor. This AWS blog posted the news about Graviton3-based general purpose (m7g) and memory-optimized (r7g) EC2 instances. At the bottom, there is a chart that compares the performance of Graviton3 with Graviton2, x86 and M6g instances. We can expect that in a few months the services that supports Graviton2 processor to start supporting Graviton3 processor.\nIn the serverless landscape, you can specify CPU architecture for Lambda function. If your runtime supports arm64 architecture, you enjoy up to 34% price performance improvement according to this post. In late 2021, AWS Fargate for ECS also started to support Graviton2 Processor with arm64 workload. As to Fargate for EKS, it has not supported Graviton2 processor as of yet, but is on track. As to Kubernetes, I\u0026#8217;ve discussed how to get container registries to support platform-specific images. So we can assume image registries all support OCI format image index(aka fat manifest), which points platform-specific images for arm64 and amd64. In this post, I\u0026#8217;ll focus on the node and workload, with EKS as an example. Since control plane is a managed service, we will focus on the worker node, where the Pods are running. Worker Node The cloudkube project uses Terraform to build our test EKS cluster. One of the node groups consists of the new m7g.large instance (Graviton3 processor). For this new node group, the AMI type must be AL2_ARM_64, so it picks up an EKS optimized AMI for arm64 during node provisioning. The IAM role of each node has SSM policy so we can use session manager and pre-installed SSM agents to connect to each node. One the m7g node, I would like to check a few things:\nThe node CPU The containerd package The kubelet executable. They all should be for the right CPU architecture, as the following commands clarifies:\n$ lscpu | grep -i arch Architecture: aarch64 $ yum list | grep containerd containerd.aarch64 1.6.6-1.amzn2.0.2 @amzn2extra-docker containerd-stress.aarch64 1.6.8-1.amzn2.0.1 amzn2extra-docker $ file -b $(which kubelet) ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=5c7a059f13f8bece4ce30f3357d57631c28bdde2, for GNU/Linux 3.7.0, stripped We can also check image pulling with containerd. Let\u0026#8217;s check what is the correct image first, by examining the image index with manifest-tool:\n$ manifest-tool inspect digihunch/colorapp:v0.1 The index in the response tells us that:\nthe digest of the whole image index starts with 0fa335; the manifest digest for the arm64 variant starts with 7479df; the manifest digest for the amd64 variant start with 1bd198; Now if we use ctr utility to pull image (\u0026#8220;sudo ctl pull image\u0026#8221;), we can see the correct digest for arm64:\nThe default behaviour of \u0026#8220;ctr image pull\u0026#8221; is to pull a platform-specific image, overridable with --platform or --all-platforms. On an amd64 node, I get the corresponding results as well.\nWorkload Let\u0026#8217;s ensure all workloads in the Namespaces are using correct images. We\u0026#8217;ll deploy the colorapp, and then examine that along with some system Pods.\nFor example, DaemonSet aws-node has one pod per node. To verify the distribution, we can get to pods\u0026#8217; command shell and check CPU architecture with uname command:\n$ kubectl -n kube-system get po -l app.kubernetes.io/name=aws-node -o name | xargs -I{} kubectl -n kube-system -c aws-node exec {} -- uname -m The command above verifies that pods scheduled to arm64 nodes correctly. It does not however, proof that the arm64-specific image is being used. I find it pretty tricky to validate a container is using intended image on arm64 node. I have not found a working kubectl command. There is a plausible tag named imageID under container status. For kube-proxy Pod in kube-system namespace, there are two values. However, for colorapp pods, there is only one value with different format, even though they are scheduled to nodes of both architectures.\n$ kubectl -n kube-system get po -l k8s-app=kube-proxy -o yaml | grep \u0026#39;imageID:\u0026#39; | sort | uniq imageID: sha256:04beb3b811d345722d689a70a30bafa27e0edd412613bee76c3648b024b25744 imageID: sha256:b9b6705d4ad6be861f0e98b7325e5106715ef21a82692f7e8a005a280f159518 $ kubectl -n default get po -l app=color -o yaml | grep \u0026#39;imageID:\u0026#39; | sort | uniq imageID: docker.io/digihunch/colorapp@sha256:0fa335fdbcc3b644d57c8debe075775b19011985b6342adfb430e7011456d12e This issue reports such inconsistency and the issue unfortunately did not get attention. The reporter also asks to have sha256 of the actual image. However, the Kubernetes developers regard this as an CRI issue. Currently we cannot tell exactly which image is used.\nI figured out a workaround, by getting on the node and dump the image on the node:\n$ sudo ctr -n k8s.io image list $ sudo ctr -n k8s.io image export /tmp/x.tar docker.io/digihunch/colorapp@sha256:0fa335fdbcc3b644d57c8debe075775b19011985b6342adfb430e7011456d12e In the export tar file review the manifest.json file which contains layer digests. We should find these layer digests match those of the platform-specific image\u0026#8217;s. Utilities Since we can ensure that Pod running on a node can always pull the correct platform-specific image, we do not need to worry about Helm chart. We just need to make sure our container registry references an index digest that points to images of multiple architecture. For the same reason, we do not need to worry about pod autoscaling. When it comes to node autoscaling, all node should support have kubernetes.io/arch and kubernetes.io/os labels (e.g. Karpenter). However, we generally prefer to expand the arm64 node group since it is cheaper. With cluster autoscaler, we can use priority based expander. With Karpenter, we can set weight so that the provisioner for arm64 node group carries higher weight. Scheduling With multi-arch image, the container runtime will pick up the right version of image. From deployment perspective, we do not worry about the difference between nodes in CPU architectures. However, in some use cases, we still want to schedule certain Pods to nodes with one CPU architecture over the other. I call these platform-specific workload.\nWe mainly needs to control scheduling behaviour. There are two mechanisms, node affinity, and taints \u0026amp; tolerations. Node Affinity is based on node labeling. From the well-known labels, annotations and taints, all Kubernetes distribution should label their nodes with the kubernetes.io/arch and kubernetes.io/os labels. The value for arch is either arm64 or amd64. When we add a node affinity of requiredDuringSchedulingIgnoredDuringExecution type to Pods, scheduler takes matchExpressions under nodeSelectorTerms into consideration, when placing Pods to Nodes. When a Pod has lots of nodeSelectorTerms, it can be brain twisting to sort through the logic. In that case we can use Taints and Tolerations. The idea is that once we taint a node, the scheduler will not schedule any Pod to the Node, unless the Pod has a matching Toleration.\nIn this post, the author customized the bootstrap script so the node provisioning process automatically taints arm64 nodes with arch=arm64:NoSchedule. Otherwise, we can manually taint a node:\n$ kubectl get no -o wide # and check KERNEL-VERSION column, taint the ones with aarch64 $ kubectl taint nodes ip-147-207-3-164.us-west-2.compute.internal arch=arm64:NoSchedule This can be a very useful technique when you\u0026#8217;re not sure if every workload image are capable of multi-arch, and you want to avoid scheduling any Pods without tolerations on the arm64 nodes. A Pod cannot get scheduled on those nodes until you confirm their container images, and add corresponding tolerations.\nSummary Given the power efficiency, a lot of workload will gradually migrate to arm64 architecture. However, software will take a while to get ready. For example, hyperkit has not supported M1 processor and I still cannot use it on newer MacOS for Minikube. Hybrid architecture is here to stay and we need to have an end-to-end examination of our supply chain.\nPrevious PostKubernetes with Multiple CPU Architectures 1 of 2 – Container Image Next PostConnect kubectl to private Kubernetes cluster in EKS and AKS ","date":"2023-05-20T01:34:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-multi-cpu.webp","permalink":"/2023/05/kubernetes-with-multiple-cpu-architectures/","title":"Kubernetes with Multiple CPU Architectures 2 of 2 – Node and Workload"},{"content":"While working on a multi-arch Kubernetes cluster, I came across quite a few issues with image logistics and decided to put these issues in a separate post here. The goal is to supply images with multi-arch support in a standard way. Therefore it is not advisable to rely only on arbitrary image tags to distinguish CPU architecture. If that was the case, each team may use different tag keys and values. The image consumers would have to build custom logics to pull the right image. The standard way is to use Manifest List (Docker\u0026#8217;s term), or Image Index (OCI\u0026#8217;s term) on the image registry, to announce the multi-platform support of the images in the repository, and to ensure that a container runtime can pick the appropriate index entry that matches the local node\u0026#8217;s architecture and platform.\nIntroducing Image Index For image index/manifest, there have been three standards, with two currently active:\nDocker Image Manifest V2, Schema 1: deprecated in 2019 Docker Image Manifest V2, Schema 2: supports attribute for platform specific image OCI Image Specification: supports attribute for platform specific image In 2016, Docker contributed the Docker V2 Image specification as the basis of the OCI image specification. Therefore, the two current active formats (V2.2 and OCI) still look alike today. They are both JSON document and shares many attribute names. However, I take OCI as the standard since it is adopted in CNCF (e.g. containerd, quay.io). Most of the registries (e.g. ECR, ACR and even Docker Hub) support both formats, and you can tell the format by requesting manifest document. When working with Kubernetes we should expect OCI format index even though the Docker Manifest List v2.2 will most likely be compatible. When examining an image manifest we should look for the following structure:\nManifest list and Manifest, source: https://www.opensourcerers.org/2020/11/16/container-images-multi-architecture-manifests-ids-digests-whats-behind/ In the Docker lingo, a \u0026#8220;fat manifest\u0026#8221; is simply a list of manifests, equivalent to image index in OCI terms. A manifest document consists of several attributes in sha256 digest, as the diagram shows. There are four types of digests and they should not be mixed:\nindex-sha256: for the index or fat manifest manifest-sha256: for the manifest of a single container image config-sha256: for the config section layer-sha256: for the image layer files If two images share a layer, then from their respective manifests, we should be able to find a layer with the same sha256 digest. If an index (fat manifest) references other manifest, we should also find that from the sha256 digest. When you reference an image, you should use the sha256 digest for the entire index or manifest list.\nTo view image index, \u0026#8220;docker inspect\u0026#8221; is insufficient. It is executed within Docker daemon, against the image pulled based on the CPU architecture of Docker daemon\u0026#8217;s server. So let\u0026#8217;s look at some tools to check index from remote registry.\nTools to view image index The most popular tool is Docker\u0026#8217;s experimental manifest inspect command. However, it is still not mature. For example, we have to use Docker CLI version 23.0.0 or later with OCI compatibility issue fixed. Prior to v23.0.0 (Feb 2023), Docker CLI were not able to correctly display a list of OCI-format manifests, and it simply says \u0026#8220;no such manifest\u0026#8221;. Even after 23.0.0, I still find it clunky. For example, it does not display the sha256 digest of the OCI image index itself (the digest on the far left of the diagram above).\nTo troubleshoot the reason, I borrowed some idea from this post, and have my shell script as below, to check the index for my colorapp image:\n#!/bin/sh ref=\u0026#34;${1:-digihunch/colorapp:v0.1}\u0026#34; sha=\u0026#34;${ref#*@}\u0026#34; if [ \u0026#34;$sha\u0026#34; = \u0026#34;$ref\u0026#34; ]; then sha=\u0026#34;\u0026#34; fi wosha=\u0026#34;${ref%%@*}\u0026#34; repo=\u0026#34;${wosha%:*}\u0026#34; tag=\u0026#34;${wosha##*:}\u0026#34; if [ \u0026#34;$tag\u0026#34; = \u0026#34;$wosha\u0026#34; ]; then tag=\u0026#34;latest\u0026#34; fi apio=\u0026#34;application/vnd.oci.image.index.v1+json\u0026#34; apiol=\u0026#34;application/vnd.oci.image.manifest.v1+json\u0026#34; apid=\u0026#34;application/vnd.docker.distribution.manifest.v2+json\u0026#34; apidl=\u0026#34;application/vnd.docker.distribution.manifest.list.v2+json\u0026#34; token=$(curl -s \u0026#34;https://auth.docker.io/token?service=registry.docker.io\u0026amp;scope=repository:${repo}:pull\u0026#34; \\ | jq -r \u0026#39;.token\u0026#39;) curl -H \u0026#34;Accept: ${apio}\u0026#34; -H \u0026#34;Accept: ${apiol}\u0026#34; -H \u0026#34;Accept: ${apid}\u0026#34; -H \u0026#34;Accept: ${apidl}\u0026#34; \\ -H \u0026#34;Authorization: Bearer $token\u0026#34; \\ -w \u0026#39;\\nResponseCode:%{http_code}\\nResponseHeader:\\n%{header_json}\\n\u0026#39; \\ -s \u0026#34;https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}\u0026#34; It appears that the sha256 digest of the OCI image index itself is provided in the response header, instead of response payload. So docker manifest tool misses the header!\nLuckily, there are some alternatives, such as skopeo or manifest-tool. I wasn\u0026#8217;t able to get the former to work with OCI index. The latter displays my OCI index in a pretty format and I was able to view a few other indexes that I wasn\u0026#8217;t able to with Docker manifest. So I have a good impression of it. So in summary, for the four tools to view image index, my recommendations are:\nDocker manifest: still glitch as of 2023 but pretty widespread manifest-tool: pretty solid, the output is pretty format, requires manual install skopeo: not straightfoward to use. easy to install Self-scripting in bash: only for troubleshooting For the rest of this post, I\u0026#8217;ll however continue to use docker manifest, given its popularity and availability. When it fails to display an index, I\u0026#8217;ll try manifest-tool.\nInspecting Image Index With docker manifest command (v23.0.1), let\u0026#8217;s take a peak at three images:\ndocker manifest inspect --verbose osimis/orthanc:22.12.2 docker manifest inspect --verbose public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0 docker manifest inspect --berbose ubuntu:23.04 In the JSON document return by the first command (osimis/orthanc image), we see the following structure:\n{ \u0026#34;Ref\u0026#34;: \u0026#34;docker.io/osimis/orthanc:22.12.2\u0026#34;, \u0026#34;Descriptor\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.distribution.manifest.v2+json\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:20413096878fb56bf8d09af08cf4055993dbcf507526f0561b26fc4d0ed7affc\u0026#34;, \u0026#34;size\u0026#34;: 11227, \u0026#34;platform\u0026#34;: { \u0026#34;architecture\u0026#34;: \u0026#34;amd64\u0026#34;, \u0026#34;os\u0026#34;: \u0026#34;linux\u0026#34; } }, \u0026#34;Raw\u0026#34;: \u0026#34;......\u0026#34;, \u0026#34;SchemaV2Manifest\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.distribution.manifest.v2+json\u0026#34;, \u0026#34;schemaVersion\u0026#34;: 2, \u0026#34;config\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.container.image.v1+json\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:9de20d8a006c6377b85dba9f817d47048982bd0f15fac7daacb64f42060d4b6d\u0026#34;, \u0026#34;size\u0026#34;: 16518 }, \u0026#34;layers\u0026#34;:[ { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.image.rootfs.diff.tar.gzip\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:025c56f98b679f70b7a54241917e56da7b59ab9d2defecc6ebdb0bf2750484bb\u0026#34;, \u0026#34;size\u0026#34;: 31412852 }, ...... ...... ...... { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.image.rootfs.diff.tar.gzip\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:a24cf4c71e6977b692bbf517eba7bf6f454d41ceab24c1b2694c3303bc718a1c\u0026#34;, \u0026#34;size\u0026#34;: 174739 } ] } } We can see that the return is a single manifest. The Descriptor key suggests that it is built for amd64 architecture. The mediaType, along with SchemaV2Manifest and schemaVersion, suggest that it is a Docker Manifest (v2.2) format. Other attributes are summarized here. In the response from the second command(public.ecr.aws/amazonlinux/amazonlinux), we see the structure below:\n[ { \u0026#34;Ref\u0026#34;: \u0026#34;public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d\u0026#34;, \u0026#34;Descriptor\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.distribution.manifest.v2+json\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:260907696498cbf078abc2f3428bf8d19faf77cded5d5459900997a1bc29903d\u0026#34;, \u0026#34;size\u0026#34;: 529, \u0026#34;platform\u0026#34;: { \u0026#34;architecture\u0026#34;: \u0026#34;amd64\u0026#34;, \u0026#34;os\u0026#34;: \u0026#34;linux\u0026#34; } }, \u0026#34;Raw\u0026#34;: \u0026#34;......\u0026#34;, \u0026#34;SchemaV2Manifest\u0026#34;: { \u0026#34;schemaVersion\u0026#34;: 2, \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.distribution.manifest.v2+json\u0026#34;, \u0026#34;config\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.container.image.v1+json\u0026#34;, \u0026#34;size\u0026#34;: 1478, \u0026#34;digest\u0026#34;: \u0026#34;sha256:d27c2e45784db13b0b2bc89a52be6661aa1d53bd25c070b41626768c9c563c3d\u0026#34; }, \u0026#34;layers\u0026#34;: [ { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.image.rootfs.diff.tar.gzip\u0026#34;, \u0026#34;size\u0026#34;: 62386320, \u0026#34;digest\u0026#34;: \u0026#34;sha256:d78505e615251c4f4af6eaa9507b67917d263d23551dcc5a1eed3c012d32a54d\u0026#34; } ] } }, { \u0026#34;Ref\u0026#34;: \u0026#34;public.ecr.aws/amazonlinux/amazonlinux:2.0.20230207.0@sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed\u0026#34;, \u0026#34;Descriptor\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.distribution.manifest.v2+json\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:7fb3183b38e1a9859374a343e72dc43731aeccaf26507da94ebc310067f39fed\u0026#34;, \u0026#34;size\u0026#34;: 529, \u0026#34;platform\u0026#34;: { \u0026#34;architecture\u0026#34;: \u0026#34;arm64\u0026#34;, \u0026#34;os\u0026#34;: \u0026#34;linux\u0026#34;, \u0026#34;variant\u0026#34;: \u0026#34;v8\u0026#34; } }, \u0026#34;Raw\u0026#34;: \u0026#34;......\u0026#34;, \u0026#34;SchemaV2Manifest\u0026#34;: { \u0026#34;schemaVersion\u0026#34;: 2, \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.distribution.manifest.v2+json\u0026#34;, \u0026#34;config\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.container.image.v1+json\u0026#34;, \u0026#34;size\u0026#34;: 1493, \u0026#34;digest\u0026#34;: \u0026#34;sha256:a1ea533a0632c6501d7848c7ed481e8fb0398c3277c0d9fddf0b0fdcd5731c09\u0026#34; }, \u0026#34;layers\u0026#34;: [ { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.docker.image.rootfs.diff.tar.gzip\u0026#34;, \u0026#34;size\u0026#34;: 64003805, \u0026#34;digest\u0026#34;: \u0026#34;sha256:71343c2791199c6e2c19c308cff6493497a02f57e225c11405e1934dc7428b3c\u0026#34; } ] } } ] Comparing this structure with the first return, we noticed that the return is a list of two manifests, each with its own platform architecture. So the amazonlinux image has multi-architecture capability. It can serve as base image for custom images for both architectures. On the \u0026#8220;image tags\u0026#8221; tab of registry page, we can see a list of published tags. Each is tied to either a single image manifest, or a manifest list:\nEach image tag is associated with a single manifest or a manifest list (aka \u0026#8220;fat manifest\u0026#8221;) When you click on \u0026#8220;image manifest\u0026#8221;, you can see both \u0026#8220;Image manifest media type\u0026#8221; and \u0026#8220;Artifact media type\u0026#8221; values. When you click on \u0026#8220;manifest list\u0026#8221;, you see the \u0026#8220;Image manifest media type\u0026#8221; value, because the \u0026#8220;fat manifest\u0026#8221; does not point to a single artifact.\nNow, let\u0026#8217;s review the third command (ubuntu:23.04) response:\n[ { \u0026#34;Ref\u0026#34;: \u0026#34;docker.io/library/ubuntu:23.04@sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0\u0026#34;, \u0026#34;Descriptor\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.manifest.v1+json\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:52293638ba652a2e8f9e1c1cfcc905839b1f2a9e671ddcc9bf77909b6bf527d0\u0026#34;, \u0026#34;size\u0026#34;: 424, \u0026#34;platform\u0026#34;: { \u0026#34;architecture\u0026#34;: \u0026#34;amd64\u0026#34;, \u0026#34;os\u0026#34;: \u0026#34;linux\u0026#34; } }, \u0026#34;Raw\u0026#34;: \u0026#34;......\u0026#34;, \u0026#34;OCIManifest\u0026#34;: { \u0026#34;schemaVersion\u0026#34;: 2, \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.manifest.v1+json\u0026#34;, \u0026#34;config\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.config.v1+json\u0026#34;, \u0026#34;size\u0026#34;: 2299, \u0026#34;digest\u0026#34;: \u0026#34;sha256:beb2152822b716b4deac2996f16bc84db0a14b7cbc549579635590438f9c0e1d\u0026#34; }, \u0026#34;layers\u0026#34;: [ { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.layer.v1.tar+gzip\u0026#34;, \u0026#34;size\u0026#34;: 26638886, \u0026#34;digest\u0026#34;: \u0026#34;sha256:db781b8aed497363312ef32499cbfac28821e0494db7f0cadc4e716853e02a12\u0026#34; } ] } }, { \u0026#34;Ref\u0026#34;: \u0026#34;docker.io/library/ubuntu:23.04@sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c\u0026#34;, \u0026#34;Descriptor\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.manifest.v1+json\u0026#34;, \u0026#34;digest\u0026#34;: \u0026#34;sha256:0c8e3367a3fe9b703c759e1c148c5809df1a2734f8f37529bd11fbcfd34b1d1c\u0026#34;, \u0026#34;size\u0026#34;: 424, \u0026#34;platform\u0026#34;: { \u0026#34;architecture\u0026#34;: \u0026#34;arm64\u0026#34;, \u0026#34;os\u0026#34;: \u0026#34;linux\u0026#34;, \u0026#34;variant\u0026#34;: \u0026#34;v8\u0026#34; } }, \u0026#34;Raw\u0026#34;: \u0026#34;......\u0026#34;, \u0026#34;OCIManifest\u0026#34;: { \u0026#34;schemaVersion\u0026#34;: 2, \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.manifest.v1+json\u0026#34;, \u0026#34;config\u0026#34;: { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.config.v1+json\u0026#34;, \u0026#34;size\u0026#34;: 2316, \u0026#34;digest\u0026#34;: \u0026#34;sha256:eb2d2fb228861107934403e776544a3f516bc7123a1275d52f1992bada8e94d6\u0026#34; }, \u0026#34;layers\u0026#34;: [ { \u0026#34;mediaType\u0026#34;: \u0026#34;application/vnd.oci.image.layer.v1.tar+gzip\u0026#34;, \u0026#34;size\u0026#34;: 25802344, \u0026#34;digest\u0026#34;: \u0026#34;sha256:29d183ded65aecf549f39ef891c21feb9034b5b10f341533b4af297bb5c60bb8\u0026#34; } ] } }, ......, ......, ...... ] I readapted the response document for simplicity. Notice that the document is also a \u0026#8220;fat manifest\u0026#8221; except that the mediaType, OCIManifest and schemaVersion keys suggest that it is an OCI format. The image supports more platforms (combination of OS and CPU architectures). The attributes for OCI index is available here.\nSingle-platform image Build At the beginning of the supply chain, we build platform-specific image with CI/CD jobs. Traditionally, the docker build process can only produce images for the platform where the build command run. For the process to work in multiple platforms, we\u0026#8217;d need multiple build agents of different platforms. Each agent runs build process and pushes artifact to the registry (with \u0026#8220;docker push\u0026#8221;). At the end we create a fat manifest that combines the images for all platforms, with \u0026#8220;docker manifest create\u0026#8221; command. A blog post on Docker from April 2020 refers to this as the \u0026#8220;hard way\u0026#8221;.\nSince then, Docker introduced a new client\u0026nbsp;Docker Buildx, a CLI plugin that extends the\u0026nbsp;docker\u0026nbsp;command with the full support of the features provided by\u0026nbsp;BuildKit\u0026nbsp;builder toolkit. One such feature is the ability to produce multi-platform images in one command run. There is also a push switch that helps you generate manifest behind the scene. In my own testing, I am able to build image for both amd64 and arm64 on MacOS (M1).\nMulti-platform image build Take the colorapp Python application as a simple example, I followed this guide to generate corss-CPU-architecture build. First, we can examine if we already have a build instance that supports our desired platform and whether that is already selected:\n$ docker buildx ls NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS vibrant_hypatia * docker-container vibrant_hypatia0 unix:///var/run/docker.sock running v0.11.3 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6 default docker default default running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6 desktop-linux docker desktop-linux desktop-linux running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6 In the example above, I\u0026#8217;ve already got one named vibrant_hypatia that supports linux/arm64 and linux/amd64 and it is selected. If that\u0026#8217;s not the crease, we can create a new build instance and tell Docker to use it. Then $ docker buildx create --use Then we can run the actual build command. In the build command we specify the platforms and tags. We also specify \u0026#8211;push so we can push the entire binary at the same time.\n$ docker buildx build --platform linux/amd64,linux/arm64 --push -t digihunch/colorapp:v0.2 ./colorapp Note that in this single command, the building for both platforms is completed and pushed with the correct OCI-compliant manifest. There is no need to build them separately and work separately on the manifest. We can use the manifest command from last section to verify the image manifest. The behaviour of defaulting to OCI-compliant image index is a change in the buildx version 0.10 in Jan 2023. With \u0026#8212;provenance=false, we can control manifest format with oci-mediatypes=true. In buildx 0.10, the default value for provenance changed from false to true, which always makes OCI the image manifest format. This change of default, along with the glitch of \u0026#8220;docker manifest\u0026#8221; prior to v23.0.0, had pretty big impact and raised confusions (such as this bug report from Ubuntu, and the issue in this post). However, since I take OCI-compliant format as the standard, I do not have a problem with this change. We can verify the image manifest with:\n$ docker manifest inspect --verbose digihunch/colorapp:v0.1. ## docker CLI version \u0026gt; 23 Platform-specific image Our Dockefile looks like this, with a base image. FROM public.ecr.aws/amazonlinux/amazonlinux:2 RUN yum update -y \u0026amp;\u0026amp; yum install -y python3 \u0026amp;\u0026amp; yum clean all \u0026amp;\u0026amp; rm -rf /var/cache/yum COPY serve.py ./ RUN chmod +x ./serve.py CMD [\u0026#34;python3\u0026#34;, \u0026#34;-u\u0026#34;, \u0026#34;./serve.py\u0026#34;] To build platform-specific images, we need the base image(amazonlinux) to support multi-arc too, so that the build process picked the correct platform specific image as base. We can verify this is the case by looking at the sha256 digest of the first layer of each colorapp variant. They are d78505 for the amd64 image, and 71343c for the arm64 image. We can find the same layer digest from the amazonlinux manifest. We now have mutli-architecture images, as well as the index. Docker Hub shows both digests with their architecture, although unlike ECR it does not show whether the tag is a manifest list or a manifest:\nIn real life, our supply chain usually has multiple levels of base images or multiple base images. It is important to ensure platform-specific image are available by examining their manifests.\nIn the Dockerfile, we consider it a best practice to reference base image by digest instead of by tag. We just need to make sure the digest actually points to an image index (manifest list), with each manifest points to the platform-specific image. Summary Now we\u0026#8217;ve build the first part of our supply chain with arm64 capability. We have build platform specific images that can use their own platform-specific base images. We\u0026#8217;ve also created an OCI-compliant image manifest to announce the image supports multi-architecture. Next, we will examine how containerd on different CPU architectures consumes the platform-specific image.\nPrevious PostLanding Zone in Azure – Introduction Next PostKubernetes with Multiple CPU Architectures 2 of 2 – Node and Workload ","date":"2023-04-15T12:17:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-multi-cpu-1.webp","permalink":"/2023/04/kubernetes-multiple-cpu-architecture-container-image/","title":"Kubernetes with Multiple CPU Architectures 1 of 2 – Container Image"},{"content":"I recently renewed my associate administrator certification, and feel it\u0026#8217;s a good opportunity to brush up on Azure landing zone. The lame part of this is the semantics. I found many similar terms across cloud service provider (CSPs). In the context of Azure, it makes sense to clarify the terms again for Cloud Adoption Framework (CAF) and Cloud Operating Models.\nCloud Adoption Framework (CAF) Similar to AWS Cloud Adoption Framework (CAF), Azure also has the concept of CAF and it means the same thing. This part may feel lofty, but it\u0026#8217;s in fact foundational. To get started on the cloud there are thousand ways to configure the foundation (right or wrong). The adopter needs CAF to navigate through the offerings and define what they can achieve. The CAF documentation is good although length. The most \u0026#8220;beefy\u0026#8221; part is Ready section. Cloud Operating Models Every cloud company has some narrative about cloud operating model. For example, Here\u0026#8217;s Hashicorp\u0026#8217;s definition, and here\u0026#8217;s AWS\u0026#8216; white paper on it. In the context of Azure, the CAF document gives some guidance on developing your own operating model in alignment with the CAF. In addition, it also gives a few example cloud operating models:\nDecentralized operations Centralized operations Enterprise operations Distributed operations There is a comparison table that highlights their differences as well as an accountability chart proposing team divisions. Another insightful table is the one that list out implementation starting point and typical path of iterations for each operating model. The table also suggests that Azure Landing Zone includes two implementation options: starting small and CAF enterprise-scale.\nLanding Zone at High Level Followed by Cloud Operating Model is the design and implementation of Azure Landing Zone. There are currently eight design areas:\nBilling and Active Directory tenant: including Azure AD tenant Identity and Access Management: including hybrid identity Network Topology and Connectivity Resource Organization: different levels of resource containers Security Management Governance Platform automation and DevOps Out of the many design areas, I fell short off IAM and Network so I\u0026#8217;ll try to discuss them in more details below in the next section. As for resource organization, apart from Resource Group and Subscription, it is also important to understand management group.\nMost cloud engineers work with subscription and resource group. That is where a lot are going on. For enterprises however, Azure has to address the requirement for the capability of top-down enforcement. Management Group provides a governance scope above subscriptions, provided that all subscriptions trust a single Azure AD account. Management groups may form a hierarchy of up to six levels to help you configure policies and access, so that the all the subscriptions under each management group have unified policy and access configuration. At the very top is root management group. Any assignment of user access or policy on the root management group applies to all resources within the directory. Because of this, all customers should evaluate the need to have items defined on this scope.\nWe can apply policy guardrails (e.g Azure Policy) at management group level so that the policies are effect across subscriptions. Azure Policy can also address operational compliance considerations by monitoring configuration drift.\nIdentity and Access Management First, we really need to distinguish AD DS on Windows Server, Azure AD and Azure AD DS. In an old post, I discussed what is a Windows domain, the key role of a domain controller (to manage user identity, as well computer identity), and the fact that Active Directory is a complete redesign of Windows Domain system since Windows 2000. So we can start with AD DS on Windows Server:\nAD DS on Windows Server: In the good old days, some common network administrative activities were to configure Active Directory (including the X.500 compatible database, the OUs, domains and forests) on Windows Servers, joining computers to the company\u0026#8217;s domain, configure group policy, configure LDAP and Kerberos, upgrading Domain controllers, etc. Over the years, Microsoft moved these activities to the cloud and offer them as a managed service, known as Azure AD DS. Azure Active Directory Domain Service (Azure AD DS): allows you to use managed domain services (e.g. Windows Domain Join, group policy, LDAP, Kerberos authentication) without having to deploy, manage or patch domain controllers. It is a SaaS offering to manage your domain controllers in the cloud, with a pay-as-you-go model. The counterpart in AWS is \u0026#8220;AWS Directory Service\u0026#8221; which lets you run Microsoft Active Directory (AD) as an AWS managed service. In summary, both AD DS on Windows Server (self-hosted) and Azure AD DS (managed service) are identity stores that operates on Windows domains. Even though the latter is a managed service, it supports LDAP or Kerberos as integration protocol for third party applications (usually on-premis) to use. Both LDAP and Kerberos came around prior to the cloud era and they are not optimized for cloud connectivity. For example, insecure bind (on port 389) in LDAP is still prevalent. Kerberos is fairly complex to configure. However, they are not phased out right away because of their established presence as well as the domain\u0026#8217;s awareness to authenticate devices. Many organizations have to keep their domain service and when they move to cloud so they still have to use Active Directory as identity store. For this, Azure has Azure AD connect. On the AWS side, there is also an AD connector tool to allow on-prem users to log into AWS applications and services. With AD connector you can also join EC2 instance to existing AD domain. Now let\u0026#8217;s examine Azure AD.\nAzure AD: is an IAM solution. It contains an identity store (with users and groups in a flat directory structure) but more importantly it integrates with external identity stores (including Domain Service, self-hosted or SaaS managed), which gives it hybrid-identity capability. A company can even sync their own on-prem identity store to Azure AD using Azure AD Connect. As an IAM solution, Azure AD also allows a company to tie their identity store to applications using modern protocols such as SAML and OAuth. Azure AD treats applications as objects, and they can represents either Microsoft Applications (Office 365, Dynamics 365, Azure) or third-party ones (Slack, Salesforce) as long as they use the supported protocol for SSO. The closest AWS counterpart of Azure AD is Amazon Cognito (arguably), even though their capabilities are not identical in every aspect. Compared to Domain Service, Azure AD alone doesn\u0026#8217;t have the concept of domain. Therefore you cannot join a server or PC to a domain and configure group policy. Azure AD\u0026#8217;s native identity store is a flat directory structure without OUs or forests. Azure AD is NOT a replacement of domain service, either self-hosted or managed.\nNow coming back to the Azure landing zone literature, the document lays out the key decision to make about identity:\nA critical design decision for enterprise organizations adopting Azure is whether to extend current on-premises identity domains into Azure or to create new identity domains.\nAzure Active Directory (Azure AD) and hybrid identity The document even includes a comprehensive identity decision guide. After this decision, we\u0026#8217;ll know what identity store to use. Then we can address the problem of platform access vs workload access. In other words, IAM of management traffic vs business traffic, which opens up topics such as RBAC, service principle and managed identities.\nNetworking Back in 2017, Azure published a white paper about V-Net and it focuses on mesh network and hub-and-spoke. Back then Azure customers run multiple lines of business (LOB) on different V-Nets. The V-Net peering feature allows early cloud adopters to organize all their V-Nets in a mesh topology, ensuring all peers have access to all other peers, or a hub-and-spoke topology to aggregate shared resources in hubs so they can be shared by the spokes in the network.\nWhen setting up a landing zone, network topology is a big decision. In the landing zone document today, clients need to consider the followings: Traditional Azure networking topologies, including: large flat V-Net multiple V-Nets connected with multiple Azure ExpressRoute circuits/connections hub-and-spoke full mesh hybrid Microsoft managed networking topology (on top of Virtual WAN) From the 2017 white paper, most organizations at that time solve their need for network isolation and connectivity by creating a mesh architecture among various V-Nets. All nodes in the network are interconnected so network traffic is fast and can be easily redirected. However, mesh topology has significant disadvantages because it requires too many connections as the footprint expands, making it very costly to operate and quick to reach limit of number of peering links. It is not scalable. The white paper is to advocate the use of hub-and-spoke topology, which I will discuss in the next section.\nIt is worth-noting that, today (Jan 2023) one can create both topologies with Azure Virtual Network Manager. It is currently a preview service but I can foresee it will eventually get integrated with landing zone.\nNo matter which topology, another issue to address is connectivity to on-prem network, and to Azure PaaS services. If the traffic is light, we can use VPN gateway to configure IPSec tunnel that goes over public internet encrypted. It is simple to configure with a good aggregate bandwidth. This connection requires a VPN device on premise as well. A faster alternative is Azure ExpressRoute, which runs a private connection with a third-party connectivity provider. ExpressRoute is more complex and expensive to set up, but it supports much higher bandwidth with direct access and better SLA. In reality, many clients configures ExpressRoute with VPN failover for connectivity to on-prem network. For connectivity to PaaS services, options are service endpoint and private link endpoint.\nHub-and-spoke topology In the traditional topologies, hub-and-spoke network topology is popular as the hub network provides a central point of management. Also it overcomes subscription limits and institutes a separation of concerns. The Azure documentation recommends hub-and-spoke architecture for larger cloud adoption efforts. If the footprint is even massive, we can even extend the model to a cluster of hubs and spokes. A cluster of multiple hub-and-spoke We can connect multiple hubs using:\nV-Net peering Azure ExpressRoute Azure Virtual WAN Site-to-site VPN Within a single hub-and-spoke model, the Hub V-Net hosts shared services and acts as central point of connectivity (to many spoke V-Nets). Often in the Hub V-Net are Azure Bastion, Azure Firewall and VPN Gateway or ExpressRoute gateway. The spoke V-Nets (in same or different subscriptions) isolates and manage workloads in prod, non-prod, etc. Since a single V-Net cannot traverse subscription boundaries, you have to use V-Net peering (preferred), ExpressRoute circuit, or VPN Gateways. V-Net peering works across regions, and across Azure AD tenants. It is low-latency but isn\u0026#8217;t transitive.\nIn some cases we also configure perimeter networks (aka DMZs) in the hub-and-spoke architecture, to handle external traffic. Perimeter networks host services such as External Load balancer, Azure Firewall, Azure Application Firewall on Azure Application Gateway or on Azure FrontDoor) , network virtual appliances (NVAs), IDS, IPS, and other security appliances. Incoming packets flow through the security appliances before reaching back-end servers. Internet-bound packets from workloads must also flow through security appliances in the perimeter network before they can leave the network. The document gives an example of a DMZ hub V-Net with two perimeter networks.\nVirtual WANs This page discusses what is WAN and SD-WAN. WAN connects multiple LANs in different geographic areas and is common with companies with multiple offices in different regions. WAN infrastructure may be privately owned or leased as a service from a third-party service provider (hybrid WAN). Companies may use IPSec VPN, SSL VPN or direct connection to build their WANs. Software-defined WAN (SD-WAN) leverages virtualization technologies, network overlays, on-site SD-WAN devices and software platforms to build hybrid WANs.\nAzure Virtual WAN (similar to AWS cloud WAN) is a managed service to build a virtual WAN with a single operational interface that brings many networking, security and routing functionalities together. It simplifies end-to-end network connectivity (within Azure, between Azure and on-prem) by creating a hub-and-spoke architecture. Virtual WAN Virtual WAN is essentially an integrated connectivity solutions (in hub and spoke), with a global transit network architecture. The configurations, including spoke setup) is automated and troubleshooting is more intuitive. Global transit network configures multiple virtual WAN hubs with hub-to-hub connectivity, which ultimately enables any-to-any connectivity, with different paths discussed here.\nThe landing zone document recommends Virtual WAN for new large or global network deployments in Azure where you need global transit connectivity across Azure regions and on-premises locations.\nSummary Landing Zone configuration involves many components and there is no way to discuss everything thoroughly. In this post I put down my notes reading Azure landing zone documentation. Overall, working on landing zones requires learning a variety of services by the CSP.\nPrevious PostA taste of IoT device tracking Next PostKubernetes with Multiple CPU Architectures 1 of 2 – Container Image ","date":"2023-03-25T01:30:00-04:00","image":"/wp-content/uploads/2025/04/feature-az-lz.webp","permalink":"/2023/03/landing-zone-in-azure/","title":"Landing Zone in Azure – Introduction"},{"content":"Last fall I worked on a demo project for IoT device tracking on AWS so I had some reading. Overview From a strategic level, as servers move away to the cloud, AWS envisions that the footprint left on premise will mainly be IoT devices. The role of the cloud therefore becomes a central point of management for IoT devices. As a result, most of the AWS services for IoT are managed services. The best resources are two white papers: IoT Lens \u0026#8211; AWS Well-Architected Framework and Securing Internet of Things (IoT) with AWS. The most important services are:\nIoT Analytics: Makes it easy to run sophisticated analytics on volumes on IoT data. It connects to the underlying IoT data store and allow you to build your own analytical queries and Jupiter notebooks. IoT Events IoT Core: Core features for IoT. IoT SiteWise IoT Device Management IoT Greengrass FreeRTOS: IoT devices usually need to be small and power efficient. The software and OS running on the device is important. FreeRTOS is a real-time operating system for micro-controllers supported by AWS. FreeRTOS provides kernel, OS and libraries to securely connect your edge device to the cloud in no time. In this post I will explore IoT architecture at high level. In real life, you program your device with AWS IoT Device SDK and AWS IoT API in different languages. In this post, I use a script to simulate GPS data, and push it to AWS IoT using SDK. Then I render the location using sample AWS code for Amazon Location.\nIoT Architecture When creating IoT services we consider registration and telemetry capturing flow. The Well architected white paper proposes registration flow as such:\nRegistration Flow Device Registry to keep track of devices (aka Things). You can find where your devices are, and filter by a common feature (e.g. ModelX device only). Registration flow usually involves a testing of communication between device and server. The authentication must be 2-way where server needs to validate device identity, and device needs to validate server identity. You can use a unique X.509 certificate per device to adhere to security best practices on AWS. This way, if one device gets hacked, the entire fleet of devices is not affected by one certificate being compromised. An alternative authentication method is Cognito. With Cognito you can sign your users into a mobile application, so you use IAM policies to authenticate them into viewing different dashboards or viewing the data that pertains to them specifically. IoT Core policies can help manage authorization.\nThe white paper also proposes a few options for capturing telemetry:\nOptions for capturing telemetry These options presents a common pub-sub pattern, where the device streams message by topic to the Message Broker in IoT core. The IoT core also involves policy and rules. A rule may involve a subscriber to consume the messages. A more comprehensive architecture from AWS IoT device connectivity workshop looks like this:\nThe message path from device to IoT core remains the same. When building an IoT solution, we first address the messaging path.\nIoT Protocols In IoT core, Device Gateway is the entry point for IoT devices connecting to AWS.It supports MQTT, WebSockets and HTTP 1.1 protocols, on top of TLS. Registration flow uses HTTP/REST protocol for provisioning, and with MQTT protocol for a message test. For pushing telemetry, we can use both but prefer MQTT (topic based) because of its advantages in IoT messaging. Here is a good article on the differences. AWS has a white paper on designing MQTT topics, with a few communication patterns and best practices. The SDK documentation also explained the communication protocols, including:\nHTTPS: publish only MQTT: publish and subscribe MQTT over WebSocket: publish and subscribe. Device Gateway will maintain long lived, bi-directional connections, enabling devices to send and receive messages at any time with low latency. Pay attention to the authentication mechanism. As to what protocol is used in a communication, they can be dynamically negotiated using the ALPN protocol. ALPN (Application-Layer Protocol Negotiation) is a TLS protocol extension that allows the application layer to negotiate which\u0026nbsp;protocol\u0026nbsp;should be performed over a secure connection in a manner that avoids additional round trips and which is independent of the application-layer protocols.\nGPS data simulator I don\u0026#8217;t have a GPS chip. To get sample GPS data, I used geojson.io website, use a pen to paint the points and collect the result in JSON format. The data looks like this:\n{ \u0026#34;type\u0026#34;: \u0026#34;FeatureCollection\u0026#34;, \u0026#34;features\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;Feature\u0026#34;, \u0026#34;properties\u0026#34;: {}, \u0026#34;geometry\u0026#34;: { \u0026#34;coordinates\u0026#34;: [ [ -119.4966304331144, 49.88901098598916 ], [ -119.4966304331144, 49.889903059931726 ], [ -119.49658503509582, 49.89066350338888 ], [ -119.49658503509582, 49.891365440561145 ], [ -119.4966304331144, 49.89219897769374 ], [ -119.4966304331144, 49.88735841200943 ], [ -119.49660773410511, 49.8880603972496 ], [ -119.4966304331144, 49.88885011844141 ] ], \u0026#34;type\u0026#34;: \u0026#34;LineString\u0026#34; } } ] } Save this file as map.geojson to later feed it to device simulation script. Rendering location data In AWS location samples project, the sample project maplibre-js-react-iot-asset-tracking is a good demo of IT. The readme document contains a walk through, using AWS amplify services. The steps includes creating certificates, configuring lambda function to add location data to tracker. The project directory also includes the device simulation script, as index.js. I slightly modified the content to this:\nconst awsIot = require(\u0026#34;aws-iot-device-sdk\u0026#34;); // Replace with your AWS IoT endpoint const THING_ENDPOINT = \u0026#34;safdsa-ats.iot.us-east-1.amazonaws.com\u0026#34;; // get from console const CLIENT_ID = \u0026#34;trackThing01\u0026#34;; const IOT_TOPIC = \u0026#34;iot/trackedAssets\u0026#34;; const DEVICE_ID = \u0026#34;thing123\u0026#34;; const GEOJSON_FILEPATH=\u0026#34;geojson/map.geojson\u0026#34;; const fs = require(\u0026#39;fs\u0026#39;) const file_raw = fs.readFileSync(GEOJSON_FILEPATH).toString(); const positions_raw = JSON.parse(file_raw).features[0].geometry.coordinates const POINTS_ON_MAP=[] for (var i=0;i\u0026lt;positions_raw.length;i++){ POINTS_ON_MAP.push({lat:positions_raw[i][1],long:positions_raw[i][0]}) } const device = awsIot.device({ host: THING_ENDPOINT, keyPath: `${__dirname}/certs/private.pem.key`, certPath: `${__dirname}/certs/certificate.pem.crt`, caPath: `${__dirname}/certs/root-CA.pem`, clientId: CLIENT_ID, keepalive: 60000, }); console.log(\u0026#34;Connecting to %s with client ID %s\u0026#34;, THING_ENDPOINT, CLIENT_ID); device.on(\u0026#34;connect\u0026#34;, async function () { console.log(\u0026#34;Connected to device %s\u0026#34;, CLIENT_ID); for (const point of POINTS_ON_MAP) { const message = { payload: { deviceId: DEVICE_ID, timestamp: new Date().getTime(), location: point, }, }; console.log( \u0026#34;Publishing message to topic %s: %s\u0026#34;, IOT_TOPIC, JSON.stringify(message) ); device.publish(IOT_TOPIC, JSON.stringify(message), { qos: 1 }); // Set timeout to sleep await new Promise((resolve) =\u0026gt; setTimeout(resolve, 10000)); } device.end(); }); We can obtain certificate ID from AWS console or by CLI command: aws iot list-certificates --output text --query \u0026#39;reverse(sort_by(certificates,\u0026amp;creationDate))[:1].[certificateId]\u0026#39; | cat When running the script, it pushes data to IoT core service. The AWS Amplify project creates Lambda function that is subscribed to the topic and trigger actions. The data are used to render points on a map, which is available on the front end.\nSummary This is an overly simplified use case but it covers the basics. IoT solution will use a lot managed service and familiar technologies (e.g. TLS, certificate). Creating an IoT solution is mostly about address the onboarding services and make use of the MQTT based workflow. AWS managed services makes these easier.\nMore IoT workshops are available on AWS workshops. Previous PostDICOM testing over TLS Next PostLanding Zone in Azure – Introduction ","date":"2023-03-03T10:53:00-04:00","image":"/wp-content/uploads/2025/04/feature-iot-device-tracking.webp","permalink":"/2023/03/a-taste-of-iot-device-tracking/","title":"A taste of IoT device tracking"},{"content":"I have two open-source projects to deploy a medical imaging application on different platforms. In both of them, I define DICOM validation scenario, and provide steps to test DICOM traffic with TLS. The steps have been working well, until a recent change in Envoy broke the testing, and led me to revisit the test scenario. In a nutshell, I changed from dcm4che to dcmtk binary builds. I\u0026#8217;ve also expanded the test case from a self-signed server certificate to one involving a self-signed CA. Although this test is about TLS for DICOM traffic, the principles apply to any traffic at TCP level. If you just need instruction for DICOM validation on Orthweb or Korthweb projects, skip the Background section. Background To test DICOM traffic with command line tool, I was investigating between dcm4che and dcmtk. Both are open-source projects with builds for multiple platforms. My DICOM test is as simple as a C-Echo command and a C-Store command using the tool, with TLS enabled. Once they work, other DICOM commands usually work as well. I have been primarily using dcm4che as I was familiar with its previous version from my old job. For example, I can issue a C-ECHO with TLS using storescu command:\n./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! To turn that test into a C-Store test, simply add a DCM file as input:\n./storescu -c ORTHANC@ec2-54-243-91-148.compute-1.amazonaws.com:11112 --tls12 --tls-aes --trust-store server.truststore --trust-store-pass Password123! MY.DCM The C-Store output tracks each DIMSE command and return codes. Note that in the command, we specify \u0026#8211;tls12 as the version, and with \u0026#8211;tls-aes switch we enabled AES or 3DES encryption. We also specified a file for trust store and password to the trust store. This is when I first frowned over dcm4che. The dcm4che utility is a Java-based program we have to take an extra step of turning certificate into Java trust store. Different JVM versions may also cause different behaviours in the test. What later prompted me to switch to dcmtk is that with dcm4che I came across a weird error since Envoy proxy version 1.23, which impacted both Orthweb (Envoy proxy) and Korthweb (Istio Ingress).\nMoreover, dcmtk is available as a HomeBrew package, Ubuntu package, and Debian package. The test case, data and the tool We can install dcmtk utility simply with brew install dcmtk, and then we need its echoscu and storescu commands with correct TLS options. The DICOM data I used for testing is a CT exam available for download here. Before mocking with dcmtk\u0026#8216;s TLS options, we first need to understand what would be a good test. Previously I have used a single self-signed certificate on server. It is an over-simplified scenario that is far from a real-life certificate chain, and also does not test client certificate. If I also self-sign the client certificate, the client and server certificates are signed by entirely different parties and have no trust relationship, making it an invalid test case for client certificate.\nFor a full-blown testing with TLS, we should have two levels of CA as below:\nServerServerRSA Private KeyRSA Private KeyX509 CertificateX509 CertificateRSA Public KeyRSA Public KeyClientClientRSA Private KeyRSA Private KeyX509 CertificateX509 CertificateRSA Public KeyRSA Public KeyIntermediate CAIntermediate CARSA Private KeyRSA Private KeyX509 CertificateX509 CertificateRSA Public KeyRSA Public KeyRoot CARoot CARSA Private KeyRSA Private KeyX509 CertificateX509 CertificateRSA Public KeyRSA Public Keysignsignsignsignsignsignself-signself-signText is not SVG \u0026#8211; cannot display\nThe chart represents a typical hierarchy of three-level certificate authorities. Sometimes we need simplicity in our testing, and it is reasonable to simplify the diagram to the following, with one CA that issues certificate for both client and server:\nServerServerRSA Private KeyRSA Private KeyX509 CertificateX509 CertificateRSA Public KeyRSA Public KeyClientClientRSA Private KeyRSA Private KeyX509 CertificateX509 CertificateRSA Public KeyRSA Public KeyTest CATest CARSA Private KeyRSA Private KeyX509 CertificateX509 CertificateRSA Public KeyRSA Public Keysignsignsignsignself-signself-signText is not SVG \u0026#8211; cannot display\nOur DICOM validation testing will be based on both approaches depending on the project and deployment option. For Orthweb project and the Helm-chart driven option in Korthweb, we have one level of CA. For the GitOps and manual option in Korthweb, we have two levels of CA. When configuring testing, it is important to have the diagram above in mind.\nIn addition, we should also be aware of the limitation of the testing. As a personal project I will not pay for the certificates. I have to self-sign the certificate of the CA so there is no way to derive trust on this CA from another level. As a result, we must tell the client and server to trust the CA. The steps to create the needed certificates are: Generate a key pair for Test CA. Generate the certificate for Test CA by self-signing its own public key Generate a key pair for the (DICOM) server. Generate the certificate for the server by signing its public key with Test CA\u0026#8217;s private key Generate a key pair for the (DICOM) client. Generate the certificate for the client by signing its public key with Test CA\u0026#8217;s private key Because the client now also has its certificate, we can test with and without client certificate, using the following dcmtck switches:\n-d (shorthand for \u0026#8211;debug): print out detailed DICOM communication log. For succinct output, use -v (shorthand for \u0026#8211;verbose) instead. +tla (shorthand for \u0026#8211;anonymous-tls): enable anonymous TLS (without client certificate) +tls (shorthand for \u0026#8211;enable-tls): enable full TLS (with client certificate), followed by client key and certificate files -rc (shorthand for \u0026#8211;require-peer-cert): \u0026#8211;require-peer-cert, require peer (server) certificate +cf (shorhand for \u0026#8211;add-cert-file): \u0026#8211;add-cert-file, add server certificate so client can trust it. For C-ECHO, the testing commands with and without client certificate look like:\n$ echoscu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 $ echoscu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 For C-STORE, the testing commands with and without client certificate look like:\n$ storescu -aet TESTER -aec ORTHANC -d +tla -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm $ storescu -aet TESTER -aec ORTHANC -d +tls client.key client.crt -rc +cf ca.crt ec2-3-98-241-51.ca-central-1.compute.amazonaws.com 11112 DICOM_Images/COVID/56364823.dcm As to how to create the certificates, in the post Creating X.509 TLS certificate in Kubernetes, I discussed different ways to create certificates for testing, including using openssl. In the next two sections, I will discuss them in further details.\nOrthweb Test The orthweb project runs Docker containers on an EC2 instance. In the cloud-init script of the EC2 instance, we self-sign a test CA with openssl. Then we create key and certificate for server and client respectively:\nIssuerComName=issuer.orthweb.digihunch.com ClientComName=dcmclient.orthweb.digihunch.com ServerComName=ca-central-1.compute.amazonaws.com openssl11 req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$IssuerComName/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ServerComName/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:$IssuerComName -keyout /tmp/server.key -out /tmp/server.csr openssl11 x509 -req -sha256 -days 365 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt openssl11 req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=$ClientComName/emailAddress=orthweb@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr openssl11 x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt When simulating DICOM activities from the client side, we supply the three files (ca.key, client.crt and client.key) to the echoscu and storescu executables, to issue DIMSE commands on top of TLS from client side, as shown in the previous section. Korthweb Test Currently, the Korthweb project deploys to Kubernetes cluster in three approaches, including two types of ingress controllers: Istio CRD (manual and GitOps deployment options) and Traefik CRD (Helm Chart driven deployment options). With both approaches, it is fairly straightforward to validate the HTTPS port. We export the CA certificate and run a curl command such as:\ncurl -HHost:web.orthweb.com -k -X GET https://web.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert ca.crt Note that by default, the curl command adds SNI (server name indication) extension by default to its TLS ClientHello Message (even without -HHost switch). It acts like a modern browser. On the ingress controller side, most ingress controllers use SNI to drive request routing (because Host field in payload is encrypted). For example, Traefik Proxy has HostSNI matching rule. With Istio, the document states that: TLS implies the connection will be routed based on the SNI header to the destination. With DICOM traffic the Ingress also expects the client to make use of SNI extension in the TLS ClientHello message. The ingress supports multiple sites so the SNI even has an impact of which TLS certificate the ingress serves to the client. We can use openssl to examine which certificate an ingress serves. For example:\nopenssl s_client -showcerts -connect dicom.orthweb.com:11112 -servername dicom.orthweb.com \u0026lt; /dev/null openssl s_client -showcerts -connect dicom.orthweb.com:11112 \u0026lt; /dev/null The second command without -servername switch constructs an ClientHello message without SNI. The ingress may not have a clue of what certificate to serve, depending on its own implementation of TLS protocol. We can also force TLS version with a switch such as -tls1_2.\nWhen it comes to open-source DICOM client, neither dcm4che or dcmtk puts SNI in the TLS request. This created some limitation with my testing. Luckily I do not have multiple routing destinations for now so I only need to direct all DICOM traffic to a service. When I use Istio ingress, I was able to set hosts to \u0026#8220;*\u0026#8221; so the Ingress does not care missing SNI extension in the client request. With Traefik proxy, I had to set sniStrict to false, and also forgo client certificate check. The workaround is different per ingress implementation. Even worse, depending on what the available workaround can achieve, the testing steps vary as well. For example, I perform DICOM validation (Istio ingress) with the steps below:\n# bhs: generate client key pair openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=dcmclient.bhs.orthweb.com/emailAddress=dcmclient@www.digihunch.com -keyout bhs.client.key -out bhs.client.csr # bhs: export intermediate CA credentials kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath=\u0026#39;{.data.tls\\.key}\u0026#39; | base64 -d \u0026gt; bhs.int.ca.key kubectl -n bhs-orthweb get secret int-ca-secret -o jsonpath=\u0026#39;{.data.tls\\.crt}\u0026#39; | base64 -d \u0026gt; bhs.int.ca.crt # bhs: get intermediate CA to sign client cert openssl x509 -req -sha256 -days 365 -in bhs.client.csr -CA bhs.int.ca.crt -CAkey bhs.int.ca.key -set_serial 01 -out bhs.client.crt # bhs: validate web request (without client certificate) curl -HHost:web.bhs.orthweb.com -k -X GET https://web.bhs.orthweb.com:443/app/explorer.html -u admin:orthanc --cacert bhs.int.ca.crt # bhs: validate DICOM c-echo request (with client certificate) echoscu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 # bhs: validate DICOM c-store request (with client certificate) storescu -aet TESTER -aec ORTHANC -d +tls bhs.client.key bhs.client.crt -rc +cf bhs.int.ca.crt dicom.bhs.orthweb.com 11112 DICOM_CT/0001.dcm On the other hand, for Traefik ingress, I have to use anonymous TLS without client certificate:\nechoscu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 storescu -aet TESTER -aec ORTHANC -d +tla -ic dicom.orthweb.com 11112 DICOM_CT/123.dcm The complexities with different test paths are consequences of the missing SNI capability in both DICOM toolkits. Unfortunately, the developers of the two DICOM tools are not aware of these consequences. I tried to contact dcmtk about this and will see what happens.\nTLS profile Another setting to pay close attention to is the security profile for TLS communication. These profiles defines the behaviours of dcmtk when it establishes TLS connection. The dcmtk has the following security profiles:\n\u0026#8211;profile-bcp195-nd (+py default): Non-downgrading BCP 195 TLS Profile \u0026#8211;profile-bcp195 (+px): BCP 195 TLS Profile \u0026#8211;profile-bcp195-ex (+pz): Extended BCP 195 TLS Profile \u0026#8211;profile-aes (+pa): AES TLS Secure Transport Connection Profile (retired) \u0026#8211;profile-null (+pn): Authenticated unencrypted communication (retired, was used in IHE ATNA) The two at the bottom have been retired. The current profiles are all based on BCP195. BCP (best current practice) are sub-series of the corresponding RFC document series. The current revision of DICOM standard discusses bcp195-nd, bcp195 and bcp195-ex profiles in DICOM standard chapter PS 3.15 (B.9-B.11). For example, bcp195-nd requires that:\nImplementation shall not negotiate TLS 1.0 or 1.1 Client and server shall prefer strict TLS configuration (as opposed to startTLS) Ciphers that should be supported. Recommend port 2762 These BCP profiles were incorporated into DICOM standard since 2018 and are all based on BCP195. BCP 195 states in section 3.6\n3.6. Server Name Indication\nTLS implementations MUST support the Server Name Indication (SNI)\nextension defined in Section 3 of [RFC6066] for those higher-level\nprotocols that would benefit from it, including HTTPS. However, the\nactual use of SNI in particular circumstances is a matter of local\npolicy.\nRationale: SNI supports deployment of multiple TLS-protected virtual\nservers on a single address, and therefore enables fine-grained\nsecurity for these virtual servers, by allowing each one to have its\nown certificate. So technically, missing SNI is considered incompliant.\nSummary I find myself switching between dcmtk and dcm4che back and forth in the past. This time, I spent some time hoping to settle with the better tool this time. The effort is insightful but not fruitful. It is unfortunate to realize that neither supports SNI so I had to compromise the feature of my deployment. Hopefully one of those tools will catch up.\nPrevious PostGitHub Action Gotchas Next PostA taste of IoT device tracking ","date":"2023-02-18T01:05:00-04:00","image":"/wp-content/uploads/2025/04/dicom-testing-feature.webp","permalink":"/2023/02/dicom-testing-with-tls/","title":"DICOM testing over TLS"},{"content":"I started with GitHub Actions a couple years ago. Recently I came across a few interesting use cases while I was trying to setup Terraform workflow with GitHub actions. These use cases prompted me to make use some new features in GitHub Action. So I put them in a post here.\nRunners can assume IAM Role in AWS In many scenarios we want to execute AWS CLI command from GitHub action. Also, executables such as terraform inherits credential from AWS CLI. The credential should be a temporary role-based credential instead of an IAM user based on access keys. There is a GitHub Action called configure-aws-credentials-for-github-actions that can help configure GitHub runner using OIDC identity provider (since Nov 2021 v1.6.0). With the action, the GitHub runner can assume an IAM role as an IAM user (with access key), or using a web identity.\nFor a GitHub runner to have a web identity thereby assume an IAM role, we should configure OIDC provider in AWS. We can do that from AWS console (i.e. under IAM), or using CloudFormation code. Below is a snippet as an example:\nResources: GitHubOIDC: Type: AWS::IAM::OIDCProvider Properties: Url: https://token.actions.githubusercontent.com ClientIdList: - sts.amazonaws.com ThumbprintList: - 6938fd4d98bab03faadb97b34396831e3780aea1 Then from the configured OIDC provider, we can obtain a thumbprint. GitHub action gives the thumbprint here. In AWS, we configure an IAM role whose AssumeRolePolicyDocument will reference the thumbprint. Here is an example. In the condition section of AssumeRolePolicyDocument, we can also specify a specific GitHub repository so that only Actions from that repository can assume the IAM role with their web identities.\n- name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v1-node16 with: role-to-assume: ${{ vars.IAM_ROLE_ARN }} aws-region: ${{ vars.AWS_REGION }} This way, we map a GitHub runner\u0026#8217;s web identity to an IAM role with a step using the Action above. We also filter what GitHub org and what repo can trigger actions that assumes the role, by the condition clause in the role statement. If the step fails, we can look at CloudTrail on the AWS side for causes. Look for entries with AssumeRoleWithWebIdentity as Event Name.\nReusable workflows For better reusability of Action steps, GitHub introduced reusable workflows (generally available since Nov 2021). It is particularly helpful when we need to run a workflow for different environments. The reusable workflows files (YML) can be placed in separate repositories, and be reference as such. This allows enterprises to centralize the management of reusable workflows.\nI have been using the act project to emulate GitHub action locally on MacBook. As of Jan 2023, act does not support reusable workflow. With the split between caller and reusable workflows, we have a new challenge of passing secrets and variables between them. It is not straightforward and GitHub documentation needs improvement to get the documentation clear. Also because the word \u0026#8220;environment\u0026#8221; is used in different contexts, it is ambiguous and therefore difficult to Google relevant information.\nPassing variables First, there are several types of variables in GitHub action:\nEnvironment variable: declared under env keyword in a workflow. To use environment variable, use the env context. For example: ${{ env.MY_VARIABLE }} Configuration variable: introduced in Jan 2023, configuration variables are defined at repository, environment and organization levels. To use configuration variable, use vars context, and ensure the workflow job specifies a value for environment attribute. Secrets: GitHub also calls it Environment secret when defined at environment level. It works the same way as a configuration variable because it is also specific to an environment. The content is not viewable once set. The reason GitHub action makes this so confusing, is that on one page, its documentation distinguishes between environment variable and configuration variable:\nOn another page, the document refers to configuration variables at environment level as environment variable:\nIt seems that \u0026#8220;configuration variable\u0026#8221; is too new for GitHub to refine its documentation as of January. This semantical confusion gave me a hard time investigating how to pass \u0026#8220;Environment variable\u0026#8221; to reusable workflows. I will stick to the meaning on the first page to distinguish environment variable and configuration variable at environment level. Passing environment variable isn\u0026#8217;t straightforward. In this discussion thread, people discussed how inconvenient it is. I used the workaround in this comment, where I had to create a job for the sake of storing variable values to output. Pass secret is easier. This is an insightful blog post (Dec 2021) about passing secret to reusable workflow. The attempt 3 in the post works for me. First, we pass the value of environment to the reusable workflow as an input, then at job level specify the environment with the value. Then in the jobs we can reference secrets as ${{ secrets.NAME }}. The job will pick up the secret based on the correct environment. It appears that since May 2022, GitHub introduced secrets: inherit keyword to address this. However, the method above still works for configuration variable.\nAuthentication of GitHub Actions By default, a GitHub action can access the code repository that triggers the action and no other repositories (with GITHUB_TOKEN). However, in many cases we need to access external repositories. For example, terraform init command from a GitHub action implicitly calls git clone to pull module code from external repositories. A GitHub workflow may also reference a workflow file from external repositories.\nThe question is how to authenticate GitHub workflow to access external repo. This post has a thorough discussion. We may create a Personal Access Token and pass it to set-git-credentials action. We are essentially sharing a personal credential (and repo access) with a GitHub action, which is not a good practice. The proper way to solve this problem, is to create a separate GitHub App and grant the access only the repo that the workflow needs to access. The GitHub App will generate a private key. Then we supply the private key to workflow-application-token-action so the workflow can act as the GitHub App, thereby access the external repos. The post has more details in the GitHub App section. Suppose we have terraform get command to clone external repo, the actions may look like this:\n- name: HashiCorp - Setup Terraform uses: hashicorp/setup-terraform@v2 - name: Get RepoReader App Token id: get_repo_reader_token uses: peter-murray/workflow-application-token-action@v2 with: application_id: ${{ vars.REPO_READER_APPLICATION_ID }} application_private_key: ${{ secrets.REPO_READER_PRIVATE_KEY }} - name: Cache Git Creds uses: de-vri-es/setup-git-credentials@v2 with: credentials: https://x-access-token:${{ steps.get_repo_reader_token.outputs.token }}@github.com/ # Terraform Get implicitly calls git clone which uses the credential cached as above - name: Terraform Get run: terraform get Another benefit of using GitHub App is that the token is a short-lived credential that expires as the job is finished, whereas a PAT will expire on a preset date. In this use case we can think of GitHub App as a service account with minimized privilege to read a short list of repos.\nFinal words I came across this reddit post against pipeline use and there are some legit points. For example, the user is frustrated with the limitation with variable passing and unclear documentation. The post wasn\u0026#8217;t specific to GitHub action but I do share some frustration while working with GitHub Actions.\nOn a positive note, since I first used GitHub actions, it has evolved quite a bit with new features, although the documentation is somewhat lagging. It is still very helpful as all of these are free to personal use. I look forward to more interesting features. Previous PostOptimize CPU and Memory for Kubernetes Pod Next PostDICOM testing over TLS ","date":"2023-01-27T01:15:02-04:00","image":"/wp-content/uploads/2025/04/gh-action-feature.webp","permalink":"/2023/01/github-action-gotchas/","title":"GitHub Action Gotchas"},{"content":"When optimizing workload performance, it is important to understand how on earth operating system allocates CPU and memory to processes. This helps understand how to set resource limit Kubernetes Pod in an optimal way.\nCPU resource assignment The OS distributes CPU resource to processes by the unit of time share of CPU time. Most of the time, many processes with CPU instructions (machine code) are waiting in the Job queue, for their share of CPU time in order to execute their instructions. As soon as CPU becomes idle, the CPU scheduler selects a process from the ready queue to run next:\nIdeally, OS should schedule CPU in a way that it should not waste any CPU cycle. It should also minimizes waiting time and response time of processes. At a high level, there are two types of CPU scheduling:\nPreemptive: OS allocate CPU resources to a process for only a limited period of time and then takes those resources back. It could interrupt a running process to execute a higher priority process. Non-preemptive: New processes are executed only after the current executing process has completed its execution. Here is more information about preemptive and non-preemptive scheduling. CPU is compressible resource in Linux In the Linux world, all scheduling is preemptive. We also call it kernel preemption. As the wikipedia entry states: the\u0026nbsp;scheduler\u0026nbsp;is permitted to forcibly perform a\u0026nbsp;context switch\u0026nbsp;(on behalf of a runnable and\u0026nbsp;higher-priority\u0026nbsp;process) on a driver or other part of the kernel during its execution, rather than\u0026nbsp;co-operatively\u0026nbsp;waiting for the driver or kernel function (such as a\u0026nbsp;system call) to complete its execution and return control of the processor to the scheduler when done.\nThe Linux scheduler implements a number of\u0026nbsp;scheduling policies, which determine when and for how long a thread runs on a particular CPU core. The scheduling policies in RHEL include real time policies such as SCHED_FIFO and SCHED_RR where processes have a sched_priority value in the range of 1 (low) to 99 (high); and normal policies such as SCHED_OTHER, SCHED_BATCH and SCHED_IDLE, where sched_priority (specified as 0) is not used in scheduling decisions.\nIt is important to understand preemptive CPU scheduling on Linux. When OS allocate CPU resource to a process for one time slot, it is not committed to the same process for the next time slot. The OS reserves the ability to revoke the next CPU use and re-assign it for processes of higher priority.\nBecause of this, we regard CPU as a compressible resource. The compressible characteristic impacts how we optimize CPU utilization for a process, including setting CPU request and limit for Kubernetes workload. Memory is non-compressible resource A few years ago, I discussed how to calculate memory usage. A process requests memory from OS using memory allocation functions (the malloc family), and return memory to OS using free functions. The design of Linux OS knows that processes have a tendency to request more memory than they use, which causes under-utilization. In combat against under-utilization, the Linux OS supports memory overcommitment (on by default), allowing processes to request more memory than what is available. The processes have access to virtual memory space and the OS may swap some pages out to disks. The overcommitment mechanism also prevents processes from crashing due to insufficient memory assignment. The kernel can also OOM kill a process when the entire system is in a crisis.\nMemory is non-compressible resource. When OS assigns memory pages to a process, the process has to right to keep those pages, until the OS takes them away. Unlike assigning CPU cycles, the assignment of memory pages to processes does not have an expiry time. This is the non-compressible characteristic of memory assignment. CPU limit and requests for Kubernetes workload It was considered best practice to set request and limit for memory and CPU. However, knowing CPU is compressible resource and memory isn\u0026#8217;t, we should re-consider this practice. In short, for CPU, we should set request only, without setting limit. For memory, we should set limit to exactly the same as request.\nA process has different level of demands for CPU at different times. Depending on the activity in the process, the level of demand can even be spiky. If there is a lot of iowait, it may not need a lot of CPU. But when there are lots of computing-bound activities, the program is CPU-thirsty as it is programmed to to more. The last thing we want is to throttle the CPU use for a process in such legit situations. When throttling happens, the process does not get sufficient time share of CPU time. At the platform level, we can\u0026#8217;t control when the Pod (process) gets busy. The best thing it can do, is trying to fit more CPU time shares to this process when it becomes CPU thirsty. When we apply a limit of CPU in workload setting, we are potentially throttling the CPU use for a process at the times it needs more CPU time shares, which is counter-productive. We should still configure CPU request, so that kube-scheduler factors it in when scheduling multiple Pods to a Node. The CPU request alone ensures the number of Pods are not excessive. This is the only thing we can do about controlling CPU assignment for Pods. We should also monitor CPU throttling. Memory limit and request for Kubernetes workload Memory is not compressible, therefore we should set both limit and request to the same value. We set memory request so that kube-scheduler has an idea assigning Pods. We set the limit so that no single Pod takes more memory than its fair share. Unlike CPU, once a Pod takes more memory than its fair share, the platform will have to be aggressive to reclaim it back, which may impacts the running of the Pod (process). In contrast, CPU scheduler never guarantees the assignment of CPU time share to a Pod beyond the end of the current CPU cycle.\nWhen we\u0026#8217;re setting memory limit and request with different values, we\u0026#8217;re sending a confusing signal. We\u0026#8217;re inviting Pods to use more memory than they requested. This increases the chance of memory shortage at the node level, and hence the need to OOM kill a Pod.\nHorizontal autoscaling and Cluster Autoscaling The native HPA is metrics-based. As I previously discussed, neither CPU nor memory metrics are good indicators of time to scale. A process or a Pod may have a temporary high demand of CPU purely due to how programmers write the code. Even if we followed the best practices as above, I would still not regard CPU and memory metrics as a reliable indicator to drive auto scaling. If a service is a potential point of congestion, we should use a queue in front and the queue size is almost always a much better indicator of the timing to scale. As to cluster autoscaler, on it FAQ, it says flat out that you should NOT use a CPU usage based scaling mechanism. I guess this is for a similar reason (compressibility). As discussed, when a Pod is pending for schedule for too long, it emits and event that drives the cluster autoscaler.\nSummary When I first worked on Kubernetes workload I did not give this much thought and proposed the use of CPU limit. As of January 2023 I still find static code analysis tools that requires CPU limit for Pods in the check (e.g. CKV_K8S_11 on Checkov), which leads me to investigate the issue further, and noticed more voices advocating the correct use of resource limit (such as this post) in 2022. For existing deployments, it is worth a review the resource limit configuration.\nPrevious PostEKS impression Next PostGitHub Action Gotchas ","date":"2023-01-13T11:47:00-04:00","image":"/wp-content/uploads/2025/04/cpu-feature.webp","permalink":"/2023/01/optimize-cpu-and-memory-for-kubernetes-pods/","title":"Optimize CPU and Memory for Kubernetes Pod"},{"content":"I\u0026#8217;ve worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an EKS cluster.\nSimilar to AKS, EKS exposes API endpoint and the control plane components are hidden from AWS users. When creating EKS cluster it does not create the underlying VPC and subnets. Therefore, you have create an existing VPC and at least two subnets ahead of time, and specify them during EKS creation. Bear in mind that there is a list of requirement for the VPC and subnets.\nIn the cluster, the CNI that EKS officially supports is Amazon VPC CNI plugin. It is available as an add-on. Similar to Azure CNI, each Pod gets its own IP address. In addition, EKS supports other compatible CNI plugins such as Calico, Cilium, Weave Net and Antrea.\nComputing Nodes in EKS There are three modes to address computing capacity: self-managed nodes, EKS managed node groups and AWS Fargate. The documentation has a comparison table.\nWith self-managed nodes, users create EC2 instances separately and then register them to the control plane. The instances must use the same IAM role and AMI. You can use Auto Scaling groups of Bottlerocket (AWS-sponsored purpose-built Linux distro for container host) nodes. The self-managed node option is mostly for AWS outpost customers who bring in their own computing capacity from data centre.\nIf you provision computing capacity from AWS, it makes sense to assign EKS managed node groups when creating EKS cluster. We can turn on Cluster Autoscaler, a Kubernetes construct to manage the auto scaling of node groups. Sometimes we want to have more than one node groups. For example, to build a multi-architecture cluster, we need one node group with amd64 nodes and the other with arm64 nodes (e.g. instances with Graviton processor). In general, arm-based CPU delivers better performance with less power consumption and the industry is slowly moving towards more arm-based CPU architecture.\nFargate is what I call managed computing service for EKS. With Fargate you do not need to tweak Cluster Autoscaler to self-manage computing capacity. The Fargate documentation has a long list of considerations. For example, Pods must match a Fargate profile (here\u0026#8216;s an example) at the time that they\u0026#8217;re scheduled to run on Fargate. So we need to build Fargate profile and Pod labelling properly. Also, Fargate does not support DaemonSet. Another big consideration is that Fargate does not support non-VPC CNI. In my opinion these are pretty significant limitations. Many workloads (system-level or application-level) would need Daemonset (e.g. kube-proxy, some CNI or CSI drivers, Dynatrace monitoring). The pro of Fargate is the serverless computing model. The construct of a Fargate profile isn\u0026#8217;t complicated. You just specify subnets, namespace and labels. However, the downside is the long list of considerations. Some teams may consider these restrictions too much. The other overhead is the need to manage Fargate profile to ensure all Pods are scheduled somewhere. To me, using Fargate alone impairs portability of workload. The good thing is that Fargate and Managed Node Group are not mutually exclusive on a cluster. In most cases, we can go partially serverless, and reap the benefits of both of them. Node AutoScaling For workloads that don\u0026#8217;t have a matching Fargate profile, we have to figure out node autoscaling ourselves. I touched on Cluster Autoscaler in \u0026#8220;Autoscaling on Kubernetes Platform\u0026#8220;. CA works on AWS as well and is triggered upon a Pod coming to unschedulable status in Scheduler. There is some limitations though. For example, CA interacts with Autoscaling Group (instead of EC2 instances directly). When it determines it\u0026#8217;s time to scale up, it bumps up the desired capacity by one at a time in the Autoscaling group. The configurations in Autoscaling group may also be at play and CA do not have direct control. For example, the \u0026#8220;scaling cooldown\u0026#8220;. The pool of nodes is homogenous as per the pre-configured launch template and CA has no control. If a Pod requires a different type of node (e.g. ARM64 CPU, spot instance, etc), then we\u0026#8217;d first have to create a node group with the desired node type. Moreover, in the worst cases, one-at-a-time scale-up does not meet the increase of demand driven by Pod increases, causing nuances such as racing conditions. Because the Cluster Autoscaler doesn\u0026#8217;t really deal with the nodes themselves, this kind of integration is clunky and slow. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive, according to this blog post. As a result, AWS launched an open-source cluster autoscaler project, Karpenter. Karpenter first only supported EKS but now the support includes other CSPs. For EKS, Karpenter directly interact with different types of EC2 instances.\nKarpenter makes node scaling work in a more cloud-native manner. In the presence of unschedulable Pods, Karpenter bypasses the Kubernetes scheduler and works directly with the Cloud provider, to launch the minimal compute resources needed to fit those Pods and immediately binds the Pods to the newly provisioned Nodes without waiting for scheduler. As Pods are removed or rescheduled to other nodes, Karpenter looks for opportunities to terminate under-utilized nodes. Karpender defines a CR called Provisioner to specify node provisioning configuration, such as instance size, zone, CPU architecture, etc. It is a manifest that describes a node group so the node scaler is aware of all the available node types. You can have multiple Provisioners for different needs, just like node groups. The Provisioner CR can also set TTL for empty Nodes, such that once a Node has no pods other than DaemonSet, Karpenter will terminate the Node on TTL expiry.\nKarpenter\u0026#8217;s idea is similar to the idea of AutoPilot cluster in GKE. The new EKS workshop has an section on how to set up CA and Karpenter in practice.\nIdentity Management for EKS For IAM, we need to be concerned with three aspects. The management traffic to the cloud service, the management traffic for Kubernetes cluster and business traffic. Traffic typeAWSAzureI. Cloud Service Endpoint (Management Traffic for Cloud Service)AWS IAM identityAzure RBACII. Kubernetes API (Management Traffic for K8s Cluster)IAM mapping or OIDCAzure RBAC (implementation of OIDC)III. Business trafficUp to Kubernetes IngressUp to Kubernetes Ingress For business traffic (type III), it is all up to the Ingress. I\u0026#8217;ve written another article on managing ingress traffic on Kubernetes platforms. We interact with cloud service endpoint (type II) with either AWS CLI or Terraform, to create any object, including resources needed for a cluster. This is generally how we work with cloud service, not specific to Kubernetes. Usually the IAM identity assumes another IAM role, which empowers it with a lot of permissions.\nFor access to Kubernetes API (type III), EKS supports OIDC and IAM mapping. AWS documentation refers to this as \u0026#8220;Cluster Authentication\u0026#8220;. There is one special scenario where your identity for type II access inherits your identity for type I access. As the document puts:\nWhen you create an Amazon EKS cluster, the AWS Identity and Access Management (IAM) entity user or role, such as a\u0026nbsp;federated user\u0026nbsp;that creates the cluster, is automatically granted\u0026nbsp;system:masters\u0026nbsp;permissions in the cluster\u0026#8217;s role-based access control (RBAC) configuration in the Amazon EKS control plane. This IAM entity doesn\u0026#8217;t appear in any visible configuration, so make sure to keep track of which IAM entity originally created the cluster.\u0026nbsp;\nThis special scenario (I call it the \u0026#8220;implicit master user\u0026#8220;) allows us to perform critical activities on the cluster, such as creating IAM mapping, or OIDC configuration. The above addressed how AWS resource access Kubernetes resource. On the other hand, to address how a Kubernetes resource access AWS resources, we need IRSA (IAM Roles for Service Account). We have a service account in Kubernetes and map it to an IAM role.\nAppMesh AppMesh is AWS\u0026#8217; Envoy based service-mesh offering supporting Kubernetes cluster, ECS service and even EC2 instance. AppMesh\u0026#8217;s control plane is a managed AWS service, with a controller running on the Kubernetes cluster. To install AppMesh on the cluster:\nOn the EKS cluster, install AppMesh Controller using Helm Associate the cluster with IAM OIDC provider Create an IAM role for the appmesh-controller service account After these steps, you can create a mesh using CloudFormation, Terraform, etc. The data plane (Envoy proxy) can run on Kubernetes (as sidecar). Traffic between control plane and data plane can go through private link (Interface VPC endpoint) for added security. Like Istio, AppMesh enables mTLS. For observability, you can export Envoy metrics with Prometheus. Coupled with XRay, AppMesh also supports distributed tracing.\nAppMesh uses a different set of CRDs than Istio. Key CRDs are:\nMesh: represents an entire service mesh. At mesh level you can configure Egress filter (to allow or deny external traffic) and set IP version (v4 vs v6) VirtualGateway: a CRD that represents an Ingress in to the Mesh. A virtual gateway allows resources that are outside of your mesh to communicate to resources that are inside of your mesh. A virtual gateway references Envoy proxy deployment by podSelector. It references GatewayRoutes by namespaceSelector, and optionally gatewayRouteSelector. You also specify listeners in the manifest to reference Envoy proxy Service (LoadBalancer Type). GatewayRoute: A gateway route is attached to a virtual gateway and routes traffic to an existing virtual service. If a route matches a request, it can distribute traffic to a target virtual service. In the manifest, you specify a list of httpRoute, each with matching condition and action. In the action section you can specify virtualService as target. VirtualService: an abstraction of a real service provided by a virtual node directly or indirectly by means of a virtual router. Dependent services call your virtual service by its virtualServiceName, and those requests are routed to the VirtualNode or VirtualRouter that is specified as the provider for the VirtualService. VirtualRouter: Virtual routers handle traffic for virtual services. In a virtual router manifest, you can define Route to direct incoming requests to virtual nodes as target. VirtualNode: A virtual node acts as a logical pointer to a particular task group (i.e. ECS service, Kubernetes deployment). It represent a Service in the AppMesh. In the manifest, you reference Pods by podSelector, specify listeners for any inbound traffic that your virtual node expects, and specify serviceDiscovery for your task group. You can configure those Custom Resources using YAML manifests (and check the API reference a lot). Alternatively, you can configure them from AWS CLI or AWS console. The console will help you visualize what can be configured. For further details on how these CRs play together, there is a workshop for AppMesh.\nEKS cluster using Terraform Officially, there is an EKS blueprint project for provisioning EKS cluster in Terraform.\nI also keep my own Terraform code in the AWS directory of cloudkube project. It works out to be a little more complex than my Terraform template to create Azure Kubernetes Cluster (Azure directory). Because I had to create Cognito resources with initial credential to allow users to connect to cluster without using the implicit master account.\nBelow is the diagram of the processes.\nCreate EKS cluster with Terraform module The template configures kubectl access on a Bastion host, which assumed the same role that our IAM user uses to create the Kubernetes cluster. Therefore, the IAM role is the master identity. Note that the IAM user (power-user) has very powerful permissions. Usually it is ideal to assign lots of permission to IAM Roles (temporary credential) instead of IAM user (long-term credential). So the role chaining would look like:\nThe IAM user that Terraform uses has no permission other than assuming a \u0026#8220;PowerUser\u0026#8221; role The PowerUser role trusts the IAM user. It also has the permission to assume the \u0026#8220;EKS-Manager\u0026#8221; role The EKS-Manager role trusts PowerUser\u0026#8217;s role session. However, the role chaining scenario above is not currently supported in Terraform. I use a Bastion host because the cluster endpoint is on private subnet. The bastion host is on a public subnet. However, if we do not like public subnet and public IP, we can place the bastion host on a private subnet, and use SSM system manager agent with SSH tunnel plugin to have SSH access to private bastion host.\nSummary I first came across this article about EKS and its awfulness and then decided to check out EKS. I\u0026#8217;m not sure all points are still valid but it\u0026#8217;s generally real-life experiences. There are also many peripheral services, such as AMP (AWS Managed Prometheus), AMG (AWS Managed Grafana), ADOT (AWS Distro for Open Telemetry), AppMesh (Another Envoy-based Service Mesh, easier to manage than Istio but less Powerful), with a lot to explore.\nPrevious PostLanding Zone in AWS – An Introduction Next PostOptimize CPU and Memory for Kubernetes Pod ","date":"2022-12-23T18:18:19-04:00","image":"/wp-content/uploads/2025/04/eks-impression-feature.webp","permalink":"/2022/12/eks-impression/","title":"EKS impression"},{"content":"Cloud adoption has gone through phases. Hashicorp\u0026#8217;s CTO Armon Dadgar has a great stream on Hashcorp\u0026#8217;s narrative of the three Phases of Cloud Adoption:\nPhase 1, with main focus on agility, app teams in wild west, account sprawl, inconsistent configuration, security \u0026amp; compliance challenge Phase 2. consistent platform layer providing an opinionated way of configurations, security and compliance control, introduction of platform team, whose customers are application team, scaling challenges with platform team capacity not keeping up with application team\u0026#8217;s demand Phase 3. self-service platform at scale for many application teams. CICD, infra-as-code New greenfield clients today should aim at phase 2 or phase 3 depending on their target operation size. Either way, they need a landing zone for consistency. This post focuses on the landing zone options on AWS and the key constructs.\nOverview AWS Prescriptive Guidance defines landing zone as a well-architected (secure, scalable, compliant, etc), multi-account AWS environment that is a secure baseline from which you can deploy workloads and applications. A landing zone may consists of:\nMulti-account architecture Identity and Access management solution Governance, compliance, logging and auditing solutions Security and networking design Landing zone reflects an enterprise opinion on how to configure networking and IAM. Establishing a landing zone can be a very involving process. In early days of cloud operation people used build landing zone by “clickops” on AWS console, which cannot keep up with the growth of landing zone and associated security services. Compared to SMB clients, some sectors such healthcare and national security have much more regulations and compliance frameworks.\nIn fact, landing zone is such a buzzword that I have learned to be very sensitive to the context. If a solution has words “landing zone” in its name, given the complexity and loose use of the words, there is a good chance that the solution only delivers some of the aspects above. Cloud consultants are still to address the gaps. Therefore I decided to write this post about what I learned about landing zone in AWS.\nMulti-account and Organization Since 2017, AWS has been officially advocating the use of multiple accounts as a best practice and security strategy. They encourage clients to view an account as a resource container, just like Resource Group in Azure (here is an article on the account structure between AWS and Azure) Client builds an account for security boundary and financial container. Many blog post came along (such as this one) on the implementation details. From tooling perspective, AWS launched AWS Organization in 2017 to facilitate multi-account management. This presentation from re:Inforce 2019 is a good material to understand multi-account environment with AWS Organizations. In 2021, AWS published a new white paper on best practices with multiple accounts. My previous coworkers authored two blog posts here and here to reflect multi-account setup as of late 2020.\nAWS Organization is an account management service that provides a vehicle to centrally manage AWS accounts by groups, which brings many benefits, such as centralized logging, compliance management, consolidated billing, etc. As an administrator, you can create accounts in your organization and invite existing accounts to join the organization.\nTypical account structure involves:\nA root organization named Root A organization hierarchy with one or more OUs under root, with each OU having one or more child OUs. Each OU can have multiple accounts, with each account having one email address. You can also configure service control policies (SCPs), a type of organization-level policy that you can use to manage permissions in your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization. SCPs are a means of implementing guardrails in your AWS organization.\nTo vend multiple account automatically, AWS introduced the Account Factory (which became part of Control Tower later in 2019) and let users create new account from AWS console and specify which OU it belongs to. It also allows users to implement customization after account creation with Service Catalog products. Some legacy orchestration solution (e.g. Augmented Account Factory) were based on this mechanism. Another orchestration solution prior to the launch of Control Tower was the AWS Landing Zone solution (ALZ, introduced in 2018), which uses AWS CodePipeline to provision accounts and deploy resources.\nFrom Landing Zone Solution (ALZ) to Control Tower The launch of Control Tower was a game changer in 2019 to bring users to multi-account best practices. Control Tower, as an AWS product (instead of a solution by some service teams at AWS) matured over several years. It gradually deprecated the previous generation orchestration solutions. Newer orchestration solutions all have to support Control Tower. According to this page, we can customize Control Tower based Landing Zone in these ways:\nAWS Control Tower console: Instead of creating OU in AWS organization, do it under Control Tower in the console, under “create required OUs”. Outside of AWS Control Tower console using Account Factory for Terraform (AFT): Terraform-based account provisioning pipeline, for heavy Terraform shops. using Customizations for AWS Control Tower (CfCT) solution. The LZ created remains in sync with Control Tower. You first launch a standardized CF stack to set up the mechanisms for customization. Then you create a custom package to define the customization. This includes a manifest file. On the other hand, the original ALZ solution is currently in long-term support and will not receive any additional features. It is deprecated. AWS advises its customers migrate to AWS Control Tower based landing zone. The ALZ page is redirected to a page about customizing Control Tower landing zone.\nThe Control Tower based landing zone configures OUs, accounts, SSO and guardrails. It however does not entail networking design except for the guardrails.\nSecurity Reference Architecture (SRA) SRA is just a reference architecture and accompanying recommendations on AWS security services, and how they work together in a multi-account environment to host a single-page application. It comes with a repository to demonstrate how to configure a secure multi-account environment with Control Tower, CfCT, as well as security services. Alternatively, you can deploy it with CloudFormation StackSets. For fully automated deployment of this architecture, check out Landing Zone Accelerator down below.\nOne important feature that reflects the notion of centralized security in a multi-VPC topology, is the use of a dedicated VPC for centralized interface endpoints. This pattern is seen in all generations of security reference architectures.\nAWS Security Environment Accelerator (ASEA) Regulated customers often find they need to add additional controls and capabilities to be defined and setup outside of Control Tower. ASEA, as an orchestration solution, aims to remove the complexity of having to develop and maintain a separate codebase to manage the additional customizations, by providing a tool to help deploy and operate secure multi-account, multi-region AWS environments on an ongoing basis.\nOvertime, as Control Tower introduces new capabilities to support the customizations required in heavily regulated environments, the capabilities will be removed from ASEA and enabled directly within the Control Tower managed service, further reducing operational burden. Read this for its relationship with ALZ and Control Tower.\nASEA was first released in late 2020. It covers more on networking design and has a fairly comprehensive installation process. ASEA primarily cater to government of Canada\u0026#8216;s PBMM Security Configuration Profile with an opinionated configuration. The first few revisions were referred to as PMBB architecture. While it provides a great reference architecture for highly regulated landing zone, it has not gotten much traction elsewhere. The recommendation going forward, is to use Landing Zone Accelerator, which incorporates the features and lessons learned from ASEA.\nLanding Zone Accelerator (LZA) First released in May 2022, the Landing Zone Accelerator on AWS solution deploys a cloud foundation that is architected to align with AWS best practices and multiple global compliance frameworks. LZA operates on top of Control Tower managed landing zone. This page in the documentation brings a good explanation of how it works. Basically it employs CodeBuild as an orchestration engine, and leverages CDK to drive resource deployment. There are two repositories: the GitHub repository for Landing Zone Accelerator itself, and a CodeCommit configuration repository provisioned during preparation.\nAs document states, LZA is a fully automated implementation of the architecture guidelines documented in the SRA. LZA also incorporates features and lessons learned from ASEA and Compliant Framework for Federal and DoD Workloads in GovCloud (US), neither of which are recommended for new deployment. In addition, LZA aims to enable iterations and extensions of a secure environment over time. The vision is to eventually replace AFT, CfCT, and ASEA. As AWS releases newer versions of LZA, client should be able to upgrade it in a pipeline run.\nBaseline architecture for LZA The Landing Zone Accelerator project also provides a samples configurations in each regulated frameworks. For example, the healthcare best practice come up in Oct 2022 for healthcare industry. The healthcare best practice sample incorporates healthcare specific configurations, such as the detective guardrails defined in the\u0026nbsp;Operational Best Practices for HIPAA Security \u0026nbsp;conformance pack. To deploy the best practices, modify the configuration in config repo, and run the pipeline again.\nLanding Zone Orchestration Options To summarize, we have the following landing zone options in AWS:\nOrchestration LayerLifecycleSummarySolutions based on account factory with Service Catalog (e.g. ALZ, Augmented Account Factory)All solutions pre-dates the launch of control tower have been or will soon be deprecated.Pre-Control Tower solutions should migrate to current alternatives.AWS Control TowerGA in June 2019To customize the landing zone, use AWS console, or alternatively, one of the following three options:\n\u0026#8211; CfCT: Customization for Control Tower\n\u0026#8211; AFT: Account Factory Terraform\n\u0026#8211; AFC: Account Factory Customizations\nFor example, CfCT can be used to deploy Security Reference Architecture (SRA)AWS Secure Environment Accelerator (ASEA)Released 2020Canadian Centre for Cyber Security (CCCS) Medium Cloud Control Profile, formerly known as PBMM. This approach will be replaced by LZAAWS Landing Zone Accelerator (LZA)Released in 2022A low-code deployment option. Samples provided in support of reference architectures that align with industry best practices or compliance frameworks. Examples for industry best practices:\n\u0026#8211; general best practice\n\u0026#8211; healthcare\n\u0026#8211; finance and tax\n\u0026#8211; education\nExample for compliance frameworks:\n\u0026#8211; US state local government\n\u0026#8211; FedRAMP for US Federal and Department of Defence (DoD)\n\u0026#8211; CCCS Medium for Canadian government\n\u0026#8211; AWS Trusted Secure Enclave (TSE) Sensitive Edition (SE), which also aligns with other medium level security profiles such as NIST 800-53, ITSG-33, FedRAMP moderate, CCCS-Medium, IRAP, etc\nIf your organization has a compliance framework that LZA supports, it makes sense to start with LZA. Otherwise, it is sufficient to use Control Tower for multi-account setup.\nNetworking constructs Landing zone involves multiple VPCs so it\u0026#8217;s important to understand VPC peering and Transit Gateway. VPC peering can only be setup between two VPCs and transitive peering relationship is not supported. For 6 VPCs to talk to all each other, we\u0026#8217;d need 15 peering setups, which is not sustainable. We have two options:\nWe can connect many VPCs to a single Transit Gateway and the VPCs will be able to talk to each other. We can also connect Transit Gateway to site-to-site VPN or Direct Connect. This is a good explanation. We can still use peering, but dedicate one VPC as “Transit VPC” in a hub and spoke model. When you attach a VPC to a transit gateway, you must specify one subnet from each AZ to be used by the transit gateway to route traffic. Specifying one subnet from an AZ enables traffic to reach resources in every subnet in that AZ.\nHere is a comparison table. Read the white-paper \u0026#8220;Building a Scalable and Secure Multi-VPC AWS Network Infrastructure\u0026#8221; for more about network design. For example, Some clients need to inspect traffic. When deploying multiple VPCs, we also need to ensure DNS resolution works across VPCs, and between on-prem networks, by configuring Amazon DNS server.\nOur VPCs also need to connect to managed AWS services. For services like S3 or DynamoDB (of the same or different account), workload in VPC can access them via public DNS. The traffic goes through Internet Gateway of VPC and then public Internet. It is neither secure nor economical. We often want all network traffic to stay on the global AWS backbone. There are three types of Endpoint under VPC to help us.\nGateway VPC Endpoint or Gateway EndpointInterface VPC Endpoint or Interface EndpointGateway Load Balancer EndpointPurposeVPC access native AWS servicesVPC access native AWS servicesVPC access your own service (aka Endpoint Service)Enabled by Private LinkNoYesYesTraffic remain on AWS networkYesYesYesMechanismUse the public IP address of the service along with configuration in routing table to access target resource. The routing table acts as gateway. You can use the public DNS name of the service.Use private IP address from the VPC to access the target service. Require endpoint-specific DNS name for the target service. Incur extra charge. S3 as example. More secure because there is ENI in the VPC controlled by security group.Your Endpoint Services is hosted in front of a fleet of network virtual appliances. You can select endpoint type as you create it in console under VPCAccess from clients on premise or from other regionDoes NOT allow access from clients on premise or in VPC from other regionAllows access from clients on premise or clients in VPC from other regionAllowed Private Link integrates with a subset of AWS services. To check the list of interface endpoint, use:\naws ec2 describe-vpc-endpoint-services --query \u0026#34;ServiceDetails[?ServiceType[0].ServiceType==\u0026#39;Interface\u0026#39;].ServiceName\u0026#34; Given the inter-VPC connectivity, most deployment centralizes interface endpoint into a dedicated VPC.\nCentralized Interface Endpoints There are several benefits to use a single VPC as dedicated provider of interface endpoint in a multi-VPC topology. First, the interface endpoints incurs a standing charge and it makes financial sense to consolidate them in one VPC. Second, this setup centralizes the configuration and security aspects as well. In most cases, interface endpoint services follow the format of com.amazonaws.\u0026lt;region\u0026gt;.\u0026lt;endpoint\u0026gt;, with the dns name looking like: \u0026lt;endpoint\u0026gt;.\u0026lt;region\u0026gt;.amazonaws.com. For example:\nEndpoint Service Name: com.amazonaws.us-east-1.ssm Private Hosted Zone: ssm.us-east-1.amazonaws.com Zone Record (A alias): ssm.us-east-1.amazonaws.com However, there are a few exceptions to that which can make it tricky to implement interface endpoint with infrastructure as code.\nException 1. Private DNS name suffix is api.aws instead of amazonaws.com:\nEndpoint Service Name: com.amazonaws.us-east-1.eks-auth Private Hosted Zone: eks-auth.us-east-1.api.aws Zone Record (A alias): eks-auth.us-east-1.api.aws Exception 2. Endpoint Service Name doesn\u0026#8217;t start with com.amazonaws\nEndpoint Service Name: aws.sagemaker.us-east-1.notebook Private Hosted Zone: notebook.us-east-1.sagemaker.aws Zone Record (A alias): notebook.us-east-1.sagemaker.aws Exception 3. Two A-records are required under the same PHZ\nEndpoint Service Name: com.amazonaws.us-east-1.ecr.dkr Private Hosted Zone: dkr.ecr.us-east-1.amazonaws.com Zone Record (A alias): dkr.ecr.us-east-1.amazonaws.com and *.dkr.ecr.us-east-1.amazonaws.com Exception 4. Two PHZs are required:\nEndpoint Service Name: com.amazonaws.us-east-1.ec2 Private Hosted Zone: ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws Zone Record (A alias): ec2.us-east-1.amazonaws.com and ec2.us-east-1.api.aws The infrastructure code that handles interface endpoints should be able to take care of the exceptions.\nSecurity Constructs AWS Network firewall is a configuration under VPC. You associate a Network Firewall with one or more subnets in the VPC. You also associate the Network Firewall with Firewall Policies. Each policy consists of rule groups. Each rule blocks or filters traffic. The log can be published to CloudWatch or S3 via Kinesis. Usecases of AWS Network firewall include:\ninspect VPC-to-VPC traffic; filter outbound traffic; prevent inbound internet traffic; secure AWS Direct Connect and VPN traffic When you associate a firewall to your VPC, you must provide a subnet for each Availability Zone where you want to place a firewall endpoint to filter traffic\nThere are many points of configuration for firewall rules in AWS. AWS Firewall Manager is a place for central management. It connects with other services such as:\nAWS WAF (L7) AWS Network Firewall (L4) AWS Shield (DDos) Amazon Route 53 Resolver DNS Firewall Security Groups Third party firewall support IAM Identity Center (previously AWS SSO) is for logging into AWS portal, giving each identity from an external identity store (such as AD) an identity on an AWS account. It addresses several challenges as a result of having multiple AWS accounts. First, a user from AD needs to access multiple accounts. Second, in each account with access, s/he needs to have an IAM role. These IAM roles are based on attributes of his/her external identity, and can be different per AWS account. With Permission Set IAM Identity Center:\nAllow you to login on different AWS account using the same credential Allow you to federate with external identity store( e.g. using SAML) Manage permission set for each account. This video has a good walk-through of how to configure Azure AD as identity store, and tie it to specific AWS accounts and specify permission set. Note that IAM Identity Center has a different purpose than Cognito. Cognito is to connect your app with an IdP. Your app can be an EC2-hosted application, serverless application on API gateway, or container-based application on Kubernetes, as long as the support open identity standards (e.g. OAuth 2.0, SAML 2.0 and OIDC). Cognito has two pools:\nUser pool for authentication. With a user pool, your app Identity pool for authorization So Cognito is for AuthN \u0026amp; AuthZ to your own app’s endpoint (business traffic) and it supports a number of standards. IAM Identity Center is for AuthN \u0026amp; AuthZ to your AWS account (management traffic). If you use it for your own application, your application user will have direct access your AWS resources. It mainly supports SAML (and OIDC but less used) and is commonly connected with Active Directory (Microsoft or AWS).\nOne can use AWS Directory Service to host a compatible and managed directory service. AWS Directory Service supports four modes:\nAWS managed Microsoft AD: actual Microsoft Active Directory in AWS Cloud Simple AD, powered by Linux-Samba Active Directory-compatible server AD Connector: a proxy for redirecting directory request to your existing Microsoft AD without caching any information in the cloud Cognito user pools If you combine two AD domains, you will need domain trust.\nOther security services In addition to the core services above, LZA involves the following four services as well.\nAWS Macie is a fully managed data security and privacy service based on ML and pattern matching. It continually evaluates your S3 environments to discovery PII and act on them. It also reports alerts on unencrypted buckets, publicly accessible buckets, etc.\nAWS GuardDuty continuously analyze S3, container \u0026amp; instance workloads, user and accounts for potential threads, across account, based on ML for intelligent threat detection. In addition, GuardDuty also acts on findings.\nAWS Config is an essential service that traces resource inventory, their changes and monitors for compliance:\nWhen you turn on AWS Config, it first discovers the supported AWS resources in your account and generates a configuration item (poin-in-time view of attributes) for each resource. AWS Config also generates configuration items when the configuration of a resource changes, and it maintains historical records of the configuration items of your resources from the time you start the configuration recorder. On an ongoing basis, AWS Config keeps track of all changes to your resources, whether or not it is initiated by the API If you are using AWS Config rules, AWS Config continuously evaluates your AWS resource configurations for desired settings. You can deploy several related rules in a pre-built compliance pack. As you may have noticed, there are many AWS services related to firewall, security and compliance. AWS Security Hub aims to be a a consolidated view of your security status in AWS. You can automate security checks, manage security findings, and identify the highest priority security issues across accounts in client environment. It can also:\nconsolidate security findings from GuardDuty, Inspector (vulnerability scanner and management), Macie, Config, Systems Manager, Firewall Manager, IAM Access Analyzer and other Integrated APN solutions Check findings against best practices Client can take action: i.e. investigate findings or take response and remediation actions Summary Landing zone deployment can take numerous iterations to finalize the requirement. It is important to have a vision of the client\u0026#8217;s cloud operating model, which is usually discussed prior to implementation. At the implementation phase, I discuss the topic on two more separate posts on control tower and landing zone accelerator.\nPrevious PostKey mapping for external PC keyboard on Mac Next PostEKS impression ","date":"2022-12-08T22:37:00-04:00","image":"/wp-content/uploads/2025/04/landing-zone-intro.webp","permalink":"/2022/12/landing-zone-in-aws/","title":"Landing Zone in AWS – An Introduction"},{"content":"This post is to document my steps to use external Windows keyboard on Mac with custom key mapping.\nBackground Apple\u0026#8217;s magic keyboard does not support multi-device so I have to repurpose my Logitech K810 keyboard with MacBook. Logitech K810 is and old model with Windows key layout even though it also supports MacOS.\nThe bottom row has the same number of keys as magic keyboard but with different keys. For example, magic keyboard has cmd key (aka GUI keys) besides space bar but K810 has Alt. Magic keyboard has Option key next to cmd key and K810 has Windows key on the left and Ctrl key on the right. The position of left Ctrl and FN is different on magic keyboard too. Since I rely on shortcuts on Magic keyboard, I therefore decide configure key mapping on the Windows key, Alt keys and right ctrl key to match magic keyboard. K810 Layout In a nutshell, I need to remap some keys on an external Windows keyboard connected to MacOS (Monterey).\nWhat is not working In MacOS, you could modify keys from keyboard preference as below. However, in my case I need to keep left ctrl key and overwrite the right ctrl key. MacOS does not distinguish them unfortunately. The native method is not flexible so I did not bother.\nUse MacOS keyboard preference to modify keys In the mean time, I hate to install a third-party application (such as Karabiner-Elements) just for the purpose of key mapping. Even though I appreciate the efforts by the community, I just don\u0026#8217;t find it a neat solution for the additional dependency.\nChallenges with key mapping I then came across a developer\u0026#8217;s post on using a utility called hidutil to map keys. This is promising after I tested using this command to swap key a/A and b/B. hidutil property --set \u0026#39;{\u0026#34;UserKeyMapping\u0026#34;:[{\u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;: 0x700000004,\u0026#34;HIDKeyboardModifierMappingDst\u0026#34;: 0x700000005},{\u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;: 0x700000005,\u0026#34;HIDKeyboardModifierMappingDst\u0026#34;: 0x700000004}]}\u0026#39; The command above takes effect immediately, on all keyboards. In my case I only need mapping on an external keyboard, not the built-in keyboard. Luckily hidutil has \u0026#8211;matching switch where I can specify ProductID and VendorID. I can find both of them from About This Mac -\u0026gt; System Report -\u0026gt; Hardware -\u0026gt; Bluetooth:\nVendor ID and Product ID From this technical note, we can find out the usage IDs of the keys in the table at the bottom of the page. Then use the Usage ID as the last two bytes of the source and destination values of the mapping expression above. For example, 0x700000004 stands for key A/a, and 0x700000005 stands for key B/b. In my case, after going through this table, my key mapping is:\nSource Key on K810Source Code on K810Destination KeyDestination CodeLeft Alt0x7000000E2Left GUI (cmd)0x7000000E3Right Alt0x7000000E6Right GUI (cmd)0x7000000E7Left Win0x7000000E3Left Alt (option)0x7000000E2Right Ctrl0x7000000E4Right Alt (option)0x7000000E6Fn+F120x700000045Globe0xFF00000003 There is even a website that helps you generate the usage ID. I put together a command, which is working, on the external keyboard.\nhidutil property --matching \u0026#39;{\u0026#34;ProductID\u0026#34;:0xB319,\u0026#34;VendorID\u0026#34;:0x046D}\u0026#39; --set \u0026#39;{\u0026#34;UserKeyMapping\u0026#34;:[{\u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E2,\u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E3},{\u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E6,\u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E7}, {\u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E3,\u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E2}, {\u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E4,\u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E6}, {\u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x700000045,\u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0xFF00000003} ]}\u0026#39; The other way to confirm that key mapping is working, is to look up key mapping with the following command:\nhidutil property --matching \u0026#39;{\u0026#34;ProductID\u0026#34;:0xB319,\u0026#34;VendorID\u0026#34;:0x046D}\u0026#39; --get \u0026#34;UserKeyMapping\u0026#34; To wipe out the mapping, we can use the following command:\nhidutil property --matching \u0026#39;{\u0026#34;ProductID\u0026#34;:0xB319,\u0026#34;VendorID\u0026#34;:0x046D}\u0026#39; --set \u0026#39;{\u0026#34;UserKeyMapping\u0026#34;:[]}\u0026#39; Note that we have to use the same matching expression. However, I noticed that the command only works while the bluetooth keyboard is connected. When I reboot the MacBook the mapping goes away as well.\nThe solution to key mapping Even though I could run this command after reboot using a plist file in ~/Library/LaunchAgents/, as the reference blog suggests, it will not work because I cannot guarantee the bluetooth keyboard is connected before MacOS executes it. Also, I might disconnect and re-connect the keyboard and I want to ensure that the system runs that command whenever the keyboard gets connected.\nIn the Karabiner Element community, some developers touched on this. Basically we can customize the plist file to tell MacOS when to fire the hidutil command, by using \u0026#8220;LaunchEvents\u0026#8221; key in the XML. There is an example for USB keyboard. For myself, I edit the plist file to be:\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;!DOCTYPE plist PUBLIC \u0026#34;-//Apple//DTD PLIST 1.0//EN\u0026#34; \u0026#34;http://www.apple.com/DTDs/PropertyList-1.0.dtd\u0026#34;\u0026gt; \u0026lt;plist version=\u0026#34;1.0\u0026#34;\u0026gt; \u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;Label\u0026lt;/key\u0026gt; \u0026lt;string\u0026gt;local.hidutilKeyMapping\u0026lt;/string\u0026gt; \u0026lt;key\u0026gt;LaunchEvents\u0026lt;/key\u0026gt; \u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;com.apple.iokit.matching\u0026lt;/key\u0026gt; \u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;com.apple.bluetooth.hostController\u0026lt;/key\u0026gt; \u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;IOProviderClass\u0026lt;/key\u0026gt; \u0026lt;string\u0026gt;IOBluetoothHCIController\u0026lt;/string\u0026gt; \u0026lt;key\u0026gt;idProduct\u0026lt;/key\u0026gt; \u0026lt;integer\u0026gt;B319\u0026lt;/integer\u0026gt; \u0026lt;key\u0026gt;idVendor\u0026lt;/key\u0026gt; \u0026lt;integer\u0026gt;046D\u0026lt;/integer\u0026gt; \u0026lt;key\u0026gt;IOMatchLaunchStream\u0026lt;/key\u0026gt; \u0026lt;true/\u0026gt; \u0026lt;/dict\u0026gt; \u0026lt;/dict\u0026gt; \u0026lt;/dict\u0026gt; \u0026lt;key\u0026gt;ProgramArguments\u0026lt;/key\u0026gt; \u0026lt;array\u0026gt; \u0026lt;string\u0026gt;/usr/bin/hidutil\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;property\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;--matching\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;{\u0026#34;ProductID\u0026#34;:0xB319,\u0026#34;VendorID\u0026#34;:0x046D}\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;--set\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;{ \u0026#34;UserKeyMapping\u0026#34;: [ { \u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E2, \u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E3 }, { \u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E6, \u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E7 }, { \u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E3, \u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E2 }, { \u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x7000000E4, \u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0x7000000E6 }, { \u0026#34;HIDKeyboardModifierMappingSrc\u0026#34;:0x700000045, \u0026#34;HIDKeyboardModifierMappingDst\u0026#34;:0xFF00000003 } ] }\u0026lt;/string\u0026gt; \u0026lt;/array\u0026gt; \u0026lt;key\u0026gt;RunAtLoad\u0026lt;/key\u0026gt; \u0026lt;true/\u0026gt; \u0026lt;/dict\u0026gt; \u0026lt;/plist\u0026gt; Store the file above as ~/Library/LaunchAgents/com.local.hidutilKeyMapping.plist (create LaunchAgents directory if it doesn\u0026#8217;t exist) and then run the following to load it:\nlaunchctl load ~/Library/LaunchAgents/com.local.hidutilKeyMapping.plist This works perfectly. The dict section (line 7 to line 23) of the file is based on this document as the GitHub comment suggests. The LaunchEvents section narrows down the type of events that will trigger the program with arguments. It triggers whenever a bluetooth device with specific ProductID and VendorID connects to Mac. We can still use hidutil command from the previous section to ensure that the custom key mapping remain in effect:\nNote that the output above has usage values converted to decimal from hexadecimal, with destination on top and source at the bottom.\nI like this solution as it is light-weight, targeting a specific event, highly customizable and resolves an issue in a very specific circumstance.\nFinal words K380 layout After all this customization effort, I came across the Logitech K380 model (current alternative to K810), whose layout is exactly the way I customized K810 to be. So I ordered the new K380 model for just $29.99 during the week of black Friday. Remembering that I had my K810 for $94.98 back in 2015, what\u0026#8217;s the point of all this\u0026#8230;.\nPrevious PostAWS serverless services and developer tools Next PostLanding Zone in AWS – An Introduction ","date":"2022-11-23T10:38:00-04:00","image":"/wp-content/uploads/2025/04/key-mapping-image.webp","permalink":"/2022/11/key-mapping-on-external-pc-keyboard-on-macbook/","title":"Key mapping for external PC keyboard on Mac"},{"content":"As discussed, serverless simply means cloud services that delegate autoscaling management to cloud platform. In my mind, the word \u0026#8220;serverless\u0026#8221; translates into \u0026#8220;managed autoscaling\u0026#8221;. As long as a service\u0026#8217;s capacity is managed automatically, we can consider it as serverless. Given that capacity scaling accounts for a good amount of work in IT operation, moving to serverless significantly reduces operation overhead. Unlike containers, the serverless ecosystem lacks standard. Since AWS is leading the charge in this field, let\u0026#8217;s take a look at its offerings.\nLambda Function Lambda functions have triggers. We can configure a trigger from an AWS service, or even a non-AWS service which supports AWS event bridge, to invoke Lambda function. Based on the trigger (e.g. SQS, S3, DynamoDB), you then specify event source mapping. For example, if trigger is SQS, you need to specify the queue name, batch size and batch window in event source mapping. If trigger is S3, the event source mapping needs to specify bucket, S3 action, etc. Event source mappings vary significantly among trigger types.\nDepending on the trigger, a Lambda function may be invoked in one of the two ways:\nSynchronous invocationAsynchronous invocationSummaryRequestor fires request and waits until it receives response before closing connection.Lambda places triggering event in a queue and immediately returns a success code (202). Then a separate process reads events off the queue and sends them to your Lambda function. When the function returns a success response or exits without throwing an error, Lambda sends a record of the invocation to an EventBridge event bus.ProsRequestor get invocation result as soon as function run is completeThe queue decouples the request and invocation.ConsWhen load is high the function requires higher capacity for concurrencyInvolves more parties at play and can be quite complex to troubleshoot. Here is a chart with the triggers that support each invocation mode. Real-life applications often need to include libraries and dependencies that requires language-specific steps. For example, NodeJS applications need bundling (often using Webpack or esbuild). We can use Lambda layers to simplify the management. The additional libraries often need to be packaged in S3 bucket or as container image.\nLambda functions are subject to cold start to cope with. When a lot of invocations occur about the same time, the order of executions might be different than the order of upstream events that invokes the function. API Gateway AWS API Gateway is an API gateway implementation for REST and WebSocket APIs. It couples with Lambda in the classic multi-tier serverless architecture pattern. In this classic pattern Lambda function often need to work with a database (e.g. DynamoDB etc), to perform CRUD operations and other custom business logics. The CRUD operations are so commonplace that it makes sense to use mapping template to configure CRUD operation instead of writing similar set of functions for each new data model. API Gateway supports such a mapping template called Velocity Template Language (VTL), a technology from Apache Velocity Project.\nAPI Gateway integrate with many other AWS services. Here are the available integration types:\nIntegration TypeDescriptionIntegration ModeHow it works AWS integrationconnects gateway to an AWS service action as the end point.AWS_PROXYthis mode only supports only one action with one service: the function invoking action for Lambda service. Therefore it is available only for Lambda integration and no other AWS services. For that reason, it is also known as Lambda proxy integration. This mode is recommended for Lambda integration and is the default mode for LambdaIntegrationOption CDK construct. It connects a method (PUT, GET, etc) to a Lambda function and pass along the request on the way in, and the response on the way out. You do not set integration request or integration response. Even if you do, there’s no effect. This “pass-along” mode is easier to understand and configure. AWSthis mode connects an API method to a broad range of supported AWS service action. Function invoking for Lambda service is a common example but not the only service action supported in this mode. When used in Lambda integration, it is also referred to as “Lambda custom integration”, or “normal (request/response mapping) integration”. This mode is good for advanced use cases (e.g. header modification) but involves more management effort. You have to control the mapping between method request and integration request, and between integration response and method response. HTTP integrationfor generic HTTP service endpointHTTP_PROXYthe pass-along mode for upstream HTTP endpoint HTTPthe request/response mapping mode for upstream HTTP endpoint MOCK integration the API gateway itself serves as the endpoint.MOCKthe API gateway itself acts endpoint without an upstream. One example use case is to return CORS-related headers upon a pre-flight OPTIONS query. The table above is a summary of API Gateway integration types as covered here. In the AWS context, proxy mode suggests that the request is not being morphed (transformed). In non-proxy mode, request or response may be modified on their ways in or out.\nFor API gateway to invoke lambda function, synchronous invocation is used by default. You can also configure API gateway to invoke Lambda function asynchronously by using headers.\nAs with other API gateway implementations, AWS API Gateway can also connects to an authorizer (either another Lambda function or Cognito service) in order to authorize the incoming request.\nAppSync In a previous post, I discussed GraphQL as a modern and efficient alternative to REST API. AppSync to GraphQL is the same as API Gateway to REST API. One of the advantages that GraphQL has over REST API is more information in the response. Oftentimes, we use a proxy that supports GraphQL in front of REST API service. AppSync can act as such proxy. I think of it as a managed Apollo since they play the the role in the architecture. Another benefit of GraphQL is the support of subscription, obviating WebSocket configuration. AppSync supports pub/sub API for real-time experience. Client application can get near real-time update as the data on the server is changed.\nWe can configure AppSync to connect to different data sources to formulate GraphQL response. The data source can be an HTTP endpoint, a Lambda function, a database (Relational or DynamoDB), or OpenSearch. App Sync Between data source and the request, AppSync uses resolver to convert GraphQL payload to the underlying protocols and executes if the caller is authorized to invoke it. Resolvers are comprised of request and response mapping templates, which contain transformation and execution logic. AppSync also uses VTL as the mapping template for resolvers.\nMessaging Services Messaging services have three patterns: queues, pub/sub, and event buses. In AWS, the corresponding services are SQS, SNS and EventBridge. Here is a good post about how to choose among them.\nQueues are temporary storage to decouple the source and destination systems. The expectation is that the actions in response to the message can be delayed. If that is not the case and the response needs to be immediate, that is by definition an event-driven pattern. A common solution is to have the AWS service invoke Lambda function. If that is not supported, we can use SNS as an intermediary. SNS is for simple event-driven pattern. In a more complicated event-driven architecture, we often need an event bus to route events in certain ways. We also need to support various event sources by different software providers. Moreover, we want the capability to register our own event schema. These are the scenarios where EventBridge can help.\nDeveloper tools In application development with serverless stack, the line between application and infrastructure is somewhat blurred. Developers often find themselves making repeated configuration on cloud resources while writing application code. For example, to test Python code (application), developer has to upload the code to S3, create Lambda function referencing the code, etc. This would require too much work with AWS CLI, or CloudFormation. The AWS SAM (AWS Serverless Application Model) is a better utility for serverless development workflow.\nSAM comes with its own CLI and developers can feed it with template that interacts with a number of serverless resources (e.g. API, SimpleTable, Function, etc). I view SAM as one layer of abstraction on top of CloudFormation that handles some resources in serverless stack. It saves developers from re-writing resources in CloudFormation templates and keeping them consistent, which would have been tedious. Once you deployed your application using SAM, it will appear as a Lambda Application in the console.\nSAM templates are also declarative. It is simple but limited in feature. AWS CDK is also a powerful utility that works with general purpose programming language for IaC and interacts with all AWS resources. It is very powerful for building serverless applications. AWS Amplify We have CDK, SAM and CloudFormation but we still have to ensure integration between those resources and our applications. For example, when creating a S3 buckets, they have to get the endpoint and reference it in the application code. So is Cognito. As a result, developers still have to spend time on resource integration. They still can\u0026#8217;t focus on business logic. We need a tool that can make opinionated configuration of cloud resources and automatically reference them from the application code. This is the purpose of AWS Amplify.\nAmplify not only help application developers create backend resources with opinionated configurations. It also provides libraries for the application to use and connect to those resources seamlessly. Developers will need to include Amplify libraries in the application code. To create cloud resources, they can use Amplify CLI or Studio (a web portal from AWS console). Amplify natively supports a number of serverless resources such as API (using API gateway or AppSync), Storage (S3 and CloudFront), Lambda function, Cognito, etc. Developer may use Amplify CLI command to create such supported resources. Amplify will prompt some guiding questions in order to configure them correctly. In addition to the natively supported resources, developers can also create custom resources. They have to declare those custom resources with CloudFormation or CDK. I came across this project as a good illustration of how Amplify works. The Amplify Studio can save developers from using Amplify CLI commands. It also can integrate with Figma, providing developers with a framework for frontend development. However, in my experience, it is still glitchy. For now I stick to Amplify CLI.\nAmplify also supports hosting capability, providing users with opinionated and customizable CI/CD pipeline configuration.\nHow about \u0026#8220;clientless\u0026#8221; Lambda function can also run client-side logics. System administrators have to create lots of client-side scripting. While Lambda functions can encapsulate those logics, we\u0026#8217;d still need a script orchestrator to invoke those functions. AWS Step Function comes to rescue. It was even regarded as the most under-utilized service. As the author states, a state machine (as design pattern) is simply a flow of actions with decision making logics.\u0026nbsp;AWS Step function helps you groom the logic flow of existing actions with Amazon States Language. There are standard and express workflows. Each step is a state. A state can be of several different types, such as Choice, Task, Succeed, Fail, End, Map, Wait, Parallel. The task can be a Lambda function, and even AWS API calls. AWS Step function integrate with Lambda functions. It is typically used for patterns with long process and the need to orchestrate the execution of several Lambda functions. For specific use cases, look at these sample projects. For example, a network professional needs to run a lot of connectivity testing, reusing the same Python script but run it from different subnets. We need to create a Lambda function for commands like \u0026#8220;nc -vz\u0026#8221; then invoke the lambda function from VPC, multiple times from different VPCs. We should use step function to drive this. It works like a Makefile on Linux, without requiring your own computer to run.\nAWS Step function also integrates with other AWS services, such as SNS, SQS, Dynamo, Batch, Glue, EMR, EKS, API gateway, event bridge.\nWhenever we need to create a script, involving custom actions, or AWS API calls, we should consider using AWS step function to organize the actions. The benefits are: it saves you a laptop or bastion host (\u0026#8220;client-less\u0026#8221;), many ways to invoke them (not just cron\u0026#8221;). Summary In this post, we started with some serverless services in AWS. For server-side application, we can use API Gateway to invoke Lambda function. For client-side, we can use step function to invoke Lambda function. We then discussed some tools to speed up application development in serverless pattern. AWS categorizes both Amplify and AppSync under Front-end Mobile. Amplify does not represent any computing resources in AWS cloud. It is a library and CLI tools to enhance developer experience. AppSync on the other hand is a cloud computing resource acting as GraphQL API for mobile or web application. Previous PostComputing services: from PaaS to Serverless Next PostKey mapping for external PC keyboard on Mac ","date":"2022-11-09T12:19:00-04:00","image":"/wp-content/uploads/2025/04/feature-server-less-devtools.webp","permalink":"/2022/11/aws-serverless-services-and-developer-tools/","title":"AWS serverless services and developer tools"},{"content":"Silicon Valley startups in mid-2000s likely do not run their own IT operations (i.e. renting their own data centre spaces, purchasing their own rack-mounted servers). Since the launch of EC2, AWS has been renting extra computing capacity to those startups, in the IaaS model. The leased infrastructure requires maintenance work, and AWS realized that many of these customers cannot afford specialized database admins, network admins, storage admins, or even server admins. As a result, they created a handful of managed services aiming to cut out admin overhead and let their customer focus on coding. This is how Platform-as-a-service (PaaS) came about. Let\u0026#8217;s take a look at what are exactly operation activities.\nOps activities IT operation team manages server provisioning, installation of operating system, tuning performance, configuring auto scaling and load balancing, configure networking and storage systems, etc. Networking can be so complex that many infrastructure teams have a dedicated Network Operation Center (NOC), who along with security team, manages key aspects of networking, such as segmentation, router configuration, load balancing, firewall configuration.\nFor client-server application, the client-side code will make outgoing connections, utilizing the TCP/IP stack on the host through an ephemeral port. The server-side code has to be wrapped as a service. A daemon ensures the process running this service stays up and listens to a TCP port in order to respond to request by invoking the functions. Application team usually assumes these activities.\nIf database is involved, then the patching, upgrade, replication, data protection are all Ops problems. If storage is involved, then Ops has to manage mass data accumulated over years, the integration between storage and database and applications, performance, replication, etc. Some larger organizations have full-time database administrator and storage administrators.\nThen comes container. Containers have their benefits but it increases the operation overhead by an order of magnitude. Running container application at scale warrants its own platform, most likely a Kubernetes platform, to address all of the problems above again at the cluster level. Some organization created platform team to manage container and VM platforms.\nIt is the Ops, that turns functional code into a running business. It is also the Ops, that becomes a pain point as a startup scales. With IaaS and PaaS models, AWS managed to convince many small businesses to delegate their IT operations to AWS. This is the humble start of cloud computing.\nElastic Beanstalk At first, I wasn\u0026#8217;t too impressed with Elastic Beanstalk, since it abstracts away too many details. However, I later realized that it has been surprisingly popular in the developer community, especially with individual developers and SMBs. It simplifies deployment to the point that their users don\u0026#8217;t need to know other AWS services, allowing them to focus on coding application logic.\nYou configure Applications and Environments (one application may have multiple environments). In the Environment layer, you can specify code platform (e.g. Python 3.8 on 64bit Amazon Linux 2, Java, Go, PHP, Ruby) and even container platform (Docker on EC2 or ECS). Behind the scene, Elastic Beanstalk configures EC2 instances, Elastic Load Balancers, etc on the selected VPC and integrate with logging and monitoring services. In the console, Elastic Beanstalk exposes a list of configurations options (e.g. AMI, instance type). This centralized configuration page is dummied down for those who don\u0026#8217;t want to deal with Ops. The downside of Elastic Beanstalk is it takes away a lot of flexibility. Many developers find Elastic Beanstalk limit their choices of deployment, as their applications scale. Elastic Beanstalk does not suit for applications that demand extensive operation efforts. Its niche market is individual developers and SMB. Few enterprise applications run on Elastic Beanstalk.\nContainerization with ECS and EKS When we containerize an application, we build container images. Then we run these images with container runtimes, is a core feature of container platform. Container platform also provides orchestration engine since we frequently take containers up and down. In addition, container platform provides mechanisms for container networking and storage.\nAWS has a couple options for container platform. ECS (Elastic Container Service) came out earlier. It organizes a group of EC2 instances as a cluster. You can manage autoscaling, networking, and persistent storage (EFS, FSx etc) on ECS. EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. Just like AKS, it provides a managed control plane along with computing nodes.\nI see ECS as a proprietary and simplified container platform, and Kubernetes as an open-source standard for full-fledged container platform with an entire ecosystem. EKS includes an upstream-certified Kubernetes distribution with a set of tools specific to AWS. Since Kubernetes is the de-facto standard container platform, I prefer EKS by default, unless I can justify the use of ECS. In fact, ECS and Kubernetes have many concepts in common. For example, a \u0026#8220;Task\u0026#8221; in ECS is equivalent to a Pod in Kubernetes. Whether it is ECS or EKS, right-sizing the computing node group is always challenging especially when the application traffic load is irregular. AWS Fargate is a technology that provides on-demand, right-sized compute capacities. It works with ECS and EKS. When integrated with EKS, we delegate the node management (e.g. scaling) to Fargate and forget about sizing the node pool.\nUsing ECS and Fargate involves quite a bit of configurations. To simplify that, we can use App Runner, which builds ECS cluster and uses Fargate to execute the container behind the scenes. App Runner helps client in a way similar to Elastic Beanstalk, but concentrate on Container workload.\nServerless with Lambda and API Gateway The services above have their limitations when it comes to scaling capability. First, they cannot scale to zero. You still pay for idling resources. Also, it is not easy to find the optimal autoscaling setting. Lambda and API Gateway together solves these challenges. AWS refers to it as serverless, which has since become a buzzword. To understand what it is, let\u0026#8217;s examine two concepts:\nFunction as a Service: service with the ability to execute code on demand. Users only pay for code execution time and do not care where the underlying runtime is Backend as a Service: service with the ability to listen to a port and respond to web request Lambda itself is a function as a service. Triggered by events, it only incurs a charge when it\u0026#8217;s invoked. It does not stay up and listening to a TCP port for incoming web request, as does a backend service. In order to act as a backend, Lambda needs to pair up with API gateway. In this configuration, API gateway listens to a web request, and it fires an event to trigger the execution of Lambda function. Lambda and API gateway together makes a backend as a service. In AWS, the coupling of API gateway and Lambda function ensures an idle service does not incur computing cost.\nSince Lambda supports many types of events as trigger, it is also used in event-driven architecture, either standalone or from a VPC. Under the hood, Lambda runs code in a container (with a quick startup time relative to a VM).\nDevelopers can release Lambda code by uploading zip package to S3 bucket, or just packaging code into container image. For deployment, apart from AWS console and CLI, one can leverage CloudFormation, SAM (serverless application model), or CDK.\nLambda vs Fargate Both Lambda and Fargate are serverless capabilities, at least from a marketing perspective. Both can be used to back web service but there are differences. They provision computing resource at different granularity. In a web service, the execution duration of a Lambda function is the response duration to an API request, in terms of seconds. While the server is waiting for a request, there is no usage of the computing resource so you\u0026#8217;re not paying for waiting for a request. However, this also creates the delay of cold-start, especially when the code size is large. There are several ways to optimize the cold start (e.g. SnapStart for Java), but none of those can completely get rid of the cold-start delay with a once-after-a-while request. A light GET call could take 5 seconds with cold start. On the Fargate side, the resource provisioning is based on container lifecycle, instead of request lifecycle. As a result, you\u0026#8217;re still paying for wait time, and it is not per-request billing. Since the container remains up, your request is not going to experience the cold-start if it\u0026#8217;s been idle for a while. Although, Fargate saves you from the effort to right-sizing the computing nodes for container execution, it is not quite the idea of \u0026#8220;scale-to-zero when idle\u0026#8221; by itself. Serverless Architecture In the white paper AWS Serverless Multi-Tier Architectures with Amazon API Gateway and AWS Lambda, AWS advocates the serverless architecture as a modern alternative to the traditional widely adopted three-tier architecture (presentation, logic and data tiers). In the three tier architecture, the scalability of three tier are managed separately. The modern serverless architecture that AWS whitepaper proposes uses API Gateway and Lambda function in place of Load Balancer and EC2 instances (e.g. in an Auto Scaling Group), as illustrated below:\nBoth API Gateway and Lambda scale automatically to support the need of application workload. It assumes the role of logic tier in three-tier architecture but requires minimal maintenance work. For presentation tier, AWS has serverless alternatives such as CloudFront, S3. For data tier, AWS has serverless alternatives such as Amazon Aurora for relational database and DynamoDB for NoSQL. However, the \u0026#8220;no request, no pay\u0026#8221; model for Lambda does not apply to the data tier in serverless architecture.\nWhilst this paradigm benefits small shop IT who wants to minimize infrastructure cost, it has downsides. There is no ability for infrastructure optimization. Since you do not manage where the code runs, client may have concerns over security (e.g. multi-tenant runtime). As business grows, keep using Lambda can result in technology lock-in. Also, a less used application usually requires warm-up time. A code start (downloading the code and preparing the environment behind the scene) can take 100ms to over a second.\nConclusion PaaS attempts to help startups simplify the \u0026#8220;grunt work\u0026#8221; of IT operation. Serverless takes it even further. The semantics of serverless computing is confusing and the Wikipedia page acknowledges it as a misnomer. The nature of serverless model, is the cloud users delegate server capacity management to cloud platforms. The users don\u0026#8217;t need to manage servers, VMs, instances, containers, etc on their own. In a previous post, I discussed the ability to scale to zero, which is just one of the many enabling technologies of serverless. Also, \u0026#8220;no request, no pay\u0026#8221; is neither an inherent nature of serverless model. Serverless service may involve storage (e.g. data service, S3, Aurora serverless) which incurs storage cost. There is a whitepaper on choosing the right AWS service to deploy your website or web application, with a decision tree. AWS pioneered serverless with Lambda release in 2014 but competitor follows. In the Azure landscape, there is an entire suite of computing services from virtual machine to serverless (also with a decision tree in documentation). Azure\u0026#8217;s counterpart for serverless architecture is Azure Function (released in 2016 for GA) with API Management. As for GCP, the serverless suite includes the event-driven Cloud Function (introduced in 2017) and Knative-based FaaS Cloud Run (introduced in 2019).\nThere are voices in advocacy of standardization of serverless model, and CNCF had since made minuscule efforts such as CloudEvents. The status quo, unfortunately, is anything but standardized.\nLastly, here is a table that summarizes the pros and cons of each computing service model.\nComputing Service ModelProConEC2\u0026#8211; Most straightforward and widespread legacy model\n\u0026#8211; Legacy\u0026#8211; Ops tasks can be heavy (e.g. patch and vulnerability management of OS)\n\u0026#8211; Utilization can be lowECS\u0026#8211; Container orchestration is managed\n\u0026#8211; Convenient to scale\n\u0026#8211; Well integrated with other AWS services\u0026#8211; Limited advanced features\n\u0026#8211; Vendor lock-inEKS\u0026#8211; Highly scalable and flexible\n\u0026#8211; Advanced, platform-neutral deployment tools available (e.g. Helm, ArgoCD, etc)\n\u0026#8211; Custom configurations (e.g. operators)\u0026#8211; Significant operation overhead\n\u0026#8211; Steep learning curve (especially for teams)Lambda\u0026#8211; automatically scale\n\u0026#8211; low ops overhead\n\u0026#8211; pay per use\u0026#8211; limited choices of runtime\n\u0026#8211; subject to latency due to cold start; yet warm start incurs cost\n\u0026#8211; not suitable for long running tasks (batch processing jobs etc) In summary, the Lambda-based serverless model is good for stateless server-side workload with short response time (\u0026lt;15s), tolerance of cold-start, no need for portability across platforms, and no complex package dependency. Previous PostGraphQL and gRPC Next PostAWS serverless services and developer tools ","date":"2022-10-21T19:31:00-04:00","image":"/wp-content/uploads/2025/04/feature-computing-paas-serverless.webp","permalink":"/2022/10/computing-from-paas-to-serverless/","title":"Computing services: from PaaS to Serverless"},{"content":"Big Picture For inter-process communication at a high level, the two styles are asynchronous and synchronous styles:\nAsynchronous event-driven style: involving an event broker as a middle man. Synchronous request-response style: including several families of technologies: RPC (Remote Procedure Call): CORBA (Common Object Request Broker Architecture) Java RMI (Remote Method Invocation) SOAP (Simple Object Access Protocol) REST (Representational State Transfer) gRPC GraphQL Apache Thrift RPCs, built on top of TCP/UDP, are usually complex to implement. SOAP improved it and can operate on HTTP. Many large companies today still used SOAP for message exchange. However, it has a limitation with complex format and specifications for XML messaging, giving rise to REST. REST is not a standard, but rather a loosely defined architectural style. Its payload can be in any format (XML, JSON, etc) specified in the header. As long as the API conforms to certain guidelines (criteria outlined in this article), we can consider the API RESTful. REST has been steadily replacing SOAP in the past few years. In this post, I start with REST, then dive into gRPC and GraphQL.\nREST The de facto method of building microservices using REST architectural style is use HTTP protocol with JSON format payload. JSON format is human readable, but not optimized for machine-to-machine communication. So there is some room for compression. To emulate a web request in REST, one can use curl, a common utility to emulate any HTTP client.\nREST has its shortcomings. For example, the interface between REST client and server is not strongly typed. You can choose to use OpenAPI/Swagger specification to define types but it is still not tightly integrated. There is no enforcement on the format of the payload either. RESTful services are quite bulky, inefficient, and error-prone. gRPC and GraphQL emerged to address different challenges with REST. GraphQL operates on HTTP and we can view it as a layer on top of REST in a broad sense. gRPC on the other hand, operates on HTTP /2, and thereby inherits many advantages from it.\nHTTP /2 HTTP/2 is the second major version of HTTP. It overcomes some issues with HTTP/1.1 on security, speed, etc. This post is a good rundown of the difference between HTTP /2 and HTTP /1.1, which account for many of the advantages of gRPC. A thorough discussion on the differences between the two HTTP versions is beyond what this post can cover. One of the important difference with HTTP/2, is that all communication between a client and server is performed over a single TCP connection that can carry any number of bidirectional flows of bytes. This makes gRPC a high-performance RPC framework. In HTTP/2, the key concepts to understand are: Stream: a bidirectional flow of bytes within an established connection. A stream may carry one or more messages; Frame: the smallest unit of communication in HTTP/2. Each frame contains a frame header, which at a minimum identifies the stream to which the frame belongs. Message: a complete sequence of frames that map to a logical HTTP message that consists of one or more frames. The request message is always triggered by the client. During the interaction, the client and server break down the message into frames, interleave them, and then reassemble them on the other side. In this way HTTP /2 multiplex the messages, and enables the following communication patterns:\nSimple RPC: a single request and a single response in the communication; Server Streaming RPC: a single request and message followed by multiple response messages; Client streaming RPC: client sends multiple messages and the server replies with one response message; Bi-directional RPC: client setups connection by sending header frames. Once connection is established, both client and server send messages simultaneously without waiting for the other to finish; The streaming communication patterns fundamentally improves performance, enabling the duplex streaming capability for gRPC.\ngRPC At the protocol level, gRPC has the following advantages over REST:\nwell-defined service interface and schema strongly typed data duplex streaming (thanks to HTTP2) built-in commodity features (e.g. authentication, encryption, resiliency, service discovery, etc) I previously touched on gRPC protocol in the context of Envoy and etcd. Envoy makes use of gRPC for its control plane, where it\u0026nbsp;fetches configuration from management server(s)\u0026nbsp;and in filters, such as for\u0026nbsp;rate limiting\u0026nbsp;or authorization checks. Etcd store implements gRPC protocol for client utility to communicate with. Since the typical use case of gRPC is internal communication, I have not been able to find a playground online. To get a taste of gRPC client-server interaction, just play with etcd on MacOS. gRPC is language neutral. To start developing, refer to the tutorial in different languages (e.g. Golang, Python). When developing a gRPC application, the first thing to do is define a service interface in IDL (interface definition language). gRPC uses protocol buffers as the IDL to define the service interface. Protocol buffers are a language-agnostic, platform-neutral, extensible mechanism to serializing structured data. Using that service interface definition, we can generate the server-side code known as a server skeleton. Also you can generate the client-side code, known as a client stub. The methods that you specify in the service interface definition can be remotely invoked by the client side as easily as making a local function invocation. gRPC also has some disadvantages. Currently, the ecosystem is still small. When we have a service interface, we have to maintain the interface across versions.\nGraphQL In most of the use cases of gRPC and GraphQL, GraphQL works for external-facing services/APIs while internal services backing the APIs are implemented using gRPC. Honeypot created good documentary for GraphQL available here. We use GraphQL for external facing services because it gives API client the ability to query. An SQL query allows client to filter requested data based on conditions, and define the interested columns in the data return. This capability is missing in the REST style guideline. One may choose to implement their RESTful service to support their client\u0026#8217;s query requirement, GraphQL standardizes this capability with typed data, thereby prevents unnecessary network round trips, over- and under-fetching of data.\nThe official site is a good reference for learning. One needs to know concepts around queries and mutations, schema and types (scalars, variable, fragment, interfaces, unions) to understand how query works.\nThis page lists a number of publicly available services in GraphQL. For example, country information service is available here. On the web page you can put in a query like this:\nquery myCountry { countries(filter:{code:{in:\u0026#34;CA\u0026#34;}}) { code, name, capital } } The query (with the name of myCountry) above asks to return entries with countries as type. Then it filters the result by the condition that the country code must include \u0026#8220;CA\u0026#8221;. The return should include code, name and capital columns. The web page looks like this:\nGraphQL is on HTTP, so I can emulate the call with curl and get the same result:\ncurl --request POST https://countries.trevorblades.com/ \\ --header \u0026#39;Content-Type: application/json\u0026#39; \\ --data-raw \u0026#39;{ \u0026#34;query\u0026#34; : \u0026#34;query myCountry { countries (filter: { code: { in: \\\u0026#34;CA\\\u0026#34; } }) {code name capital} }\u0026#34; }\u0026#39; The return prints the same result:\nThe document has a page on library options for different languages, such as Graphene and Ariadne for Python. You can use Ariadne and Flask to build GraphQL API as this post suggests.\nThe example on this page has a better example that highlights how GraphQL allows client to ask multiple questions at the same time, and how the response includes different pieces of answers. GraphQL backend often needs to connect to different downstream system. This requires a proxy service to connect to disparate data sources. Examples of such services with such capabilities include Apollo, AWS AppSync, or ApiGee. Summary For API protocols, we often compare among REST, gRPC and GraphQL. REST is loosely defined and widely adopted for the past few years. The use case of REST diverged into two areas: external facing API and internal service-to-service communication. gRPC and GraphQL are relatively new and they each are good for one of the use cases.\nMicroservice pattern based on gRPC and GraphQL (source: \u0026#8220;gRPC up \u0026amp; running\u0026#8221; by Indrasiri \u0026amp; Kuruppu) The gRPC protocol on HTTP2 is often used in internal communication between microservices. GraphQL offers query capability and is therefore often used to face external client.\nPrevious PostBuild and Manage Kubernetes Clusters Next PostComputing services: from PaaS to Serverless ","date":"2022-10-07T14:49:00-04:00","image":"/wp-content/uploads/2025/04/feature-graphql-grpc.webp","permalink":"/2022/10/graphql-and-grpc/","title":"GraphQL and gRPC"},{"content":"There are numerous options to build a Kubernetes cluster. If your company has a multi-cloud strategy, most likely you will have to deal with cluster creation on multiple cloud platform or on virtual machines on premise. Most likely, the chosen cloud platform already make it simple for us. However, it is still important to understand what it really takes to build a Kubernetes cluster. In general, we need to figure out these tasks:\nDecide where to host the computing infrastructure (i.e. Node) : on premise or public cloud; Choose a Kubernetes release: either the vanilla release or one of the third-party distributions; Install Kubernetes to the computing environment, and integrate it with the cloud platform; Determine required add-ons (e.g. Istio or Linkerd for Service Mesh, dashboard utility, etc); Deploy application workload to Kubernetes platform; A public cloud platform provider usually can assist you with task 1 through 3, and partially 4, depending on the provider. If your Kubernetes resides on private cloud or on-prem environment, you can use a Platform solution such as VMware Tanzu or Openshift, which usually covers task 1, 3 and 4. There is no standard about what task these platform solution must address. Therefore it is important to have this list of tasks in mind in order to make a good comparison. I will discuss each of the tasks in this post.\nHosting environment Nodes are the building blocks of a Kubernetes cluster. We need master nodes as well as worker nodes. In addition, a working cluster also requires storage, and networking infrastructure. Public cloud platforms typically provides control plane as a service, obviating administrator\u0026#8217;s effort to provision master nodes. For example, the control plane of Azure AKS has two levels of uptime commitment: a free tier of 99.5% SLO and a paid tier with an SLA of 99.95% (using AZs) and 99.9% (without using AZs). This uptime commitment applies to control plane only and do not apply to worker nodes. The management of etcd store is also a responsibility of the cloud provider, which frees up the cluster administrator from managing etcd store. However, they cannot access etcd store either. This is not very convenient because as the size of the cluster grows it is a common requirement to connect to etcd store for troubleshooting purpose.\nThe deployment APIs for public cloud allow the cluster administrator to define the instance size, count and availability zone for the worker nodes. They also automatically register the worker nodes to control plane so that the cluster administrators do not have to do so by themselves. As to storage, the public cloud usually provide some default storage classes based on their storage as service. For networking device, the cluster provision process automatically configures the cloud API so the cluster can manage cloud resources such as network load balancer. With private cloud or data centre, we usually use virtual machines, or bare-metal servers. Cluster administrators will need to make their own control plane with master nodes. and install worker nodes and register them to the master nodes. The Kubernetes Installation section below will discuss this.\nKubernetes release If you have to install Kubernetes, you have to think about the Kubernetes release being used. You can use the binary from official Github repository. For example, the release note of version 1.24.3 points to the change log file for download links to server binaries, node binaries. This is the vanilla Kubernetes release.\nApart from the vanilla release, many developers build their own distributions, based off forks of the Kubernetes project. CNCF has a page to keep track of certified Kubernetes distributions. Some of the distributions are open source and can be used for on-prem infrastructure. Here is a list of top players:\nDistribution NameRepoDescriptionEKS DistroLinkUsed in EKS managed service or EKS Anywhere for on-prem infrastructureAKS EngineLinkUsed in Azure Stack for on-prem infrastructure. Google Kubernetes EngineN/AUsed in GKE managed service only. OpenShift Kubernetes Engine\nLinkCommunity distribution (OKD, or OpenShift Kubernetes Distribution) is the open-source upstream.Rancher Kubernetes Engine (RKE)Linkstill using Docker as container runtime. Supported CNI include: Canal, Flannel, Calico and WeaveK3sLinkLightweight distro without small resource requirement. Great for Edge, IoT, ARM etcRKE2LinkOriginally named RKE government. Supports deployment via Cluster API. Supports containerd as container runtime. Supported CNI include: Cillium, Calico, Canal and Multus. LightweightVMware TanzuLinkVMWare Tanzu Grid and VMWare Tanzu Community Above is just a very incomplete list of Kubernetes distributions. There are many more distributions that are not on this list, such as CoreOS Tectonic, Docker Kubernetes, Heptio, Mesosphere, Mirantis, Platform9, Stackube, Telekube. For full details of how each distribution is different, you will need to go over their documents. With the selected distribution, we still need to deploy the binaries to the nodes. We can do this with a cluster management platform, or standalone installers. Cluster management platform can also help us with baseline configuration (e.g. IAM integration, CNI plugin), in addition to the binary installation. Cluster Management Platform These platforms are also sometimes referred to as container management platform.\nFor example, OpenShift container platform is a self-managed platform based on OpenShift Kubernetes Engine and can run on a variety of hosting environment, public cloud, or private cloud. The installation steps varies depending on the hosting environment. When running on public cloud such as AWS (aka ROSA), the public cloud only provides computing nodes and associated infrastructure. Many corporate with multi-cluster strategy use this option on public cloud to keep their Kubernetes cluster fleet consistent across cloud vendors. The Openshift container platform also packages some useful open-source add-ons with corporate support, for example:\nOpenShift Service Mesh: Istio Ceph Storage Gluster Storage OpenShift GitOps (ArgoCD) OpenShift Pipelines\u0026nbsp;(Tekton) Quay (Quay Image Registry) OpenShift Streams for Apache Kafka OpenShift Serverless (Knative Serving) Red Hat\u0026#8217;s strategy is to pick the most renowned open-source project in each domain and add enterprise support to it. However, for management portal, Red Hat developed its own Advanced Cluster Management tool for Kubernetes, and open-sourced it in 2020 in the upstream project Open Cluster Management.\nSimilar to OpenShift, VMware Tanzu also attempts to cover the domains, with a smaller product portfolio:\nService Mesh: compatible with Istio Mission Control: management portal Observability Google Anthos is also a container platform. Their product line include, but not limited to:\nAnthos Config Management (ACM) Anthos Service Mesh (ASM, an Istio distribution) SUSE, the developer of RKE, RKE2, and K3s) offers Rancher as multi-cluster management platform. Apart from the engines, SUSE also offers Lonhorn as a storage solution. However, they do not have offerings for service mesh or GitOps. So there is no doubt that Red Hat OpenShift has the most complete portfolio for Kubernetes.\nThere are also companies that only offers management platforms without their own Kubernetes distribution. For example:\nPlatform9 Rafay Product capabilities in this category vary a lot and you should refer to their specific documentation to understand. You will probably see a stack chart from each of the platform provider (e.g. SUSE Enterprise Container, OpenShift, Tanzu, Anthos, Rafay) with all technology integrations.\nCluster Installation Tools As we saw in the installation steps for OpenShift, they are highly dependent on platform. With public cloud, the provisioning process also applies only to a specific platform. Since Kubernetes Installation process is tedious, some tools emerged to help, for example: kubespray, kubeadm, kops and Cluster API. These are governed by SIG cluster lifecycle special interest group. Here are some traditional options to install a Kubernetes clusters:\nkube-up: the first tool to build cluster from 2015. It has been deprecated. Kubeadm: a tool built to provide best-practice \u0026#8220;fast paths\u0026#8221; for creating Kubernetes clusters that are minimum viable, and secure. Kubeadm\u0026#8217;s scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools. It is first released in Sep 2016. The high level configuration steps goes through initialization (kubeadm init), control plane (kubeadm join control plane), and node (kubeadm join node). Kubeadm does not integrate with cloud providers and it does not install addons (auth, monitoring, CNI, storage class) Kubespray: runs on bare metal or VMs using Ansible for provisioning and orchestration. The first release was in Oct 2015. Since v2.3 (Oct 2017) kubespray started to use kubeadm internally. In addition to kubeadm, kubespray configures CNI, storage class, other CRI. It supports cloud providers and air-gap environment. However it does not support infrastructure management. The options above are official options. You may use kubeadm and kubespray to quickly (i.e. in an hour) spin up clusters for education purposes. However, with their limitations, it typically requires a lot of efforts to build a production-grade cluster with the needed addons and integrations. Apart from the official options, there are also unofficial tools such as kubicorn, which was first introduced in 2018 as a cluster management framework with modular support for cloud providers. However it appears to be short-lived.\nIn the next two sections, we introduce kops and cluster API, two most recent projects to install cluster.\nKops The kops utility directly perform the provisioning and orchestration via API to the cloud deployment engine. Kops, with first release in Oct 2016, is tightly integrated with the unique features of the cloud providers (e.g. AWS: ASG, ELB, EBS, KMS, S3, IAM). However, kops is only CLI without controller-style reconciliation. It does not support baremetal or vsphere. It also bundles addons with fixed version.\nWhen picking a tool to install cluster, we need to strike a balance between how much simplification the tool brings, and how many different platform the installer can work with. Kops appears to be such a good compromise. It works with a number of cloud platforms using different set of APIs, although most are in alpha and beta stages today. Here is how to install cluster on AWS. Both kops and Cluster API have good momentum but they work differently. Cluster API was first released in Mar 2019, and is currently less mature than kops. However, it is declarative and may reflect the direction of where cluster lifecycle management is heading.\nCluster API Cluster API focuses on following areas:\nManage cluster lifecycle declaratively Infrastructure abstraction (e.g. computing, storage, networking, security, etc) Utilizing existing tools (e.g. kubeadm, cloud-init) Modular and pluggable: to be adaptable to different infrastructure providers. It involves a number of CRs as illustrated in its diagram. We should be clear on the providers for Bootstrap, Infrastructure and Control Plane.\nThe biggest benefit is the controller pattern to manage the entire lifecycle of a cluster. This allows managing clusters with GitOps, and rolling upgrade of the cluster. It also allows for declarative node scaling, self healing and multi-cluster management.\nThe client utility for is clusterctl, and with that along with the manifest, we can create a cluster in a few commands. A lot of workflows are still in development but we can take a look at its quick start guide to get a taste of how it works. The installation steps vary a lot based on the environment and the cluster. Also it introduces the separation of management cluster and workload cluster.\nWorkload cluster is the target cluster being created, as per the manifests. Management cluster is where you keep track of the workload cluster being managed. You can manage multiple workload clusters from a single management cluster. Note that this management cluster will store credentials about workload clusters, and may become a single point of failure. Although Cluster API reflects a great initiative to standardize the provisioning of Kubernetes cluster, whether it will succeed has to do with the level of complexity. In the next section, we will get a taste of how it looks to deploy a Kubernetes cluster in a lab.\nManagement cluster vs workload cluster In the lab, I use my MacBook to create a management cluster with KinD. Then we configure a workload cluster in AWS from the management cluster. Cluster API Lab Note that the steps here are based on the quick start guide on Cluster API document. Also, there is a bug with the AWS provider so the end of the lab will report a warning. The main purpose of this lab is to demonstrate how Cluster API is supposed to work, even though it still has yet to mature.\nTo start, I install clusterctl (the cluster API client utility), clusterawsadm (the utility specific for AWS) on MacBook, then start a simple KinD cluster.\ncurl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.0/clusterctl-darwin-amd64 -o clusterctl chmod +x ./clusterctl sudo mv ./clusterctl /usr/local/bin/clusterctl clusterctl version curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.4.1/clusterawsadm-darwin-amd64 -o clusterawsadm chmod +x clusterawsadm sudo mv clusterawsadm /usr/local/bin clusterawsadm version kind create cluster So far, I installed the required utility and a KinD cluster on MacBook. Then I use clusterawsadm to create InstanceProfile, ManagedPolicy and IAM Roles required for cluster creation. The AWS region and access are configured as environment variables:\nexport AWS_REGION=us-east-1 export AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXX export AWS_SECRET_ACCESS_KEY=J8ByduiofpwuisDjDoijOISDs clusterawsadm bootstrap iam create-cloudformation-stack This runs a CloudFormation stack to create the permission related resources:\nThen I initialize the management cluster with the clusterctl utility, specifying AWS as a provider. I also need to assign the environment variable AWS_B64ENCODED_CREDENTIALS with proper value: export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile) clusterctl init --infrastructure aws Now I use clusterctl to generate the manifest for the workload cluster. In environment variables, I specify cluster and node sizes, SSH key name, control plane machine type and node machine type:\nexport AWS_SSH_KEY_NAME=cskey export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large export AWS_NODE_MACHINE_TYPE=t3.large clusterctl generate cluster myekscluster --kubernetes-version 1.24.3 --control-plane-machine-count=3 --worker-machine-count=3 \u0026gt; capi-quickstart.yaml kubectl apply -f capi-quickstart.yaml At the end I tell the management cluster to create a workload cluster as per the manifest, by simply declaring the CRs. It will take some time for the cluster to create, and there are a number of ways to monitor the progress. You can monitor the log on the controller pods in their respect namespaces. You can also check the cluster status with:\nkubectl get kubeadmcontrolplane clusterctl describe cluster myekscluster Currently there is a bug and the commands at the end will report as below:\nHopefully the bug will be fixed shortly. To delete the cluster, simply delete the resources in the manifest with kubectl delete -f capi-quickstart.yaml\nSummary There are numerous ways to build a Kubernetes cluster. Before deciding on the approach, I recommend having a full understanding of the hosting environment. This is because installation approach and hosting environment are still tightly coupled. This is the status quo and is not going to change in the near future. Both kops and cluster API reflects initiative to decouple the two but both are still in early stage and already facing growing complexity. Cluster API manages complexity with CRDs to abstract system resources and infrastructure, as illustrated here:\nCRDs and providers to abstract system resources and infrastructure The diagram is from the \u0026#8220;Cluster API and declarative Kubernetes Management\u0026#8221; white paper. Here is a stream with more about the same topic.\nPrevious PostMinIO for S3-compatible Object Storage Next PostGraphQL and gRPC ","date":"2022-09-23T11:50:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-cluster.webp","permalink":"/2022/09/build-a-kubernetes-cluster/","title":"Build and Manage Kubernetes Clusters"},{"content":"I reviewed some storage technologies on Kubernetes but they are all for block and file storage. In this post, I will discuss the current available options for container workload to use object storage. I will also touch on MinIO as an object storage solution.\nObject storage Block and file system are more native to operating system because they present themselves to the OS as a block device or file system attached to the OS. In other words, application processes running on the OS will be able to access the storage by address expressed as a POSIX-compatible path. On the contrary, object storage is a REST API service, operating at the application layer in the TCP/IP stack. Therefore, we can think of object storage as \u0026#8220;storage as a web service\u0026#8221;.\nObject storage can be made very cheap. However, the application protocol may vary depending on the object storage provider. Amazon S3 is a forerunner in object storage market and its protocol has emerged as the de-facto standard for object storage. When building an application and if there is one object storage protocol to support, it should be S3. For non-S3 object storage services, we can front them with an S3 interface, if the provider itself does not have one. For example Ceph storage has its Gateway S3 API. Container Object Storage Interface If we use S3 as the universal object storage protocol, does that also address object storage access with container workload on Kubernetes? Absolutely. Nonetheless, for a number of reasons using REST API from containers are not the best option. From platform\u0026#8217;s perspective, it is the platform that should define how to access object storage, instead of leaving it with an application-layer protocol. When a pattern (for storage, or networking, etc) turns out very common, the platform layer should incorporate it as an infrastructure service, manage it with its own standard, and provide it to application so that developer can focus on business features. With that vision, the community brought up the Container Object Storage Interface (COSI) initiative. It is currently in very early stage, but the idea is to commoditize object storage in Kubernetes platform with a unified interface. For more background about this initiative, refer to the post \u0026#8220;Beyond block and file \u0026#8211; COSI enables object storage in Kubernetes\u0026#8220;.\nCOSI is the ultimate cloud native solution but it is still in pre-alpha phase as of mid 2022. Unfortunately, it is not a recommended solution to any real-life project in 2022, and we are stuck with the unified API approach until COSI matures.. The unified API approach is by no means cloud native, but has come to maturity for adoption. S3 Rest API is our friend, regardless of whether the client process is in a container or not.\nUpdate: on Sept 2, 2022, Kubernetes introduced COSI as alpha feature.\nMinIO Introduction In order to use S3 protocol without using Amazon S3 storage, we can use MinIO to build our own object storage service serve client via a S3-compatible REST API interface. The main developer of the MinIO project is MinIO Inc, a startup from 2014. Having learned the lessons from GlusterFS, the founders and developers make MinIO very simple. MinIO operates in two modes: gateway mode (soon to be legacy) and server mode.\nIn the Gateway mode, MinIO as a gateway between client and destination storage, and does not persist data to itself. In the past, the destination storage can be Azure Blob and Google Cloud Storage (GCS) and HDFS as backend. However, these supports are deprecated now. The current release (July 2022) only supports S3 and NAS backend. According to MinIO\u0026#8217;s blog post from February 2022, the entire MinIO Gateway feature will be removed in August, leaving server mode the only option for MinIO.\nIn the Server mode, the MinIO service will persist data to itself in a file system (or volume). You can specify that file system (or volume) as you launch the server. As one of the quick-start guides shows, we can host MinIO server using a single executable. For administrative tasks, MinIO has a web console and a client utility called mc.\nMinIO Deployment Options For storage service, there are a number of deployment options: SNSD (single-node, single-drive): single MinIO server with a single storage volume or folder. SNMD (signle-node, multi-drive): single MinIO server with four or more storage volumes. MNMD (multi-node, multi-drive, aka distributed): multiple MinIO servers with at least four drives across all servers. This should be considered for production grade configuration. The deployment options above describes the node and volume topology. No matter which topology option, there are also a number of ways to host the MinIO service process: on Linux OS, Windows OS, MacOS, Docker Container, and on Kubernetes platform. In addition, MinIO Inc ships the software under different business models. For example, there are fully managed applications in Azure Marketplace, AWS Marketplace, and GCP Marketplace all hosted on virtual machines with extra charges. Clients not willing to pay can host MinIO storage all on their own, either on virtual machines, or on managed Kubernetes environment provided by each cloud provider. MinIO Hosting solutions MinIO lists these hosting solutions under multi-cloud products. These hosting solutions (or \u0026#8220;products\u0026#8221; in MinIO\u0026#8217;s term) vary in terms of where peripheral services and data tiers are hosted. Here is the list of the supported platforms:\n(generic) Kubernetes; VMWare Tanzu; OpenShift; SUSE Rancher EKS AKS GKE To illustrate how these solutions are different, I put some details on a few options together for an incomplete comparison below:\nKubernetesEKSAKSGKEHot StorageDirect PV (NVMe)EKS EBS CSIAzure CSI GKE Standard SSDWarm StorageDirect PV (HDD)S3 IAAzure BlobStoreGCSCold StoragePublic Cloud storageGlacierAzure Cool BlobGCS for Data ArchivingEncryptionHashiCorp VaultKMSAzure Key VaultCloud Key ManagementObservabilityElastic Stack and GrafanaManaged ElasticSearch PrometheusAzure MonitorStack DriverIdentity ProviderKeyCloakLDAP, SSOAzure Active DirectoryGCP Cloud IdentityLB and Cert MgmtNginx, Let\u0026#8217;s EntryptAWS Cert Mgr, ELBAzure Load Balancer, JetStack, Let\u0026#8217;s EncryptGCP Cloud LB and Managed Cert Note that all of these hosting solutions are based on some flavour of Kubernetes. The hot tier is usually based on storage options available to the platform. MinIO service access this hot tier via Kubernetes persistent volume. The warm and cold tiers are backed by different object storage service. Between MinIO and storage client, it always use the same S3 compatible Rest API.\nMinIO also has tiering capability. While the hot storage destination has to be either a file system or Kubernetes persistent volume, remote tiers can be S3 , Azure Blob, or GCS. MinIO supports encryption at rest (SSE-KMS, SSE-S3, SSE-C) and in transit (TLS) for security, as well as many other useful features such as object replication, versioning, locking, events, Prometheus metrics, lifecycle management etc. Connect to MinIO server with S3 client To validate that the client is compatible, we use MinIO\u0026#8217;s client utility (mc) to connect to an AWS S3 bucket. Then we use AWS CLI to connect to a MinIO server, similar to this instruction. To do so, we first install client and server utilities:\nbrew install minio/stable/minio brew install minio/stable/mc minio --version mc --version Then, we start MinIO server and store an object using AWS CLI\u0026#8217;s S3 tool. In our working directory, we create a new directory called minio_data and launch MinIO server with it:\nmkdir minio_data minio server minio_data --console-address :9090 Once the server is up, the screen should display the details, including the portal URL and the default username and password will be used as Access Key ID and Secret Key:\nNote that the MinIO service does NOT have TLS enabled by default, on the console or API service. At this point, we can browse to the console web page using the given credential. Then, we can configure AWS CLI with a new profile just to act as a client to communicate with the MinIO server:\n$ aws configure --profile minio-cli AWS Access Key ID [None]: minioadmin AWS Secret Access Key [None]: minioadmin Default region name [None]: us-east-1 Default output format [None]: json $ aws configure set default.s3.signature_version s3v4 --profile minio-cli At this point, the AWS CLI is configured to communicate with MinIO server. Then, we can create bucket, list object in the bucket, copy an object to the bucket, etc\n$ aws --endpoint-url http://127.0.0.1:9000 s3 ls --profile minio-cli # list all bucket, should return empty $ aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://hehebucket --profile minio-cli # create new bucket make_bucket: hehebucket $ aws --endpoint-url http://127.0.0.1:9000 s3 cp README.md s3://hehebucket --profile minio-cli # copy a file to bucket as a new object upload: ./README.md to s3://hehebucket/README.md $ aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://hehebucket --profile minio-cli # list objects in the bucket 2022-07-09 00:30:23 631 README.md The created bucket and object are also visible in MinIO web console, under \u0026#8220;Bucket\u0026#8221;:\nThe steps above validate that AWS CLI can talk to MinIO server. Because of that, MinIO server can emulate an S3 service in any development environment so users do not always have to use S3 from AWS. This makes sense for both cost and security reasons for the organization. Connect to S3 with MinIO client In this lab, we create an S3 bucket and use mc utility to store an object to it. In order to consistently create S3 bucket and associated permissions, I use the CloudFormation template in this repo. The output of the CloudFormation stack returns the Access Key ID and Secret Key required for the client to access the bucket. Once we cloned the repo, let\u0026#8217;s enter the obj-store-helper directory, and run aws cli command to launch the CloudFormation template, assuming it has been configured:\nBUCKET_NAME=c0sas2dsadigihunch S3_STACK_NAME=$BUCKET_NAME-stack aws cloudformation create-stack --template-body file://aws-s3-stack.yaml --stack-name $S3_STACK_NAME --parameters ParameterKey=S3BucketName,ParameterValue=$BUCKET_NAME --capabilities CAPABILITY_IAM # to delete stack after test, run: aws cloudformation delete-stack --stack-name $S3_STACK_NAME In the AWS console, we should see the configuration information as below:\nSupposed the bucket name is vna-tst-c0sas2dsadigihunch as shown above, this allows us to configure the client utility MC as below:\nmc alias set awss3 https://s3.amazonaws.com # Fill in access key ID and Secret key at the prompt mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, should return empty mc cp README.md awss3/vna-tst-c0sas2dsadigihunch/README.md # upload and object to bucket mc ls awss3/vna-tst-c0sas2dsadigihunch # list objects in the bucket, the uploaded object should be there mc rm awss3/vna-tst-c0sas2dsadigihunch/README.md # delete the object mc alias remove awss3 # remove awss3 alias Once we emptied the bucket, we can delete the CloudFormation stack. This test only needs client utility mc to verify that MinIO client is able to talk to AWS S3 server.\nErasure Coding For scalable production use, we should deploy MinIO in distributed mode. When MinIO is configured in distributed deployment (MNMD, or multi-node, multi-drive), it implicitly enables an important feature called erasure coding. This erasure coding feature further unlocks a number of other MinIO features:\nObject Versioning Server-Side Replication Write-Once Read-Many (WORM) Locking Erasure coding is MinIO\u0026#8217;s data redundancy and availability feature that allows MinIO deployments to automatically reconstruct objects on-the-fly despite the loss of multiple drives or nodes in the cluster. Erasure coding provides object-level handling with less overhead than adjacent technologies such as RAID. The key concept is Erasure Set, a set of drives in a MinIO deployment that supports Erasure Coding. MinIO evenly distributes object data and parity blocks among the drives in the Erasure Set. Two important variables are M and N: for a given erasure set of size M, MinIO splits objects into N parity blocks, and M-N data blocks. MinIO uses the EC:N notation to refer to the number of parity blocks (N) in the deployment. To determine optimal erasure set size for the cluster, use MinIO\u0026#8217;s Erasure Coding Calculator tool.\nTo help client to specify per-object parity with Erasure Coding, MinIO uses storage classes. Note that the storage class concept in MinIO is distinct from AWS S3 storage class or Kubernetes storage class. In MinIO, a storage class defines parity settings per object. The STANDARD storage class (default) defines EC:N based on M, which can be overridden. In addition, there is REDUCED_REDUNDANCY storage class, whose parity must be less than or equal to that of STANDARD storage class. The erasure coded backend also protects the storage against Bit Rot with HighwayHash algorithm. More Features Authentication and authorization between MinIO client and MinIO server have a number of options. MinIO client may use the built-in standalone identity management in MinIO server. This is the default mode. In addition, one may delegate IAM to external service. To Active Directory via LDAP, or any Identity provider that supports OIDC (JWT with Authorization Code Flow). As to Object Lifecycle Management (OLM), MinIO allows you to define a remote tier storage for each local target (bucket). The remote tier can be Amazon S3, Google Cloud Storage or Azure Blob storage. We can use mc utility to administer the remote tier and OLM. Configuration steps (e.g. Azure Blob, AWS S3) usually include:\nConfigure required permissions on the MinIO bucket, create user account for OLM activities. Configure the Remote Storage Tier Create and Apply an ILM Transition Rule. The rule can be expressed in a json document. Validate the creation of ILM transition rule Validate the effect of transition rule. As for encryption, MinIO can support encryption at rest. It can also work with etcd store to store encrypted IAM assets if KMS is configured. Conclusion Even though we watch for the progress of COSI initiative, we still use Rest API to access object storage from container, which is no different than from a virtual machine. If we develop an application, then we should make it support S3 protocol, a de-facto standard protocol for object storage. As for the storage backend, if we want to be vendor neutral, the feature-rich MinIO is the best bet. We can use MinIO to build our own Object storage as a service compatible with S3. We can also lifecycle our object to remote object storage tier backed by Azure, GCP or S3. In this post we validated the S3 compatibility, and discussed some advanced MinIO features.\nPrevious PostKubernetes Storage on Azure 3 of 3 – Ceph by Rook Next PostBuild and Manage Kubernetes Clusters ","date":"2022-09-09T09:00:00-04:00","image":"/wp-content/uploads/2025/04/feature-minio.webp","permalink":"/2022/09/minio-object-storage/","title":"MinIO for S3-compatible Object Storage"},{"content":"In the last two posts, I covered the native storage options on Azure Kubernetes Service, as well as Portworx as an example of a proprietary Software Defined Storage (SDS) solution. There are also a number of open-source alternative SDS solutions. Ceph has nearly a decade of history from prior to containerization, and is the most widely adopted storage platform. In this post, we continue to explore Ceph as an open-source storage solution on Azure Kubernetes. Ceph by Rook Ceph is an open-source SDS platform for distributed storage on a cluster and provides object, block and file storage. Installation of Ceph SDS can be complex, especially on Kubernetes platform. Rook is a graduated CNCF project to orchestrate storage platform. Rook by itself is not SDS and it supports:\nCeph: configure a Ceph cluster. Think of this as the equivalent of cephadm on Kubernetes platform. NFS: configure an NFS server. Think of this as the equivalent of nfsd daemon on Kubernetes platform. Cassandra: an operator to configure a Cassandra database cluster. It is now deprecated. We play with Rook Ceph. I also refer to it as Ceph by Rook. The contribution of Rook project is it simplifies the installation as a matter of declaring custom resources using CRDs. Here are some high-level CRDs to know:\nCephCluster: creates a Ceph storage cluster CephBlockPool: represents a block pool CephFilesystem: represents a file system CephObjectStore: represents an object store CephNFS: spins up a NFS Ganesha server to export NFS shares of a CephFilesystem or CephObjectStore. As with typical Kubernetes resources in controller pattern, Ceph by Rook needs an operator along with custom resources. We can use YAML manifest for both of them, and the manifests are usually very tediously long. We can also use Helm to install both of them, by providing a value file. Now we will install Ceph on AKS.\nInstall Ceph Operator on AKS The steps are influenced by two relevant posts (here and here). However, I\u0026#8217;ve incorporated the cluster configuration in the Azure directory of the cloudkube project, a modular Terraform template to configure AKS cluster and facilitate storage configuration. The node group and instance sizes are selected to be just enough to run a ceph POC cluster with minimum cost. One of the node groups is tainted with storage-node, as if the following command were run:\nkubectl taint nodes my-node-pool-node-name storage-node=true:NoSchedule You will only need to taint the nodes with the command above if you choose not to use the cloudkube template. The taint ensures that only Pods with corresponding toleration and effect can be scheduled to those nodes.\nWe use Helm to install Rook Operator. We need a value file (e.g. rook-ceph-operator-values.yaml) with content as below:\n# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/operator-chart.md crds: enabled: true csi: provisionerTolerations: - effect: NoSchedule key: storage-node operator: Exists pluginTolerations: - effect: NoSchedule key: storage-node operator: Exists agent: # AKS: https://rook.github.io/docs/rook/v1.7/flexvolume.html#azure-aks flexVolumeDirPath: \u0026#34;/etc/kubernetes/volumeplugins\u0026#34; Then we install the operator with Helm:\nhelm install rook-ceph-operator rook-ceph --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-operator-values.yaml kubectl -n rook-ceph get po -l app=rook-ceph-operator After installing the operator, we check the Pod status to make sure it is running. Then we can install the actual Ceph Cluster in one of the two ways. We can declare a CephClusterCRD ourself, or we can use Helm again to declare the CRD. Helm Chart gives us a lot of useful default values and saves us from editing a large body of YAML manifest.\nInstall Ceph CR on AKS We use Helm to install CephCluster CRD. We create a value file (e.g. rook-ceph-cluster-values.yaml) with content as below:\n# https://github.com/rook/rook/blob/master/Documentation/Helm-Charts/ceph-cluster-chart.md operatorNamespace: rook-ceph toolbox: enabled: true cephObjectStores: [] # by default a cephObjectStore will be created. Setting this to null disables it #cephBlockPools: # by default a cephBlockPool will also be created with default values #cephFileSystems: # by default a cephFileSystem will also be created with default values cephClusterSpec: mon: count: 3 volumeClaimTemplate: spec: storageClassName: managed-premium resources: requests: storage: 10Gi resources: limits: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;1Gi\u0026#34; requests: cpu: \u0026#34;100m\u0026#34; memory: \u0026#34;500Mi\u0026#34; dashboard: enabled: true storage: storageClassDeviceSets: - name: set1 # The number of OSDs to create from this device set count: 3 # IMPORTANT: If volumes specified by the storageClassName are not portable across nodes # this needs to be set to false. For example, if using the local storage provisioner # this should be false. portable: false # Since the OSDs could end up on any node, an effort needs to be made to spread the OSDs # across nodes as much as possible. Unfortunately the pod anti-affinity breaks down # as soon as you have more than one OSD per node. The topology spread constraints will # give us an even spread on K8s 1.18 or newer. placement: topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchExpressions: - key: app operator: In values: - rook-ceph-osd tolerations: - key: storage-node operator: Exists preparePlacement: tolerations: - key: storage-node operator: Exists nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: agentpool operator: In values: - storagenp topologySpreadConstraints: - maxSkew: 1 # IMPORTANT: If you don\u0026#39;t have zone labels, change this to another key such as kubernetes.io/hostname topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchExpressions: - key: app operator: In values: - rook-ceph-osd-prepare resources: limits: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;4Gi\u0026#34; requests: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;2Gi\u0026#34; volumeClaimTemplates: - metadata: name: data spec: resources: requests: storage: 100Gi storageClassName: managed-premium volumeMode: Block accessModes: - ReadWriteOnce During the cluster provisioning, there will be a number of preparing Pods. We want those Pods to run on nodes with label agentpool=storagenp. In real life, we need to orchestrate where to run each workload, by restricting the nodes to schedule certain types of workload.\nThen we can install the cluster using Helm:\nhelm install rook-ceph-cluster rook-ceph-cluster --namespace rook-ceph --create-namespace --version v1.9.6 --repo https://charts.rook.io/release/ --values rook-ceph-cluster-values.yaml After running the Helm install, it may take as long as 15 minutes for all resources to settle. Watch the Pod status in rook-ceph namespace. At the end, make sure that the cluster is created successfully:\nkubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get CephCluster NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL rook-ceph /var/lib/rook 3 15m Ready Cluster created successfully HEALTH_OK kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephBlockPools NAME PHASE ceph-blockpool Ready kubeadmin@pro-sturgeon-bastion-host:~$ kubectl -n rook-ceph get cephFileSystems NAME ACTIVEMDS AGE PHASE ceph-filesystem 1 20m Ready In my case it took 15 minutes before the cluster comes up as created successfully. You should notice that two storage classes were also created as a part of the install. It however did not create a storage class or CRD for object storage, because we explicitly disabled it in the Helm value file by setting cephObjectStores value to null.\nDashboard We enabled dashboard. To configure the dashboard view properly, we would need an ingress. For a quick view here, we can play port forwarding tricks. First we fetch the admin password for use in the next step. Then expose the dashboard to the bastion host:\n$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath=\u0026#39;{.data.password}\u0026#39; | base64 -d $ kubectl -n rook-ceph port-forward svc/rook-ceph-mgr-dashboard 8443:8443 Since I don\u0026#8217;t have UI on the bastion host, I use the port forwarding trick again from my own MacBook. Start a new terminal and SSH to the bastion host with port-forwarding switch:\n$ ssh -L 8443:localhost:8443 kubeadmin@20.116.132.8 The command above suppose the public IP of the bastion host is 20.116.132.8. Then from my MacBook I can browse to localhost:8443 (with Safari browser which gives me the option to bypass certificate error). At the web portal, provide username (admin) and password (as retrieved above):\nCeph console for Kubernetes Apart from the dashboard, we can also use ceph admin tool from a toolbox pod, following this instruction. For monitoring, Ceph by Rook can expose metrics for Prometheus to scrape.\nPerformance With default ceph configuration on AKS, I ran quick performance test using kube-str . The result is as follows:\nread_iopswrite_iopsread_bwwrite_bwceph-blockIOPS=464.507294 BW(KiB/s)=1874IOPS=243.296143 BW(KiB/s)=989IOPS=509.928162 BW(KiB/s)=65797IOPS=248.530762 BW(KiB/s)=32338ceph-filesystemIOPS=438.701324 BW(KiB/s)=1770IOPS=226.270660 BW(KiB/s)=920IOPS=405.936340 BW(KiB/s)=52456IOPS=208.869293 BW(KiB/s)=27229 The metrics reflects performance under default configuration. It should not be considered as the best performance that Ceph can deliver on Azure Kubernetes.\nSummary I discussed three storage options for Azure Kubernetes but the idea applies to other Kubernetes platform hosted on a CSP. The native storage has significant limitation. NFS has latency. Block storage does not address high availability at the storage layer. Portworx and LINSTOR fill that gap as a commercial solution. Ceph is based on Object storage.\nPrevious PostKubernetes Storage on Azure 2 of 3 – Portworx Next PostMinIO for S3-compatible Object Storage ","date":"2022-08-26T19:43:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-storage-3.webp","permalink":"/2022/08/storage-solution-on-aks-2-of-3-ceph-by-rook/","title":"Kubernetes Storage on Azure 3 of 3 – Ceph by Rook"},{"content":"In the previous post, we have discussed built-in storage classes on Azure Kubernetes. Further to that, we will examine some third-party software defined storage (SDS) options that are compatible with Azure Kubernetes Service in this post. Then we take Portworx on Azure as an example. Although, these options are specific to Azure, most of the players also have solutions for other managed Kubernetes platforms. Also, the methodology to study storage options remain the same regardless of cloud service provider.\nIn fact, I touched on software defined storage (SDS) in the context of general non-containerized workload in a separate post. In short, storage providers decouple the SDS appliance from the full storage solution in order to lower the cost and increase flexibility. To the storage consumer (e.g. a process running on Linux OS), SDS can present a block disk or file system. There are also SDS solution that can host your own object storage and we will discuss that later. In fact, SDS has gained significant popularity in recent years. For example, the report \u0026#8220;Validating Software-Defined Storage Operating Models for the Enterprise\u0026#8221; by archiectingit divided the evolution into four phases and cited that Gartner predicts the SDS revolution to reach 50% of the storage market by 2024, from 15% in 2020.\nThis post discusses SDS in the context of container storage. Then we will install Portworx on Azure Kubernetes.\nSDS for Kubernetes Many SDS appliances also developed the capability to present storage volumes to containerized workload. I put together a list of SDS products that works on Azure with their supported access modes: SolutionLicenceDevelopment and SupportAccess ModePortworxClosed source. Free Essential tier. Enterprise features on License. Commercially supported by PureStorageRWO, RWXCeph by RookOpen source. Rook is a graduated CNCF project.Developed and commercially supported by Red Hat, Canonical and SoftIronRWO, ROX, RWXOpenEBSOpen source. Sandbox CNCF project.Developed and commercially supported by MayaData et al.RWOLonghornOpen source. Incubating CNCF project.Originally developed by Rancher, and commercially supported by SUSERWOStorageOSClosed source. License required.Commercial support by Ondat.RWOLINBITOpen-source with enterprise plansEnterprise support RWO is the most commonly supported mode. The report \u0026#8220;Performance Benchmarking Cloud Native Storage Solutions for Kubernetes\u0026#8221; makes a comparison of performance among some of the options in early 2021. Another potentially opinionated comparison list is by LINBIT.\nPortworx is a leading player with commercial SDS solution and I will test its free Essential tier in the rest of this post. Ceph is one of the most mature leading open-source offering and I will test it in the next post.\nFor Portworx, we can use the terraform template cloudkube for Azure. The template assigns the kubelet\u0026#8217;s managed identity as contributor of the node resource group. The template also creates a bastion host with direct SSH access to the nodes.\nPortworx Operator on Azure We use Portworx Operator to configure storage cluster. Portworx has an instruction for AKS but it is not tailored to specific identity model. For simplicity, use my cloudkube Terraform template to create the AKS cluster, and skip the \u0026#8220;Prepare Your AKS Platform\u0026#8221; page. Instead, follow the \u0026#8220;Deploy Portworx using Azure managed identity on new AKS cluster\u0026#8221; page starting at step 7. At that step, we need to create a secret with the client ID of the managed identity for node agent. The terraform template outputs the BYO identity\u0026#8217;s client ID. After cluster creation, we simply SSH to the bastion host and create the secret using the output.\nTo install Porworx using operator, we can follow a wizard in PX-central. If this is the first time, we need to create an account and log in to the portal. If this is not the first time and you have previously created a cluster, you need to detach that cluster by going to Profile from bottom left corner on the portal page. Follow the guide in \u0026#8220;Install Portworx on AKS using the Operator\u0026#8220;. In the wizard, click on \u0026#8220;Portworx Essentials\u0026#8221; for free tier, or \u0026#8220;Portworx Enterprise\u0026#8221; for the 30-day trial. Then select operator with latest version. In the rest of the wizard steps, select options applicable to Azure environment. The last step will present two kubectl commands to install operator and install the CR. Run the command to install operator and verify result:\nkubectl apply -f \u0026#39;https://install.portworx.com/2.9?comp=pxoperator\u0026#39; kubectl -n kube-system get deployment portworx-operator Portworx Custom Resource To install the CR, we need to customize the given manifest in order to use our managed identity. We can download the YAML manifest (portworx_essentials.yml) and modify it in text editor. As the page \u0026#8220;Deploy Portworx using Azure managed identity on new AKS cluster\u0026#8221; suggest at step 9: in the\u0026nbsp;env\u0026nbsp;section, remove the AZURE_CLIENT_SECRET and AZURE_TENANT_ID sections but keep the AZURE_CLIENT_ID section. My CRD declaration looks like this:\n# SOURCE: https://install.portworx.com/?operator=true\u0026amp;mc=false\u0026amp;kbver=\u0026amp;oem=esse\u0026amp;user=myuserid\u0026amp;b=true\u0026amp;kd=type%3DPremium_LRS%2Csize%3D150\u0026amp;s=%22type%3DPremium_LRS%2Csize%3D150%22\u0026amp;c=my-very-long-px-cluster-id\u0026amp;aks=true\u0026amp;stork=true\u0026amp;csi=true\u0026amp;mon=true\u0026amp;tel=false\u0026amp;st=k8s\u0026amp;promop=true kind: StorageCluster apiVersion: core.libopenstorage.org/v1 metadata: name: my-very-long-px-cluster-id namespace: kube-system annotations: portworx.io/install-source: \u0026#34;https://install.portworx.com/?operator=true\u0026amp;mc=false\u0026amp;kbver=\u0026amp;oem=esse\u0026amp;user=myuserid\u0026amp;b=true\u0026amp;kd=type%3DPremium_LRS%2Csize%3D150\u0026amp;s=%22type%3DPremium_LRS%2Csize%3D150%22\u0026amp;c=my-very-long-px-cluster-id\u0026amp;aks=true\u0026amp;stork=true\u0026amp;csi=true\u0026amp;mon=true\u0026amp;tel=false\u0026amp;st=k8s\u0026amp;promop=true\u0026#34; portworx.io/is-aks: \u0026#34;true\u0026#34; portworx.io/misc-args: \u0026#34;--oem esse\u0026#34; spec: image: portworx/oci-monitor:2.10.2 imagePullPolicy: Always kvdb: internal: true cloudStorage: deviceSpecs: - type=Premium_LRS,size=150 kvdbDeviceSpec: type=Premium_LRS,size=150 secretsProvider: k8s stork: enabled: true args: webhook-controller: \u0026#34;true\u0026#34; autopilot: enabled: true monitoring: prometheus: enabled: true exportMetrics: true featureGates: CSI: \u0026#34;true\u0026#34; env: - name: AZURE_CLIENT_ID valueFrom: secretKeyRef: name: px-azure key: AZURE_CLIENT_ID --- apiVersion: v1 kind: Secret metadata: name: px-essential namespace: kube-system data: px-essen-user-id: aaaabbbbccccddddmyverylongpxessenuserid px-osb-endpoint: ssssssyyyyyyyzzzzzzmyverylongpxosbendpoint In this custom resource manifest, I specify to get the cluster to create disk from Azure, even though it has the capability of using an existing disk. Then apply the CRD manifest:\nkubectl apply -f portworx_essentials.yml This will take up to 10 minutes to create several related resources. There are several check points to ensure the cluster is created successfully. Here are some useful commands:\nkubectl -n kube-system get po # all Pods related to portworx should be ready and running, especially the portworx-api ones kubectl -n kube-system get storagecluster # the status should report online PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath=\u0026#39;{.items[0].metadata.name}\u0026#39;) # get the name of one of the portworx Pod for storage cluster kubectl -n kube-system exec -c portworx -it $PX_POD --tty --stdin -- /opt/pwx/bin/pxctl status # should report \u0026#34;Status: PX is operational\u0026#34; with valid license loaded kubectl get sc # portworx related storage classes are available The pre-built storage classes are not CSI based. However, CSI should be automatically enabled in recent operator versions. We should create our own CSI storage classes and PVCs using our own storage classes. Here are two examples:\nkind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: px-csi-database provisioner: pxd.portworx.com parameters: repl: \u0026#34;2\u0026#34; priority_io: \u0026#34;high\u0026#34; io_profile: \u0026#34;db\u0026#34; --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: px-csi-artifact provisioner: pxd.portworx.com parameters: repl: \u0026#34;2\u0026#34; priority_io: \u0026#34;medium\u0026#34; io_profile: \u0026#34;sequential\u0026#34; That is a bare minimum Portworx installation. Check out their documentation for the full features. With this minimum install we can go to the section \u0026#8220;Validate Persistent Storage\u0026#8221; from the previous post to validate the persistent volume.\nFor troubleshooting purpose, pxctl is the utility and it is available on Portworx Pods.\nPerformance We care not only the functionality, but also the performance. So I ran a quick performance test using kube-str, using all default configuration. The result is as follows:\nread_iopswrite_iopsread_bwwrite_bwpx-csi-databaseIOPS=969.614136 BW(KiB/s)=3894IOPS=729.698059 BW(KiB/s)=2935IOPS=1172.772827 BW(KiB/s)=150639IOPS=691.626526 BW(KiB/s)=89053px-csi-artifactIOPS=780.681946 BW(KiB/s)=3139IOPS=682.522766 BW(KiB/s)=2746IOPS=773.548584 BW(KiB/s)=99549IOPS=659.015320 BW(KiB/s)=84890 Note that those numbers reflect performance under default configuration, and they should not be considered as the best performance that Portworx can deliver on Azure Kubernetes. Before moving to production, it is important to establish your own test parameters that best represents the container workload, and then iterate through different parameters for the storage class based on the requirement and performance output.\nPrevious PostKubernetes Storage on Azure 1 of 3 – built-in storage and NFS Next PostKubernetes Storage on Azure 3 of 3 – Ceph by Rook ","date":"2022-08-12T15:23:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-azstorage.webp","permalink":"/2022/08/kubernetes-storage-on-azure-2-of-3-portworx/","title":"Kubernetes Storage on Azure 2 of 3 – Portworx"},{"content":"In the previous post, we understand that to host stateful workload, we need to manage persistent storage to the Kubernetes platform. In this post, I will explore the different storage options. These options are specific to Azure Kubernetes service. However, the principals apply to any Kubernetes platform regardless of cloud vendor.\nIn another old post, I discussed in-tree and CSI storage classes, and from a developer\u0026#8217;s perspective, how to mount volumes statically and dynamically once the storage class is available. Here in this post we are concerned with how to make storage classes available, from a platform specialist perspective.\nStorage in Azure Kubernetes As soon as the Azure Kubernetes cluster is launched, a number of built-in storage classes are available. Unlike third-party storage classes, they do not require kubelet identity to be contributor for node resource group. As discussed we shall use CSI based storage classes. managed-csi managed-csi-premium azurefile-csi azurefile-csi-premium The main difference between them is the backing technology. However, form Kubernetes workload\u0026#8217;s perspective, the Pods as storage consumers are concerned with the access mode instead of backing technology. Here are supported access modes:\nReadWriteOnce: read-write by a single node ReadOnlyMany: read only by many nodes ReadWriteMany: read-write by many nodes ReadWriteOncePod: new in Kubernetes 1.22 to restrict volume access to a single Pod The storage classes managed-csi and managed-csi-premium support ReadWriteOnce. The storage classes azurefile-csi and azurefile-csi-premium support ReadWriteMany.\nApart from these built-in options, Azure also suggests a few more options based on other types of Azure resources. For example, AKS can integrate with HPC cache and it for HPC. We can also self-manage a virtual machine configured as NFS server, and use the NFS subdir external provisioner to configure storage class. Despite of the overhead with managing a VM, you have more configurability. My previous client reports that they gain better performance than the built-in options. Another alternative is Azure NetApp Files. However, being a full enterprise grade solution (similar to FSx ONTAP), Azure NetApp Files costs an arm and a leg. In this comparison, it cost 60 to 100 times as the cost by built-in options.\nAnother option is Azure Ultra Disk, which needs to be enabled at cluster level. You can provision performance target (DiskIOPSReadWrite and DiskMBpsReadWrite) in the storage class. Ultra Disk is a good middle ground between the pricey NetApp files and the less performant built-in options.\nTerraform Template for AKS To explore the storage options, I use my own terraform template to create an AKS cluster. The template is in the azure directory of the cloudkube repo. The template consists a few configurations with Azure Kubernetes to facilitate storage configuration. First, it configures an SSH key pair to use between the bastion host and the node. Users can SSH to Kubernetes nodes from bastion host as soon as terraform apply is completed. Second, the third party storage options installed after the cluster creation need their Pod to instruct Azure to create Azure disks. This requires that a Kubernetes node agent have the permission to provision resources in the node resource group. This is important to understand because there are a couple of managed identities at play (refer to this post) when building an AKS cluster. In Azure Kubernetes, it is the managed identity of kubelet, that needs to have contributor permission over the resource group for the nodes (not the one for the AKS cluster itself). A managed identity is expressed by a client ID, an object ID (aka principal ID), and the identity ID. We can find them out with an AZ CLI command as below:\nWe can also tell that the kubelet managed identity represents node agent, by connecting to a node and looking at the argument (kubernetes.azure.com/kubelet-identity-client-id) of kubelet process:\nIn the template, I also chose to designate the same BYO identity for both the cluster and for kubelet (node agent), in order to minimize my requirement on permission. If I had left it with a system assigned identity for node agent, I would have to assign that identity as a contributor for the node resource group, either as a user, or via Terraform\u0026#8217;s identity. Either way, it is beyond what a Contributor is allowed to do.\nBenchmarking with kubestr I used fio utility for storage benchmarking from virtual machines. However, fio utility is not for container. For fio testing on Kubernetes, I\u0026#8217;d have to use a Docker image, and test with target volume attached. Fortunately, the Kasten team shared their initiative in the open source project Kubestr. The kubestr release is available as an executable on common platforms. It connects to the cluster the same way as kubectl and here is a demo. To begin with, download the utility to bastion host, and run it without any argument, which prints the storage classes and volume snapshot classes:\ncurl -L -o kubestr.tar.gz https://github.com/kastenhq/kubestr/releases/download/v0.4.31/kubestr_0.4.31_Linux_amd64.tar.gz tar -xvf kubestr.tar.gz \u0026amp;\u0026amp; rm kubestr.tar.gz \u0026amp;\u0026amp; chmod +x kubestr ./kubestr # if kubectl is configured, this command will print out the details of storage classes and volume snapshot classes In addition to outputting details, it is also very simple to perform storage benchmarking with kubestr. All we need to do is giving it the storage class name and it will run four tests by default with common global options (ioengine=libaio verify=0 direct=1 gtod_reduce=1). The four tests are:\nJobNameblock_sizefilesizeiodepthrwread_iops4k2G64randreadwrite_iops4k2G64randwriteread_bw128k2G64randreadwrite_bw128k2G64randwrite During each test, it measures and reports IOPS and bandwidth (throughput). If your I/O profile falls out of the four jobs, you can even customize your test by supplying a fio config file. For example, you need a longer test duration, or you need a larger total size for the test. Before the test, kubestr automatically mount their test volumes using the storage class being tested.\n./kubestr fio -s my-storage-class # benchmarking a storage class For read_iops and write_iops, we mainly look at the IOPS. For read_bw and write_bw, we mainly look at the bandwidth. The iops and bw based on samples are reported as first line of result, followed by min, max and average.\nMetrics With kubestr I ran a performance test amongst the native storage classes with Azure Kubernetes Service, with results as below:\nread_iopswrite_iopsread_bwwrite_bwmanaged-csiIOPS=314.729797 BW(KiB/s)=1275IOPS=297.071136 BW(KiB/s)=1204IOPS=315.311188 BW(KiB/s)=40887IOPS=261.048645 BW(KiB/s)=33941managed-csi-premiumIOPS=493.395844 BW(KiB/s)=1990IOPS=426.731812 BW(KiB/s)=1723IOPS=455.950348 BW(KiB/s)=58894IOPS=422.888855 BW(KiB/s)=54662azurefile-csiIOPS=259.333282 BW(KiB/s)=1053IOPS=283.985779 BW(KiB/s)=1152IOPS=240.447403 BW(KiB/s)=31298IOPS=230.689804 BW(KiB/s)=30048azurefile-csi-premiumIOPS=394.044739 BW(KiB/s)=1586IOPS=371.181793 BW(KiB/s)=1494IOPS=380.360535 BW(KiB/s)=49018IOPS=491.313446 BW(KiB/s)=63310 These numbers are based on default test parameters (e.g. 100Gi PVC size). The numbers indicate that block storage generally performs better in default setting. So we should use managed disk instead of azure file unless multiple pods needs to access the same volume.\nValidate Persistent Storage We can use mysql as a quick and dirty test of storage classes. We can deploy the following manifest:\napiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc spec: storageClassName: managed-csi-premium # the storage class being tested accessModes: - ReadWriteOnce resources: requests: storage: 5Gi --- apiVersion: v1 kind: Secret metadata: name: mysql data: password: eHl6 # base64 code of xyz --- apiVersion: apps/v1 kind: Deployment metadata: name: mysql labels: app: mysql spec: replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql key: password ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pvc --- apiVersion: v1 kind: Service metadata: name: mysql-service labels: service: mysql spec: selector: app: mysql ports: - name: tcp-mysql protocol: TCP port: 3306 targetPort: 3306 Once the Pod has been created, then we use a throw-away Pod to connect to mysql service and build some data:\n$ kubectl run mysql-cli --rm -i --tty --image imega/mysql-client -- /bin/sh If you don\u0026#39;t see a command prompt, try pressing enter. / # mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz Welcome to the MariaDB monitor. Commands end with ; or \\g. Your MySQL connection id is 9 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type \u0026#39;help;\u0026#39; or \u0026#39;\\h\u0026#39; for help. Type \u0026#39;\\c\u0026#39; to clear the current input statement. MySQL [(none)]\u0026gt; From the mysql shell, we create a test database with dummy data populated:\nCREATE DATABASE `testdb`; USE testdb; CREATE TABLE IF NOT EXISTS tasks ( task_id INT AUTO_INCREMENT PRIMARY KEY, title VARCHAR(255) NOT NULL, description TEXT, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) ENGINE=INNODB; INSERT INTO tasks (title, description) VALUES (\u0026#39;Job A\u0026#39;,\u0026#39;Morning Standup\u0026#39;); INSERT INTO tasks (title, description) VALUES (\u0026#39;Job B\u0026#39;,\u0026#39;Latte with two shots of espresso\u0026#39;); INSERT INTO tasks (title, description) VALUES (\u0026#39;Job C\u0026#39;,\u0026#39;Coding coding and coding\u0026#39;); INSERT INTO tasks (title, description) VALUES (\u0026#39;Job D\u0026#39;,\u0026#39;git commit\u0026#39;); Each SQL command should return with number of rows affected and then we can exit the MySQL shell and Pod shell. Once we exit out of the Pod shell, the Pod is deleted. We can re-connect to validate the data are still present:\n$ kubectl run mysql-tester --rm -i --tty --image imega/mysql-client -- mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz --database testdb --execute=\u0026#39;SELECT * FROM tasks;\u0026#39; If you don\u0026#39;t see a command prompt, try pressing enter. Got error: Access denied for user \u0026#39;root\u0026#39;@\u0026#39;147.206.3.15\u0026#39; (using password: NO) +---------+-------+----------------------------------+---------------------+ | task_id | title | description | created_at | +---------+-------+----------------------------------+---------------------+ | 1 | Job A | Morning Standup | 2022-06-22 20:29:12 | | 2 | Job B | Latte with two shots of espresso | 2022-06-22 20:29:12 | | 3 | Job C | Coding coding and coding | 2022-06-22 20:29:12 | | 4 | Job D | git commit | 2022-06-22 20:29:13 | +---------+-------+----------------------------------+---------------------+ Session ended, resume using \u0026#39;kubectl attach mysql-tester -c mysql-tester -i -t\u0026#39; command when the pod is running pod \u0026#34;mysql-tester\u0026#34; deleted The output validates the persistent storage of the data. When testing a different storage class, simply start over with a different storage class specified in the PVC part of the manifest.\nSummary This post focuses on storage options for persistent volumes on Azure Kubernetes service. I use my own Terraform template with some custom configuration. I also covered kubestr as benchmarking tool and ran it against the built-in storage classes. In some situations, the built-in options do not suit your needs. For example, you might want your application to use persistent volume in a consistent way across multiple cloud vendors. We will have to resort to third-party software defined storage layer. In the next post, I will explore a couple of SDS-based options, namely Portworx and Ceph. The custom configurations in the Terraform template will be helpful when we configure Portworx.\nPrevious PostIntro to Ceph storage Next PostKubernetes Storage on Azure 2 of 3 – Portworx ","date":"2022-07-31T15:22:00-04:00","image":"/wp-content/uploads/2025/04/feature-aks-storage-1.webp","permalink":"/2022/07/kubernetes-storage-on-azure-1-of-3-built-in-storage-and-nfs/","title":"Kubernetes Storage on Azure 1 of 3 – built-in storage and NFS"},{"content":"Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. In this post, I will introduce Ceph and explain how it stands out from traditional enterprise storage technology. Software defined storage In the realm of enterprise storage, I discussed PowerScale (Isilon) from Dell EMC, and touched on ONTAP by NetApp as an alternative. These solutions usually include both enterprise grade hardware, and the software layer that manages those expensive hardware. As the competition with cloud storage arises, those vendors start to decouple the software layer from the hardware to sell them separately. As a result, clients have the options to use commodity hardware. On the other hand, the software layer is built to be more accommodative to different hardware options. Eventually, the software layer evolves into Software Defined Storage (SDS) with the purpose of supporting cheaper storage hardware.\nThis table shows the full solution offering and SDS offering from NetApp and Dell EMC:\nFull solution offeringSDS offeringNetAppONTAPONTAP SelectEMCPowerScalePowerFlex It is not easy to make a proprietary SDS appliance support commodity hardware. For example, PowerFlex currently supports (and bundles with) DELL\u0026#8217;s commodity hardware only. It is most likely an involuntary move. Then, why would these commercial providers even be motivated to support a broader range of hardware by moving to SDS? It is because they face fierce competition from open-source SDS technologies, which were born to support commodity hardware. In this family of technologies, Ceph is a rising star. This family also includes other technologies such as Gluster and HDFS.\nNote that the performance of a storage based on SDS still has to do with the underlying hardware. Therefore, comparing Ceph storage with PowerScale is apple to orange, without identical storage hardware. Now that we decoupled SDS and hardware, let\u0026#8217;s take a look at two important aspects of SDS: the distributed technology to manage hardware, and the interface it provides to storage clients.\nDistributed storage The reason to use an SDS layer to manage hardware in a distributed architecture is for better scalability and high availability. The soul of this SDS layer is the ability to manage distributed system. However, a distributed storage introduces problems of its own, such as coordinating consistency. Different storage technologies have their own way to tackle these problems. For example, with PowerScale, OneFS has its own Group Management Protocol. Ceph uses CRUSH for data distribution. GlusterFS uses DHT(Distributed Hash Table) Translator. Storage architects usually do not need to know these technologies in detail. It is not the intention of this post to cover the details of any distributed technology in any of the storage options above. However, storage architects needs to know supported API very well.\nAccess API The supported access API of a storage system determines its compatibility with client systems. One good example is NFS for file storage, which defines the protocol for file share without defining the underlying implementation. Most GNU/Linux distributions come with nfsd (NFS server) which exports directories on XFS or ext4 FS as a file share with NFS protocol. In order to transfer data over network, NFS uses RPC, a request-response protocol. With object storage, S3 is a widespread protocol. Below is a list of storage implementations and their supported access API:\nCeph supports librados, S3, Swift and FUSE GlusterFS supports SMB, NFS, FUSE, PowerScale supports NFS, SMB/CIFS, HDFS, Object, POSIX CephFS is distributed file system built on top of Ceph RADOS. It is also a client-server architecture. A Ceph Client, via librados, interacts directly with OSDs to store and retrieve data. In order to interact with OSDs, the client app must invoke librados and connect to a Ceph Monitor. For compatibility, CephFS namespaces can be export over NFS protocol using NFS-Ganesha NFS server.\nCeph Architecture Ceph is a high-performance, distributed storage platform. It provides object storage, block storage and distributed file system, all backed by a single, reliable storage cluster running on commodity server hardware. A Ceph Storage Cluster consists of Ceph Nodes on a network. A Ceph Storage cluster requires at least one Ceph monitor (ceph-mon), Ceph Manager (ceph-mgr) and Ceph OSDs (ceph-osd). For file system clients, it also requires Ceph Metadata Server (MDS, ceph-mds) to allow user to execute basic commands on POSIX file system (e.g. ls, find)\nUnder the hood, Ceph stores data as objects within logical storage pools. Using the\u0026nbsp;CRUSH\u0026nbsp;algorithm, Ceph calculates which placement group (PG) should contain the object, and which OSD should store the placement group. The CRUSH algorithm enables the Ceph Storage Cluster to scale, rebalance, and recover dynamically.\nCeph is based on RADOS (reliable autonomic distributed object store), a self-healing system that distributes and replicates data across nodes. It then layers CephFS (a distributed file system), block storage service (RADOS Block Device or RBD), and s3-compatible object storage (RADOS Gateway or RGW) on top of RADOS. For a better description, refer to this page. The chart above shows how Ceph interacts with different kinds of client. For CephFS, the client can interact with the file system via metadata daemon, as illustrated below. This diagram looks similar to the diagram for NFS.\nIn a RADOS cluster, each server runs some daemons (i.e. OSD, MON or MDS). When an I/O request occurs, it needs to be mapped to the specific OSD that keeps the storage units. Here is an illustration of the mapping:\nAs typically observed in distributed system, there is quite some communication overhead to serve a file.\nCeph Cluster Installation Installing a VM-based Ceph cluster is no trivial effort and there are several methods. The recommended method is Cephadm. Here is a good instruction, where you will notice a lot of steps on each nodes, such as configuring NTP, installing docker, configuring hostname, Linux user and SSH, etc. You may also check this video for how involving it is. Red Hat adopts Ceph project as a product and has an installation guide on its documentation.\nPreviously, there was a legacy tool ceph-ansible to help administrators with server configuration. It is similar to the way kubespray helps administrators configure Kubernetes cluster. However, the document suggests that ceph-ansible is not integrated with new orchestrator APIs and therefore is not a viable option anymore. Also I did not find a way to install a single-node ceph cluster just for a quick demo. It involves tweaking the CRUSH map configuration. If we deploy Ceph on Kubernetes for Kubernetes workload, we use Rook, an orchestrator running on Kubernetes, to integrate storage to a cluster.\nCloud Native Storage Moving to cloud native storage, instead of presenting storage to operating system, we need to configure storage classes for Pods to use persistent volumes dynamically, using storage provisioners. Ceph also shows good presence in cloud native storage ecosystem. In a self-managed Kubernetes cluster, Ceph gives us the capability to configure storage classes to access connected storage. In public cloud, Ceph allows us to configure storage classes connecting to disks attached to the Nodes, an alternative to the cloud vendor provided native storage classes with high availability across availability zones. This layer enables the organization to normalize how their application connects to persistent volumes, a capability particularly helpful in the multi-cloud strategy of the cluster.\nRook is a CNCF project to orchestrate storage system on Kubernetes. It automates storage administrative tasks such as deployment, bootstrapping, configuring, provisioning and monitoring, using declarative templates. It supports Ceph and a number of other storage backends such as Cassandra, NFS, MinIO. Previous PostKick the tires on ArgoCD Next PostKubernetes Storage on Azure 1 of 3 – built-in storage and NFS ","date":"2022-07-21T18:55:00-04:00","image":"/wp-content/uploads/2025/04/feature-ceph.webp","permalink":"/2022/07/intro-to-ceph-storage/","title":"Intro to Ceph storage"},{"content":"Background In January, I wrote about FluxCD, and adopted it in the Korthweb project. I like the simple design of FluxCD, and I am comfortable with commands without using a web UI. Half a year later, I am re-visiting this choice, with ArgoCD in mind.\nAfter reading numerous recent posts that compare the two (such as this one from the new stack), I started to give more thoughts on ArgoCD, for a couple reasons. First, ArgoCD has more contributing companies and public references. Red Hat adopted Argo CD as the underlying technology for OpenShift GitOps (and Tekton for pipeline). Second, ArgoCD has a mature UI. In corporate collaboration, especially with those not well-versed with command line, a UI is extremely helpful. Bundled with the UI, Argo CD also uses its own RBAC independent of Kubernetes RBAC, making it a great choice for continuous deployment for enterprise applications.\nUser Interface You can choose to install just the core components, without UI, SSO and multi-cluster features. To me, it does not make sense because those features are exactly the reason to choose Argo CD. Argo CD also has an eponymous CLI tool, similar to flux for Flux CD. Since Argo CD uses its own RBAC independent of Kubernetes RBAC, Argo CD will have its own credential. The default username is admin and password needs to be retrieved from Secrets. In the tutorial we use this user\u0026#8217;s credential to interact with the target cluster. This is different from Flux where the client simply uses kubectl configuration. In the \u0026#8220;Getting Started\u0026#8221; guide, we create a namespace \u0026#8220;argocd\u0026#8221; and install it with manifests. Alternatively, we can use CLI tool to install it. With Argo CD, we can consider hosting it on a different port than the workload\u0026#8217;s port, such as 8443. This port serves as \u0026#8220;GitOps management port\u0026#8221; separated from the application port (e.g. 443) for business workload. Just like any application on Kubernetes, we usually need to configure an Ingress on the management port.\nIn the argocd namespace, we can see a few services. The application set controller acts as the main controller that reconciles between actual state and desired state. There is also a redis server, for storing data in Argo CD. Another service to note is the dex server, indicating that the SSO capability is provided by the Dex project.\nFeatures In FluxCD, they use a CRD Kustomization with kustomize.toolkit.fluxcd.io/v1beta2 as version. This name is confusing. Luckily, in ArgoCD, the CRDs are Application (with argoproj.io/v1alpha1 version) and ApplicationSet (with argoproj.io/v1alpha1 version). The controllers monitors CRs created with these CRDs. ArgoCD defines Application as a group of Kubernetes resources as defined by a manifest. Argo CD Application resource deploys resources from a single Git repository to a single destination cluster/namespace. On the other hand, ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once. ApplicationSet controllers monitors ApplicationSet resources. We can define a template within the declaration of an ApplicationSet and it allows for parameter substitution.\nAnother key capability for a GitOps utility is the support of multiple templating tools. FluxCD supports Helm and Kustomize. So do Argo CD. In addition, ArgoCD also supports jsonnet (ksonnet is not supported anymore). Jsonnet is a templating tool with many useful operators. Kubectl can directly consume Jsonnet\u0026#8217;s JSON output as if it were YAML. In FluxCD, there are different CRDs for Helm and Kustomize (HelmRelease and Kustomization). In ArgoCD, the Application CRD covers Helm, Kustomize, and Jsonnet by embedding them as attributes in the declaration. This makes it even simpler than Flux CD. Secret Management It\u0026#8217;s a big no-no to put secret in a code repository. Therefore any GitOps solution must solve the problem with secret logistics. What baffles me is ArgoCD remains un-opinionated on this matter. There appears to be some context for this stance. As a result, ArgoCD only points to a few third-party secret management solutions.\nSecret logistics is a tricky problem with GitOps workflow. One of the techniques is sealed secret. In the repo we store the secret as encrypted by public key. The controller keeps the private key which decrypts the secret at the time of deployment. A few engineers don\u0026#8217;t find it feasible for operation, as explained in this post and this post.\nAn alternative is to use external secret operator. Such operators will be able to sync a secret from external secret store such as Hashicorp Vault, AWS secret manager, Azure Key Vault, etc. This is much neater because we only store reference to secret in the repository. However, when the cluster assesses external secret store, it still requires either a secret, or a permission. Another technique that drawing attentions is using sops for secret and kustomize-sops to integrate with ArgoCD. Mozilla\u0026#8217;s sops (SecretOPerationS) project releases a binary utility called sops to encrypt and decrypt YAML manifest. It can encrypt only the values in YAML manifest and not the attributes (keys). We use sops to encrypt YAML files in order to store them in Git repo. We also use sops to decrypt YAML right before we deploy it with kubectl. The key pair for encryption and decryption can be stored in Azure Key Vault, AWS KMS, GCP KMS, and even age and pgp. Here is a simple tutorial. Example In a few command we can configure a minimal example. I first use KinD or Minikube to create a cluster following the steps in real-quicK-cluster repo. Then we configure an application using YAML manifest.\n$ kubectl create namespace argocd $ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml $ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\u0026#34;{.data.password}\u0026#34; | base64 -d; echo $ kubectl port-forward svc/argocd-server -n argocd 8443:443 From MacOS, I can now browse to https://localhost:8443/ for ArgoCD UI, and login with admin user and the password as printed above. We can then use the UI to create an Application. Alternatively, we can configure an application using CLI utility:\n$ brew install argocd $ argocd login localhost:8443 $ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default $ argocd app sync guestbook On the GUI we should see the application is sync\u0026#8217;ed. On the left panel, click on the gear icon for settings. Under Clusters, we can see the cluster URL, which we use as dest-server value above. We can add more clusters here. Pick the right open-source project My experience with ArgoCD and FluxCD raised an issue to reflect on. For the same problem, there are multiple choices of open-source project. What would be a good methodology to choose the right open-source technology? I would first look at the following factors:\nHistory Community size License model Enterprise support Governing model The history and community size give a good idea of the technical maturity level. The License model has legal implications as to where you can use the technology. License model can change even after the project has been launched and even become not open-source any more. One example is Elasticsearch license change in Jan 2021, from the very open APLv2 to under dual license of both Elastic License (by Elastic) and Server Side Public License (SSPL, introduced by MongoDB). Neither license is certified by OSI as compliant with open source definition. For enterprises, apart from licensing, I\u0026#8217;d also consider if any organization provides commercial support for the open-source product. What is often neglected is the governing model. The governing model determines how product decisions are made that will shape the future of the project. For example, CNCF is a vendor neutral foundation for cloud computing and if an organization contributes a project to CNCF, then the member projects have to align with CNCF\u0026#8217;s governing practices. Google also introduced an OUC (Open Usage Common) governing model.\nAnother good way is to just follow Red Hat\u0026#8217;s choice. Red Hat built its business model around open-source technologies. They have to pick open-source project to provide enterprise level support so presumably they have a rigour selection process that one can piggyback off. Products supported by Red Hat are composed of open source components often vetted by multiple upstream communities, and changes made to these components are pushed to their respective upstream projects, often before they land in a supported product from Red Hat. Here are more insights in this business model.\nHad I gone with the Red Hat rule, I would have picked Argo CD in the first place because that is the base of OpenShift GitOps! Verdict ArgoCD and FluxCD are tools for GitOps. However, just using a GitOps tool does not guarantee that all workload are deployed continuously. First, these GitOps controllers cannot replace Operators, which has domain knowledge about how to orchestrate their workloads. Second, resources not directly managed by these operators (such as those installed by Helm) are not being monitored continuously. For continuous deployment over all workloads, I recommend use operator to install third party applications, and use Argo CD or Flux CD to manage the operators. GitOps makes use of the controller pattern to manage deployment in a continuous matter. Controller works closely with CRDs. ArgoCD uses similar set of CRDs to manage continuous deployment. What sets it apart is the user friendly web UI, the IAM integration and multi-cluster capabilities. These capabilities make ArgoCD well adapted in enterprise IT eco-system.\nPrevious PostChaos Mesh – Cloud Native Chaos Engineering Next PostIntro to Ceph storage ","date":"2022-07-10T00:10:00-04:00","image":"/wp-content/uploads/2025/04/feature-argocd-tire.webp","permalink":"/2022/07/kick-the-tires-on-argocd/","title":"Kick the tires on ArgoCD"},{"content":"In this post, we discuss the resilience test problem and why chaos mesh emerged. Then we go over a lab of chaos mesh with a few experiments. The Problem I used to support a server application installed in customer\u0026#8217;s data centre. The server application receives data from client application, cleanse the data and put them on customer-managed storage. Unfortunately, the customer-managed storage system hang up from time to time. When this happens, the application does not get an I/O failure to begin with so it continues to serve client traffic with many threads pending for I/O completion. All the threads will fail eventually after a long, unresponsive pause. What is really bad, is that the server application first acknowledges the client application of the recipient of their data, and then asynchronously archive them to the customer\u0026#8217;s storage. In this case, the client applications think they securely send out the data upon receiving acknowledgement. However, the server application lost the data afterwards because it does not interact with storage system in a fail-safe fashion. As a result, I often find myself in the business of restoring client data from the cache directory on the servers, and grumbling about how the quality process allowed this issue slipped into production.\nTo be fair, this is not an easy catch in the quality assurance process. It is caused by a hardware issue, which is technically the customer\u0026#8217;s own problem. However, it is the vendor\u0026#8217;s responsibility to design fault-tolerant software. Further, if we put on the site reliability engineering goggle, we see vendor\u0026#8217;s software and customer\u0026#8217;s hardware as a whole, instead of two disparate silos. The problem is however, how we emulate this kind of fault in house? Chaos Engineering A software engineer could have unmounted the storage while the software is active during test. This approach is faulty because it has a couple of issues. First, unmounting the storage does not produce exactly the same symptom. Unmounting the storage in most cases gives a clear failure upfront and the application would have caught it. What we need is a delay in storage long enough to cause a time out. Second, we\u0026#8217;d have to perform this activity with an operating system command with sufficient privileges. The software being tested usually do not have such privileges. We need to systematically inject failures of particular kind.\nChaos are interruptions along with the cascading effects, that happen in production but are impossible or extremely difficult to emulate in lower environments through automated quality process. To improve robustness, we need the capability to emulate various failure scenarios in a systematic manner. Chaos Engineering aims to improve such capability. Chaos Engineering emerged as a discipline to improve software\u0026#8217;s ability to tolerate failures while still delivery adequate quality of service. This is known as the resiliency of the software. Suppose the storage has 99.9999% SLA, or 31 seconds of downtime every year. Chaos Engineer requires the functional parts of our software do what they can to protect the data, during the 31 seconds of turbulence each year, instead of just falling apart. In regulated industries, this is critical. Resilience testing is difficult in quality assurance process, but we can\u0026#8217;t avoid it.\nBy definition, Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. It is particularly useful in distributed system, where network faults are normal. The one-page chaos engineering website has a good summary of such intent. At Netflix, they built Chaos Monkey, a tool to randomly cause failures on certain computing instances in the cloud. It is one of the first Chaos Engineering tools and it is a big step forward. Chaos Monkey can perform one type of experiment (faulty server) and requires writing custom code. Chaos Mesh Moving to Kubernetes platform, where everything operates on network, it is even more important to have a suite of resilience testing tools. Thankfully, we have Chaos Mesh, a cloud-native Chaos Engineering platform that helps platform engineers and software engineers simulate faults in a variety of scenarios. There are other chaos engineering tools (such as Litmus, also a CNCF project) but I will just focus on Chaos Mesh in this post because it has longer development history.\nWith Chaos Mesh, we can run experiments to simulate certain types of faults, including faults on Pod, Network, File I/O, DNS, Time, JVM, Linux Kernel, HTTP. We can also emulate incidents on cloud platforms (Azure, AWS, GCP) and emulate stress. We can even orchestrate multiple experiments into a workflow. Experiments can start ad hoc or on schedule. In addition to Kubernetes platform, the Chaos Mesh team also provides a tool Chaosd to emulate faults on a physical nodes. In the rest of this post, we will install Chaos Mesh and perform four simple experiments using the bookinfo sample application from Istio\u0026#8217;s installer.\nDemo Platform In this lab, we install chaos mesh on Minikube on my Mac environment. I have install the platform in a particular way so the experiments will all be successful. I use this script in my real-quick-cluster project to create minikube cluster. This script configures a 3-node minikube cluster, with containerd as CRI, and enables plugins metallb and metrics server. Metallb provides a load balancer for the cluster. Metrics server allows scaling based on metrics, a test scenario for stressor in Chaos Mesh.\nI will also install Istio in this lab, because I use it for Ingress and service routing. I also use a sample application called bookinfo which comes with Istio code repository. It is highly recommended to have an ingress as it is used in production, although you may use your own choice of ingress and service routing tool such as ingress-nginx. To install istio, we run this simple script:\n#! /bin/bash curl -L https://istio.io/downloadIstio | sh - export PATH=$(realpath istio*/bin):$PATH if istioctl x precheck; then echo ready to install istio and label namespace for istio-injection istioctl install -f istio-operator.yaml -y --verify kubectl label namespace default istio-injection=enabled else echo failed precheck exit 1 fi Below is the content of istio-operator.yaml apiVersion: install.istio.io/v1alpha2 kind: IstioOperator metadata: name: istio-operator spec: profile: default hub: docker.io/istio tag: 1.13.2 namespace: istio-system meshConfig: accessLogFile: /dev/stdout components: pilot: k8s: hpaSpec: maxReplicas: 7 minReplicas: 1 nodeSelector: beta.kubernetes.io/os: linux overlays: - kind: Deployment name: istiod patches: - path: spec.template.metadata.labels.version value: 1.13.2 ingressGateways: - name: istio-ingressgateway enabled: true label: istio: ingressgateway k8s: hpaSpec: maxReplicas: 5 minReplicas: 1 service: ports: - name: http port: 80 targetPort: 20080 protocol: TCP - name: http-dashboard port: 8080 targetPort: 28080 protocol: TCP overlays: - kind: Deployment name: istio-ingressgateway patches: - path: spec.template.metadata.labels.version value: 1.13.2 - path: spec.template.spec.containers[name:istio-proxy].lifecycle value: preStop: exec: command: [\u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;sleep 5\u0026#34;] For more about istio installation, refer to my previous post. Next, we\u0026#8217;ll create the namespaces, install bookinfo and install chaos mesh using Helm:\nkubectl create ns chaos-testing kubectl create ns bookinfo kubectl -n bookinfo apply -f istio/samples/bookinfo/platform/kube/bookinfo.yaml kubectl -n bookinfo apply -f istio/samples/bookinfo/networking/bookinfo-gateway.yaml helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-testing --version 2.2.0 --set dashboard.service.type=ClusterIP --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock As soon as we install bookinfo, we should be able to access it at http://192.168.64.16/productpage, suppose the IP address is 192.168.64.16. We will simply use HTTP because certificate is not the point of this demo. Note that when we install Chaos Mesh we have to specify the runtime (containerd). Otherwise, we might run into issues when running experiments (e.g. \u0026#8220;unable to flush ip sets\u0026#8221; for network faults). Before fault injections, we\u0026#8217;ll start a curl command and observe the output:\nwhile true; do curl -I http://192.168.64.16/productpage --connect-timeout 4 --max-time 5 sleep 5 done We can watch the output of the command above as we inject each of the following faults later on.\nChaos Mesh UI Since we installed chaos mesh as cluster IP service, we need to expose it using Istio Gateway and Virtual Service with the following manifest. If you have your own choice of Ingress, use a corresponding ingress resource.\napiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: chaos-dashboard-gateway namespace: chaos-testing spec: selector: istio: ingressgateway servers: - port: number: 8080 name: http protocol: HTTP hosts: - \u0026#34;*\u0026#34; --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: chaos-dashboard-vs namespace: chaos-testing spec: hosts: - \u0026#34;*\u0026#34; gateways: - chaos-dashboard-gateway http: - route: - destination: port: number: 2333 host: chaos-dashboard Once we configured Virtual Service and Gateways, and waited all Pods in chaos-testing namespace to come up, then we should be able to access the web portal of chaos mesh at 192.168.64.16:8080. Follow the on-screen instruction to fetch the token for access at Cluster level.\nChaos Mesh Web Portal Now, we can create experiments, either using web portal, or using their CRDs. We will use CRDs in this demo. Network Latency Injection In this section we artificially introduce a network latency of 3 seconds for a period of 45 seconds. It takes effect as soon as we create the NetworkChaos CRD as below:\napiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: network-delay spec: action: delay # the specific chaos action to inject mode: one # the mode to run chaos action; supported modes are one/all/fixed/fixed-percent/random-max-percent selector: # pods where to inject chaos actions namespaces: - bookinfo labelSelectors: app: productpage delay: latency: \u0026#39;3s\u0026#39; duration: \u0026#39;45s\u0026#39; Watch for the curl output, which normally returns 200 code every 5 seconds, now fails for about 45 second window.\nAfter the window of latency injection, the curl test returns normal. During the experiment, we can check the status of the fault injection by describing the NetworkChaos object. Alternatively, we can look at Experiments and Events in Chaos Mesh web portal. When completed, we can delete the NetworkChaos object. Alternatively, we can archive the experiment from web portal.\nHTTP Failure Injection The NetworkChaos API injects faults at Pod\u0026#8217;s network layer. HTTPChaos API introduces HTTP error. We can configure an HTTPChaos object following the instruction: apiVersion: chaos-mesh.org/v1alpha1 kind: HTTPChaos metadata: name: test-http-chaos spec: mode: all selector: namespaces: - bookinfo labelSelectors: app: productpage target: Request port: 9080 #method: GET path: \u0026#39;*\u0026#39; abort: true duration: 2m This object intercepts HTTP calls to the Pods as labelled and aborts the request. If we monitor the curl output at the same time, we can see that it starts to receive 503 error as soon as we injected the fault.\nOnce we remove the HTTPChaos object, the return code becomes 200 again.\nPod Failure Injection We can emulate a fault that kills a Pod with the following manifest:\napiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: pod-failure-example spec: action: pod-kill mode: one duration: \u0026#39;30s\u0026#39; selector: namespaces: - bookinfo labelSelectors: app: productpage In the mean time, watch for the Pods in the bookinfo namespace, we\u0026#8217;ll notice the old Pod being terminated.\nBecause the Pod is managed by a replica set, the replica set brings up a new Pod to ensure the desired number of replicas.\nStressor In this experiment, we add an HPA (Horizontal Pod Autoscaler) to the productpage deployment, emulate a memory stress to the Pod and watch it scale up. Before the testing, we need to first configure a resource request for memory for the deployment. To configure memory request, edit the deployment productpage-v1 by adding memory request:\nThen we add an HPA as below:\napiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: productpage-deploy namespace: bookinfo spec: maxReplicas: 4 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: productpage-v1 metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 70 Now if we describe the HPA, we shall see that HPA is active:\nThen we create a StressChaos object as below:\napiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: memory-stress-example spec: mode: one selector: namespaces: - bookinfo labelSelectors: app: productpage stressors: memory: workers: 2 size: 128MiB It will take some time for the HPA to detect the memory stress and scale up. We can tweak this with scaling policy. After scaling activity, the HPA status will report with the new desired number of replica:\nWe can also confirm that by checking the deployment size. If we want the size to reduce after we remove the stressor, we need to configure HPA accordingly with scale down triggers.\nChaos Mesh uses stress-ng utility to emulate memory or cpu stress on a Pod.\nSummary Chaos mesh is a cloud native resilience testing tool. It supports many fault experiments not only on Kubernetes, but also on virtual machines. I wish I came across this tool sooner in my professional life. Had I had chaosd in the past, I could have incorporate it in our quality assurance process on Linux platform. Without Chaos Mesh, we\u0026#8217;d have to use different utilities for different types of faults (e.g. stress or stress-ng for cpu and memory stress). In the domain of platform engineering on Kubernetes, it helps not only software developers, but also platform engineers with correct configuration for workloads on Kubernetes. In this post we explored four types of faults. For more about all fault types and their attributes, go to their documentation for more details.\nPrevious PostEtcd – the key-value store for Kubernetes Next PostKick the tires on ArgoCD ","date":"2022-06-30T21:49:00-04:00","image":"/wp-content/uploads/2025/04/feature-chaos-monkey.webp","permalink":"/2022/06/chaos-mesh-cloud-native-chaos-engineering/","title":"Chaos Mesh – Cloud Native Chaos Engineering"},{"content":"Etcd in Kubernetes In Kubernetes architecture, etcd is the data store. It stores the desired state of Kubernetes object. API server is the only client that connects to etcd (via gRPC protocol). Cluster builder specifies the endpoint of etcd as a parameter to the kube-api-server process. Other Kubernetes components, whether in the control plane or from the nodes, connect to API server. API server translates their request into etcd query, and then translates etcd query result into what its clients ask for. For this reason, communication with etcd accounts for a lot of network traffic in a Kubernetes cluster.\nThe etcd store is a CNCF project for \u0026#8220;a distributed, reliable key-value store for critical data in a distributed system\u0026#8221;, developed by CoreOS team. So it is essentially a distributed key-value store for any distributed application. If an application runs on Kubernetes, it can leverage etcd store, by keeping their configurations in ConfigMap and Secret objects. One key feature is to watch for specific keys or directories for changes, and react to the changes. Voila! This is the underlying mechanism for controller!\nA Kubernetes cluster may have stacked etcd deployment or connect to an external etcd store.\nstacked etcd architecture external etcd architecture In managed Kubernetes services such as EKS in AWS and AKS in Azure, users usually do not directly access etcd store. However, it is still a very important component to understand. Its use case includes:\nConfiguration sharing Service discovery Consistency Watching mechanism Expiry and extension of key The consistency use case is based on Raft protocol for distributed consensus.\nRaft protocol I am not an expert in distributed consensus protocols and nor do I intent to cover it in depth. At a high level, I have heard of three of them so far:\nEtcd uses Raft protocol Zookeeper uses ZAB protocol Cassandra uses paxos protocol Here is a good intro to the three protocols. Instead of getting into the fine details, I would like to discuss why we need such a consensus protocol (or consensus mechanism) in distributed systems, which are also decentralized systems.\nCentralized, Decentralized, Distributed systems The reason a distributed system needs consensus protocol, is that a distributed system lacks a single source of truth as centralized systems do. Different parts of the distributed system may receive different signals but they must come to agreement of a single plan to act. Lamport studies this with an analogy of Byzantine Generals problem, and first proposed Paxos protocol. Paxos has been an important foundation to modern distributed systems. In Paxos, consensus is achieved in two phases, which creates the problem of livelocks. Raft is an alternative to Paxos, and is widely adopted today. Here is a link to an animated illustration for Raft protocol. The Raft protocol is also used in Redis. It has three roles: Leader, Candidate, and follower. ZAB protocol is similar to Raft, where it needs to select a leader.\nEtcd Lab In troubleshooting, if we suspect that the response from API server is inconsistent with etcd store, we want to directly connect to it.\nManaged Kubernetes services do not expose their etcd store. We can use KinD or Minikube. There are two types of jump box to access etcd store: using etcd Pod, or SSH to a Node. To connect to etcd, we also need the X509 key, certificate and CA\u0026#8217;s certificate, in addition to the endpoint, usually an IP with port 2389. When I connect to Pod shell, I find the command shell not easy to use. They might miss basic command such as ls, or do not support auto completion.\nTake KinD for example, we first create a secret, then we can connect to the node with docker CLI command:\nkubectl create ns myns kubectl -n myns create secret generic mysecret --from-literal key1=value1 kubectl -n myns get secret mysecret -o jsonpath=\u0026#39;{.data.key1}\u0026#39; | base64 -d docker exec -it control /bin/bash From the node, apt update \u0026amp;\u0026amp; apt install etcd-client etcdctl version nc -vz localhost 2379 cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd export ETCDCTL_API=3 export ETCDCTL_CERT=/etc/kubernetes/pki/apiserver-etcd-client.crt export ETCDCTL_KEY=/etc/kubernetes/pki/apiserver-etcd-client.key export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt export ETCDCTL_ENDPOINTS=\u0026#39;https://127.0.0.1:2379\u0026#39; etcdctl member list write out=table Now we can see the secret object directly with etcd store:\netcdctl get /registry/secrets/myns/mysecret With get query, when using \u0026#8211;prefix, we can use \u0026#8211;keys-only switch to list keys without values:\netcdctl get --prefix /registry/api --keys-only etcdctl get --prefix /registry/namespace -wjson We can write key-value with put command:\netcdctl put myloc 0 etcdctl get myloc -wjson In Kubernetes, all the key names start with / which makes the key looks like a POSIX path. Every Kubernetes object is stored in etcd with a unique key following a self-explanatory naming pattern. To display the path, we can also use debug log that records the call to API server:\nkubectl get ns myns -v9 Look for curl command such as:\nI0523 22:51:43.517728 32347 round_trippers.go:466] curl -v -XGET -H \u0026#34;Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json\u0026#34; -H \u0026#34;User-Agent: kubectl/v1.23.6 (darwin/amd64) kubernetes/ad33385\u0026#34; \u0026#39;https://127.0.0.1:64081/api/v1/namespaces/myns\u0026#39; From there we can see the etcd query as the URI is namespaces/myns, which we use in etcdctl query path:\netcdctl get /registry/namespaces/myns Every type of Kubernetes object has a storage.go file in their implementation that defines how api server should write object. Here is an example for Pod object.\nEtcd also supports watch command to watch for changes. For example:\netcdctl watch --prefix /registry/namespace # watch output k create ns newns Now we create a namespace with kubectl:\nkubectl create ns myns The output from etcdctl will reflect the change. The communication between etcdctl and etcd is gRPC protocol. The output is based on stream, as we can see from the watch result.\nEtcd Maintenance Like any distributed store, etcd needs maintenance and operation work. For example, we can check endpoint status with endpoint command:\netcdctl endpoint status We can also backup and restore etcd store with etcdctl command:\netcdctl snapshot save /tmp/backup.db This was an question in CKA exam. In real life, when the workload scales up, the etcd store may come across many pitfalls, such as degraded performance, unresponsiveness, some etcd member going down, network partition on etcd store causing split brain. It is important to ensure efficient communication between API server and etcd store. The etcdctl provides defrag and compact commands for common maintenance activities.\nPrevious PostHosting database on Kubernetes Next PostChaos Mesh – Cloud Native Chaos Engineering ","date":"2022-06-14T00:10:00-04:00","image":"/wp-content/uploads/2025/04/feature-etcd.webp","permalink":"/2022/06/etcd-the-key-value-store-for-kubernetes/","title":"Etcd – the key-value store for Kubernetes"},{"content":"Background \u0026#8220;We want to host Postgres database on Kubernetes. Can you help us?\u0026#8221;. The client appears assertive and reluctant to resort to managed services. So I did some homework and went through this tutorial. My thought: it\u0026#8217;s doable, but don\u0026#8217;t do it unless operating database as a service is your main business.\nI believed that was the client\u0026#8217;s best interest, until I came across this the blog post A Case for Databases on Kubernetes from a Former Skeptic. The author explained his journey from being a skeptic, to grudging acceptance, and eventually to an evangelist on running database on Kubernetes. The same voice came from the author of the upcoming book Managing Cloud Native Data on Kubernetes, who also advocates hosting database on Kubernetes. While the points in the chapters are valid, the book also includes a good amount of technical details which might lead reader to believe the opposite view.\nJust a few years back, Kubernetes was not mature to host database. This is changing in 2022. Nowadays, for clients with their own Kubernetes platform, technological maturity is no longer the main reason that keeps them from hosting database on Kubernetes, it is the operational cost. The operational cost has to do with whether the client has in-house expertise in database and Kubernetes. If they do, the hard path makes economical sense.\nIn this post, we discuss what we need to be aware of in order to host database on Kubernetes.\nBenefit with Kubernetes The first few versions of Kubernetes only supported stateless workload (reference documentary). That is what Kubernetes was born to solve. Built-in objects such as replicaSet, deployment, horizontalPodAutoscaler are abstractions of operations particular to stateless workload. Pods for stateless workload are ephemeral: they crash and get replaced at any time. Because they don\u0026#8217;t carry persistent data themselves, they are expendable. Kubernetes\u0026#8217; orchestration capability are driven by controllers. As discussed, the controller pattern is adopted in all controller implementations. They are the engines of the platform that works tirelessly in a control loop to ensure desired states matches their declared states. This is a key feature of Kubernetes as container platform. Let\u0026#8217;s examine a web service that requires 5 instances behind load balancer. With traditional hosting model on Linux servers, you\u0026#8217;d have it installed on all five VMs. If the process on one of the VMs dies, the VM has to be removed from the load balancer\u0026#8217;s target pool. One may wrap the process with process monitor and control utility such as supervisord, and re-install the application using automation utility (e.g. Ansible). However, each server is unaware of the status of its peer. Without a central \u0026#8220;Control Plane\u0026#8221;, there is no coordination between the activities of each VMs. Kubernetes controller solved all these operational problems. Control Loop Kubernetes comes with a set of build-in controllers that run inside the kube-controller-manager. Here is a good page about how controllers work. Controller is what is missing in many automation tools other that Kubernetes. Even though Red Hat now brands Ansible as Automation Controller, it does not involve a control loop or controller pattern. If there\u0026#8217;s one thing that sets Kubernetes apart from other hosting platforms and automation platforms, it is the implementation of controller pattern. Stateful workload Does the controller pattern also benefit stateful workload? Yes. How to orchestrate Pods for stateful workload is usually more tricky. CRD can define a custom object type for controller to consume. In this case, an operator is an implementation of the controller pattern. This pattern is also known as the operator pattern. In a replicaSet, Pod names have extensions of randomly generated numbers. A statefulSet names its the Pods by sequential numbers. For Postgres database, Bitnami built a good Helm Chart to install the database automatically. However, it does not have a control loop. If someone changes the workload after initial installation, the change is not monitored or controlled by any controller. This is a disadvantage of Helm chart as compared with operators. For PostgreSQL, there are a number of operators, the most notable being PGO (Postgres Operator) from Crunchy Data. To install an instance of PostgreSQL database, we need to install the operator, and then declare a Custom Resource using the PostgresCluster CRD. The operator will set up the cluster according to the declaration made in the PostgresCluster CR. I used the quick start guide to bring Postgres up real quick on an Azure Kubernetes cluster. The operator (v5) supports common cloud Kubernetes platforms (GKE, EKS, AKS), VMware Tanzu, Openshift, Rancher, Kubernetes. It does not explicitly indicate whether PGO supports Minikube or kind.\nSo far, I\u0026#8217;ve discussed the pros of running PostgreSQL on Kubernetes using Postgres Operator. We can describe the database deployment in a CR and the controller (operator) will monitor the resource incessantly to ensure the actual state matches the state defined in the CR. Not only is it doable to host database in Kubernetes, it makes our lives even easier. Persistent storage Database is not only a stateful workload, it also has special requirement on storage. It needs to persist data, support ACID transaction, and make optimal use of disks. When we operate everything on premise, we use fibre cable with a SAN as the storage media for database file. The operating system allows the database process to interact with blocks on the storage volume via device mapper.\nIn Kubernetes, we need to give Pods persistent volumes. There are a few APIs: Storage Class, Volume Storage Class, Persistent Volume and Persistent Volume Claims. Storage Class represents how Pod can connect to a storage. Pods will need PVCs in order to read and write on PVs. However, since Pods are ephemeral \u0026#8211; a Pod may crash any time, even if it is in the middle of writing to a PV, during an ACID transaction. The scheduler may reschedule the crashed Pod to a different node. Then it will need to pick up the PV from where it left off, on the new Node. Take Azure Kubernetes Service for example, a few storage classes are available by default, backed by Azure managed disk (managed-csi) or Azure file storage (azurefile-csi):\nStorageClassAzure storage service in-treedefaultManaged Disk using Azure StandardSSD managed-premiumManaged Disk using Azure Premium Storage azurefileAzure File Share using Azure Standard Storage azurefile-premiumAzure File Share using Azure Premium Storage csimanaged-csiManaged Disk using Azure StandardSSD managed-csi-premiumManaged Disk using Azure Premium Storage azurefile-csiAzure File Share using Azure Standard Storage azurefile-csi-premiumAzure File Share using Azure Premium Storage If we use storage class based on Azure disks to create a PV, only one Pod can use the PV. If we use storage class based on Azure files to create a PV, then the storage is mounted as NFS (Linux) or SMB (Windows) share. File storage is not a valid use case for database workload and it can significantly degrade database performance. When I tried to use a file-storage based CSI with PGO, the Pod reports an error and will not start properly. We should use Azure disk based CSI storage classes. That leaves us with two options: managed-csi and managed-csi-premium. High Availability Even with these to options left, we still have to investigate how database Pods interact with persistent volume for high availability, in order to determine whether any of the options are suitable. The two storage classes differ by disk performance but both have its own limitation with multi-AZ support on Azure managed disks. When the cluster operates across zones, the Kubernetes scheduler may reschedule a Pod crashed in one zone to a Node in a different availability zone (a different data centre). Even though the managed disks, when attached to VMs, can be configured as zone-redundant, when they are used as Kubernetes volume, they are NOT zone-redundant. So the node in a different zone will not be able to attach PV to the new Pod. There are SDS (software-defined storage) solution such as Portworx that solves the limitation of Azure disk for cross-region storage volume. The SDS layer brings managed disks from multiple availability zones into a pool. This storage pool acts as a highly available, cross-zone storage tier presented to AKS as persistent volumes. We can install Portworx as the SDS layer using Portworx operator. To do so, we first have to configure grant the cluster the permission to provision resources in Azure, because the Portworx operator will use node\u0026#8217;s identity (kubelet identity) to provision Azure resources on behalf of the nodes. Portworx will provision Azure disks and acts as the intermediary layer.\nApart from cross-zone high availability enabled by PX-Store, Portworx can also help with cross-region replication of persistent volumes. The PX-DR component can perform asynchronous replication across Azure regions. The destination region needs to have its own cluster because a single AKS cluster cannot span across regions.\nStorage Class Once we have portworx installed, the following storage classes are available by default:\npx-db px-db-cloud-snapshot px-db-cloud-snapshot-encrypted px-db-encrypted px-db-local-snapshot px-db-local-snapshot-encrypted px-replicated px-replicated-encrypted The steps for installing porworx on AKS are documented here. This blog post has more details in the installation process on a different platform. We can also built CSI based storage classes with different IO priority and replication factors.\nIn summary, Kubernetes operator pattern makes it easier to manage stateful workload. However, database performance depends largely on storage. To host database on Kubernetes, one will have to also manage the storage volumes on their own. There has not been a study on the impact to performance by moving database to Kubernetes platform. However, I only expect a degraded performance due to the layers introduced.\nExample In this section we configure a (minimally viable) PostgreSQL cluster using Crunchy Data pgo to demonstrate the idea. The steps are based on its tutorial but it works on a local KinD cluster. As discussed in a previous post, I use KinD for testing workload requiring persistent storage because Minikube has this open issue with permissions on PVs with multiple nodes.\nTo prepare the cluster, we can use kind-config.yaml file from my real-quicK-cluster repo:\nkind create cluster --config=kind-config.yaml # to delete cluster after testing: kind delete cluster --name kind We use Helm to install the operator. Since the Helm chart is not hosted in a public repo, we\u0026#8217;d have to download the directory of the Helm Chart.\ngit clone https://github.com/CrunchyData/postgres-operator-examples cd postgres-operator-examples helm install -n postgres-operator --create-namespace crunchy-pgo helm/install kubectl -n postgres-operator get po --watch kubectl explain postgresclusters Now we can create a YAML file for the Custom Resource and let\u0026#8217;s call it test-cluster.yaml with the following content:\napiVersion: postgres-operator.crunchydata.com/v1beta1 kind: PostgresCluster metadata: name: hippo namespace: postgres-operator spec: backups: pgbackrest: image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.38-1 repos: - name: repo1 volume: volumeClaimSpec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: \u0026#34;standard\u0026#34; image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.3-0 instances: - dataVolumeClaimSpec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: \u0026#34;standard\u0026#34; name: instance1 replicas: 3 minAvailable: 2 postgresVersion: 14 In the manifest, we specified a cluster, using storageclass named \u0026#8220;standard\u0026#8221;, with 3 replicas and requiring 2 available. We assume a storage class named \u0026#8220;standard\u0026#8221; already exists and optimized for database workload. In the manifest, we also configured a backup job. We can apply the manifest and watch for the Pods to come up in a few minutes.\nkubectl apply -f test-cluster.yaml kubectl -n postgres-operator get po --watch kubectl -n postgres-operator describe postgresclusters hippo The Pods in the postgres-operator namespace should report something like this:\nNAME READY STATUS RESTARTS AGE hippo-backup-mwpm-ps8wk 0/1 Completed 0 21s hippo-instance1-6mls-0 4/4 Running 0 3m35s hippo-instance1-hjp6-0 4/4 Running 0 3m35s hippo-instance1-k4qf-0 4/4 Running 0 3m35s hippo-repo-host-0 2/2 Running 0 3m35s pgo-548d5f48bc-9w4z4 1/1 Running 0 8m41s pgo-upgrade-566b9cc98f-d7gkr 1/1 Running 0 8m41s Three Pods for PostgreSQL are all up. The first backup run has completed already. We can connect to the cluster using psql following the quick start guide. We can also configure an application. A good example application that uses PostgreSQL database is KeyCloak. We briefly mentioned it in OIDC discussion. Currently the keycloak example on Crunchy pgo\u0026#8217;s quick start guide is outdated. Instead, use the following content as keycloak.yaml:\napiVersion: apps/v1 kind: Deployment metadata: name: keycloak namespace: postgres-operator labels: app.kubernetes.io/name: keycloak spec: selector: matchLabels: app.kubernetes.io/name: keycloak template: metadata: labels: app.kubernetes.io/name: keycloak spec: containers: - image: quay.io/keycloak/keycloak:latest name: keycloak args: [\u0026#34;start-dev\u0026#34;] env: - name: DB_VENDOR value: \u0026#34;postgres\u0026#34; - name: DB_ADDR valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: host } } - name: DB_PORT valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: port } } - name: DB_DATABASE valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: dbname } } - name: DB_USER valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: user } } - name: DB_PASSWORD valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: password } } - name: KEYCLOAK_USER value: \u0026#34;admin\u0026#34; - name: KEYCLOAK_PASSWORD value: \u0026#34;admin\u0026#34; - name: PROXY_ADDRESS_FORWARDING value: \u0026#34;true\u0026#34; ports: - name: http containerPort: 8080 - name: https containerPort: 8443 readinessProbe: httpGet: path: /realms/master # https://stackoverflow.com/questions/70577004/keycloak-could-not-find-resource-for-full-path port: 8080 initialDelaySeconds: 30 restartPolicy: Always Once we apply keycloak.yaml, in a minute we should see and be able to port-forward web traffic:\n$ kubectl apply -f keycloak.yaml $ kubectl -n postgres-operator get po -l app.kubernetes.io/name=keycloak NAME READY STATUS RESTARTS AGE keycloak-7995d78d7c-zjp4d 1/1 Running 0 4m29s $ kubectl port-forward deploy/keycloak -n postgres-operator 8080:8080 After using the port-forward command, we can browse to web portal on my MacBook by http://localhost:8080 and configure an initial password, as shown here:\nIn real life system we would need a proper Ingress. After testing, delete the cluster with kind command and specify the cluster name (kind).\nOperation Cost Operation cost is an important consideration. Troubleshooting on Kubernetes platform is in general more complicated than just on a Unix system. Hosting database on Kubernetes requires skills not only on the Kubernetes platform, but also on database. There used to be database administrator positions where someone has to maintain the upgrade, the storage, the replication, the multi-tenancy and the performance optimization of database. With a database hosted on Kubernetes, the database administrator will have to perform all these activities on a containerized platform. This is not an easy undertaking, and in many occasions warrants a full-time position on its own. Therefore, don\u0026#8217;t host your database on Kubernetes, unless that is your main business. It is not the technology that shots down this option. It is the operation cost, such as complexity of configuration, and staff skillset, that makes this option not worth it.\nPrevious PostFSx ONTAP – Enterprise storage on AWS Next PostEtcd – the key-value store for Kubernetes ","date":"2022-05-29T11:01:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-database.webp","permalink":"/2022/05/hosting-database-on-kubernetes/","title":"Hosting database on Kubernetes"},{"content":"Even though object storage has gained a lot of popularity, file storage is still prevalent. AWS has Elastic File System but the performance is insufficient for enterprise workload. The FSx product line has enterprise storage options and on Sept 2, 2021, AWS launched FSx ONTAP. This post is my impression about FSx ONTAP. As previously discussed, FSx ONTAP\u0026nbsp;is a managed NetApp storage service by AWS. Essentially, AWS installs NetApp arrays in their data centres, so that users can provision ONTAP volumes from AWS console, or using AWS CLI.\nFSx ONTAP ONTAP (or Data OONTAP) has been a very successful operating system to manage storage arrays. It was so successful that NetApp uses ONTAP as brandname for their storage arrays. This is similar to Isilon, a name of BSD based operating system to manage storage and later becomes brandname of EMC\u0026#8217;s storage products. ONTAP has been in competition with other enterprise storage players such as EMC Isilon, HP 3PAR, etc and AWS landed on ONTAP as their partner for enterprise storage. It appears that NetApp still owns their ONTAP storage technology. AWS operates the data centre and provides capability via the CLI layer and console.\nSometimes, people confuses FSx ONTAP with NetApps offering Cloud volumes ONTAP. The two are fundamentally different. Cloud volumes ONTAP works with Cloud Manager (available as self-hosted or SaaS) as the management UI. It manages volumes provisioned from cloud vendor such as AWS, Azure, etc. Clients often configure these \u0026#8220;Cloud volumes\u0026#8221; as extension to an existing on-premise ONTAP storage deployment. When their on-prem ONTAP volumes fall short of space, the Cloud Manage is aware of a remotely available, cloud backed volume to move cold data off to. Client Tool FSx ONTAP is essentially an ONTAP storage cluster sitting in AWS data centre. Users have the options of using AWS CLI or ONTAP CLI to manage the cluster. Users with storage administrator background are likely to prefer the latter. In my professional service experience, I have taken some iterations to come to best practice to use the right tool to interact with ONTAP resources. In a nut shell, it depends on the level of resource that we are interacting with. I categorize those resources into two classes:\nCategoryExampleIdentificationAWS-level resourceFSx ONTAP file system, Storage Virtual MachineThese resources are from ONTAP but they are identified as AWS resources (with ARN). They are also exposed to AWS SDK and can be managed by AWS CLI.ONTAP-native resourceVolume, Snapshot policy, Schedule, Snapmirror relationship, VserverThese resources come from ONTAP and can only be managed using ONTAP CLI. The AWS CLI cannot manage these resources simply because they are not exposed to AWS SDK. Some type of resource such as Volume, may be managed by AWS CLI with very limited options. So we still prefer ONTAP CLI to manage resources. The AWS CLI (v 2.2.37) has very limited options when creating volumes. For example, the create-volume documentation states that the OntapVolumeType section of output can display types of RW, DP (data protection), or LS. However, it doesn’t allow users to create a volume other than the default RW type. When we configure SnapMirror destination, we need a volume of DP type. We had to use ONTAP CLI to achieve that.\nOur practice works out to be: use AWS CLI to create a file system and storage virtual machine. Then we use ONTAP CLI to create everything else. Even though AWS CLI intend to support volume creation, we prefer ONTAP CLI for full functionality support, and alignment with the ONTAP literature.\nAdministration Tasks You should configure most of the administrative tasks with ONTAP CLI. We use the documentation by ONTAP as reference. For example, when a volume runs out of Inode, the AWS CLI reports that there is no space left. We need to increase the inode limit and this is, again, not something that AWS CLI can manage.\u0026nbsp; We’d have to use the volume modify command from ONTAP CLI.\nNFS version The AWS document states that FSx ONTAP supports NFSv3.0, v4.0 and v4.1. However, FSx ONTAP is currently backed by NetApp ONTAP 9.10.0, which partially supports NFSv4.2, with basic protocol and Labled NFS feature. NetApp’s ONTAP Best Practices and Implementation Guide suggests a method for clients to mount as NFSv4.2. In the POC we mount as NFS v4.2 in all of our testings.\nClarity of terminology The ONTAP storage system has been around for a while and many of its concepts are well known in the storage community. For example, A “Snapshot copy” is a read-only, point-in-time image of a volume. (ref: ONTAP 9 documentation -\u0026gt; ONTAP concepts -\u0026gt; Replication -\u0026gt; Snapshot copies) This concept becomes “Snapshot” in AWS literature.\u0026nbsp; It took us some research to come to realize that “Snapshot” in AWS document, essentially maps to “Snapshot copy” in ONTAP literature.\nThis creates confusion, because we use ONTAP documentation for operation guidance because we can’t get enough help from AWS documentation. The terminology in AWS documentation should align with ONTAP.\nThe other example is the difference between “backup” and “snapshot” in AWS documentation. It is my understanding that they both use the same underlying Snapshot technology on the ONTAP side. I’m not exactly sure what their difference is.\nONTAP can create a Snapshot copy nearly instantaneously. However, when taking a snapshot using web console in AWS, it takes up to 10 minutes to update the status. This is confusing because it creates a perception that it takes 10 minutes to complete snapshot.\nCross Region Replication There is a document page on AWS about using SnapMirror at a very high level. It points to two documents: using NetApp Cloud Manager and ONTAP CLI.\nThe former is not a viable option as we started natively on FSx ONTAP and do not have NetApp Cloud Manager. As to the latter, we managed to configure cross-region replication with ONTAP CLI following the document and identified some gaps in the documentation. Specifically, it would be helpful if AWS documentation calls out that:\ninter-cluster network connectivity is a prerequisite (e.g. via VPC peering, transit gateway) Port 10000, 11104-11105 must be added to security group for inter cluster communication. The ONTAP CLI command to validate connectivity between clusters (using the ping command from ONTAP CLI). With AWS CLI alone it is not possible to configure cross-region replication.\nFinal words As someone who lived with enterprise storage for more than a decade, I\u0026#8217;m glad to see that cloud vendors brings enterprise storage into their data centre, acknowledging that consumer grade file storage are just insufficient for heavy storage use cases such as medical imaging. FSx ONTAP seems to be in early maturity level. However, since AWS exposes ONTAP CLI access to users, ONTAP professionals are able to leverage its full potential.\nPrevious PostKnative Eventing Introduction Next PostHosting database on Kubernetes ","date":"2022-05-14T22:28:00-04:00","image":"/wp-content/uploads/2025/04/feature-fsx-ontap.webp","permalink":"/2022/05/fsx-ontap-enterprise-storage-on-aws/","title":"FSx ONTAP – Enterprise storage on AWS"},{"content":"In the previous post, I mentioned that Knative Serving and Knative Eventing should be seen as two different projects. The former is supposed to be widely used as a serving layer for microservices, whereas the latter has a narrower customer base. There are a dozen companies who need to build Platform as a Service, and will benefit from event-driven architecture. They are the niche customer for Knative Eventing. In this post, we go through two demos for Eventing.\nEvent Driven Architecture Before jumping into Knative eventing, let\u0026#8217;s first understand the value of event-driven architecture. As covered in Knative presentation in 2020, a growing micro-service architecture involves lots of service-to-service communications that looks like a spider web in a diagram. Two services communicating with each other are tightly coupled: changing one requires a change of the other. Event-driven architecture evolves to decouple those interdependent services. Knative eventing tries to standardize the event data with Cloud Event and also proposes a framework to drive this architecture. The documentation uses different words to express the same concepts. So let\u0026#8217;s take a note first: source = producer which emits events, and sink = subscriber = consumer which is the destination of events. Knative Eventing supports two models of in even-driven architecture.\nBrokers and Triggers provides an \u0026#8220;event mesh\u0026#8221;. Event producers delivers events to a Broker. Triggers then distribute them uniformly by consumers. Broker is a \u0026#8220;hub\u0026#8221; for events. It is a central location to receive and send events for delivery. We can choose from the following broker types:\nMT (multi-tenant) channel-based broker (which a channel implementation, such as Kaka channel or an in-memory channel) GCP broker Apache Kafka broker RabbitMQ broker Triggers represents a desire to subscribe to events from a specific broker. It can act as a filter\nchannel-subscriber model Channels and Subscriptions provide a \u0026#8220;event pipe\u0026#8221; model which transforms and routes events between Channels using Subscriptions. This model makes more sense when events out of one system needs to be transformed and then routed to another process.\nA channel can be either a generic channel object, or a custom channel implementation (such as Kafka channel or in-memory channel)\nThe subscription consists of a Subscription object, which specifies the Channel and the Sink (aka the Subscriber) to deliver events to. You can also specify some Sink-specific options, such as how to handle failures.\nThe key difference is Broker-Trigger model allows you to filter events, where as Channel-Subscription model allows you to transform events.\nNote that a Knative Service can act as both a Source and a Sink for events, and for good reason. You may want to consume events from the Broker and send modified events back to the Broker, as you would in any pipeline use-case. Between these components, Knative Eventing uses\u0026nbsp;CloudEvents\u0026nbsp;to send information back and forth. Knative Eventing Lab We can create a lab cluster using Kind as per the instruction in real-quicK-cluster. In docker-desktop settings, I configured 6 CPU and 8GB memory as the lab involves a number of nodes. The creation takes a couple minutes. This Knative eventing lab has four paths one can take, based on different model and different implementations:\nLab A demonstrates the configuration of broker-trigger model using Kafka based broker. Lab B demonstrates the configuration of channel-subscription model, using in-memory channel. The two labs share some common steps. We first install Knative Eventing using the provided manifests:\n$ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-crds.yaml $ kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/eventing-core.yaml $ kubectl get pods -n knative-eventing The Pods should come to READY fairly quickly. In the mean time, we can configure the consumer as well using the manifests below:\napiVersion: v1 kind: Namespace metadata: name: event-example --- apiVersion: apps/v1 kind: Deployment metadata: name: hello-display namespace: event-example spec: replicas: 1 selector: matchLabels: \u0026amp;labels app: hello-display template: metadata: labels: *labels spec: containers: - name: event-display image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display --- kind: Service apiVersion: v1 metadata: name: hello-display namespace: event-example spec: selector: app: hello-display ports: - protocol: TCP port: 80 targetPort: 8080 --- apiVersion: apps/v1 kind: Deployment metadata: name: goodbye-display namespace: event-example spec: replicas: 1 selector: matchLabels: \u0026amp;labels app: goodbye-display template: metadata: labels: *labels spec: containers: - name: event-display image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display --- kind: Service apiVersion: v1 metadata: name: goodbye-display namespace: event-example spec: selector: app: goodbye-display ports: - protocol: TCP port: 80 targetPort: 8080 The consumers (aka sinks) will be able to process events at the validation step at the end.\nLab A \u0026#8211; Install Kafka and configure Kafka-based Broker This lab is for broker-trigger model. We first configure a Kafka Broker, which requires a Kafka cluster. The document uses Strimzi operator to quickly install Kafka cluster. Instead, here we use Helm to install the Kafka cluster:\nhelm install my-cluster-kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3 At the end of installation, the Helm notes indicate that the bootstrap server is my-cluster-kafka.kafka.svc.cluster.local:9092. All Kafka pods should come ready in about 5 minutes. After cluster install, we need to install Kafka controller and the Broker-layer data plane which allow you to map a Kafka instance to Broker CRD later:\nkubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-controller.yaml kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.0/eventing-kafka-broker.yaml kubectl get pods -n knative-eventing kubectl -n knative-eventing edit cm kafka-broker-config As the command above suggests, once data plane is installed, we edit the default broker config by updating the bootstrap.servers with the bootstrap server address noted above. We also need to make sure the replication factor is no greater than number of partitions, which should not be a problem in our lab as the replica count was set to 3. Now we can create a Broker in the same namespace with consumer:\napiVersion: eventing.knative.dev/v1 kind: Broker metadata: name: default namespace: event-example annotations: eventing.knative.dev/broker.class: Kafka spec: config: apiVersion: v1 kind: ConfigMap name: kafka-broker-config namespace: knative-eventing Once the manifest above is applied, we should see the status of broker (named default) to be READY. We can also validate broker creation with kn CLI command:\n$ kubectl -n event-example get broker default NAME URL AGE READY REASON default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m True $ kn broker list -n event-example NAME URL AGE CONDITIONS READY REASON default http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/default 19m 7 OK / 7 True Then we create Triggers that connects both to the broker with filters, and to the consumers by referencing service name. The manifest to create triggers is provided below:\napiVersion: eventing.knative.dev/v1 kind: Trigger metadata: name: hello-display namespace: event-example spec: broker: default filter: attributes: type: greeting subscriber: ref: apiVersion: v1 kind: Service name: hello-display --- apiVersion: eventing.knative.dev/v1 kind: Trigger metadata: name: goodbye-display namespace: event-example spec: broker: default filter: attributes: source: sendoff subscriber: ref: apiVersion: v1 kind: Service name: goodbye-display We can confirm the setups above by monitoring the resources (like we did for broker), or use knative CLI:\n$ kn trigger list -n event-example NAME BROKER SINK AGE CONDITIONS READY REASON goodbye-display default service:goodbye-display 3m24s 6 OK / 6 True hello-display default service:hello-display 3m24s 6 OK / 6 True Now we can go to the validation step to send event and validate result.\nLab B \u0026#8211; Install in-memory channel and configure Channel This lab is for channel-subscription model. we first install an in-memory channel:\nkubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.3.0/in-memory-channel.yaml Then we can map this channel to a Channel resource in Eventing. We can do so by applying the manifest below:\napiVersion: messaging.knative.dev/v1 kind: Channel metadata: name: demo-channel-1 namespace: event-example --- apiVersion: messaging.knative.dev/v1 kind: Channel metadata: name: demo-channel-2 namespace: event-example The creation of Channel resources can be verified as below\n$ kn -n event-example channel list NAME TYPE URL AGE READY REASON demo-channel-1 InMemoryChannel http://demo-channel-1-kn-channel.event-example.svc.cluster.local 6m34s True demo-channel-2 InMemoryChannel http://demo-channel-2-kn-channel.event-example.svc.cluster.local 6m34s True Now we need to create Subscription which connects Channels to Consumers. Subscriptions can be created with the manifest below:\napiVersion: messaging.knative.dev/v1 kind: Subscription metadata: name: demo-subscription-1 namespace: event-example spec: channel: apiVersion: messaging.knative.dev/v1 kind: Channel name: demo-channel-1 subscriber: ref: apiVersion: v1 kind: Service name: hello-display --- apiVersion: messaging.knative.dev/v1 kind: Subscription metadata: name: demo-subscription-2 namespace: event-example spec: channel: apiVersion: messaging.knative.dev/v1 kind: Channel name: demo-channel-2 subscriber: ref: apiVersion: v1 kind: Service name: goodbye-display We can confirm subscription:\n$ kn -n event-example subscription list NAME CHANNEL SUBSCRIBER REPLY DEAD LETTER SINK READY REASON demo-subscription-1 Channel:demo-channel-1 service:hello-display True demo-subscription-2 Channel:demo-channel-2 service:goodbye-display True Now we can go to validation step.\nValidation For validation, we need to first configure an event source. The source in our lab is a throw-away Pod with curl utility. I use my favourite nicolaka netshoot to fire off the events using. To launch the Pod and get to command shell:\n$ kubectl run tmp-shell -n event-example --rm -i --tty --image nicolaka/netshoot -- /bin/bash If you don\u0026#39;t see a command prompt, try pressing enter. bash-5.1# We can POST message to the corresponding endpoint URL to fire events. The endpoint is different for each lab:\nLabURLA (Kafka-based broker)http://kafka-broker-ingress.knative-eventing.svc.cluster.local/event-example/defaultB (channel)http://demo-channel-1-kn-channel.event-example.svc.cluster.local\nhttp://demo-channel-2-kn-channel.event-example.svc.cluster.local With the correct URL, we can build a payload for POST method. Below is three curl commands with different payload POST to the same endpoint (taking lab A as an example):\n# curl -v \u0026#34;http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default\u0026#34; \\ -X POST \\ -H \u0026#34;Ce-Id: say-hello\u0026#34; \\ -H \u0026#34;Ce-Specversion: 1.0\u0026#34; \\ -H \u0026#34;Ce-Type: greeting\u0026#34; \\ -H \u0026#34;Ce-Source: not-sendoff\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{\u0026#34;msg\u0026#34;:\u0026#34;Hello Knative!\u0026#34;}\u0026#39; # curl -v \u0026#34;http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default\u0026#34; \\ -X POST \\ -H \u0026#34;Ce-Id: say-goodbye\u0026#34; \\ -H \u0026#34;Ce-Specversion: 1.0\u0026#34; \\ -H \u0026#34;Ce-Type: not-greeting\u0026#34; \\ -H \u0026#34;Ce-Source: sendoff\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{\u0026#34;msg\u0026#34;:\u0026#34;Goodbye Knative!\u0026#34;}\u0026#39; # curl -v \u0026#34;http://broker-ingress.knative-eventing.svc.cluster.local/event-example/default\u0026#34; \\ -X POST \\ -H \u0026#34;Ce-Id: say-hello-goodbye\u0026#34; \\ -H \u0026#34;Ce-Specversion: 1.0\u0026#34; \\ -H \u0026#34;Ce-Type: greeting\u0026#34; \\ -H \u0026#34;Ce-Source: sendoff\u0026#34; \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{\u0026#34;msg\u0026#34;:\u0026#34;Hello Knative! Goodbye Knative!\u0026#34;}\u0026#39; All the requests should receive 202 return code. As highlighted above, we specified type and source for each event fired to the broker. On the consumer side, we can simply check the pod log to verify the events received. Below is an example of events received as a result of the events sent above (take broker-trigger model as example):\n$ kubectl -n event-example logs -l app=hello-display --tail=100 2022/03/09 17:05:59 Failed to read tracing config, using the no-op default: empty json tracing config ☁️ cloudevents.Event Context Attributes, specversion: 1.0 type: greeting source: not-sendoff id: say-hello datacontenttype: application/json Extensions, knativearrivaltime: 2022-03-09T17:14:19.196589801Z Data, { \u0026#34;msg\u0026#34;: \u0026#34;Hello Knative!\u0026#34; } ☁️ cloudevents.Event Context Attributes, specversion: 1.0 type: greeting source: sendoff id: say-hello-goodbye datacontenttype: application/json Extensions, knativearrivaltime: 2022-03-09T17:14:42.977803608Z Data, { \u0026#34;msg\u0026#34;: \u0026#34;Hello Knative! Goodbye Knative!\u0026#34; } $ kubectl -n event-example logs -l app=goodbye-display --tail=100 2022/03/09 17:05:58 Failed to read tracing config, using the no-op default: empty json tracing config ☁️ cloudevents.Event Context Attributes, specversion: 1.0 type: not-greeting source: sendoff id: say-goodbye datacontenttype: application/json Extensions, knativearrivaltime: 2022-03-09T17:14:27.430702588Z Data, { \u0026#34;msg\u0026#34;: \u0026#34;Goodbye Knative!\u0026#34; } ☁️ cloudevents.Event Context Attributes, specversion: 1.0 type: greeting source: sendoff id: say-hello-goodbye datacontenttype: application/json Extensions, knativearrivaltime: 2022-03-09T17:14:42.977803608Z Data, { \u0026#34;msg\u0026#34;: \u0026#34;Hello Knative! Goodbye Knative!\u0026#34; } This is the gist of Knative evening. Note that the format of HTTP payload when we fire off events needs to conform to CloudEvents format. The container image (event_display) used in the container is also created for the purpose of processing events received and display them in stdout.\nConclusion Persistent storage allows for stateful applications on Kubernetes. It also enables architectural patterns that requires persistent storage. Knative eventing is a framework that enables event driven architecture. It supports many messaging channels (including NATs, a cloud-native messaging system) and broker types (MT channel based, GCP, RabbitMQ). Previous PostKnative Serving Introduction Next PostFSx ONTAP – Enterprise storage on AWS ","date":"2022-04-29T10:16:00-04:00","image":"/wp-content/uploads/2025/04/feature-knative-eventing.webp","permalink":"/2022/04/knative-introduction/","title":"Knative Eventing Introduction"},{"content":"Background As per IBM\u0026#8216;s definition, Knative enables serverless workloads to run on Kubernetes clusters, and makes building and orchestrating containers with Kubernetes faster and easier. It has drawn a lot of attention recently. It released version 1.0 in November 2021, and was accepted as a CNCF incubating project in March 2022. Glories aside, the value it delivers is the ability to go serverless on a Kubernetes platform. Originally, Knative was built with three components: build, serving and eventing. The build component was deprecated in favour of the Tekton project. Tekton is cloud native CI/CD pipeline. It connects to source code repo and build artifacts. It can also deploy applications with pipeline as code. Tekton\u0026#8217;s documentation includes a quality tutorial here linked to interactive terminal. On the other hand, we\u0026#8217;ll focus on Knative Serving in this post, and Knative Eventing in the next.\nKnative can be seen as a serverless framework with a number of open-source technologies as building blocks, such as Istio for ingress gateway, Kafka or Google Pub/Sub as event-streaming engine, and Prometheus for observability to name a few. Despite of using the same CLI tool (named kn), Knative Serving and Eventing are considered separate capabilities using different groups of CRDs. They are installed separately, using their respective YAML files, or operator. Serving vs Eventing ? After reading the article \u0026#8220;Did we market Knative wrong\u0026#8221; from Ahmet Balkan, my impression is the two are not related. They could have been two separate projects without sharing a common name. As the author puts, these two shared some core logics. But beyond that, they don\u0026#8217;t have anything in common. Serving is lightweight and requires Istio. It provides the capability to scale from N to 0 when the system is idle, and from 0 to N when requests come in. It is the missing serving layer for running microservices on Kubernetes and Ahmet position it as the first thing that people installed after creating a cluster.\nThe target user of Eventing is much smaller. Eventing is for event-driven architecture and is more complex than Serving. Ahmet admits that they over-estimated how many people on the planet want to build a Heroku-like PaaS layer on top of Knative. There are a couple of dozen companies who would work through the complexity and build their own Kubernetes-based internal PaaS or even public-facing FaaS using Knative. They are the niche audience of Knative Eventing. For most platform builders who just need to run micro-service, Knative Serving is all they need.\nA Demo on Serving The definition of term \u0026#8220;serverless\u0026#8221; is very loose. Instead, Knative documentation promotes the ability to scale to zero and refer to it as \u0026#8220;some people call this Serverless\u0026#8221;. This suggests that \u0026#8220;the ability to scale to zero\u0026#8221; should be the better term to describe this capability. To deploy it, we need the Service object with apiVersion serving.knative.dev/v1. Below is a guide for a quick hands-on, with some steps modified from the Knative Serving tutorial and installation guide, with optional steps (extensions, DNS) skipped.\nFollow the first part of this post or this guide in real-quicK-cluster project, to install minikube with metal LB and configure istio (using istioctl as per instruction). Then Knative serving can be installed with two YAML manifests:\n$ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-crds.yaml $ kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.2.0/serving-core.yaml Then we can install Knative istio controller to integrate them so we can see the knative gateways:\n$ kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.2.0/net-istio.yaml $ kubectl -n knative-serving get gateway Now we can install the dummy service, either using kn CLI tool or using YAML manifest. The manifest is given below:\napiVersion: serving.knative.dev/v1 kind: Service metadata: name: hello namespace: default spec: template: metadata: name: hello-world annotations: autoscaling.knative.dev/class: \u0026#34;kpa.autoscaling.knative.dev\u0026#34; autoscaling.knative.dev/metric: \u0026#34;rps\u0026#34; autoscaling.knative.dev/target: \u0026#34;50\u0026#34; autoscaling.knative.dev/min-scale: \u0026#34;0\u0026#34; autoscaling.knative.dev/max-scale: \u0026#34;10\u0026#34; spec: containerConcurrency: 0 containers: - env: - name: TARGET value: World image: gcr.io/knative-samples/helloworld-go name: user-container ports: - containerPort: 8080 protocol: TCP We can notice that in the annotations there are some settings to overwrite the default configuration. They are explained in the autoscaling section of the Knative serving documentation. Here we use request per second (rps) as metric, and have 50 as target. The scale range is between 3 and 10 replicas. Once the resource is created, we can validate it with one of the two commands:\n$ kubectl get services.serving.knative.dev $ kn service describe hello The creation of knative service also created a number of Kubernetes resources, such as deployment and services:\n$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello ExternalName \u0026lt;none\u0026gt; knative-local-gateway.istio-system.svc.cluster.local 80/TCP 52s hello-world ClusterIP 10.110.127.231 \u0026lt;none\u0026gt; 80/TCP 108s hello-world-private ClusterIP 10.111.27.73 \u0026lt;none\u0026gt; 80/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP 108s kubernetes ClusterIP 10.96.0.1 \u0026lt;none\u0026gt; $ kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE hello-world-deployment 1/1 1 1 71s Before testing, let\u0026#8217;s mock the DNS entry to resolve to the Istio ingress service IP like below in /etc/hosts:\n192.168.64.16 hello.default.example.com Now we can confirm that the service returns the expected result (after a brief pause): $ curl hello.default.example.com Hello World! Wait for a short period without curl or connection from browser, the service should scale down to zero:\n$ kubectl get deploy -l serving.knative.dev/service=hello -w NAME READY UP-TO-DATE AVAILABLE AGE hello-world-deployment 0/0 0 0 78s From a separate terminal, emulate HTTP request with hey command, so the service receives more than 50 rps per pod:\n$ hey -n 200 -c 20 -z 20s -m GET http://hello.default.example.com We can watch the deployment get scaled up to 10. Once the hey command completes its task, the deployment will scale back down:\n$ kubectl get deploy -l serving.knative.dev/service=hello -w NAME READY UP-TO-DATE AVAILABLE AGE hello-world-deployment 0/0 0 0 78s hello-world-deployment 0/1 0 0 3m54s hello-world-deployment 0/1 0 0 3m54s hello-world-deployment 0/1 0 0 3m54s hello-world-deployment 0/1 1 0 3m54s hello-world-deployment 1/1 1 1 4m2s hello-world-deployment 1/10 1 1 4m4s hello-world-deployment 1/10 1 1 4m4s hello-world-deployment 1/10 1 1 4m4s hello-world-deployment 1/10 10 1 4m5s hello-world-deployment 2/10 10 2 4m9s hello-world-deployment 3/10 10 3 4m10s hello-world-deployment 4/10 10 4 4m10s hello-world-deployment 7/10 10 7 4m10s hello-world-deployment 9/10 10 9 4m10s hello-world-deployment 10/10 10 10 4m11s hello-world-deployment 10/5 10 10 5m18s hello-world-deployment 10/5 10 10 5m18s hello-world-deployment 5/5 5 5 5m18s hello-world-deployment 5/2 5 5 5m20s hello-world-deployment 5/2 5 5 5m20s hello-world-deployment 2/2 2 2 5m20s hello-world-deployment 2/1 2 2 5m22s hello-world-deployment 2/1 2 2 5m22s hello-world-deployment 1/1 1 1 5m22s hello-world-deployment 1/0 1 1 5m52s hello-world-deployment 1/0 1 1 5m52s hello-world-deployment 0/0 0 0 5m52s Autoscaling in Knative serving is backed by KPA (Knative Pod Autoscaler, used in the example above) or HPA. Also note that the trigger to scale up and down is client connection (RPS). This is different than KEDA. KEDA also supports the capability to scale to zero. However, with KEDA, it is an event that scales a deployment up from zero. In Knative serving, it is the connection to the Service itself that wakes up the service. Cost of scale-to-zero As discussed, both KEDA and Knative serving supports the ability to scale to zero and wake up from zero. The trigger to wake up from zero is different. With KEDA, workload wakes up by an event. With Knative Serving, workload has to wake up upon receiving a connection to the service. In the design to solve this problem, Knative has four sub-components :\nActivator \u0026#8211; When a service is scaled to zero, its request are routed to the\u0026nbsp;activator\u0026nbsp;which waits for a Pod to become ready (wake up) and proxies the traffic while editing the underlying\u0026nbsp;Endpoints\u0026nbsp;to route the traffic directly to the Pod(s) Autoscaler \u0026#8211; KPA Controller \u0026#8211; The main component responsible for watching API objects for Knative CRDs (KService, Configuration, Route, Revision) and manage their lifecycle, create the underlying Kubernetes resources and garbage-collect old objects. Webhook \u0026#8211; a Kubernetes Admission Webhook acting both as validating admission controller and mutating admission controller. The key player is activator, which receives requests when a service is scaled to zero. The wake-up process is summarized as follows:\nAfter receiving request, activator will buffer (hold onto) the request Then, it will look at request’s hostname to find which KService it is for. Then, it will scale up the Kubernetes Deployment and wait for a Pod to become ready. In the meanwhile, it updates Kubernetes Service to point to Pod IP addresses (so that activator gets out of the network path if the KService is awake). Finally, activator proxies the request to the started pod So the \u0026#8220;plumbing\u0026#8221; of the service connection managed by the Activator pod is the cost of scale to zero. This is also called Load Balancing, and the behaviour can be tweaked using two parameters: activator capacity and target burst capacity:\nTarget burst capacity: once the target deploy has one more more Pod, service requests may still route via activator or bypassing the activator. Target burst capacity determines at what point service request should start to bypass activator. Activator capacity: determines how many requests can activator hold on to. Considering the service-to-service connection also routes via Service CRD, there can be a lot of connections via Activator so its capacity needs to be adjusted. The additional overhead of managing and optimizing the \u0026#8220;Load Balancer\u0026#8221; is also part of operational cost for the ability to scale to zero.\nIn comparison, KEDA now has an HTTP add-on still at beta but allows connection-based wake-up. The design is a little different. As discussed in this previous post.\nConclusion Knative consists of two disparate components: Serving and Eventing. Serving uses KPA to provide scaling based on service request, and the ability to scale to zero. It is often compared with KEDA, a single-purpose lightweight tool for autoscaling. Here is a blurb on their difference by KEDA. The takeaways is that KEDA is more focused on scalability, whereas Knative serving covers more aspects. For example, the Service object of Knative also supports Traffic Splitting. Knative serving can integrate with istio.\nPrevious PostKubernetes Operator Next PostKnative Eventing Introduction ","date":"2022-04-17T11:33:00-04:00","image":"/wp-content/uploads/2025/04/feature-knative-serving.webp","permalink":"/2022/04/knative-introduction-serving/","title":"Knative Serving Introduction"},{"content":"Kubernetes has a number of tools to automate the deployment of a single workload. In previous posts, we had covered Helm and Kustomize. What are left unresolved is how to maintain the status of workload after deployment is completed. In this post, I will give an introduction to Kubernetes Operator. Compared with Helm (templating approach) and Kustomize (patching approach), Kubernetes Operator follows the operator pattern. Operators are usually provided by the developer of the application.\nOperator Pattern In Kubernetes, we know that a controller takes care of routine tasks to ensure that desired state expressed by Kubernetes resource types matches the current state. One example is that the Deployment controller ensures the number of pods running matches the amount specified in the replica field. Controller is the key to ensure that resources can be managed by declarative manifests for Kubernetes resources. Kubernetes makes use of controller pattern throughout its own design. One of its key component, Controller Manager, is a collection of many controllers. Each controller is in charge of a control loop, responsible for listening the object it manages. Another component, Kube-scheduler, is also a special type of Controller. The kube-scheduler monitors unscheduled Pod and health of nodes and determines the best Node to schedule the new Pod to. Then it writes the decision to etcd store for kubelet to execute.\nThis controller pattern is fairly successful in what it does and we can extend the use of it. Beyond the built-in resource types, we can create our own custom resource definitions (CRDs), and create controllers that watches for the manifest that declares custom resources (CRs). The controller ensures that the resource status matches their specifications. This is also known as reconciliation, which is implemented as a control loop. Operator pattern can be illustrated in the diagram below:\nOperator Pattern Technically, there is no difference between a controller and an operator. What makes an Operator (used to install workload) different than a native Kubernetes controller, are two things. First, an Operator usually needs CRDs because the built-in resource types are insufficient. Second, the operator reflects the domain knowledge to keep the target workload running. For example, stateful workloads such as database needs their operational steps executed in certain orders.\nOn Operator Pattern, CNCF published a whitepaper with a deeper review. This white paper is the best reference for a good understanding of the Operator Pattern.\nCustom Resource Definition The built-in controllers work with built-in objects (pre-defined APIs). Custom operators usually need their own APIs to function. To extend Kubernetes API, we define the schema of these APIs in the form of CRDs (custom resource definitions) using OpenAPIv3 standard. Then, we can declare Custom Resources (CRs) in compliance with the schema. The OpenAPIv3 schema in the CRD resource tells validating web hook (admission control) how to validate the schema when we send an CR in to API server.\nWhen we work with third-party operators, they usually provide CRDs along with the operator implementation. For example, in my operator example project, we have a minimalist CRD WordPress with one property: sqlRootPassword and we can declare a CR as in this example. For a more realistic use case, we can take a look at Kiali CRD. In the next section, we\u0026#8217;ll use it along with Kiali operator to install Kiali. Operator Usage Like Artifact Hub to Helm, OperatorHub is a public registry of most used Kubernetes Operators. In this section, we will take an example of using Operators. We will install Kiali as an add-on to Istio using Kiali CR and operator, which also depends on Prometheus to be installed using Prometheus Operator first. Note that the Kiali installation outlined in this section is not the the quick-start install manifests from Istio\u0026#8217;s sample directory. For Kiali on production system we have to customize the installation. Suppose we have installed Istio, we can then install Prometheus operator using Helm. The Prometheus operator will install Prometheus. Then we use Helm again to install Kiali operator. The Kiali operator will watch for creation of Kiali CRD, to deploy services:\n$ helm install -f prometheus-values.yaml --namespace istio-system --repo https://prometheus-community.github.io/helm-charts --version 13.6.0 istio-prometheus prometheus --insecure-skip-tls-verify $ helm install -f kiali-operator-values.yaml --namespace kiali-operator --repo https://kiali.org/helm-charts --version 1.45.0 kiali-op kiali-operator --create-namespace $ kubectl apply -f kiali-cr.yaml I include example content for each file in the commands above on Github gist (prometheus-values.yalm, kiali-operator-values.yaml and kiali-cr.yaml). For more options for installing Kiali, refer to their documentation.\nI use this example to install Kiali and it includes two Operators, the Prometheus Operator and the Kiali Operator. The Prometheus Operator is one of the first ever written Kubernetes Operator. As soon as the operator is deployed, it starts to deploy the operator service. For the Kiali operator, we need to deploy Kiali CR after the Kiali Operator has been deployed. Both are valid patterns.\nOperator Development Operator is powerful. However, authoring an Operator is not a trivial effort. One usually start with a framework. A framework creates a body of boiler plate code that has the pattern implemented and allows developers to enrich the functions following the pattern. The white paper introduced three frameworks:\nCNCF Operator Framework \u0026#8211; aims at Operator Developers with an SDK, a scaffolding tool and a test harness. It currently supports three project types: Golang, Helm and Ansible. CNCF Operator framework consists of SDK and OLM. Kopf (Kubernetes Operator Pythonic Framework) \u0026#8211; an easy-to-use framework in Python that abstracts away most of the low-level Kubernetes API communications hassle. kubebuilder \u0026#8211; helps build a Manager similar to the native kube-controller-manager. For difference with OperatorSDK, read here. Metacontroller: lightweight Kubernetes Controller as a Service In CNCF Operator Framework, the Operator SDK supports development using Ansible, Helm and Golang. The author of this post makes a general comparison as follows:\nType Best use caseUnderlying technologyAmt of EffortHelmStateless workloadHelm ChartsMedAnsibleStateless workloadAnsible Roles and PlaybooksMedGolangStateful workloadCode developed in GolangHigh The aforementioned Kiali operator is an example of Operator developed in Ansible. The prometheus operator, is developed in Golang as the workload can be stateful depending on configuration. One needs to know how to develop operator in Golang in order to tackle the most complicated situations. This is requires some serious development effort. The documentation with a quick start section is available here. Even that is not very straightforward. RedHat, the maintainer of the CNCF Operator framework has a good blog post on how to develop an Operator in Golang. The example requires some development knowledge to go through. On my MacOS (Intel) I have to configure the following prerequisites:\nInstall gcc, using command: xcode-select \u0026#8211;install Install the right version of golang. You can find the version here. The MacOS has a version of golang installed already so I had to install version 1.17 and link to it: brew install go@1.17 \u0026amp;\u0026amp; brew link \u0026#8211;force go@1.17 Install operator-sdk with home brew: brew install operator-sdk When you run \u0026#8220;operator-sdk version\u0026#8221;, ensure the result shows a golang version that matches your installation. If you need to push docker image, also connect to docker registry by running: docker login Then we can create our working directory, initialize the repository and create boilerplate code (scaffolding) with these commands:\n$ mkdir wordpress-operator \u0026amp;\u0026amp; cd wordpress-operator $ operator-sdk init --domain digihunch.com --repo github.com/digihunch/wordpress-operator $ operator-sdk create api --group wordpress --version v1 --kind WordPress --resource --controller With the repo initialized, we can go to the section \u0026#8220;Defining the API\u0026#8221; and \u0026#8220;Implementing the Controller\u0026#8221;. The blog post does not cover every code editing needed to bring up wordpress. You are supposed to go to the author\u0026#8217;s repository to fit the changes into your own repo. The author\u0026#8217;s repo has a few more controllers such as common.go and mysql.go. At the end of the lab, you should be able to run the controller and bring up wordpress. I used my own repository for this lab and have made the code changes for this lap in a couple commits. To test locally with the code:\n$ git clone git@github.com:digihunch/wordpress-operator.git $ cd wordpress-operator $ make install run Then we can validate wordpress install from a new terminal as the instruction shows:\n$ kubectl create -f config/samples/wordpress_v1_wordpress.yaml $ minikube service wordpress --url For Developers that requires more details, RedHat has an eBook for Kubernetes Operators, in supplement to the documentation. As DevOps professional, I\u0026#8217;m mainly concerned with understanding how Operator works and using Operators correctly.\nToo many Tools? Now we seem to have too many choice of tools when it comes to deploying workload on Kubernetes. Kustomize and Helm can deploy simple workloads. Operator can deploy stateful workloads, as well as keep the workload status in check. Further, we have FluxCD and ArgoCD based on GitOps workflow.\nWhen assessing a tool, we should think about the complexity of the workload deployed. If it is a single stateless workload, Kustomize or Helm should be sufficient. If it is not very simple but still stateless, we can consider using Helm charts developed by the community. For multiple workloads, we can build our own top-level chart to combine existing sub-charts created by the community.\nHelm is essentially a package manager. It does not follow controller pattern and therefore will not monitor the current status of deployment. Helm has other limitations compared to Operator. For example, as a templating scheme, it reaches limitation when dealing with complex logic, even with the help of its helper functions. It is also hard to reason through the template code when we have to troubleshoot a deployment. Refer to this blog post for the author\u0026#8217;s experience with Helm.\nIf we want our deployment to be fully declarative and continuous, then we will follow the Operator pattern by using a Kubernetes Operator. When we have many workloads of different levels of complexity, we can combine them with GitOps tool. Operator is one of the underlying technologies behind GitOps.\nWorkload profileJust InstallationInstallation and Maintain StatusSingle stateless workloadHelm or KustomizeOperator (using Ansible or Helm)Single stateful workloadHelm or KustomizeOperator (using Golang)Multiple workloadsHelm (e.g. build parent chart)GitOps in combination with Operator, Helm and Kustomize The table above helps refine deployment requirement. It\u0026#8217;s not a recommendation, but rather a model of analyzing deployment requirement.\nPrevious PostAutoscaling on Kubernetes Platform Next PostKnative Serving Introduction ","date":"2022-04-07T09:39:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-operator.webp","permalink":"/2022/04/kubernetes-operator/","title":"Kubernetes Operator"},{"content":"\nIntroduction The concept of autoscaling on Kubernetes platform dates from the era where virtualization first became widespread and the overhead of provisioning a new server became lightweight through the use of cloud-init. With public cloud, customers operate on usage-based billing. Autoscaling allows workload to scale down during idle times to reduce cost, and scale up during peak time to meet the demand of business traffic. Vertical autoscaling replaces a VM with one of higher capacity, which is usually interruptive. Horizontal autoscaling adds or removes VMs to adjust capacity, and works in conjunction with load balancing mechanism to assign load to a specific target in the group. Unless otherwise specified, we simply refer to horizontal autoscaling as autoscaling. Depending on what triggers autoscaling, it can be metrics based or event driven.\nMetric-based autoscaling is based on VM (or Pod) metrics in the autoscaling group. The metrics are mostly about CPU usage, memory, IOPS, number of connections, etc. For example, when the average CPU utilization across all VMs in the last five minutes hits 70% threshold, then the scaler introduces a new VM into the autoscaling group. The trigger can factor in a variety of metrics. Advanced autoscaling APIs can also support lifecycle hooks, i.e. custom activities upon creation of new VMs during scale-up, or upon deletion of existing VMs during scale-down. Other aspects of custom behaviours include a cool-off period, i.e a no-activity window after the previous scaling activity. Since scaling activities are re-active. Using real-time metrics as a trigger of scaling is not always a good idea. For example, a buggy order processing program may consume 100% of CPU due to an infinite loop, or 99% of memory due to memory leak. Metric-based scaling may fire off even though there is currently no order pending in the queue. Event-driven approach is more flexible. Event can fire from any type of source. For example, in Kubernetes, when scheduler fails to schedule a Pod due to constraints, it is an event This event can trigger scaling. In some case, a metric hitting a threshold fires an event. For example, scale up when size of order queue reaches 20. In this sense, metric-based autoscaling is a special case of event-driven autoscaling. With Kubernetes, let\u0026#8217;s examine node group autoscaling (aka cluster autoscaling) and workload autoscaling (Pod autoscaling). Node Group Autoscaling Cluster autoscaler is the mechanism to auto-scale node groups for Kubernetes. As per its documentation, any metric-based cluster/node group autoscalers are NOT compatible with CA. They are also not particularly suitable for Kubernetes in general. Take AKS for example, the events to trigger scale-up and scale-down are as below:\nThe cluster autoscaler component can watch for pods in your cluster that can\u0026#8217;t be scheduled because of resource constraints. The cluster then automatically increases the number of nodes. The cluster autoscaler decreases the number of nodes when there has been unused capacity for a period of time. Pods on a node to be removed by the cluster autoscaler are safely scheduled elsewhere in the cluster. Both are in essence event driven. The behaviours can be fine-tuned with a number of parameters as below:\nscan-interval scale-down-delay-after-add scale-down-delay-after-delete scale-down-delay-after-failure scale-down-unneeded-time scale-down-unready-time scale-down-utilization-threshold max-graceful-termination-sec balance-similar-node-groups expander: random, most-pods, least-waste, priority skip-nodes-with-local-storage skip-nodes-with-system-pods max-empty-bulk-delete new-pod-scale-up-delay max-total-unready-percentage max-node-provision-time ok-total-unready-count The parameters above constitute the autoscaler profiler, and are effective if cluster autoscaler is enabled. For many implementations, cluster autoscaler can be enabled and disabled even after the cluster has been created, and the parameters can be changed. The overhead of provisioning a new node should not be overlooked, because that is usually the window that a Pod needs to wait to get scheduled. As stated in CA\u0026#8217;s FAQ, the main purpose of CA is to get pending pods a place to run, instead of pre-emptively accommodating to increasing workload.\nThe delay in pod scheduling while adding a new node can be controlled to a certain degree with one of the two workarounds below:\nWith HPA or KEDA, set lower threshold so the workload level scaling acts more aggressive than the increase of demand. This buys some buffer time Use a tool to puff up utilization, such as cluster overprovisioner, which deploys pods that request enough resources to reserve virtually all resources for a node consume no actual resources use a priority class that causes them to be evicted as soon as any other Pod needs it. In practice the cluster autoscaler setup should be conservative and keep node size as stable as it can. For example, a 20 minutes idle-window (low utilization) on a node is not worth the overhead to remove a node and add it back in 20 minute later.\nWhen the cluster do need to scale down by removing a node, one common symptom is failing to scale down because some Pods have nowhere else to schedule to. Here is a list of possible causes as troubleshooting tips.\nIf node scaling should be triggered sparsely, then pod scaling is by design very dynamic. Cloud native applications should assume that pod scaling occurs very frequently.\nIn late 2021, AWS released the open-source project Karpenter for cluster autoscaler. Karpenter addresses some challenges with native Cluster Autoscaler on EKS. Karpenter is gaining momentum and now adding support for other cloud service providers including Azure. Workload Autoscaling Stateless workload are controlled by a Deployment object, which is associated with a replicaSet object. For stateless workload we can use HorizontalPodAutoscaler, or HPA. There is a VerticalPodAutoscaler (VPA) which is much less common. HPA is metrics based with flexible options such as specifying an object, depending on what metrics are available via metrics API. There are two versions of HorizontalPodAutoscaler: autoscaling/v1 and autoscaling/v2. The latter supports scaling policies, such as adjusting downscale stabilization window, and limiting scale down rate. No matter which API version, the metric-based triggers in HPA are fairly limited.\nWe already know that metrics are not always the best indicator to trigger scaling. We need an option to trigger scaling based on the status of other components such as queue size. KEDA (Kubernetes Event Driven Autoscaling) is a great option to consider for horizontal workload scaling. KEDA works with HPA, and significantly enriches trigger options. Apart from metrics, KEDA can use a number of external mechanisms as triggers, for example:\nRabbitMQ/Kafka/SQS: scale based on queue size Azure Log Analytics: scale based on a kusto query result against Azure Log Analytics AWS CloudWatch, Azure Monitor: scale based on metrics from Azure Monitor/AWS CloudWatch Azure Pipelines: scale based on agent pool queues of Azure Pipeline Elasticsearch: scale based on elasticsearch query result Kubernetes Workload: scale based on the count of running pods of a specified workload MSSQL, MySQL, Postgres, Cassandra: scale based on a query result Prometheus: scale based on prometheus query result KEDA is a single-purpose and lightweight component. With KEDA, we don\u0026#8217;t need to explicitly define HPA. It allows us to select from a longer list of triggering mechanisms for our auto scaler. We shall not underestimate the work needed to select the most suitable trigger because having an incorrect trigger (e.g. bad metrics) is costly. Let\u0026#8217;s take Java applications as an example. Java workload operates in a JVM inside of the container. JVM request the entire heap size from operating system. The garbage collection activities also consumes a good portion of CPU cycles. This pattern makes CPU and memory metrics inaccurate as an indicator for scaling activity. Because of this we need to find out what is the best scaler for Java application, based on understanding of how the entire solution stack works as a whole.\nThe other aspect that KEDA beats HPA is its ability to scale to 0. This can be helpful when a service is idle most of the time but cannot shut down.\nKEDA lab let\u0026#8217;s use Kafka as an example to configure KEDA for a dummy workload. We create a mock cluster using Kind with a simple configuration file. Then, let\u0026#8217;s start with the following dummy workload with replica count set to 1:\napiVersion: v1 kind: Namespace metadata: name: workload --- apiVersion: apps/v1 kind: Deployment metadata: name: aks-helloworld-one namespace: workload spec: replicas: 1 selector: matchLabels: app: aks-helloworld-one template: metadata: labels: app: aks-helloworld-one spec: containers: - name: aks-helloworld-one image: neilpeterson/aks-helloworld:v1 ports: - containerPort: 80 env: - name: TITLE value: \u0026#34;Welcome to Azure Kubernetes Service (AKS)\u0026#34; --- apiVersion: v1 kind: Service metadata: name: aks-helloworld-one namespace: workload spec: type: LoadBalancer ports: - port: 80 selector: app: aks-helloworld-one We need to install KEDA and Kafka using Helm:\nhelm repo add kedacore https://kedacore.github.io/charts helm install keda kedacore/keda -n keda --create-namespace helm repo add bitnami https://charts.bitnami.com/bitnami helm install kafka bitnami/kafka -n kafka --create-namespace --set volumePermissions.enabled=true --set replicaCount=3 Watch for all Pods to come up. Also read the notes from Kafa installation and confirm the Kafka service address. Now, we will apply KEDA scaled object, defined as below:\napiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: kafka-scaledobject namespace: workload spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: aks-helloworld-one pollingInterval: 10 cooldownPeriod: 30 idleReplicaCount: 0 minReplicaCount: 2 maxReplicaCount: 5 fallback: failureThreshold: 3 replicas: 1 advanced: restoreToOriginalReplicaCount: true horizontalPodAutoscalerConfig: behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 100 periodSeconds: 15 triggers: - type: kafka metadata: bootstrapServers: kafka-0.kafka-headless.kafka.svc.cluster.local:9092 consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics topic: test # Optional lagThreshold: \u0026#34;5\u0026#34; offsetResetPolicy: latest The field definitions are on KEDA deploy documentation. Items under Kafka trigger are on the trigger documentation. In this lab we set the idelReplicaCount to 0. It will scale up with average lag of all partitions reaching 5. In the next few steps, we\u0026#8217;ll mock up some messages posted to the Kafka topic named \u0026#8220;test\u0026#8221; for consumer group my-group. We can watch deployment size grow along with the growth of lags.\nTo emulate Kafka client activity, we can spin up a Kafka test Pod:\nkubectl run kafka-client --restart=\u0026#39;Never\u0026#39; --image docker.io/bitnami/kafka:2.8.1-debian-10-r73 --namespace kafka --command -- sleep infinity kubectl exec --tty -i kafka-client --namespace kafka -- bash From within the Pod, we can leverage the client-side scripts located in /opt/bitnami/kafka/bin/. For example, to post message to a topic (e.g. named test):\nkafka-console-producer.sh --topic test --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092 The Helm installer also gives the command with broker list. To consume messages from a topic (e.g. test) to a given consumer group:\nkafka-console-consumer.sh --topic test --bootstrap-server kafka.kafka.svc.cluster.local:9092 --group my-group We can have two command terminals, post test messages on one terminal and watch it consumed nearly immediately on the other terminal.\nThe Kafka trigger documentation suggests that the number of replicas will not exceed the number of partitions on a topic when a topic is specified. To make this lab work, we need to have set 5 partitions:\nkafka-topics.sh --alter --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test --partitions 5 kafka-topics.sh --describe --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test Once we confirm five partitions, we can spin up two command terminals, one to produce message and the other to consume messages. If working, we can stop the consumer and use the command below to watch for the lag for each partition. kafka-consumer-groups.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --describe --group my-group Now we can artificially trigger scaling by increasing average lag. We keep posting messages on the producer (each carriage return posts a message), and we can check the lag after posting:\nGrowth of average lags The size of deployment starts with 0 as defined in the scaled object. As the average exceeds 5, we can see deployment size growing.\nGrowth of deployment size This lab is an oversimplified scenario to illustrate the idea of scaling. Kafka is a typical queue construct and other queue configuration such as RabbitMQ or AWS SQS works in very similar ways. Real life use case involves more aspects to consider, such as multiple topics, and authentication.\nConnection triggered wake-up KEDA uses Events to scale workload from zero to one (wake up). There is no way to scale (wake up) based on an incoming web request. In many cases, such as serverless configuration, we need to scale the deployment size from zero to N once the service receives incoming web request. This is not supported by KEDA. By definition KEDA uses Events to wake up. There is an HTTP-add-on for KEDA still at beta but it is trying to address this problem. This page shows the design. Suppose a service has scaled down to zero, the followings will happen to wake it up:\nThe incoming request is routed to an interceptor behind the service interceptor keeps track of number of pending HTTP request The scaler periodically watches for the size of the pending queue on the interceptor Based on the queue size, the scaler reports scaling metrics as appropriate to KEDA As the queue size increases, the scaler instructs KEDA to scale up as appropriate The periodical check activity is the key to make it work and also what makes it a pseudo-trigger. Summary Fine-tuning autoscaling is important to the performance of workload on Kubernetes. At node level, we briefed on cluster autoscaler and suggest that we only use it sparsely. At pod level, we introduced native HPA as well as KEDA, with an example. We also discussed KEDA has limitations and the HTTP-add-on. In the next post, we\u0026#8217;ll explore Knative\u0026#8217;s autoscaling capability.\nPrevious PostIstio Operation Gotchas Next PostKubernetes Operator ","date":"2022-03-28T13:14:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-autoscaling.webp","permalink":"/2022/03/autoscaling-in-kubernetes-from-metric-based-to-event-driven/","title":"Autoscaling on Kubernetes Platform"},{"content":"In this post I discuss a few aspects when putting istio in operation.\nInstallation Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to Istio\u0026#8217;s perceived complexity. Currently, the recommended installation methods are istioctl or Helm. Using Istio Operator manifest (with out istioctl) is discouraged. As to the Helm chart installer, it was once deprecated (around early 2020), but was later re-introduced in the fall of 2021. This back-and-forth had caused some aversion. As of Nov 2021, their re-introduced Helm charts version dropped alpha tag. The Helm repo consists of separate Helm charts, for Istio CRD (base), control plane (istiod), each gateway and CNI respectively. A typical deployment therefore requires multiple Helm Releases (example here). Istio document still considers Helm support as alpha, so I assume the most reliable method to install Istio is istioctl. The istioctl utility can be used with many options to customize Istio install. For example, we can supply a YAML declaration input (to -f switch) to customize installation behaviours. The YAML file declares a CRD. Two different CRDs have been used: IstioOperator and IstioControlPlane. According to this blog and this post, since Istio 1.5 in early 2020, we\u0026#8217;re supposed to IstioOperator CRD exclusively. The IstioControlPlane CRD is left only for legacy support. As stated in the current documentation: The\u0026nbsp;istioctl\u0026nbsp;command supports the full\u0026nbsp;IstioOperator\u0026nbsp;API\u0026nbsp;via command-line options for individual settings or for passing a yaml file containing an\u0026nbsp;IstioOperator\u0026nbsp;custom resource (CR).\nWith IstioOperator CRD, we still have a number of options to tweak the install behaviours. Here is a summary of potential options:\nuse IstioOperator API via IstioOperator CRD (without using \u0026#8220;values\u0026#8221; or overlay fields) specify an attribute value in argument, including a pre-built profile e.g. \u0026#8211;set meshConfig.accessLogFile=/dev/stdout, \u0026#8211;set profile=demo use K8sObjectOverlay by using \u0026#8220;k8s/overlays/patches\u0026#8221; field in IstioOperatorCRD use Helm API by using \u0026#8220;values\u0026#8221; field in IstioOperatorCRD The key-value specified in \u0026#8211;set switch overrides the same key-value supplied in the IstioOperator CRD. So option 2 overrides option 1. The value for profile can also be empty if you\u0026#8217;d rather start from scratch. However too many \u0026#8211;set switches makes the command wordy so we should build our own IstioOperator CRD\nFor option 4, the document hyper-links Helm API to a section from version istio 1.4, and I appears to exist only for legacy (pre-2020 Helm support) compatibility. Option 3 (K8sObjectOverlay) would be helpful when a field cannot be conveniently customized with option 1 and we have to patch the object like in Kustomization.\nSo the most practical approach is IstioOperator CRD for per-component customization, potentially with K8sObjectOverlay. No matter which option, istioctl compiles the installation manifest before applying it against Kubernetes API. This manifest can be previewed using \u0026#8220;istioctl manifest\u0026#8221; command, so that you can take a look before installation.\nThank you Isito for so much confusion just to land on a working installation method. Below is the content of az-istio-operator.yaml file that I use for my installation:\napiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: name: istio-install-customization spec: profile: default hub: docker.io/istio tag: 1.13.1 revision: 1-13-1 namespace: istio-system meshConfig: accessLogFile: /dev/stdout outboundTrafficPolicy: mode: \u0026#34;REGISTRY_ONLY\u0026#34; components: pilot: k8s: hpaSpec: maxReplicas: 7 minReplicas: 1 nodeSelector: beta.kubernetes.io/os: linux ingressGateways: - name: istio-ingressgateway #namespace: istio-ingress enabled: true label: istio: ingressgateway k8s: hpaSpec: maxReplicas: 11 minReplicas: 1 serviceAnnotations: service.beta.kubernetes.io/azure-load-balancer-internal: \u0026#34;true\u0026#34; service.beta.kubernetes.io/azure-load-balancer-internal-subnet: \u0026#34;my-lb-subnet\u0026#34; egressGateways: - name: istio-egressgateway enabled: true I can preview the install, run the install and validate installation status:\n$ istioctl manifest generate -f az-istio-operator.yaml $ istioctl install -f az-istio-operator.yaml -y --verify $ istioctl verify-install -f az-istio-operator.yaml $ kubectl -n istio-system get IstioOperator installed-state-istio-install-customization-1-13-1 -o yaml | less Note that in the IstioOperator declaration I marked the revision. This is helpful when I run multiple versions of control plane (e.g. during upgrade). I can check revisions of istiod with:\n$ istioctl x revision list We can delete installed istio components\n$ istioctl x uninstall -f az-istio-operator.yaml $ istioctl x uninstall --revision 1-11-5 In practice, it is helpful to use separate operators each for a different component (pilot, ingressGateways, egressGateways). This makes maintenance and upgrade easier. Debugging The documentation has a page for common problems that one needs to be familiar with. It covers not only problems, but also steps to troubleshoot each kind of problem (e.g. authorization policy).\nAlthough istioctl is pretty confusing as an installation tool, it is a good utility for many troubleshooting activities. We should probably add its path to PATH environment variable and add the export command (e.g. export PATH=\u0026#8221;$HOME/istio/bin:$PATH\u0026#8220;) to .zshrc or .bashrc. Istioctl has a few useful subcommands, some of which are only available as experimental and therefore needs to be following an x. Some common commands are given below:\nTo analyze Istio problems:\n$ istioctl analyze -n istio-system To look at proxy configuration of an Envoy instance at different levels, use proxy-config sub-command or pc for shorthand:\n$ istioctl proxy-config \u0026lt;clusters|listeners|routes|endpoints|bootstrap|log|secret|stats\u0026gt; $ istioctl pc cluster deploy/istio-ingressgateway -n istio-system # see what Envoy cluster an ingress gateway knows about $ istioctl proxy-config log deploy/httpbin --level \u0026#34;rbac:debug\u0026#34; $ istioctl pc log \u0026lt;pod_name\u0026gt; -n \u0026lt;namespace\u0026gt; --level connection:debug $ istioctl pc secret -n istio-system deploy/istio-ingressgateway # check certificates loaded to a gateway $ istioctl proxy-config listeners deploy/istio-ingressgateway -n istio-system # query envoy listener configuration $ istioctl pc routes deploy/istio-ingressgateway -n istio-system --name http.8080 # query envoy route configuration The last two commands set logging level to the specified workload. If we want to set logging level at mesh level, we can use these commands:\n$ istioctl admin log --level authorization:debug $ istioctl admin log To look at synchronization status of each envoy in the mesh, use proxy-status sub-command, or ps for shorthand:\n$ istioctl ps # ensure data plane is in sync If an item for a workload shows STALE instead of SYNCED, it means that the configuration has not been pushed from control plane to that instance of Envoy proxy. Check if the Istio configuration change is valid. If the system is newly installed and there is no ingress or egress gateway resources declared, the RDS column for ingress or egress may show \u0026#8220;NOT SENT\u0026#8221;. The far right column displays the version of istiod connected.\nTo describe applied istio config:\n$ istioctl describe \u0026lt;pod|service\u0026gt; $ istioctl describe po workload1 -n my-workload # detect misconfigurations on workload To view dashboard:\n$ istioctl dashboard \u0026lt;envoy|grafana|prometheus\u0026gt; To check authorization policy on a Pod,\n$ istioctl x authz check mypod -n workload To validate istio configuration in a file:\n$ istioctl validate -f resource_authorization_policy.yaml Sometimes we need to turn on access logging just on the envoy proxy on the Gateway Pod. In that case, we will need to apply Istio\u0026#8217;s Envoy filter object. This filter is applied to Pods labelled as gateway. It patches the existing filter chain with the additional defined in the manifest:\napiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: hcm-gw-access-log namespace: istio-system spec: workloadSelector: labels: istio: ingressgateway configPatches: - applyTo: NETWORK_FILTER match: context: GATEWAY listener: filterChain: sni: demo.digihunch.com filter: name: \u0026#34;envoy.filters.network.http_connection_manager\u0026#34; patch: operation: MERGE value: typed_config: \u0026#34;@type\u0026#34;: \u0026#34;type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\u0026#34; access_log: - name: envoy.access_loggers.file typed_config: \u0026#34;@type\u0026#34;: \u0026#34;type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog\u0026#34; path: /dev/stdout format: \u0026#34;[%START_TIME%] \\\u0026#34;%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\\\u0026#34; %RESPONSE_CODE% %RESPONSE_FLAGS% \\\u0026#34;%UPSTREAM_TRANSPORT_FAILURE_REASON%\\\u0026#34; %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \\\u0026#34;%REQ(X-FORWARDED-FOR)%\\\u0026#34; \\\u0026#34;%REQ(USER-AGENT)%\\\u0026#34; \\\u0026#34;%REQ(X-REQUEST-ID)%\\\u0026#34; \\\u0026#34;%REQ(:AUTHORITY)%\\\u0026#34; \\\u0026#34;%UPSTREAM_HOST%\\\u0026#34; %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\\n\u0026#34; The object above enables access logging on the Gateway Pods only, without impacting other Envoy proxies.\nMulti-tenancy In large enterprises, Istio is usually installed by a platform/operation team. The entire platform is shared by multiple application teams. Istio should not be seen as a responsibility of a single party. It is necessary to break down the Istio CRDs and define the responsibility of each CRD. For example:\nGateways are placed in the istio-system namespace (or a dedicated istio-ingress namespace in some case). Platform team can decide whether a Gateway is considered a shared infrastructure, or each tenant (application team) uses their own Gateway. In the servers/hosts field of Gateway declaration, add namespace before host to suggest that the routing behaviour of that host must be defined in a certain namespace. Virtual Services can be managed in two models as well. They can be managed individually and are all placed in each tenant namespace. Alternatively in a shared responsibility model, Virtual Service can be created in istio-system namespace, and the processing of each match can be delegated to a Virtual Service in each tenant namespace, using the Delegate feature of Virtual Service. Another field that can help with multi-tenancy is the \u0026#8220;gateways\u0026#8221; field, you can specify a value of mesh to indicate the virtual service is available to the entire mesh. Destination Rules are usually placed in each Tenant\u0026#8217;s workspace Peer Authentication also depends on the configuration. If a mesh level configuration is enforced, it is easier for the Platform Team to manage it and this should be the setup for new clusters. If for historical reasons Peer Authentication is enforced per tenant namespace, it can be delegated to each application team. It can make communication troubleshooting more complex. Control Plane and observability workloads such as Kiali are usually the responsibility of Platform team. Request Authentication and Authorization Policy should be the responsibility of individual application team. In a multi-tenancy management model, Istio related resource should be subject to admission control policy, as well as scrutiny by the security and platform teams. Google Anthos has a good page on the constraint templates for Istio resources.\nAs to restricting traffic between namespaces, apart from the measures from this previous post, we can also use Sidecar CRD to restrict outbound traffic. As its documentation states:\nSidecar\u0026nbsp;describes the configuration of the sidecar proxy that mediates inbound and outbound communication to the workload instance it is attached to. By default, Istio will program all sidecar proxies in the mesh with the necessary configuration required to reach every workload instance in the mesh, as well as accept traffic on all the ports associated with the workload. The\u0026nbsp;Sidecar\u0026nbsp;configuration provides a way to fine tune the set of ports, protocols that the proxy will accept when forwarding traffic to and from the workload. In addition, it is possible to restrict the set of services that the proxy can reach when forwarding outbound traffic from workload instances.\nThe documentation page also includes two examples. The first is a sidecar at the mesh level that restricts outbound traffic to the same namespace that the sidecar is in, and the istio-system namespace:\napiVersion: networking.istio.io/v1beta1 kind: Sidecar metadata: name: default namespace: istio-config spec: egress: - hosts: - \u0026#34;./*\u0026#34; - \u0026#34;istio-system/*\u0026#34; The second example overrides the mesh level default above, and allows egress traffic to three specified namespaces:\napiVersion: networking.istio.io/v1beta1 kind: Sidecar metadata: name: default namespace: prod-us1 spec: egress: - hosts: - \u0026#34;prod-us1/*\u0026#34; - \u0026#34;prod-apis/*\u0026#34; - \u0026#34;istio-system/*\u0026#34; As to inbound control, we can expose a Virtual Service to other namespaces by using the exportTo field to specify which other namespaces the Virtual Service should be exported to. If no namespaces are specified then the virtual service is exported to all namespaces by default.\nSometimes we want a virtual services to expose to both outside of the mesh via Ingress, and within the mesh, and we hope to use the same hostname. For this requirement, we can use the ServiceEntry CRD. ServiceEntry enables adding additional entries into Istio’s internal service registry.\nPrevious PostService Proxy – from Nginx to Envoy Next PostAutoscaling on Kubernetes Platform ","date":"2022-03-19T11:09:00-04:00","image":"/wp-content/uploads/2025/04/feature-istio-ops.webp","permalink":"/2022/03/istio-operation-gotchas/","title":"Istio Operation Gotchas"},{"content":"Update (Nov 20, 2022): 1. Envoy\u0026#8217;s configuration schema can be hard to get used to. It is lacking examples because the documentation is mostly generated. Use its examples directory to find real-life configuration examples. 2. the Envoy implementation in the example project has been reverted in favour of Nginx.\nEnvoy proxy is the underlying technology for Istio, as well as a number of other service mesh products, such as AppMesh (AWS), Consul (Hashicorp) and OpenServiceMesh (Azure). Most of the capabilities of Isito is ultimately provided by Envoy proxy. Envoy has a page outlining its differences with similar technologies. I decided to take a look into Envoy by replacing Nginx with it.\nRate limiting and Circuit Breaker In most SDLC, it is application developers that create backend APIs or server applications. Most developers specializes in application features, and cannot fathom all the nuances with the TCP/IP network stack. Nginx allows them to push network concerns (non-business features) to a dedicated proxy to handle the dynamics in network connection. Nginx can be configured as both a reverse proxy (handling incoming connection on behalf of the process) and a forward proxy (handling outgoing connection on behalf of the process). This is the prototype of sidecar pattern, an important idea behind service mesh. For example, when a sudden increase in connection to the server-side application, the server process could be either unresponsive (refer to \u0026#8220;the queuing knee\u0026#8220;, and Little\u0026#8217;s Law), or OOM killed. When such interruptions are not automatically recovered, a downtime is caused. This traditionally requires some congestion control strategy for TCP/IP queue but two features provided by a network proxy can help circumvent this situation: rate limiting, and circuit breaking. Rate limiting keeps more requests above threshold from entering the queue. Circuit breaker releases downstream pressure by cutting out existing in-queue request. Nginx added both over the years but Circuit breaker still remains a premium feature exclusive to Nginx Plus. Envoy on the other hand has them free when it was launched.\nEnvoy also supports other advanced traffic management such as traffic shaping, and mirroring. It is on top of those features that Istio introduces its own abstraction such as virtual service, destination rules to its users. In that sense, we can think of Istio as a configurator (control plane) for Envoy proxy (data plane), similar to Ansible to Nginx proxy instances.\nDynamic Configuration via API I used Nginx previously with traditional environment and loved its flexibility. As the system grows, I started to feel the pain of management overhead. With one of the production system, there were 25 + instances of Nginx each running on a VM and I managed configuration files with Ansible. Ansible pushes out configuration files and triggers a reload from each Nginx instance. In the cloud-native era where Pods are ephemeral, this kind of overhead would snowball to an unmanageable level. Envoy was designed for cloud-native applications, with all these kinds of problems in mind. Envoy has dynamic configuration. The majority of the configurations can be pulled from xDS API, or file system. Updating configuration drains connections gracefully without runtime having to reload the file. The idea of centrally managing Nginx instances with Ansible, also evolved into the concept of control plane.\nTLS origination In the Orthweb project, I used Nginx to proxy TLS and HTTP traffic, and performed TLS termination on both ports. This is know as TLS offloading. The traffic between the proxy and the upstream service takes place in the clear, even though they do not travel across different network interfaces in most cases. For a true end-to-end encryption, it is helpful to also encrypt the traffic between proxy and upstream server. This requires the capability of securing TCP traffic to upstream server. With Nginx, the ability to secure HTTP traffic to upstream server is offered in open-source. The ability to secure TCP traffic is available in Nginx Plus, or with self-compiled binary. In Envoy, both are available using the UpstreamTlsContext configuration.\nMore pros In another post, I also discussed Nginx as a LDAP proxy to front services such as Kibana and Nifi. It requires a proxy service (ldap-auth in this case), to defer auth to third party. Envoy has this capability using a filter with extension for external authorization. Istio also exposes this capability, an enabler for the configuration proposed in my previous post. Envoy also uses WebAssembly for its extensibility.\nAnother useful feature is protocol detection. It can use filters to detect protocol (TLS or regular TCP) and route traffic to predefined destination.\nPerformance wise, this benchmark from 2018 ran a comparison among the popular options where Envoy leads by a margin.\nObservability (logging, metrics and tracing) are well supported in Envoy. User can configure format of logs that takes effect immediately. There are many metrics that works with Prometheus and they are expandable using filters. On the tracing side, Envoy supports integration with jaeger, zipkin and datadog.\nBasics of Envoy The configuration of Envoy is more involving. There is an Envoy course by Tetrate, as well as two blog entries for envoy 101: Envoy as gateway proxy and File-based dynamic configuration. Another good way to get started is the Sandboxes projects, which covers a number of different areas of configuration. The admin port (by default at port 9901. Istio\u0026#8217;s default is 15000) provides helpful information. If we need to turn on debug on some features, we can do so with curl:\ncurl -X POST http://localhost:9901/logging?client=debug Stats are exposed at the same port:\ncurl -X GET http://localhost:9901/stats When packets are received at a listener, the is first processed by listener filters. Then, depending on filter match, one or more network filter chains will further process the packet, including further actions, as illustrated below:\nEnvoy supports dynamic configuration, which uses a set of discovery services (xDS) APIs. Some of the important xDS APIs include:\nLDS (Listener Discovery Service) \u0026#8211; allows you to add listeners dynamically while Envoy is running RDS (Route Discovery Service) \u0026#8211; allows you to dynamically update routes for HTTP connection managers CDS (Cluster Discovery Service) \u0026#8211; allows you to update cluster definitions dynamically EDS (Endpoint Discovery Service) \u0026#8211; allows you to add or remove endpoints dynamically Secret DS The relation can be illustrated in this diagram below:\nThis post from Tetrate has more examples.\nNginx to Envoy For the advantages of Envoy, I decided to migrate from Nginx to Envoy on my Orthweb project. Using Envoy as service proxy is not where Envoy is mostly used (as sidecar), but it is how Envoy was originally used at Lyft to replace ELB in 2015.\nThe original Nginx configuration was referenced in this old blog post. The Envoy setup also covers both TCP (DICOM) and HTTP (HTTPS) traffic. For HTTP traffic, it also encrypts the traffic to upstream. Below is what it looks like:\nadmin: address: socket_address: { address: 0.0.0.0, port_value: 9901 } static_resources: listeners: - name: https_listener address: socket_address: address: 0.0.0.0 port_value: 443 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager codec_type: AUTO stat_prefix: ingress_http route_config: name: local_route virtual_hosts: - name: app domains: - \u0026#34;*\u0026#34; routes: - match: prefix: \u0026#34;/\u0026#34; route: cluster: service-https http_filters: - name: envoy.filters.http.router transport_socket: name: envoy.transport_sockets.tls typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificates: - certificate_chain: {\u0026#34;filename\u0026#34;: \u0026#34;/etc/ssl/certs/site.pem\u0026#34;} private_key: {\u0026#34;filename\u0026#34;: \u0026#34;/etc/ssl/certs/site.pem\u0026#34;} - name: dicomtls_listener address: socket_address: address: 0.0.0.0 port_value: 11112 filter_chains: - filters: - name: envoy.filters.network.tcp_proxy typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy stat_prefix: downstream_cx_total cluster: service-dicomtls transport_socket: name: envoy.transport_sockets.tls typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificates: - certificate_chain: {\u0026#34;filename\u0026#34;: \u0026#34;/etc/ssl/certs/site.pem\u0026#34;} private_key: {\u0026#34;filename\u0026#34;: \u0026#34;/etc/ssl/certs/site.pem\u0026#34;} validation_context: allow_expired_certificate: true trusted_ca: {\u0026#34;filename\u0026#34;: \u0026#34;/etc/ssl/certs/site.pem\u0026#34;} require_client_certificate: false clusters: - name: service-https type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: service-https endpoints: - lb_endpoints: - endpoint: address: socket_address: address: orthanc-backend port_value: 8042 transport_socket: name: envoy.transport_sockets.tls typed_config: \u0026#34;@type\u0026#34;: type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext - name: service-dicomtls type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: service-dicomtls endpoints: - lb_endpoints: - endpoint: address: socket_address: address: orthanc-backend port_value: 4242 layered_runtime: layers: - name: static_layer_0 static_layer: envoy: resource_limits: listener: https_listener: connection_limit: 1000 overload: global_downstream_max_connections: 5000 From Nginx to Envoy, to achieve nearly the same functionalities, it takes 100 lines of configuration instead of less than 30. The configuration also appears more abstract, which is one of the cons of Envoy to consider before the migration.\nUpdate from Jan 2025 \u0026#8211; To align with what most uses in Orthanc community, the project switched back to using Nginx. The envoy configuration is kept here.\nPrevious PostIstio External Authorization via OIDC Next PostIstio Operation Gotchas ","date":"2022-03-09T21:36:00-04:00","image":"/wp-content/uploads/2025/04/feature-nginx2envoy.webp","permalink":"/2022/03/from-nginx-to-envoy-proxy/","title":"Service Proxy – from Nginx to Envoy"},{"content":"Istio service mesh allows application developers to offload non-core features to infrastructure layer. We explored authentication and authorization with Istio in a basic lab. In this post we continue to explore its capabilities with OIDC integration. This capability is made available thanks to the CUSTOM action in authorization policy, supported since the release of 1.9. It enables any workload on Istio to integrate with an external IAM solution.\nAuthorization\nPolicyAuthorization\u0026#8230;UserUserIstio\nIngressIstio\u0026#8230;External AuthorizationExternal\u0026#8230;OAuth2-ProxyOAuth2-ProxyApp\nHelloWorldApp\u0026#8230;GCPGCPdemo1.digihunch.comdemo1.digihunch.comText is not SVG \u0026#8211; cannot display\nThe rest of this post, provides the step-by-step instruction to configure OIDC integration, based on Istio\u0026#8217;s External Authorization use case. My work is influenced by two blog posts from jetstack and elastisys on similar topic, with my own additions, simplifications and clarifications. In this lab I use my own DNS hostname demo1.digihunch.com and to follow along you need to bring your own hostname as well. The prerequisites of this lab is summarized as below:\nAn identity provider: we use Google in the lab but it can be anything with OIDC capabilities (e.g. Azure AD, Facebook). Read my old post for more details on OIDC. A public IP address for Cluster\u0026#8217;s Ingress service: Since we use Google as identity provider on the Internet, we need our service to be exposed on the Internet because the OIDC integration involves a two-way HTTP redirect. Public DNS resolution to the service\u0026#8217;s public IP: during authentication, the identity provider will call back with JWT. So we need our hostname demo1.digihunch.com resolved to the service\u0026#8217;s public IP address. This is also required in other scenarios. For example, cert-manager needs to automatically configure X.509 certificate using Let\u0026#8217;s Encrypt. An application workload: this is the tenant workload without its own IAM capability. We use a dummy workload base on a hello-world image, for simplicity. The files required in this lab is in istio-oidc repo under the envoy directory. It is also important to understand the Authentication Code flow in OIDC authorization code flow to make sense of the integration between OIDC provider and Istio on our platform. For example, the client application needs a one-time registration with identity provider. Let\u0026#8217;s start with this configuration.\nConfigure Identity Provider We need a Google account with a GCP project and log on to APIs \u0026amp; Services to register our application. At the console, from the left side bar, click on \u0026#8220;OAuth consent screen\u0026#8221; and create an App. Provide the followings:\nStep 1. OAuth consent screen. Under App Information, provide App name and User support email. Step 1. OAuth consent screen. Under App domain, provide Application home page Step 1. OAuth consent screen. Under Authorized domains, provide an Authorized domain. Step 3. Test users. Add the emails for Test users. The summary page looks like this:\nOAuth Consent Screen Then on the sidebar, click on Credentials. We create new credentials for OAuth 2.0 Client ID. Select Web application and add \u0026#8220;https://demo1.digihunch.com/oauth2/callback\u0026#8221; as an Authorized redirect URI. The credential created includes a Client ID and a Client secret, which can be downloaded as a JSON file. Here is my screen:\nAPI Credential Note that the redirect URIs cannot be localhost. Because the browser will consume the redirect URI and it has no clue where localhost is. It must resolve to the public IP of the ingress.\nWe need to keep the Client ID and Client secret for later use. According to Google\u0026#8217;s documentation, there is a discovery document from which we can understand the keys in their Open ID configuration. We need the value of two keys (issuer and jwks_url) to use later in the configuration.\nConfigure Infrastructure Services My real-quicK-cluster project has a few different ways to quickly bring up a cluster. For this lab, I use Azure CLI command to create a simple, three-node cluster:\n$ az aks create \\ -g AutomationTest \\ -n orthCluster \\ --node-count 3 \\ --enable-addons monitoring \\ --generate-ssh-keys \\ --vm-set-type VirtualMachineScaleSets \\ --network-plugin azure \\ --network-policy calico \\ --tags Owner=MyOwner Then we configure kubeconfig credential to connect to the cluster, so that kubectl and helm CLI commands to connect to the newly created cluster.\n$ az aks get-credentials --resource-group AutomationTest --name orthCluster Then we will install the infrastructure services and workload. We can start with loading environment variables for use later:\nOIDC_DISCOVERY=$(curl \u0026#34;https://accounts.google.com/.well-known/openid-configuration\u0026#34;) OIDC_ISSUER_URL=$(echo $OIDC_DISCOVERY | jq -r .issuer) OIDC_JWKS_URI=$(echo $OIDC_DISCOVERY | jq -r .jwks_uri) COOKIE_SECRET=$(openssl rand -base64 32 | tr -- \u0026#39;+/\u0026#39; \u0026#39;-_\u0026#39;) WEB_HOST=\u0026#34;demo1.digihunch.com\u0026#34; CLIENT_ID=\u0026#34;ThisIsTheClientIDFromGoogle\u0026#34; CLIENT_SECRET=\u0026#34;ThisIsTheClientSecretFromGoogle\u0026#34; For WEB_HOST, CLIENT_ID, CLIENT_SECRET, you need to bring your own values. The COOKIE_SECRET value is randomly generated. The variables OIDC_ISSUER_URL and OIDC_JWKS_URI are parsed from Google OpenID configuration and they should remain static. We can validate the value with echo commands. Also, to use helm v3 later, we need to add the repositories we need. These steps can be skipped if they have been performed in the client environment:\n$ echo $OIDC_ISSUER_URL https://accounts.google.com $ echo $OIDC_JWKS_URI https://www.googleapis.com/oauth2/v3/certs $ helm repo add jetstack https://charts.jetstack.io $ helm repo add istio https://istio-release.storage.googleapis.com/charts $ helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests $ helm repo update Now, with Helm ready, we can install Cert Manager, Istio CRD, Control Plane, Gateways as well as OAuth2-Proxy using Helm. We run each of the following commands from the envoy directory, where the required files are stored:\n$ helm install cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --create-namespace \\ --version v1.7.1 \\ --set installCRDs=true $ helm install -n istio-system istio-base istio/base --create-namespace $ helm -n istio-system install istiod istio/istiod -f istiod-values.yaml --wait $ helm -n istio-system install istio-ingress istio/gateway -f ingress-gateway-values.yaml $ kubectl -n istio-system get po $ kubectl create ns oauth2-proxy \u0026amp;\u0026amp; kubectl label ns oauth2-proxy istio-injection=enabled $ helm install -n oauth2-proxy \\ --version 6.0.1 \\ --values oauth2-proxy-values.yaml \\ --set config.clientID=$CLIENT_ID \\ --set config.clientSecret=$CLIENT_SECRET \\ --set config.cookieSecret=$COOKIE_SECRET \\ --set extraArgs.oidc-issuer-url=$OIDC_ISSUER_URL \\ --set extraArgs.whitelist-domain=$WEB_HOST \\ oauth2-proxy oauth2-proxy/oauth2-proxy $ kubectl -n oauth2-proxy get pods -l app=oauth2-proxy Note that the istiod installation step uses a value file that includes extensionProviders. This is where we tell Istio to connect to external authorization provider:\nmeshConfig: accessLogFile: /dev/stdout extensionProviders: - name: oauth2-proxy envoyExtAuthzHttp: service: oauth2-proxy.oauth2-proxy.svc.cluster.local port: 4180 includeRequestHeadersInCheck: - cookie headersToUpstreamOnAllow: - authorization headersToDownstreamOnDeny: - set-cookie When we install istio ingress gateway, we need to expose port 80 as well as 443. Port 80 is used in ACME protocol for certificate configuration. We also installed oauth2-proxy with some configurations from oauth2-proxy-values.yaml as well as some argument set imperatively.\nOnce configuration is successful, we should be able to confirm that oauth2-proxy service is running. We should also be able to tell the External IP address of the Ingress:\n$ kubectl -n oauth2-proxy get svc $ kubectl -n istio-system get service istio-ingressgateway -o jsonpath=\u0026#39;{.status.loadBalancer.ingress[0].ip}\u0026#39; Now we can go to our DNS configuration portal, to populate the DNS A-record for demo1.digihunch.com with this IP address:\nAdd DNS A-record It may take several minutes to a couple of hours, for the A-record update/creation to take effect, depending on the TTL. Sit tight and query DNS with nslookup command, until it is set. In the next step, we will need successful DNS resolution for certificate configuration.\nInstall workload and configure Certificate Now we install the workload, along with the necessary constructs such as Gateway CRD, Virtual Service, and certificates. We can use kubectl kustomize command to preview the changes, then apply the changes with -k:\n$ kubectl kustomize demo $ kubectl apply -k demo When you have your own DNS hostname, modify the kustomization.yaml accordingly before applying. This command uses Kustomization to apply numerous YAML manifests in the demo directory, including the following activities:\nCreate a Namespace named demo Create Deployment and Service Set up PeerAuthentication for the mesh Configure Ingress class, ClusterIssuer and Certificate using cert-manager Configure Virtual Service and Gateway This configures everything we need and we need to verify from several aspects. The demo Service is a ClusterIP service exposed within the cluster at port 80. Then we check the status of certificate:\n$ kubectl -n istio-system get certificate $ kubectl -n istio-system describe certificate demo The column of READY should have a value of True. Check the logs from cert manager Pods if that\u0026#8217;s not the case. Common reasons include:\nThe automatic ACME validation is just not ready The automatic ACME validation is still waiting for DNS resolution The Istio ingress gateway port 80 is not open for ACME validation The let\u0026#8217;s encrypt server applies rate limiting Note that for the ACME server, we can use staging server\u0026#8217;s URL or productions. The former provisions a certificate that may come off as insecure as the CA is not fully trusted by browser. The latter doesn\u0026#8217;t have this issue, but the server applies rate limiting. For more details of how validation with ACME work, check out my previous post.\nOnce the certificate is applied, our website is open to any visitor with HTTPS access:\nThen we will use Request Authentication and Authorization Policy to tighten up the access by requiring visiting user to log in with Google identity.\nRequestAuthentication and AuthorizationPolicy We configure a CUSTOM action in the AuthorizationPolicy, and specified the provider by name oauth2-proxy. The YAML manifests for RequestAuthentication and AuthorizationPolicy looks like this:\napiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: istio-ingressgateway namespace: istio-system spec: jwtRules: - issuer: https://accounts.google.com jwksUri: https://www.googleapis.com/oauth2/v3/certs selector: matchLabels: app: istio-ingressgateway --- apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: istio-ingressgateway namespace: istio-system spec: action: CUSTOM provider: name: oauth2-proxy rules: - {} selector: matchLabels: app: istio-ingressgateway Note that we cannot apply Request Authentication and Authorization Policy before we confirm that the Certificate has been created. This is because when letsencrypt use HTTP resolver to provision certificate with ACME protocol, the request from letsencrypt should get to the validating URI without being asked to provide a Google identity. So do not perform this step yet before confirming the previous step:\n$ kubectl apply -f oidc-auth.yaml Now launch your browser in incognito mode and browse to our URL (https://demo1.digihunch.com) and you will be re-redirected to log on to Google (accounts.google.com). Once logged on, you will have access to the site. If you run into errors, check the oauth2-proxy log from Pod stdout, which should give a reason of 4xx errors. If oauth2-proxy log indicates no activity, confirm if the request has been forwarded to the proxy. Check the service object of the proxy and make sure it is exposed to the correct port, as indicated in the meshConfig. To check mesh config, examine the configmap named istio in the namespace of istio-system. To check if Istio\u0026#8217;s authorization is unable to speak with oatuh2-proxy, inspect the log of istiod Pod.\nTo clean up the lab, remove the app registration from Google, and then delete the cluster from Azure:\n$ az aks delete -g AutomationTest -n orthCluster In this lab we used a very open AuthorizationPolicy. We can polish it up with more conditions such as:\napiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: istio-ingressgateway namespace: istio-system spec: action: CUSTOM provider: name: oauth2-proxy rules: - when: - key: request.auth.audiences values: - $CLIENT_ID selector: matchLabels: app: istio-ingressgateway Under the rules section, we can explore the claims from the JWT and make creative use of it to achieve our specific goals for IAM.\nBottom line In this lab, we registered our app with Google and obtained client ID and secret for it to identify our application. We configured oauth2-proxy as our IAM application, carrying the credential validated by Google. We also configured Istio to delegate authorization to oauth2-proxy as external authorization provide, making up an entire OIDC integration.\nThe third party service (IdP) we integrate with is GCP, which natively supports OIDC protocol. However, if the third party IdP does not support OIDC natively, such as Active Directory via LDAP, then we would need one more proxy to sit between OAuth2 and the external IdP service. One good choice is the Dex connector. Another more complex yet powerful alternative connector is Key Cloak. Consider Dex or KeyCloak as an OIDC proxy, when: The external IdP does not support OIDC; or We choose not to use the OIDC capability of the external IdP because we want to managed OIDC centrally in our own proxy The external identity provider does support OIDC, but the configuration is managed by a different team (e.g. IAM team) other than the platform team AuthorizationPolicy is the key piece in this integration, and it is executed at the http filter in envoy sidecar proxy. It can be done with alternatives to OAuth2-Proxy such as the authservice project. A slightly different approach to implement OIDC integration is to use leverage extensibility of WebAssembly (a sandboxing technology to extend Envoy), which is suggested in this example for WasmPlugin. However, Wasm is still considered experimental.\nPrevious PostIstio Lab – Authentication and Authorization Next PostService Proxy – from Nginx to Envoy ","date":"2022-02-25T01:29:00-04:00","image":"/wp-content/uploads/2025/04/feature-istio-external-auth.webp","permalink":"/2022/02/istio-external-authorization/","title":"Istio External Authorization via OIDC"},{"content":"My previous blog discussed as service mesh what Istio can offer in terms of authentication and authorization capabilities. Istio can authenticate an incoming HTTP request, ensuring the JWT issued has not been tampered somewhere in the middle. The fields in the JWT allows for more flexibilities at the point of authorization. This combination allows Istio to integrate with identity providers that can issue JWT.\nWe\u0026#8217;ve also discussed how JWT works, and pointed out that the two key element of request authentication is the JWT (payload signed with private key) itself as well as the JWK (carrying public key). In this post, we will test it in a lab. To start with this lab, we need a test cluster (e.g. Minikube) with Istio installed.\nPreparation I have not find a native Bash way to produce JWT. We will do that with python packages python_jwt and jwcrypto in Python3. Let\u0026#8217;s install the modules and import them in Python environment.\n$ python3 -m pip install python_jwt jwcrypto datetime $ python3 \u0026gt;\u0026gt;\u0026gt; import python_jwt as jwt, jwcrypto.jwk as jwk, datetime Now we\u0026#8217;re in the Python3 shell with needed modules loaded. We can take the following steps to produce the JWT as well as the JWK:\nRSAkey = jwk.JWK.generate(kty=\u0026#39;RSA\u0026#39;, size=2048) private_key = RSAkey.export_private() public_key = RSAkey.export_public() raw_payload = {\u0026#39;iss\u0026#39;:\u0026#39;digihunch.com\u0026#39;,\u0026#39;sub\u0026#39;:\u0026#39;DIGIHUNCH\u0026#39;,\u0026#39;role\u0026#39;:\u0026#39;reader\u0026#39;,\u0026#39;permission\u0026#39;:\u0026#39;read\u0026#39;} ## use private key to generate jwt token. HTTP request will bear this token jwt_token = jwt.generate_jwt(raw_payload, jwk.JWK.from_json(private_key), \u0026#39;RS256\u0026#39;, datetime.timedelta(minutes=50)) print(jwt_token) ## The JWKS keeps public key and is referenced by Istio RequestAuthentication object jwks=\u0026#39;{\u0026#34;keys\u0026#34;:[\u0026#39;+public_key+\u0026#39;]}\u0026#39; print(jwks) # Helpful command to print the key in PEM format: ## RSAkey.export_to_pem(private_key=False) # Helpful command to verify JWT token: ## header, claims = jwt.verify_jwt(jwt_token, jwk.JWK.from_json(public_key), [\u0026#39;RS256\u0026#39;]) To complete this lab, we need the values of jwt_token and jwks.\nNext in the preparation is a local cluster, istio with metallb installed, which is covered in a previous post. We should be able to get the ingress IP address:\n$ export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath=\u0026#39;{.status.loadBalancer.ingress[0].ip}\u0026#39;) $ echo $INGRESS_HOST We create our own namespace and use the httpbin application:\n$ kubectl create ns web $ kubectl label namespace web istio-injection=enabled $ kubectl apply -n web -f httpbin.yaml $ kubectl apply -n web -f httpbin-gateway.yaml $ curl -I -XGET $INGRESS_HOST/headers The curl command above (without any token) should return HTTP 200 code, indicating that no authorization token is needed to connect to the service. Note that the Pod for httpbin has the label app=httpbin which will be used in the request authentication. Also note that traffic is served over a named port called http in the Service object for http, which will implicitly enable HTTP based conditions for authorization policies we will build later. Request Authentication Let\u0026#8217;s create a request authentication object with the following manifest:\napiVersion: \u0026#34;security.istio.io/v1beta1\u0026#34; kind: \u0026#34;RequestAuthentication\u0026#34; metadata: name: jwt-req-authn namespace: web spec: selector: matchLabels: app: httpbin jwtRules: - issuer: \u0026#34;digihunch.com\u0026#34; jwks: | ## jwks output from previous step ## Replace the last line with the jwks output from the preparation step and store it to jwt-req-authn.yaml. It should look like this:\njwt-req-authn.yaml Then apply it to the web namespace:\n$ kubectl -n web apply -f jwt-req-authn.yaml Now the request authentication resource is applied to the httpbin workload. We first test it with a random authentication token and it should be denied of 401 error:\n$ curl -I -XGET $INGRESS_HOST/headers --header \u0026#34;Authorization: Bearer randomstring\u0026#34; HTTP/1.1 401 Unauthorized www-authenticate: Bearer realm=\u0026#34;http://192.168.64.16/headers\u0026#34;, error=\u0026#34;invalid_token\u0026#34; content-length: 79 content-type: text/plain date: Sun, 13 Feb 2022 16:13:32 GMT server: istio-envoy x-envoy-upstream-service-time: 31 Then we take the jwt_token from the preparation step and present it to the request authentication resource by sending an HTTP request with the appropriate authorization token. It should return an HTTP 200 code this time:\n$ curl -I -XGET $INGRESS_HOST/headers --header \u0026#34;Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDQ3NzE0NjUsImlhdCI6MTY0NDc2ODQ2NSwiaXNzIjoiZGlnaWh1bmNoLmNvbSIsImp0aSI6Im1faUhla2pNbWRmUmlsWEdCaTFBR3ciLCJuYmYiOjE2NDQ3Njg0NjUsInBlcm1pc3Npb24iOiJyZWFkIiwicm9sZSI6InJlYWRlciIsInN1YiI6IkRJR0lIVU5DSCJ9.sVppwmvDqKvSsVdB05a_mDHymZq7Okvnwu-caTywXQgsUvOA6HfaySp_WXMyTp1HQ4WcTqKE4frZm7QNtrZsPso4bdD_4mEDYTswTCWhblaPy236NJBEH3ilB2BVySBVQKsjyxd94F1KV24SFWiR6lUxk52wKKE3ipBwR79jPizhAu9xxrfJ2Lfi5ypNa_kjBdJi63KCt2Y0eW94Fjq3PZs4ZalHJyaXYSx5Gxyei5f7QdpEOBpvs13mSdi9RqkgVQOjE0V1uBRpMckMyZs-IijknJcSu4fkrjgfNmXsrm__-vlM9UjUl2Jlj0x8bRC8l20IZ6t1ml-GFkwj39JC0g\u0026#34; HTTP/1.1 200 OK server: istio-envoy date: Sun, 13 Feb 2022 16:13:54 GMT content-type: application/json content-length: 589 access-control-allow-origin: * access-control-allow-credentials: true x-envoy-upstream-service-time: 17 In the example above, jwtRules can be used with other keys such as jwksUri to reference the jwks by Uri. More fields in JWTRules can be found here.\nNow we have tested three curl commands:\nWithout authentication token at all: http server (istio-envoy) returns 200 code. With an invalid authentication token: http server (istio-envoy) returns 401 code for error. with a valid authentication: http server (istio-envoy) returns 200 code. So we have the capability to validate token, but it is not yet mandatory to present the token. We can change this behaviour by tweaking authorization policy.\nAuthorization Policy According to Istio documentation, to restrict access to authenticated requests only, this should be accompanied by an authorization rule. We start with the following policy:\napiVersion: \u0026#34;security.istio.io/v1beta1\u0026#34; kind: \u0026#34;AuthorizationPolicy\u0026#34; metadata: name: auth-pol namespace: web spec: selector: matchLabels: app: httpbin action: ALLOW rules: - from: - source: requestPrincipals: [\u0026#34;*\u0026#34;] Applying the manifest above to namespace web to it applies to workload httpbin. Then we $ curl -I -XGET $INGRESS_HOST/headers HTTP/1.1 403 Forbidden content-length: 19 content-type: text/plain date: Sun, 13 Feb 2022 16:29:58 GMT server: istio-envoy x-envoy-upstream-service-time: 35 The requestPrincipals clause makes it mandatory to present a token. The RequestAuthentication validates the token. We can beef up the authorization policies by adding claims to the conditions, for example:\napiVersion: \u0026#34;security.istio.io/v1beta1\u0026#34; kind: \u0026#34;AuthorizationPolicy\u0026#34; metadata: name: auth-pol namespace: web spec: selector: matchLabels: app: httpbin action: ALLOW rules: - from: - source: requestPrincipals: [\u0026#34;*\u0026#34;] to: - operation: methods: [\u0026#34;GET\u0026#34;] when: - key: request.auth.claims[iss] values: [\u0026#34;digihunch.com\u0026#34;] - key: request.auth.claims[role] values: [\u0026#34;reader\u0026#34;] Both the to and when conditions are for HTTP traffic only, and we must tell Istio to inspect the traffic as HTTP, which is done implicitly with named ports on the Service object. Refer to this common problem from Istio\u0026#8217;s documentation. The request principals, if a none wildcard value is specified, will be a SPIFFE format identity, the same one used for peer authentication, as discussed in the previous post.\nThe when clause above contains two key-value pairs. The first looks for the value of a standard claim (iss), the second for a custom claim (role). In the preparation step, we created the claims with those claims in Python and they will match the condition here. Should any of the conditions above not match, a 403 (Forbidden) error code will be returned by the workload\u0026#8217;s istio-envoy proxy.\nVerify mTLS connection In some cases, we need to audit whether the TLS traffic between workloads actually take place in mTLS. As I was developing my korthweb project, I don\u0026#8217;t find a straightforward way of validating TLS. We can validate that mTLS mode on a workload using the following istio CTL command:\n$ istioctl x describe pod my-workload-pod -n [namespace] The output shows Effective PeerAuthentication and Applied PeerAuthentication. This verifies configuration. But how can we ensure that traffic are indeed using mTLS? For most of mTLS traffic, we can use Kiali\u0026#8217;s observability feature. In Graph, we need to ensure \u0026#8220;Security\u0026#8221; is checked in the display drop-down. The pad lock will indicate the traffic is mTLS. To get reliable results, we have to artificially create some live traffic between workloads (e.g. curl from one Pod to another) so Kiali can pick up the update.\nValidate mTLS traffic in Kiali Unfortunately, even artificial traffic does not make Kiali the most reliable way to detect mTLS. There are three other approach to verify TLS traffic.\nThe first approach is to let Envoy proxy emit TLS related traffic. We can apply the following annotation line to a Pod, to tell its Envoy proxy to emit measurements related to tls_inspector:\nsidecar.istio.io/statsInclusionPrefixes: \u0026#34;tls_inspector,listener.0.0.0.0_15006\u0026#34; The metrics will be exposed to Envoy\u0026#8217;s admin port (15000 on istio-proxy) with the path /stats. For example:\n$ kubectl -n orthweb exec orthanc-7f4c9b759-lxnrb -c istio-proxy -- curl localhost:15000/stats | grep tls_inspector % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 22194 0 22194 0 0 21.1M 0 --:--:-- --:--:-- --:--:-- 21.1M tls_inspector.alpn_found: 13 tls_inspector.alpn_not_found: 0 tls_inspector.client_hello_too_large: 0 tls_inspector.connection_closed: 0 tls_inspector.read_error: 0 tls_inspector.sni_found: 13 tls_inspector.sni_not_found: 0 tls_inspector.tls_found: 13 tls_inspector.tls_not_found: 0 We still need some live traffic to bump up the measurement. Comparing between tls_found and tls_not_found is a good way to determine if a Pod is receiving both TLS and plaintext traffic in PERMISSIVE mode.\nThe second approach to tell mTLS is via the connection_security_policy metric label. It is set to mtutual_tls if the connection is in mTLS. We need the dashboard for Prometheus:\n$ istioctl dashboard prometheus In the prometheus dashboard, click on \u0026#8220;Graph\u0026#8221; at the top, then search for \u0026#8220;istio_tcp_connections_closed_total\u0026#8221; for \u0026#8220;istio_tcp_connections_opened_total\u0026#8220;, the result should include a metrics called connection_security_policy which is labelled as mtutual_tls, as illustrated below:\nmTLS metric For more about collecting and querying metrics from Prometheus, check out Istio\u0026#8217;s documentation here and here.\nThe third approach is to utilize the AUDIT feature of Authorization Policy. When a rule in Authorization Policy has a source with namespace or notNamespace field, it requires the incoming connection to have an SPIFFE identity and use mTLS. We can set the Authorization Policy\u0026#8217;s action to AUDIT and use RBAC access logging, the same way we would do to troubleshoot RBAC access issues. From the log we can check RBAC failures due to missing SPIFFE identity.\nPrevious PostIstio Authentication and Authorization Next PostIstio External Authorization via OIDC ","date":"2022-02-13T13:21:13-04:00","image":"/wp-content/uploads/2025/04/feature-istio-lab.webp","permalink":"/2022/02/istio-lab-authentication-and-authorization-in-jwt/","title":"Istio Lab – Authentication and Authorization"},{"content":"Applications running on Kubernetes platform seeks to offload common non-business features to the platform. Istio helps Kubernetes bridge that gap. It can enforce mTLS communication, which is known as Peer Authentication. It can help with two other things with the use of JWT token: when a web request presents a JWT token, it can validate whether it is authentic. Then, it can use the claims in JWT token to drive authorization decision on whether the specific request is allowed or denied. Both will use Istio CRDs.\nPeer Authentication Istio can enforce mTLS for TCP traffic between Pods. According to its documentation, enforcing mTLS at mesh level is as simple as applying a Peer Authentication resource to the root-level namespace:\napiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: istio-system spec: mtls: mode: STRICT The role of mTLS is so Pods can validates each other\u0026#8217;s identity and then encrypt the TLS traffic in between. Each workload must first have an identity and Envoy proxy addressed this issue by adopting SPIFFE framework. It gives each workload an identity in the format of \u0026lt;TRUST_DOMAIN\u0026gt;/ns/\u0026lt;NAMESPACE\u0026gt;/sa/\u0026lt;SERVICE_ACCOUNT\u0026gt;. For example: spiffe://cluster.local/ns/myapp-dev/sa/default. It is also important to understand that only Pods with injected Envoy sidecar have SPIFFE workload identity and therefore is able to speak in mTLS. For new services, this is usually not an issue. For migrating workload without sidecar, a Pod without sidecar may connect with one in the mesh (with sidecar) if the mtls mode is PERMISSIVE in Peer Authentication. Otherwise, the connect is reset at layer 4 with the following error:\ncurl: (56) Recv failure: Connection reset by peer command terminated with exit code 56 Therefore, it is advisable to start with PERMISSIVE mode for a precautionary migration of workload to mTLS. With mTLS all effective at the mesh level, there is no need to natively configure TLS between services.\nThe SPIFFE identity used in PeerAuthentication can also be used in Request Authorization as rule conditions. I will discuss request authentication before request authorization. To understand request authentication, let\u0026#8217;s first warm up on JWT.\nJSON Web Token JSON Web Token (JWT, RFC 7519) is a format to carry JSON payload with optional signature and/or encryption. It can be thought of as a document (in JSON format) with signature for web servers to exchange information. The signature portion makes it friendly for document consumers to validate the authenticity. It is also URL-safe, and thereby adopted in web-browser SSO context, to pass identity of an authenticated user between and identity provider and a service provider.\nJWT enables token-based authentication, a significant improvement from traditional session-based authentication. The traditional session-based authentication can be illustrated as below:\nsession-based authentication This authentication model has major drawbacks. First, a mechanism to validate the authenticity of Cookie is missing. Second, the server has to keep the session information, making itself not stateless, unless a state store such as memcached is introduced.\ntoken-based authentication In token-based authentication such as using JWT, a token is issued. The authenticity of the token are validated before the server provides data, and it can be validated by any backend server. The payload of JWT consists of claims, which are statements about an identity (such as name, role, email). There are custom claims as well as standard reserved claims, such as iss (issuer), sub (subject), aud (audience), iat (issued at time), exp (expiration time), and jti (JWT ID). When a program produces a JWT, it turns the raw payload into standardize payload by adding the required reserved claims and may sort the claims alphabetically. The JWT consists of three parts with a period as delimiter:\nThe third part is a signature in the format of JWS (JSON Web Signature, RFC 7515) for the JWT consumer to validate its authenticity. The first and second parts, as you can tell, are the claims in the document. Their base64 encoding can be decoded with no effort and should therefore be considered exposed. Although JWT addresses the authenticity of information, it does not intend to address the confidentiality of the payload at HTTP layer. The payload should not carry sensitive information and should always be used with secure HTTPS port. To tackle this issue, there is JWE (JSON Web Encryption, RFC 7516) which is an implementation similar to JWT which also encrypts the payload.\nSome IAM protocols are built on top of JWT. For example, the OpenID Connect specification also defines a set of standard claims that it uses while still allow custom claims.\nRequest Authentication Istio can perform request authentication using its CRD. It is important to distinguish request authentication and user authentication. In user authentication, the identify provider typically looks up an identity store and compares password hash results to check whether the identity of the visiting user is authentic or not. This is outside of Istio\u0026#8217;s capability but many off-the-shelf solution excels at it, such as Azure AD. Once the user\u0026#8217;s identity is validated by identity provider, and a JWT is issued for downstream service providers to consume. Istio\u0026#8217;s CRD can front the service provider and validate that the presented JWT is authentic. It authenticates the identity of a request (as truly issued by the trusted issuer without being tampered). This process does not involve checking user\u0026#8217;s identity, even though user\u0026#8217;s identity could be stored in the payload by the JWT issuer. Istio uses the RequestAuthentication CRD to perform this function. The JWT issuer signs with its private key and stores the signature in the JWT. When it is presented to Istio, Istio\u0026#8217;s RequestAuthentication CRD needs the public key of the issuer in order to validate the JWT. The public key usually comes in as a JWK (JSON Web Key, RFC7517), a format convertible to and from PEM format. The JWK can be provided either inline in the RequestAuthentication\u0026#8217;s YAML manifest, or via a URI. Below is an example of a basic RequestAuthentication declaration:\napiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: httpbin namespace: foo spec: selector: matchLabels: app: httpbin jwtRules: - issuer: \u0026#34;issuer-foo\u0026#34; jwksUri: https://example.com/.well-known/jwks.json In this example (from the documentation), the jwtRule requires that the issuer be issuer-foo, and the JWK (containing public key) is provided by a given URI address. Istio will pass the authentication once the signature in the presented JWT is verified with the JWK.\nAuthorization Policy Istio\u0026#8217;s Authorization Policy by itself can operate at both TCP or HTTP layers and is enforced at the envoy proxy. The result is an ALLOW or DENY decision, based on a set of conditions at both levels. If the traffic is HTTP then you should consider use some HTTP level information as it provides a lot more flexibility. Even when operating at HTTP layer, AuthorizationPolicy does not have to work in conjunction with RequestAuthentication. The rules can use path, methods, etc to drive an authorization decision, for example:\napiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: authz-policy-orthanc spec: selector: matchLabels: app: orth action: ALLOW rules: - from: - source: namespaces: [\u0026#34;istio-system\u0026#34;,\u0026#34;orthweb\u0026#34;] to: - operation: methods: [\u0026#34;GET\u0026#34;,\u0026#34;POST\u0026#34;,\u0026#34;PUT\u0026#34;,\u0026#34;HEAD\u0026#34;,\u0026#34;DELETE\u0026#34;] ports: [\u0026#34;8042\u0026#34;] - from: - source: namespaces: [\u0026#34;istio-system\u0026#34;,\u0026#34;orthweb\u0026#34;] to: - operation: ports: [\u0026#34;4242\u0026#34;] The claims in the JWT payload can also be used to drive authorization decision, as exemplified in the Istio documentation, by using a when keyword in a rule and specifying the claim as a key:\nwhen: - key: request.auth.claims[iss] values: [\u0026#34;https://accounts.google.com\u0026#34;] The when clause requires that the iss claim in the JWT must carry a specific value in order to ALLOW the HTTP request. While the claims in JWT is just an additional factor to drive authorization decision, using authenticated information to drive authorization decision makes the overall workflow more secure, and should therefore be used when applicable.\nWhen using AuthorizationPolicy CRD, keep in mind:\nUse correct selectors so it only applies to workloads labelled as such. Without selector the policy takes effect in the entire namespace which is a recipe for issues When multiple policies (each with multiple rules) are applied to the same workload, be aware of the policy precedence. Troubleshooting may get tricky. For an AuthorizationPolicy to use HTTP fields in rules, it first needs to identify the traffic as HTTP. To tell Authorization Policy to treat the traffic as HTTP, we need to understand Protocol Selection. In a nutshell, we can name the port as http or http-* in the Service manifest of the workload. Otherwise all HTTP-based rules will be missed. For troubleshooting, we can check authorization policies effective on a Pod with:\n$ istioctl x authz check orthanc-6c9679d8c7-2ttlf -n dev-orthweb This returns the effective policies but does not necessarily indicate which rule is matched when a request is denied or allowed. To find out further information, you will need to follow Istio FAQ to set RBAC logging to debug, and then monitor the log in the istio-proxy sidecar.\nApart from HTTP fields, path, authenticated claims in JWT, Istio Authorization can also integrate with an Open Policy Agent (OPA) to drive actions, in advanced use cases.\nBottom line While Istio itself does not perform user authentication, its support of JWT in RequestAuthentication allows a workload to integrate with external identity provider. This capability, along with creative use of claims in JWT, also empowers authorization capability. Previous PostTraffic Segmentation on Kubernetes Platform Next PostIstio Lab – Authentication and Authorization ","date":"2022-02-05T23:04:00-04:00","image":"/wp-content/uploads/2025/04/feature-istio-auth.webp","permalink":"/2022/02/authentication-and-authorization-with-istio/","title":"Istio Authentication and Authorization"},{"content":"When operating Kubernetes as a platform for multiple tenants, one of the concerns is controlling the network traffic. This is sometimes referred to as traffic segmentation. This initiative involves a broad range of technical topics from networking to containerization. By no means I am an expert on each of those topics. I have however developed some best practices in how to break down this challenge and hence bringing the thought into this post.\nTenant Isolation Kubernetes has the concept of namespace to logically separate resources allocated for each tenant. Each tenant only operates within their given namespaces. The isolation of computing resources such as CPU and memory can be managed via ResourceQuota objects, and they are enforced at the kernel level, leaving networking isolation the main discussion in the topic of tenant isolation. If the platform hosts a lot of stateful workload then we also needs to address tenant isolation at the storage layer. In this post we focus on the network aspect of resource isolation, aka traffic segmentation.\nControlling network traffic can require a significant amount of efforts depending on the goal. That is why we need to first assess the multi-tenancy models:\nSoft multi-tenancy: usually a platform is shared by multiple teams within the same organization. Tenants are incentivized to be good neighbours. Hard multi-tenancy: usually a platform shared by multiple customers from different organizations. There is no trust between different tenants, or between tenant and platform operator. Reality may sits somewhere in between, but we often have to come back to this model when making a technical decision, because it determines the degree of tenant isolation, or the amount of effort we are willing to put in on tenant isolation. At the tough end, is zero-trust network, which usually have the following requirement:\nRequirement 1:\u0026nbsp;All network connections are subject to enforcement (not just those that cross zone boundaries). Requirement 2: Establishing the identity of a remote endpoint is always based on multiple criteria including strong cryptographic proofs of identity. In particular, network-level identifiers like IP address and port are not sufficient on their own as they can be spoofed by a hostile network. Requirement 3: All expected and allowed network flows are explicitly allowed. Any connection not explicitly allowed is denied. Requirement 4: Compromised workloads must not be able to circumvent policy enforcement. Requirement 5: Many Zero Trust Networks also rely on encryption of network traffic to prevent disclosure of sensitive data to hostile entities snooping network traffic. This is not an absolute requirement if private data are not exchanged over the network, but to fit the criteria of a Zero Trust Network, encryption must be used on every network connection if it is required at all. A Zero Trust Network does not distinguish between trusted and untrusted network links or paths. Also note that even when not using encryption for data privacy, cryptographic proofs of authenticity are still used to establish identity. As you can see there\u0026#8217;s a lot of efforts involved in building a zero-trust network. The cost of building a zero-trust network is worth it only when we determines that the overall business requirement demands it.\nPod Networking It is important to understand Pod networking before developing a traffic segmentation strategy. Pod networking has to do with the CNI driver used for the cluster. There are in general two categories:\nOverlay network: Pods are placed on a VXLAN configuration. This is mostly seen in basic Kubenet mode or CNI drives such as Flannel. NAT is required for Pods to communicate across nodes, which might introduce performance issues when deployed at scale. Pods do not use IP address from the host network. Regular network: In this mode Pods are on the same network as the nodes are. For example, Azure CNI assigns Pods with IP address from a given V-Net. The AWS-VPC CNI integrates VPC networking with Pods. Since Pods are on a corporate network, the traffic control must also consider measures at the whole network level. The main benefit of the first approach, is that IP exhaustion is less likely due to the introduction of a VxLAN. The other benefit from a networking perspective is that the Pod networking is born separated from the corporate network. In the second approach, by assigning Pods with a corporate IP address (which brings the risk of IP exhaustion), Pods are also potentially exposed to all corporate traffic at layer 3. To tackle this additional risk, network security group should be used in the V-Net for Azure AKS, or security groups for Pods should be considered with AWS EKS. Although we will discuss Network Policy in the rest of this essay, Network Policy mostly addresses the traffic segmentation issue within a Kubernetes cluster. A Pod placed on the corporate network needs traffic segmentation strategies from the perspective of the whole network.\nAnother network-level traffic segmentation strategy is on the corporate firewall. For example, with AKS you can specify outbound type as user-defined routes (UDR) to direct all outbound traffic through a corporate firewall where traffic will be inspected. There are firewall products dedicated for managing highly dynamic pod traffic from Kubernetes. This strategy can be used in conjunction with network security groups.\nNetwork Policy Kubernetes’s default behaviour is to allow traffic between any two pods in the cluster network. This is undesirable. NetworkPolicy is the native Kubernetes construct for platform operators and application developer to control network traffic at layer 3/4. It uses namespace and pod selectors, and is defined based on allow rules, which is good for general use. Further to the native Network Policy, you can adopt third party policies for advanced features. For example, Azure has Azure Network policy (works for Azure CNI only) and Calico Network policy (works for Calico CNI, Azure CNI or Kubenet). The third party network policies usually provides advanced features such as:\nDeny rules multiple types of endpoints in addition to Pods, for example, VMs, network interfaces which can be useful in network-level traffic control ordering and priority of rules Flexible matching rules Calico network has a page that summarizes its features and how it extends the Kubernetes NetworkPolicy. Below is an example of a Calico\u0026#8217;s network policy:\napiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-tcp-6379 namespace: production spec: selector: color == \u0026#39;red\u0026#39; ingress: - action: Allow protocol: TCP source: selector: color == \u0026#39;blue\u0026#39; namespaceSelector: shape == \u0026#39;circle\u0026#39; destination: ports: - 6379 It is as self-explanatory as Kubernetes Network Policy. No matter which kind of network policy, this approach takes effect at layer 3/4. The rules are eventually implemented in the kernel on the node (Iptables). The management of this layer is usually by the platform team and they need to have some application knowledge.\nAuthorization at Application Layer Traffic above layer 4 is considered application layer traffic. At application layer, the decision to allow or deny a request is by definition an authorization decision. Another layer of protection can be placed at layer 4 is mTLS which ensures that each request to have an identity. The authorization can be built in the application, but it is also very common to offload these functions to the service mesh layer. For example, Istio has constructs such as PeerAuthentication, Request Authentication and Authorization Policy. We will those in more details in a few coming blog posts. Below is a simple example of Istio\u0026#8217;s Authorization Policy:\napiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: \u0026#34;details-viewer\u0026#34; namespace: default spec: selector: matchLabels: app: details action: ALLOW rules: - from: - source: principals: [\u0026#34;cluster.local/ns/default/sa/bookinfo-productpage\u0026#34;] to: - operation: methods: [\u0026#34;GET\u0026#34;] The rule is also fairly self-explanatory. Compared to Network Policy, the point of enforcement of these Authorization policies are at the envoy proxy. The management of policies at this layer can be debatable if department boundaries are not clear, but it should in general be owned by personnels with good application knowledge.\nConsistency between Policies In-cluster traffic can be controlled with both Network Policy (Calico or Kubernetes) operating at layer 3-4, and Authorization Policy (Istio) at layer 4-7. This brings another challenge of maintaining consistency between the two types of policies. This is especially challenging when they are managed by different teams in a corporate and therefore many operators for soft multi-tenant platform choose not to implement Network Policy or only implements a baseline.\nSome network solution providers builds a solution for this. For example, Calico has the capability to enforce network policy for Istio. This integration requires some configuration, but the enhanced GlobalNetworkPolicy supports HTTP methods, eliminating the need to define a separate Authorization Policy in Istio and worry about its consistency with NetworkPolicy. The platform build however, still needs to determine who owns this policy construct. Below is an example from Calico documentation:\napiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: customer spec: selector: app == \u0026#39;customer\u0026#39; ingress: - action: Allow http: methods: [\u0026#34;GET\u0026#34;] egress: - action: Allow One of the benefits of using this integration is a unified policy language based on GlobalNetworkPolicy CRD. In the mean time, organization should also develop strategy to ensure that, once Calico is integrated with Istio, then there is no need to separately build authorization policies, which may come in conflict with Global network policy.\nSummary Controlling network traffic is difficult on Kubernetes platform. In this article I proposed a few angles to approach this issue for enterprise clients.\nPrevious PostFluxCD: Continuous Deployment with GitOps Next PostIstio Authentication and Authorization ","date":"2022-01-27T13:54:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-traffic-segmentation.webp","permalink":"/2022/01/traffic-segmentation-on-kubernetes-platform/","title":"Traffic Segmentation on Kubernetes Platform"},{"content":"This post explains why I land on FluxCD GitOps for my project. Let\u0026#8217;s star\nBackground In the Korthweb project, I landed on Istio for the Ingress Gateway technology. I first attempted to expand the orthanc Helm Chart to bring Istio as dependency (sub-chart). One of the external chart for Istio gateway needs to be referenced multiple times (for ingress and egress). However, it cannot even be used as dependency (sub-chart) because of this issue. Istio didn\u0026#8217;t re-introduce Helm Chart as a supported deployment until September 2021. So I\u0026#8217;m not too confident about it.\nThis leads to a second thought of the \u0026#8220;Big Helm Chart\u0026#8221; approach to deploy all tiers in Korthweb. Helm is based on templating, and having two layers of charts brings complexity. I do need to bring a number of Helm charts together, but not necessarily by another Helm Chart. The Kubernetes documentation also mentions Kustomize as an alternative to Helm. Below is a quick exploration of it.\nKustomize A challenge with managing declarative object is to organize numerous manifest files to maintain consistency and readability. Kustomize is a standalone tool to customize Kubernetes objects through a Kustomization file. This page provides a good list of fields that can be used:\nvars and replacements: copy fields from one source into any number of specified targets. namePrefix, namespace, nameSuffix: customize namespace and names. configMapGenerator, secretGenerator, and generatorOptions: create configuration entries from literal, files, or environment variables. resources: indicates another kustomization directory (e.g. as base) patches (also called overlays) add or override fields on resources. patchesStrategicMerge: modifies values in known (loaded) resources images: modifies the name, tags and/or digest for images, without creating patches. openapi: use Kubernetes OpenAPI data to get merge key and patch strategy information about resource types. The base -\u0026gt; overlay pattern ensures readability as well as portability. Typical use pattern is create one base kustomization, and several overlay kustomizations each representing an environment, such as dev, qa and production. Here is a good tutorial. Helm and Kustomize are two approaches to deploy Kubernetes workload. Both tackle the challenge of managing many YAML declarations. Helm is template driven, and is commonly used by application developers as a means of packaging application releases while orchestrating the dependencies. Kustomize follows a base-overlay pattern, and is more commonly used by cluster operators for re-using manifests across multiple environments (e.g. dev, staging and prod).\nIn my use case, my deployment needs both. I need third party Helm chart to configure components such as PostgreSQL for HA. I also need Kustomize for the orthanc workload. With the need for both, there should be a higher level deployment technology that ingrate with both mechanisms.\nGitOps GitOps is originally brought up by Weaveworks in 2017. It is a methodology to deploy workload continuously, using a Git repository as source of truth. The point of GitOps is not about tooling, or specific platform, but rather to ensure the workload state matches the declaration in repository. For example, you can implement GitOps with Ansible for VM environment.\nWhen it comes to managing Kubernetes workload, a GitOps tool must handle the challenge with managing many YAML declarations. Therefore most GitOps tools seek to support many well-adopted mechanisms such as Helm and Kustomize as discussed above, instead of simply taking an enormous amount of raw YAML declarations. The amount of deployment mechanisms supported, is a key indicator of how powerful a GitOps tool is. The most notable tools are ArgoCD and FluxCD. Both are currently CNCF incubating projects. ArgoCD is powerful with many tools supported, such as Kustomize, Helm, Ksonnet, Jsonnet, etc. It also contains a user interface and aims to manage an entire deployment workflow. On the other hand, FluxCD has controllers mainly for Kustomize and Helm, with a jsonnet extension from third party. This post draws a comparison of them (along with JenkinsX) based on earlier versions (from mid 2020). There is even a standard (OpenGitOps) in CNCF landscape but still at Sandbox level.\nFor Korthweb project, I chose FluxCD. It is simple, and provides just enough types of controller for what I do.\nFluxCD The diagram below illustrates the components:\nGit RepositoryGit Rep\u0026#8230;flux-systemflux-s\u0026#8230;Kustomize:\ninfrastructureKustom\u0026#8230;Kustomize:\ndependencyKustom\u0026#8230;Kustomize:\napplicationKustom\u0026#8230;Kustomize:\ndevKustom\u0026#8230;flux-system\nnamespaceflux-sys\u0026#8230;application\nnamespace\ndevapplicat\u0026#8230;flux-system\ncustom resource\ncontrollersflux-syst\u0026#8230;reconcilereconcileText is not SVG \u0026#8211; cannot display\nThe configurations starts with a bootstrapping process, which creates directory in Git repository (if not exist), and installs flux-system components in the target Kubernetes cluster. The sync process starts as soon as bootstrapping is completed. The process in charge of syncing declarations to target cluster, confusingly, is also called Kustomization. Therefore there are two Kustomizations. According to the FAQ on FluxCD website:\nThere are two Kustomization types. the kustomization.kustomize.toolkit.fluxcd.io is a Kubernetes custom resource while kustomization.kustomize.config.k8s.io is the type used to configure a Kustomize overlay. The kustomization.kustomize.toolkit.fluxcd.io object refers to a kustomization.yaml file path inside a Git repository or Bucket source.\nInside of the Git repository, with a kustomization.kustomize.toolkit.fluxcd.io obejct, the flux-system points to Kustomization file (representing kustomization.kustomize.config.k8s.io object) at root level. The kustomization.yaml file organizes resources in the same directory. A kustomize directory may also reference other kustomize directory, forming a hierarchy. Implementation FluxCD has a command \u0026#8220;flux check \u0026#8211;pre\u0026#8221; to check the prerequisite, such as kubectl. The code is stored in the GitOps directory of Korthweb repository.\nTo configure deployment, we need to first create a personal access token. For GitHub, here is the instruction. Export the token to environment variable, and launch bootstrapping:\n$ export GITHUB_TOKEN=xxx_yyy55555XXXodr7ABBBB234CCccw $ flux bootstrap github \\ --owner=digihunch \\ --repository=korthweb \\ --branch=main \\ --personal \\ --path=gitops/environment/dev A deploy key is configured during the bootstrapping process. As soon as bootstrapping is completed, the sync (aka kustomization, or reconciliation) has started, which can be monitored using:\nflux get kustomizations --watch Running this command without \u0026#8211;watch switch returns the overview of all kustomizations. Once reconciliation is completed, it should display something like:\nNAME READY\tMESSAGE REVISION SUSPENDED application True Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tmain/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tFalse dependency True Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tmain/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tFalse flux-system True Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tmain/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tFalse infrastructure\tTrue Applied revision: main/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tmain/98f3c771d4ab23f5cb5fd8c7aee325f6490000c7\tFalse FluxCD introduced a number of CRDS. For example, to check configured source repository, check gitrepositories CRD:\nkubectl get gitrepositories -n flux-system To check the detail of one kustomization (e.g. infrastructure), check kusomization CRD in flux-system namespace:\nkubectl -n flux-system get kustomizations flux-system -o yaml | less Other commonly used custom resources include:\nhelmcharts helmreleases helmrepositories A Helm release can be created imperatively, for example:\n$ flux create source helm istio \\ --interval=1h \\ --url=https://istio-release.storage.googleapis.com/charts $ flux create helmrelease istio-base \\ --interval=1h \\ --release-name=istio-base \\ --target-namespace=istio-system \\ --create-target-namespace=true \\ --source=HelmRepository/istio \\ --chart=base \\ --chart-version=\u0026#34;1.12.0\u0026#34; $ flux create helmrelease istiod \\ --interval=1h \\ --release-name=istiod \\ --target-namespace=istio-system \\ --source=HelmRepository/istio \\ --chart=istiod \\ --chart-version=\u0026#34;1.12.0\u0026#34; \\ --values=istiod-values.yaml However, in a GitOps approach, they should be stored as code (use \u0026#8211;export to export declaration). For example, the infrastructure kustomization keeps HelmReleases for installing Istio and PostgreSQL. This kustomization is referenced by a Flux Kustomization from higher level.\nYou can also manually reconcile one of the kustomizations (or other CRDs) with flux reconcile command, for example:\nflux reconcile kustomization dependency Since Nov 2021, FluxCD (0.20) supports reconciliation based on server-side apply. This increases performance and help address issues such as applying large config map, which is the equivalent of adding \u0026#8211;server-side flag to the kubectl apply command.\nLimitation Troubleshooting the FluxCD repo can be involving and counter-intuitive. I had to commit a lot of changes to the repo because it serves as source of truth. Even though the commits can be made to a branch, it still involves a lot of code pushes. Traditionally I commit a change after testing. In the GitOps workflow, I commit a change then to test.\nMore flexible troubleshooting options are still to be desired. For example, there is no way to run one (FluxCD\u0026#8217;s) Kustomization object at a time (and disable the rest), unless you remove their YAML files from the repo.\nThe next limitation is the ordering of resources in Kustomization. Arguably this is a limitation from Kustomize, instead of FluxCD\u0026#8217;s. For example, I need the following manifest to be executed:\napiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-issuer spec: selfSigned: {} The resource is a CRD that needs to be first installed using Helm chart (cert-manager). However there\u0026#8217;s no way to control the sequence (Helm release executed first before declaration using CRD). Although there\u0026#8217;s some workaround, it is not convenient. Alternatively, we can separate the resource creation and CRD creation into separate kustomization objects with dependency relationship, as suggested in this example.\nSummary Deployment in Kubernetes can get complicated with a lot of manifests. Helm, Kustomize and the like are means to handle the complexity due to numerous manifests. GitOps tools such as FlexCD brings these tools under a single framework, and more importantly, implements the idea of using Git repository as source of truth for continuous deployment.\nPrevious PostKubernetes Admission Control Next PostTraffic Segmentation on Kubernetes Platform ","date":"2022-01-15T18:49:00-04:00","image":"/wp-content/uploads/2025/04/feature-flux-pipeline.webp","permalink":"/2022/01/fluxcd-continuous-deployment-with-gitops/","title":"FluxCD: Continuous Deployment with GitOps"},{"content":"This post discusses admission control, and its implementation \u0026#8211; the OPA Gatekeeper. I also discuss Azure Policy as a different Gatekeeper implementation.\nAdmission Webhooks Admission controller intercepts requests to the Kubernetes API server after the request has been authenticated and authorized, and prior to persistence of the object into etcd store. There are many compiled-in controllers, which can be turned on and off on the node with the arguments of kube-apiserver process. For example, the ImagePolicyWebhook can be enabled with value ImagePolicyWebhook added to the \u0026#8211;enable-admission-plugins switch. Its configuration can be provided via the \u0026#8211;admission-control-config-file switch. In addition to the compiled-in admission plugins (which must be configured for kube-apiserver process on the node), admission plugins can be developed as extensions and run as webhooks configured at runtime. This allows users to configure webhooks via API access, dynamically without having to restart kube-apiserver process on the Node, which is usually hard to do with managed Kubernetes platforms. They are therefore called Dynamic Admission Control.\nYou can define two types of admission webhooks in dynamic admission control: validating admission webhook, and mutating admission webhook. Their interaction with API server can be illustrated in the diagram below:\nThe mutating admission hook takes action to change the API request, whereas the validating admission hook accepts or denies the request. A good example of mutating webhook is Istio\u0026#8217;s sidecar injector. We can view the configuration with this command:\n$ kubectl get MutatingWebhookConfiguration istio-sidecar-injector -o yaml | less From the manifest returned, we can see that in this configuration the request is forwarded to istiod service on port 443, at path /inject for processing. We can also see some matching rules to find the target Pod creation API request.\nValidating webhook can be display with the following call:\n$ kubectl get ValidatingWebhookConfiguration The output of validating webhook is a yes or no decision. We usually use validating webhook in conjunction with a policy engine to decide whether the request should be accepted or denied.\nOpen Policy Agent Open Policy Agent (OPA) is an open-source general-purpose policy engine that applies policies written in Rego language to ingested JSON document and returns a result. It is usually integrated with system which requires a policy engine. For example, Kyverno is a policy engine designed for Kubernetes. Styra (one of the OPA contributors) develops policy engines to integrate with Istio\u0026#8217;s authorization policy. They have online courses on OPA policy authoring and microservice authorization with their product.\nOPA is build to be a general-purpose, unified way of solving policy and authorization problem. With microservice authorization, the activities includes decision making (determine action based on input, aka Policy Decision Point, PDP), and decision enforcement (issue 400 code or 200 code depending on decision, aka Policy Enforcement Point, PEP). OPA is introduced to decouple these two activities. OPA\u0026#8217;s input is a JSON payload and it uses Policy in Rego language to come to decision.\nThe team that developers Open Policy Agent also created their controller (with OPA as the core component) to run validating web hook and mutating web hook. The original version is OPA-Kubernetes that uses kube-mgmt. This original version is also dubbed Gatekeeper v1.0. When OPA starts, the kube-mgmt sidecar container will load Kubernetes Namespace and Ingress objects into OPA. You can configure the sidecar to load any kind of Kubernetes object into OPA. The sidecar establishes watches on the Kubernetes API server so that OPA has access to an eventually consistent cache of Kubernetes objects. It has gone through a couple of major version changes as summarized in this section. As of today, when we deploy Gatekeeper we should use version 3.\nGatekeeper v3 Currently, Gatekeeper v3 is the most popular choice for Kubernetes Policy Controller. The diagram bellow illustrate how Gatekeeper integrates with Kubernetes API server.\nGatekeeper and Kubernetes We can follow this guide to install Gatekeeper but the key step is as simple as to apply the correct version of manifest. Alternatively it can be installed using Helm. After the installation, we should see a Service named gatekeeper-webhook-service in the gatekeeper-system namespace. We can also inspect the newly created validationg web hook configuration\nk get validatingwebhookconfiguration gatekeeper-validating-webhook-configuration -o yaml | less The result indicates that the configuration forwards incoming manifests to the gatekeeper-webhook-service web service at the path /v1/admin for validation, and then at /v1/admitlabel for labelling. The configuration also stores rules as matching criteria.\nWe can smoke test Gatekeeper v3, with the basic example in its directory. Apply the template, constraint and then the manifests in resources. The pod creation will fail with an error like:\nError from server ([pod-must-have-gk] you must provide labels: {\u0026#34;gatekeeper\u0026#34;}): error when creating \u0026#34;resources/bad_pod_namespaceselector.yaml\u0026#34;: admission webhook \u0026#34;validation.gatekeeper.sh\u0026#34; denied the request: [pod-must-have-gk] you must provide labels: {\u0026#34;gatekeeper\u0026#34;} The gatekeeper document also covers the details of using ConstraintTemplate and Constraints. However, Writing your own a policy in Rego still takes time and we want to piggyback on the community for commonly used policies. OPA\u0026#8216;s gatekeeper-library projects keeps a handful of those in its library directory. We can test the privileged container example:\n$ cd gatekeeper-library/library/pod-security-policy/privileged-containers $ kustomize build . | kubectl apply -f - constrainttemplate.templates.gatekeeper.sh/k8spspprivilegedcontainer created $ kubectl apply -f samples/psp-privileged-container/example_disallowed.yaml pod/nginx-privileged-disallowed created $ kubectl delete -f samples/psp-privileged-container/example_disallowed.yaml pod \u0026#34;nginx-privileged-disallowed\u0026#34; deleted $ kubectl apply -f samples/psp-privileged-container/constraint.yaml k8spspprivilegedcontainer.constraints.gatekeeper.sh/psp-privileged-container created $ kubectl apply -f samples/psp-privileged-container/example_disallowed.yaml Error from server ([psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {\u0026#34;privileged\u0026#34;: true}): error when creating \u0026#34;samples/psp-privileged-container/example_disallowed.yaml\u0026#34;: admission webhook \u0026#34;validation.gatekeeper.sh\u0026#34; denied the request: [psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {\u0026#34;privileged\u0026#34;: true} Currently the library directory contains two sub-directories, general and pod-scurity-policy. The latter is to regulate Pod creation, while the former includes more common usecases such as disable node port, enforce https, and enforce probes. This is the place I start with when building a policy.\nThe policy constraints take effect cluster wide. When we have multiple clusters, we would like a unified place to manage policies. Azure Policy with AKS We take Azure Policy with AKS as an example to illustrate how public cloud platform can simplify policy management. When building AKS cluster, an addon profile for Azure Policy can be installed. This allows Azure Policy to connect to the AKS cluster. Azure Policy contains many built-in policies definitions (as well as initiative definitions which are groups of related policies). We can simply search by Kubernetes keyword and look for the built-in policies. For example, there is a built-in policy definition \u0026#8220;Kubernetes clusters should not allow container privilege escalation. The definitions (policy or initiative) can be assigned to a resource group with enforcement action set to denied, and with excluded namespaces, as shown in the screenshot below\nThe assignment can take as long as 10 minutes to push down to the cluster. Then we should be able to confirm by checking the constraint CRDs. We can see this This setup brings a centralized policy management system that can be easily hooked up to multiple clusters.\nOther benefits of this architecture includes the ability to report compliance. As per CIS report for Azure AKS recommendation 4.3:\nAzure Policy extends Gatekeeper v3, an admission controller webhook for Open Policy Agent (OPA), to apply at-scale enforcements and safeguards on your clusters in a centralized, consistent manner. It covers many basic resource types but does not cover any well-known CRDs. Azure Policy makes it possible to manage and report on the compliance state of your Kubernetes clusters from one place.\u0026nbsp;\nChecks with Azure Policy service for policy assignments to the cluster. Deploys policy definitions into the cluster as constraint template and constraint custom resources. Reports auditing and compliance details back to Azure Policy service. As of February 2022, AWS EKS doesn\u0026#8217;t seem to have the equivalent of this capability to integrate with a policy management. The only option would be to install Gatekeeper v3 yourself on the cluster, or host it separately. Bottom line Admission control should be a standard setup in Kubernetes deployment. When building gatekeeper system on your own, it can be set up separately on a different cluster. When Kubernetes is provided as a platform, it is very helpful for platform operator to manage their tenants. If the tenant is application development team, it also makes sense for them to develop their own policies for the developers in their team.\nPrevious PostFrom Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms Next PostFluxCD: Continuous Deployment with GitOps ","date":"2022-01-07T22:21:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-admin-control.webp","permalink":"/2022/01/kubernetes-admission-control/","title":"Kubernetes Admission Control"},{"content":"Update: also read my other article here on the different generations of ingress technologies.\nIn my Korthweb project I was researching for the best ingress mechanism for HTTP and TCP workload, both of which need to be secured. I started with Kubernetes Ingress but eventually decided to go with Istio Gateway. This blog post is about the justification. In this essay, I will make the distinction between Ingress and Gateway and explain why a Kubernetes platform needs the latter going forward.\nThe word ingress can be used either to indicate ingress resource (in conjunction with ingress controller) in the context of Kubernetes cluster, or more generally to indicate the technology (e.g. provided by service mesh) to direct north-south traffic, originated from the outside into the workload running within the cluster. I will use both meanings of the word ingress throughout the article. Background In my previous post, I discussed Kubernetes Service object and native Ingress. In short, a Service object addresses the problem of exposing a workload (Pod or Deployment), operating at layer 3-4. A ClusterIP type of service exposes workload within the cluster only, and therefore is only used by internal services that are consumed by workload in the same cluster. A NodePort type of service exposes workload to outside of the cluster, using the node\u0026#8217;s IP address and port range available on the node, a great step forward but still quite inconvenient and subject to the limitation of using Node\u0026#8217;s IP and Port. A LoadBalancer type of Service brings separate IP address and port for running a service. To back up this type of service, the platform has to provide a load balancer with its own IP address and port range to manage. The implementation is platform specific e.g. Metal LB for Minikube, Azure Load Balancer for AKS, NLB for AWS EKS. In this type of service, Node Ports still exists but are not exposed to the outside world. Instead, they are only exposed to the backend of Load Balancer, whose front-end port is exposed to the outside world. In both NodePort and LoadBalancer types of Service, the kube-proxy process plays a role on each node to direct traffic into NodePort to target Pods.\nKubernetes Ingress, on the other hand, targets issues above layer 4, for example, path-based routing and is therefore mostly used for HTTP and HTTPS traffic. It is implemented by an Ingress Controller and now there has been a diverse ecosystem of those ingress controllers. In many implementations (e.g. Azure Application Gateway Ingress Controller, AGIC), the ingress themselves usually come with a load balancer managed by themselves. This eliminates the need for a separate LoadBalancer type of Service just for L4 capability. As a result, Ingress (with both L4 and L7 capabilities) is usually deployed along with ClusterIP type Service. Here is an example. The declaration of the Ingress in this case usually contains numerous lines of annotations in order to communicate the specification to the implementation.\nRequirement Using Kubernetes Ingress along with Service of ClusterIP type seems perfect to address the majority of use cases. However, it has its blind spots. My Korthweb project deals with DICOM traffic (a protocol on top of TCP) over TLS as well as HTTP traffic, which can be broken down as:\nThe ability to proxy TCP traffic over an arbitrary TCP port The ability to terminate TLS/SSL encryption for TCP traffic The Ingress documentation of Kubernetes clearly states that: an Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type\u0026nbsp;Service.Type=NodePort\u0026nbsp;or\u0026nbsp;Service.Type=LoadBalancer. The stance of community-driven Nginx ingress controller can be found in the documentation, which suggests the use of Service object for arbitrary TCP port. There is some unofficial claims of workaround available but I\u0026#8217;m not confident.\nI also went through a number of other Ingress providers but to my disappointment, the only product that supports it seems to be Traefik:\nProductRequirement #1Requirement #2F5 driven Nginx Ingress ControllerSupportedThere have also been requests for TLS termination for TCP traffic but the request has not been closed as of yet.HA Proxy Ingress controllerSupportedThere\u0026#8217;s no mention of this in documentation.Kong\u0026#8217;s Kubernetes Controller (with TCPIngress CRD)SupportedDocumentation claims support through SNI-based routing. However, the example demonstrates it using self-signed certificate only. BYO cert not supported according to this GitHub issue.Traefik Lab\u0026#8217;s IngressRoute (with IngressRouteTCP CRD)SupportedDocumentation seems to suggest that it is supported by IngressRoute. My general impression is that Req #2 isn\u0026#8217;t very popular so the providers either don\u0026#8217;t support it or delaying the implementation. Even if they do, the CRD used for TCP ingress is different than HTTP\u0026#8217;s. For example with Traefik Lab, the CRD for TCP is IngressRouteTCP, and for HTTP it is IngressRoute.\nIntroduction to Gateway In addition to functional shortages, according to this blog post, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. In Kubecon 2019, a group of contributors discussed the evolution of Ingress into Gateway. Below are the two key slides stolen from their presentation:\nThis sums up how the concept of Gateway is different from Ingress. Gateway is an instantiation of a given LB. It works along with Route to achieve the functions brought by an Ingress. Gateway as a resource type of its own makes management easier at L4-L6 by a separate team. Routing at L7 is offloaded to \u0026#8220;Route\u0026#8221; resource type.\nThis conceptual evolution gives rise to the Gateway API open source project managed by the SIG-NETWORK community. Gateway API is a collection of resources that model service networking in Kubernetes, including GatewayClass, Gateway, HTTPRoute, TCPRoute, Service etc. The aim of this initiative is to evolve Kubernetes service networking through expressive, extensive and role-oriented interfaces that are implemented by many vendors and have broad industry support.\nThe diagram above stolen from Gateway API website illustrate the management model for each resource.\nKubernetes Gateway Implementations The Gateway API is still a fairly young initiative and all the reference implementations are either work-in-progress or in early stages. In the last section, we learned that in the journey from Ingress to Gateway, the standardization initiative comes a bit behind the implementation efforts. For many supporters, the natural strategy is to continue with existing proprietary ingress controller implementation, and retrofit their technology to the emerging Gateway API along the way.\nBecause of that, we can see many market players with multiple flavours of implementations, typically one that evolves from their original product offering, with higher adoption rate and maturity level, and one that conforms to Gateway API specification. For example, Traefik Labs has its Gateway API implementation in experimental stage. Its own implementation consists of (standard) Kubernetes Ingress and Kubernetes IngressRoute (based on custom resource with support of advanced features such as TCP route). Another example is the HAProxy Ingress, a community driven ingress controller implementation for HAProxy. Starting from its version 0.13, it partially supports the Gateway API\u0026#8217;s v1alpha1 specification. Here is the conformance statement. Contour as a CNCF project for ingress controller has support for Gateway API at alpha version. Kong\u0026#8217;s Kubernetes Ingress Controller follows the same path.\nThe Gateway functionality is also provided as part of Service Mesh product offering, and we can see some reference implementation by service mesh providers. Istio has its own Gateway implementation but tries to adapt to Kubernetes Gateway API. Hashicorp Consul also claims to be building support for Kubernetes Gateway API.\nUntil the Gateway API project matures, it is not recommended to use Gateway implementations that conforms to it, unless you intend to be their Guinea Pig. For my project, I chose Istio Gateway.\nIstio Gateway implementation As discussed, when the Gateway standard is still in its infancy, I chose a non-standard implementation of Gateway, even though it may include some CRDs. Out of the many Gateway implementations, I choose Istio mainly for two reasons.\nFirst, I need a Gateway as part of service mesh because service mesh provides many other features that are needed in the platform. One of the goals of Service Mesh is to provide commonly used features (traffic management, observability, security, extensibility) in a commodity layer on top of Kubernetes. Gateway is not the only thing I need out of this commodity layer. With one install of Istio, many common platform-level problems are also addressed (e.g. mTLS, traceability, etc). I\u0026#8217;ve compared three major service mesh technologies as below:\nLinkderdIstioConsul\u0026#8211; lightweight\n\u0026#8211; uses linkerd2-proxy\n\u0026#8211; the original service mesh\n\u0026#8211; no Gateway implementation (deal breaker in my use case)\u0026#8211; feature rich\n\u0026#8211; uses Envoy proxy\n\u0026#8211; complex but getting better\u0026#8211; initially a service discovery and distributed key-value store\n\u0026#8211; lacks observability features but getting better\n\u0026#8211; uses Envoy proxy It is worth-noting that there are some initiatives to standardize service mesh (e.g. SMI) with reference implementation such as OpenServiceMesh. The standard is too weak to be considered important for now compared with maturity and stability. Istio has strong community support. The risk with Istio to acknowledge, is that it is not following an open-governance model (unlike many other CNCF projects), which could potentially causes vendor lock-in. Second, The Istio Gateway features the separation between Gateway and Virtual Service CRDs. The former defines entry point and the latter defines routing rules. This design separates entry points from routing rules, enabling the flexibility of reusing the same Virtual Service for different gateways.\nIstio uses Envoy proxy to manage traffic between Pods. Unlike the kube-proxy pattern, the Envoy proxies are side-car containers centrally managed by Istio Control Plane, which also enables other features such as traceability, as illustrated in the diagram below (stolen from this post).\nIn addition to Virtual Service, Istio Gateway also has the concept of Destination Rules. Virtual Service defines how to route traffic to different destinations. Destination rule defines how to split traffic to different subset at the routing destination. The concepts are documented on this page. For HTTP traffic, the relevant entities can be represented in the diagram below:\nhostsgatewayshttptlstcpexportTohostsgatewayshttptls\u0026#8230;matchrouteretriestimeoutrewriteredirectmatchrouteretriestim\u0026#8230;urischememethodheadersporturischememethodheade\u0026#8230;VirtualServiceVirtualServiceHTTPRouteHTTPRouteHTTPMatchRequestHTTPMatchRequestdestinationweightheadersdestinationweighthea\u0026#8230;HTTPRouteDestionationHTTPRouteDestionationhostsubsetstrafficPolicyexportTohostsubsetstrafficPo\u0026#8230;DestionationRuleDestionationRulenamelabelstrafficPolicynamelabelstrafficPolicySubsetSubsetloadBalancerconnectionPooloutlierDetectiontlsportLevelSettingsloadBalancerconnectionPoo\u0026#8230;TrafficPolicyTrafficPolicyselectorserversporthoststlsnameselectorservers\u0026#8230;GatewayGatewayViewer does not support full SVG 1.1\nHere is an example of using Gateway and Virtual Service resources (from Korthweb sample project). Also note that in the Istio literature, there is neither a CRD name called \u0026#8220;Ingress\u0026#8221;, nor a resource type named \u0026#8220;Ingress Gateway\u0026#8221;, although they may be loosely used to refer to \u0026#8220;Gateway resources configured to manage ingress traffic\u0026#8221;. Istio Gateway Installation There are more than one ways to install Istio. In the past, Istio Operator was used to install Istio. The Operator calls IstioOperator API. Today the use of Istio Operator is not recommended anymore but the IstioOperator API is used implicitly by Istioctl installer. The two recommended approaches to install Istio today is by Istioctl and Helm. With istioctl, you can specify an option imperatively, or using an overlay file as did in this lab. With Helm, istio has multiple charts, and requires multiple steps:\ninstall CRDs using the base chart install istiod using the istiod chart. Here is an example of values provided to Helm installer. install ingress gateway using the gateway chart. Here is an example of values provided to Helm installer for ingress gateway. Note that egress gateway also requires the same helm chart with different value definition. This instruction contains how to manually install Istio gateways using Helm, including installation of multiple Helm charts. I attempted to create a single chart to consolidate the multiple charts required for istio. It was not successful because of an error when trying to reference the same gateway chart dependency for multiple times.\nPrevious PostAKS Lessons Learned 2 of 2 Next PostKubernetes Admission Control ","date":"2021-12-29T22:50:16-04:00","image":"/wp-content/uploads/2025/04/feature-ingress-crd.webp","permalink":"/2021/12/from-ingress-to-gateway-why-you-need-istio-gateways-on-kubernetes-platforms/","title":"From Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms"},{"content":"Even though Azure Kubernetes Service (AKS) is a managed service, building a cluster is not trivial. For help resources, I would start with the webinar \u0026#8220;Configure Your AKS cluster with Confidence\u0026#8221; from April 2021, which focuses on a set of working best practices (convention over configuration) but obviously not every recommendation suits every use case. For a deeper technical tour, the John Savill\u0026#8217;s Technical Training channel has good videos (from 2020) on AKS overview, high availability and networking. Lastly, there is also an AKS checklist to remind you of the implementation details to consider.\nAll the references aside, I need to write down some gotchas from my implementation experience in the last two month.\nIdentity and Access Management AKS is a special type of Azure resource in the sense that it manages other Azure services on user\u0026#8217;s behalf. Therefore the access management needs to consider several aspects:\nAccess TypeMechanisms involvedExampleUser access Kubernetes APIAzure AD, Azure RBAC and Kubernetes RBAC. \u0026#8211; Azure AD is for authentication\n\u0026#8211; Azure RBAC for Kubernetes\n\u0026#8211; Kubernetes RBACA user connects to Kube-API server using kubectlAKS access other Azure resourceThere are several identities that represents different components of AKS. For example, the AKS cluster, the node agent pool, and each add-on.\nThe AKS cluster can be represented as a service principal, or managed identity (system assigned or user assigned). The node agent pool can be represented as a managed identityAKS cluster connects to a VNet in a different resource group. (requiring cluster\u0026#8217;s identity to have network contributor role on the target network resource group)\nAKS node agent pulls images from ACR (requiring the node agent pool\u0026#8217;s identity to have ArcPull role on the target ACR)Pod access other Azure resourceAAD-Pod Managed IdentityBusiness workload connects to managed database service such as PostgreSQL on Azure.Pod access Kubernetes APIAccess Kubernetes API using Service Account. This issue is solved completely by Kubernetes native mechanisms. Roles and ClusterRoles defines permissions. RoleBindings and ClusterRoleBindings associates Service Accounts with permissionsWorkload access ConfigMap, Secret etc. In the first access type, for RBAC with user to access Kubernetes API, there is an overlap between Azure RBAC and Kubnernetes RBAC. Azure RBAC has four built-in roles and three of them (reader, writer, admin) are namespaced. When you use Azure CLI to manage to assign one of those roles, the rolebinding and cluster rolebinding record stored in etcd will be updated accordingly. RBAC mechansimUse caseAzure RBAC for KubernetesManage RBAC programmatically using Azure CLI, or infrastructure as codeKubernetes RBACManage RBAC declaratively with more granularity for all types of Kubernetes resources including CRD For ease of operation it is advised to use Kubernetes RBAC whenever possible. Azure RBAC is still used for RBAC at the level of Azure resource but not at the level of Kubernetes resource.\nIn the the second access type, AKS cluster may use managed identity or service principal. Azure\u0026#8217;s recommendation is managed identity over service principal. Managed Identity is a wrapper around Service Principal with less overhead. Managed Identity can be system assigned (created at the time of cluster creation), or user assigned (can be created ahead of time by Azure administrator and imported to the cluster\u0026#8217;s context).\nThe second access type can be further broken down because there are several components in AKS that uses their own identities. I list the related managed identities as below:\nNamePurposeBYO identity with Terraformcluster identityThis identity represents the clusterSpecify in identity block in kubernetes_cluster resourceagent pool identityThis identity represents kubelet running in the agent poolSpecify in kubelet_identity block. addon: azurepolicyThis identity represents azure policy addon to access the policyN/Aaddon: omsagentThis identity represents OMS agent to access monitoring etcSpecify in oms_agent_identity block addon: secretThis identity represents to the secret addon, to access AKVSpecify in secret_identity block addon: ingress gateway This identity represents the ingress application gatewayingress_application_gateway_identity block By default, the system creates a new managed identity for each of the required identity above. For simplicity with identity management, we may create a managed identity and use it for all the occasions where an identity is needed and user assigned (BYO) identity is supported.\nIn the \u0026#8220;az aks show\u0026#8221; command return (a JSON document), the identity section (root level) reports the cluster identity, the identityProfile section (root level) reports the agent pool (kubelet) identity. Other identities such as omsagent, are reported in their own child document.\nNode Networking In Azure, a subnet can span across multiple availability zones. Therefore an AKS cluster can put its nodes on a single subnet with nodes evenly distributed across three AZs for high availability. The AZ of each node is indicated in the node label, and can be displayed with kubectl command.\nWithin a single AZ, a good practice to minimize latency between nodes is to place the nodes in a proximity placement group (PPG). However, only a single PPG can be associated with a node group. You can\u0026#8217;t have three PPGs, one in each AZ, for a single subnet. Pod Networking The default Pod networking model is kubenet, which involves overlay network. Pod-to-Pod traffic across nodes requires Network Address Translation (NAT). To overcome this performance tax, Azure introduces Azure CNI which gives each Pod an routable IP address from the VNet\u0026#8217;s CIDR. This requires advanced IP planning to prevent IP exhaustion. A risk introduced in Azure CNI is that all Pods are exposed on the V-net, which needs to be protected by Network Security Group and/or outbound firewall.\nDNS On the DNS side, when AKS cluster integrate with an external node network, it may create weird issues that are hard to troubleshoot. Another example is with DNS. If the V-Net uses an external DNS server (which is common for enterprises with hybrid network to use an on-premise DNS server), then the cluster creation failed with time-out with misleading error messages (for example, this\u0026nbsp;comment). This is because the DNS name of the newly created cluster is not resolvable within the V-NET, which points to the on-prem DNS server. The fix to that is:\nUse a BYO DNS zone (in Azure) for AKS cluster creation; The AKS cluster will publish the A-record to the zone. To allow this to happen, the AKS cluster’s managed identity needs to have DNS contributor permission for the zone; Configure the on-prem DNS for conditional forwarding to the DNS zone This fix will allow AKS to resolve its name and therefore confirm its own creation. Here is a good blog about the DNS zone\u0026nbsp;options.\nAnother potential issue introduced with the use of on-prem DNS server, is the resolution of single-label hostname of the nodes. This is not just an issue in the context of AKS. It is a generic issue with VMs running on a V-Net pointing to on-prem DNS, as explained in detail\u0026nbsp;here.\nIn this situation, we should use the fully qualified hostname instead of single-label hostname. The fully qualified hostname with DNS suffix can help the on-prem server to configure conditional forwarding. For example, when the DNS suffix is *.internal.cloudapp.net, then forward it to Azure’s virtual internal DNS server 168.63.129.16 which can resolve the hostname.\nIf only the Pods need to resolve those FQDNs, then we can configure Core-DNS with\u0026nbsp;conditional forwarding, which will take effect only at the cluster level without the need for changing the on-prem DNS. The Core-DNS configuration looks like this:\napiVersion: v1 kind: ConfigMap metadata: name: coredns-custom # this is the name of the configmap you can overwrite with your changes namespace: kube-system data: cloudapp.override: | # you may select any name here, but it must end with the .override file extension log rewrite continue { name regex ^(.*[0-9]{7}-vmss[0-9]{6})$ {1}.internal.cloudapp.net answer name ^(.*)\\.internal\\.cloudapp\\.net$ {1} } forward internal.cloudapp.net 168.63.129.16 cloudapp.server: | internal.cloudapp.net:53 { errors log cache 10 forward . 168.63.129.16 } Alternatively, use Pod\u0026nbsp;DNS policy\u0026nbsp;so that the Pod can use an external DNS server.\nInitial Service Account When a cluster is created, an Azure AD user or group can be assigned as cluster administrator. For a CI/CD pipeline to interact with the newly created cluster, a service account in Kubernetes is needed. Suppose we use Terraform to create the AKS cluster, we can create such service account automatically with the Kubernetes provider. This requires that the Terraform execution environment to have network access to the cluster. If the AKS cluster is located in a private network, then the agent where Terraform CLI runs should also be on the network. Alternatively, use Terraform Enterprise hosted in an environment with access to the cluster\u0026#8217;s network.\nIntegration with Azure KeyVault Azure Key Vault can store several types of secrets, key value pair, X509 keys and certificate. When AKV is integrated with an AKS cluster, the Kubernetes workload can access the secrets as mounted volumes, using CRD named SecretProviderClass. Further, they can be presented as Kubernetes Secret, using a Pod to sync between mounted content and Secret. AKV has three types of entries: key, certificate and secret (key-value). The certificate entry requires both key and certificate are stored, with optional certificate chain. In my opinion this is an over design. Unless we need Azure to manage the certificate (e.g. rotation) I would simply use secret to store my own X509 key and certificate.\nPrevious PostAKS Lessons Learned 1 of 2 Next PostFrom Ingress to CRD: why my solution needs Istio Gateways on Kubernetes platforms ","date":"2021-12-18T01:18:00-04:00","image":"/wp-content/uploads/2025/04/feature-aks-lession-2.webp","permalink":"/2021/12/aks-lessons-learned-2-of-2/","title":"AKS Lessons Learned 2 of 2"},{"content":"Digi Hunch contributes to open source community focusing on application deployment. Check out my GitHub page. Here are some recent projects:\nOrthweb Orthweb is a cloud-based mini-PACS solution based on Orthanc and AWS. Orthanc is an open-source medical imaging application. Orthweb automates the infrastructure provisioning and configuration management. With the Orthweb artifact, users bring up a fully functional, scalable and secure mini-PACS in 30 minutes.\nKorthweb Korthweb is an initiative to run Orthweb on Kubernetes platform. Korthweb brings up Orthanc service using a number of different approaches, making Orthanc a single-command deployment on existing Kubernetes platform.\nCloud Kube CloudKube is Infrastructure as Code project for provisioning production grade Kubernetes clusters in common Cloud platforms (Azure and AWS).\nKubelab (Archived) Kubelab is an Infrastructure as Code project (in AWS CDK v1, now archived) to deploy a Kubernetes cluster with self-managed nodes. It is also a demonstration of using CDK in Python. Kubelab brings up a number of EC2 instances and automatically configures them as a Kubernetes cluster. ATlab (Archived) ATLab (Ansible Tower Lab) is an infrastructure-as-code project (in AWS CDK v1, now archived) to deploy an AWX (open-source alternative of Ansible Tower). It is also a demonstration of using CDK in TypeScript. AT Lab brings up an AWX server and automatically configures it.\n","date":"2021-12-05T22:53:05-04:00","image":"/wp-content/uploads/2025/04/menu-projects.webp","permalink":"/projects/","title":"Projects"},{"content":"Security is one of the most important aspects in cloud architecture design and implementation. Security concerns data privacy, an important aspect of platform compliance. With regard to security, we perform security review with threat model assessment on the infrastructure stack, mostly looking at the following aspects:\nIdentity and Access Management Authentication (Identity Management) and Authorization (Access Management) is a foundational design aspects. We need to consider issues such as identity store, integration, SSO, attributes at all layers such as application (business traffic), container platform (e.g. Kubernetes admin traffic), and cloud platform (e.g. cloud admin traffic).\nEncryption and Certificate Management All security standards mandates the encryption of data in transit and at rest. Data in transit are encrypted by standards at different network layers. Transport Layer Security (TLS) is the most important standard in this regard and it operates on X.509 certificates, which is managed by the Public Key Infrastructure (PKI) of the organization.\nCompliance Most of the enterprise cloud deployment should target certain compliance programs as part of the security initiative. Common compliance frameworks and programs include:\nDoD SRG (Department of Defense Cloud Computing Security Requirements Guide) FedRAMP (Federal Risk and Authorization Management Program) HIPPA (Health Insurance Portability and Accountability Act) GDPR (General Data Protection Regulation) PCI-DSS (Payment Card Industry Data Security Standard) CIS (Center for Internet Security) Benchmarks The main cloud service providers provides tools to help client assess the compliance status of their cloud deployment.\nMore on security IAM Roles for any workload - Background A few month back a client of mine wanted to use GitLab pipeline to deploy infrastructure on AWS with Terraform. The key question is how to authenticate the Terraform process running in the pipeline to AWS with temporary credential. Having worked it out on GitHub, my proposal at time\u0026hellip; Managing EC2 instances across accounts with Ansible - I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is\u0026hellip; WordPress Security Basics - Background In 2019, I moved this site to WordPress hosted on an Amazon Lightsail instance. There were few visits at that time so I lived with the single-server architecture. The website traffic has since been in steady growth but I have been too busy to catch up with the WordPress\u0026hellip; EKS impression - I've worked on a few AKS projects previously. Since I joined AWS I wanted to put aside some time to check out EKS (Elastic Kubernetes Service). Here in this post, I put down my first impression on EKS, and also share my Terraform template in cloudkube project to create an\u0026hellip; Istio Operation Gotchas - In this post I discuss a few aspects when putting istio in operation. Installation Istio installation can be confusing, due to architectural and guideline changes as well as renaming of operator CRDs since its release, and especially since 2020. This left lots of information outdated on the web, adding to\u0026hellip; Contact Digi Hunch for Professional Services.\n","date":"2021-12-05T22:33:26-04:00","image":"/wp-content/uploads/2025/04/menu-security-fence.webp","permalink":"/cloud-security/","title":"Security"},{"content":"Professional IT Service Hunch Digital Services Inc (Digi Hunch) delivers professional IT services. We have more than a decade\u0026#8217;s experience in different aspects of enterprise IT, including application development, customer support, infrastructure, solution architecture and implementation.\nReview the service categories and contact for a free estimate.\nContact The best way to contact Hunch Digital Services for professional service is by email, or via LinkedIn message.\nCertifications ","date":"2021-12-05T22:19:07-04:00","image":"/wp-content/uploads/2025/04/menu-contact.webp","permalink":"/contact/","title":"Contact"},{"content":"In the 60s, automobiles manufactured in Japan consistently beats their competitors in American market. Many refers to the lean manufacturing methodology in the automation as the secret sauce. The software industries borrowed a lot of similar methodologies from TPS (Toyota Production System) into software development industry, which brought about agile software development.\nFor software to deliver value, it is not just about developing software in agile methodologies. A full SDLC (software development life cycle) includes build, release and upgrades too, some of which are managed in a different department in the organization. DevOps extends agile methodology across departments. DevOps has now become a buzzword. Some even refers to it as a culture but none of these are possible without automation.\nAutomation Pipelines The power horse of the DevOps tooling is automation pipeline (e.g. Jenkins, Azure DevOps, GitHub). These pipelines expedites iterations with frequent feedback about software quality, whether it is common conventional SDLC workflow or more recent infrastructure as code worklfow. For SDLC, the goal is to establish continuous integration (CI) and ultimately continuous deployment (CD). Serverless Deployment With serverless deployment, the operation of managing computing resources is abstracted away. Serverless deployment models further simplifies SDLCs and are ideal for some common use cases such as API services, IoT, scheduled and event-driven tasks.\nDataOps Another creative use of automation pipelines is the data pipelines. Data engineering tasks includes ingestion, ETL, integration, and storage and automation pipelines are ideal automation tools for these tasks.\nObservability Observability setup enables instant feedback, an important construct of DevOps. An observability stack consists of metrics collection, log shipping, performance monitoring, request tracing and visualization etc.\nMore on automation The Leanest Web and Email Hosting - This site has been quiet for a while. During this time, I migrated the hosting platform again, and refactored email solution. This post, is another note about how I finally came to the most cost-effective web and email solution for a small business, with a solid security posture, and at\u0026hellip; Debating between count and for_each in Terraform - In Terraform, we often have to create an array of resources of the same type but similar attribute values. For code reusability, manageability and for DRY principle, it's better to use loop. Terraform HCL supports loop via the use of meta-argument. Currently, there are two options to drive a loop:\u0026hellip; Test Open ID Connect Flows Locally - Earlier this year, I had to integrate an application with an identity provider. Both claim to be compliant with Open ID Connect. But when they don't get along, I must find out where it breaks to determine which party isn't compliant. Therefore, I had to really get to the transaction-level\u0026hellip; IAM Roles for any workload - Background A few month back a client of mine wanted to use GitLab pipeline to deploy infrastructure on AWS with Terraform. The key question is how to authenticate the Terraform process running in the pipeline to AWS with temporary credential. Having worked it out on GitHub, my proposal at time\u0026hellip; Managing EC2 instances across accounts with Ansible - I regard AWS Systems Manager as omnipotent. Nonetheless, there are a few reasons that makes Ansible still a prevalent VM (EC2) management tool over Systems Manager (SSM). First, organizations already vested in their custom Ansible roles and playbooks want to reuse, and expand their assets in Ansible. The benefit is\u0026hellip; Contact Digi Hunch for Professional Services.\n","date":"2021-12-05T22:11:18-04:00","image":"/wp-content/uploads/2025/04/menu-devops-automation.webp","permalink":"/automation-consulting/","title":"Automation"},{"content":"In general, troubleshooting Kubernetes is tricky. That is because one has to get in and out of pods. I took two days to troubleshoot some networking issues with private AKS cluster. For the amount of of tricks I had to employ, I need to take some notes.\nThe issue After writing the Terraform code, I used the following dummy service to test the private AKS cluster:\napiVersion: apps/v1 kind: Deployment metadata: name: aks-helloworld-one spec: replicas: 1 selector: matchLabels: app: aks-helloworld-one template: metadata: labels: app: aks-helloworld-one spec: containers: - name: aks-helloworld-one image: neilpeterson/aks-helloworld:v1 ports: - containerPort: 80 env: - name: TITLE value: \u0026#34;Welcome to Azure Kubernetes Service (AKS)\u0026#34; --- apiVersion: v1 kind: Service metadata: name: aks-helloworld-one spec: type: LoadBalancer ports: - port: 80 selector: app: aks-helloworld-one The expected behaviour, is that the service object will tell cloud API to provision a load balancer, with public IP listing at port 80. I should be able to curl to the IP address and connect to the site in the Pod. However, I was not able to. On the bastion host, I was able to curl to nodePort of the node address. But anything on public IP does not work, no matter where I ran curl from. This feels like a basic issue, but is is quite annoying because the native troubleshooting tool for Azure Load Balancer is horrible. In and out of a bunch of components named \u0026#8220;insights\u0026#8221;, \u0026#8220;diagnostic log\u0026#8221;, or \u0026#8220;Metrics\u0026#8221;, I can\u0026#8217;t simply find a way to trace whether it received an HTTP request. Most of the information I was able to see was irrelevant or useless.\nThe approach The hard way to troubleshooting infrastructure as code, is configuration comparison approach: revert to a baseline configuration, and see if the expected function works. Then from the baseline, change one configuration at a time and see where it starts to break. This approach is very time consuming, and AKS cluster as a relatively large resource, with numerous attributes, takes this effort to extreme. The baseline configuration I started with is:\naz aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner With this baseline, I simply use kubectl to apply the YAML file above. Then I can tell that the port is working. With a good start point, I started to apply one change at a time and repeat the test. I ran into a snug when I\u0026#8217;m using the following configuration:\naz aks create -g AutomationTest -n orthCluster --generate-ssh-keys --node-count 1 --tags Owner=MyOwner --enable-private-cluster --network-plugin azure --network-policy calico With the cluster created from the command above, the variable introduced is \u0026#8211;enable-private-cluster. This puts the cluster on a private network. I cannot connect to the cluster via a public endpoint anymore, and thus have to figure out some tricks to run the kubectl commands. I had to play with the Command Run feature of AKS cluster because I don\u0026#8217;t have a bastion host when using AZ CLI command. The Command Run feature would not allow me to use any file from bastion host. So i had to create my test objects, the Deploy and the Service objects all by imperative commands. The equivalent commands I worked out is:\nkubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1 --replicas=1 --port=80 kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type=\u0026#39;LoadBalancer\u0026#39; Then I realized a limitation with Command Run feature: it only supports basic command switches and doesn\u0026#8217;t like switches such as \u0026#8211;replicas. So I used the following commands:\naz aks command invoke -g AutomationTest -n orthCluster -c \u0026#34;kubectl get no\u0026#34; az aks command invoke -g AutomationTest -n orthCluster -c \u0026#34;kubectl create deployment aks-helloworld-one --image=neilpeterson/aks-helloworld:v1\u0026#34; az aks command invoke -g AutomationTest -n orthCluster -c \u0026#34;kubectl get deploy\u0026#34; az aks command invoke -g AutomationTest -n orthCluster -c \u0026#34;kubectl expose deploy aks-helloworld-one --port 80 --target-port 80 --type=LoadBalancer\u0026#34; az aks command invoke -g AutomationTest -n orthCluster -c \u0026#34;kubectl get svc\u0026#34; This trick allows me to continue with the testing eliminate Azure CNI and Calico policy as the cause. Testing after each cluster creation is painful because the cluster creation can take 10 minutes.I had to temporarily minimize the size of the cluster to speed up provisioning. I finally came to the point that I can reproduce the issue using TF template. I realized that when I set the vnet_subnet_id attribute of azurerm_kubernetes_cluster\u0026#8217;s default_node_pool, the problem came back. That\u0026#8217;s the smoking gun that the node subnet is the issue. The Network Security Group on Node Subnet The node subnet has an associated network security group. I discovered that once I add an allow rule for port 80 to the security group, the curl test will work. I also noticed the security group rule change will take 60 sec to come to effect and load balancer will also take 60 sec to warm up.\nThis confuses me because port 80 is only listened by the load balancer and not by any of the nodes. It\u0026#8217;s most likely when public load balancer is used the load balancer is placed on the node subnet. According to this note: Inbound, external traffic flows from the load balancer to the virtual network for your AKS cluster. The virtual network has a Network Security Group (NSG) which allows all inbound traffic from the load balancer. This NSG uses a service tag of type LoadBalancer to allow traffic from the load balancer.\nThe packet coming from external source can travel up to the VNet, but it was blocked at the NSG of node subnet.\nLessons Learned We always need to have some dummy service ready to test what we need. We can use nginx dummy service like:\napiVersion: apps/v1 kind: Deployment metadata: name: my-nginx spec: selector: matchLabels: run: my-nginx replicas: 2 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: my-nginx labels: run: my-nginx spec: type: LoadBalancer ports: - port: 80 protocol: TCP selector: run: my-nginx As discussed above, it\u0026#8217;s also important to have a Bastion host able to access the control plane when the AKS cluster is private. Azure touts about CloudShell (and its ability to run in specified V-Net) but it\u0026#8217;s pretty useless in troubleshooting. CloudShell sessions run inside of Kubernetes cluster and lacks common network troubleshooting tool such as nc. Azure has a managed service for Bastion but it requires a subnet with the exact name of AzureBastionSubnet.\nWe will explore more issues in the next post.\nPrevious PostFrom Microservice to Service Mesh Next PostAKS Lessons Learned 2 of 2 ","date":"2021-12-04T02:11:06-04:00","image":"/wp-content/uploads/2025/04/feature-aks-lesson-1.webp","permalink":"/2021/12/aks-troubleshooting-lessons-learned/","title":"AKS Lessons Learned 1 of 2"},{"content":"We all know what microservice is now but how does service mesh assist with microservice development.\nMicroservice Microservice as an architecture was firstly conceptualized in this article by Martin Fowler in 2014. It covers the pros (strong module boundaries, independent deployment, technology diversity) and cons (dealing with distributed system, eventual consistency, operational complexity). The reality is, many teams develops their product with the microservice architectural pattern. The implementation of microservice architecture involves a lot of programming patterns, and tools. The creation of these patterns and tools are usually done in a separate dedicated project so developers can focus on business logics. When building software, developers only need to interact with libraries and frameworks. Libraries (e.g. log4j) provides dependencies, and developers needs to write code to call the libraries. On the other hand, frameworks (e.g. Spring, Flask) not only provides tools, but also implements a pattern. It addresses a set of common problems such as authentication, expose http service, logger and database connectivity. Once set up, the framework will call the code that developers write (unlike in libraries).\nSpring \u0026#8211; an example of Microservice framework When it comes to microservice, a well-known appliction framwork is the Spring framework. It solves problems such as:\nApplication context and dependency injection (for Inversion of Control, or IOC) Database access and transaction management Expose rest APIs (using spring MVC) There\u0026#8217;s an entire ecosystem of projects under Spring framework. This framework is a huge system requiring a lot of configuration efforts. This is where Spring Boot helps.\nSpring Boot makes it easy to create stand-alone, production-grade Spring based applications that you can just run. It features the \u0026#8220;convention over configuration\u0026#8221; paradigm to save programmers from boiler plate configuration. SpringBoot gives you a standalone application ready to run without complicated deployment steps.\nManaging configuration in property files does not scale in the time of microservice. Spring Cloud provides configuration as a service (in line with everything else microservice framework). It doesn\u0026#8217;t necessarily have to be hosted in the cloud. It is comparable to Apachee Zookeeper, Etcd (distributed key value store), Hashicorp Consul and Netflix OSS (Eureka, Ribbon, Hystrix). You can pull from Git repo. The mission of Spring Cloud is to eliminate boilerplate associated with distributed systems problems for Spring Boot applications.\nMany developers use Spring Boot along with Spring Cloud to build microservices. This page contains a diagram for such architecture. In this architecture, Spring cloud helps with service discovery, traffic routing, circuit-breaking, distributed tracing and monitoring. It can also act as API gateway (in place of Nginx).\nAPI Gateway and API Management Microservices relies on API. Let\u0026#8217;s distinguish API gateway, and API management (this long post has some good information).\nAPI Gateway is a microserivce pattern. The idea is a single point of entry for all clients. The API gateway either proxy an incoming request to the appropriate service, or it may fan out a request to multiple services. The other important aspect is the API gatway can expose a different API for reach client. API Gateway Pattern A variation of this pattern is the Backends for frontends pattern, where it defines a separate API gateway for each kind of client. The API Gateway may authenticate user and pass an Access Token containing information about the user to the services. It may use a circuit breaker to invoke services. To summarize, the key functions of an API Gateway in this pattern is:\nUnified entry point for multiple API implementations Protocol transformation Request morphing Client specific logics This page from Azure has a good comparison between API Gateway pattern vs direct connection between client and microservice. Note that API Gateway can also refers to API Gateway product, which implements the functions above. For example\nSpring Cloud Gateway Solo.io Gloo Netflix Zuul API management acts as a proxy for an existing API implementations. Typical functions include:\nAuthN and AuthZ Service Discovery Ingreation Load balancing (e.g. L7 path based routing) Logging, tracing (track user), correlation Response Caching Retry policies, circuit breaker, QoS Enforce policy Track usage and monetization metrics (duration) rate limiting and throttling Request morphing (header, query string and claims transformation) IP whitelisting API management are usually implemented as tightly controlled shared infrastructure owned by either a \u0026#8220;platform team\u0026#8221;, \u0026#8220;integration team\u0026#8221;, or other API infrastructure teams. Examples of API management product (including SaaS) are:\nGoogle Cloud Apigee Mulesoft Kong In real world, people often use API management produce and API gateway product interchangeably. Service Mesh Many dub Service Mesh the next generation of Microservice. So what is the relationship between microservice and service mesh. The Microservice architectural pattern creates the need for API gateway pattern. To address this pattern, the API Gateway products first emerged. Service mesh emerged later. Service mesh and API gateway have a common set of features. In this presentation (a tale of two frameworks) from 2018 (early days of service mesh), two teams discussed microservice (spring cloud) and service mesh (istio) approaches. There is a slide about when to use which. Many teams since have moved to Service Mesh for feature richness. This is a case study from 2021.\nThere are a number of service mesh technologies, such as Consul, Isito and Linkerd. Here is a comparison chart. Even though all those projects are open-source, there is some competition already. Linkerd is the first to bring up the concept of service mesh in this blog. It also purportedly has better performance than Istio. However, it does not use Envoy proxy. Istio is good at marketing. It has higher adoption rate and is feature rich. However, Google did not donate Istio project to CNCF as many expected. Instead, it created its own governing body, the Open Usage Commons. The Istio is not an open-governance project, which potentially diverge from CNCF in the future [1]. Hashicorp Consul initially was built for service discovery and distributed key/value store. It supports Kubernetes and VM. However, it still lacks observability features. Apart from the three major technologies, other players tries to push for standardization of service mesh. The most influential initiative is the SMI (service mesh interface), pushed by Microsoft. The idea is a separation of standard and implementation, so late players will have a chance. OpenServiceMesh is Microsoft\u0026#8217;s reference implementation of SMI. The SMI is something to watch for but it remains pretty week thus far. Google\u0026#8217;s platform has Anthos Service Mesh which is a commercial distribution based on Istio. AWS has its own AppMesh technology, also using Envoy proxy.\n[1] Update from Apr 25, 2022 \u0026#8211; Istio applied to become CNCF project.\nPrevious PostIstio Lab – Ingress and Egress Next PostAKS Lessons Learned 1 of 2 ","date":"2021-11-25T00:31:19-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-networking.webp","permalink":"/2021/11/from-microservice-to-service-mesh/","title":"From Microservice to Service Mesh"},{"content":"Istio is a popular open-source service mesh implementation using Envoy proxy. One of the benefit of using Istio is the ingress and egress it brings to native Kubernetes platform. This article is a hands-on guide to test Istio ingress and egress gateways on Minikube. It was tested on my MacBook. All the information in this post can be found in several different pages from Istio and Minikube documentation. The purpose of this article is a single cheat sheet for myself to quickly set up Istio ingress and egress testing.\nInstall Istio on Minikube We start with configuring Minikube on MacBook (or change the command accordingly on other platform), and install metal LB load balancer. The steps with Istio also works with a regular Kubernetes platform.\nminikube start --memory=12288 --cpus=6 --kubernetes-version=v1.20.2 --nodes 3 --container-runtime=containerd --driver=hyperkit minikube addons enable metallb Then we need to give load balancer an IP address range before setting up the load balancer. Minikube adds an interface named bridge100 to MacOS host environment. The IP address of the host can be found via the following command:\nminikube ssh \u0026#34;ping host.minikube.internal -c 1\u0026#34; For example, my host IP address is 192.168.64.1, so I dedicate the IP range 192.168.64.64 to 192.168.64.80 to the MetalLB load balancer. Then I need to provide this range when configuring the load balancer:\nminikube addons configure metallb kubectl -n metallb-system get po After the configuration, make sure the MetalLB pod is up. Then we can install istio to the Kubernetes cluster. The steps are completed using istioctl following the instruction, I created a quick and dirty script to put the steps together:\n#! /bin/bash curl -L https://istio.io/downloadIstio | sh - export PATH=$(realpath istio*/bin):$PATH ln -sfn $(ls -d -- istio-*) istio if istioctl x precheck; then echo ready to install istio and lable namespace for istio-injection istioctl install --set profile=demo -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY kubectl label namespace default istio-injection=enabled else echo failed precheck exit 1 fi The script above download the istio zip file, unzips it, installs Istio (in demo mode and set outboundTrafficPolicy to REGISTRY_ONLY), labels default namespace for proxy injection. To use istioctl without spelling out full path, we can run the following command again separately outside of script:\nexport PATH=$(realpath istio/bin):$PATH To validate, check istioctl version command.\nInstall Sample application For simplicity, the rest of this article uses httpbin to test Istio\u0026#8217;s ingress and egress. For completeness, this section gives the steps to install bookinfo application as it is used in other Istio testing as well. It also help understand how to ingress to an application with Istio using an IP provided by the load balancer. Skip this section if just need to test with the simple httpbin application. The Bookinfo application comes within the istio 1.12 directory. In the following steps from the istio directory, we install some observability add-ons as well as the bookinfo application. Then we test access to the website.\nkubectl apply -f samples/addons/prometheus.yaml kubectl apply -f samples/addons/kiali.yaml kubectl apply -f samples/addons/jaeger.yaml kubectl apply -f samples/addons/grafana.yaml kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml kubectl exec \u0026#34;$(kubectl get pod -l app=ratings -o jsonpath=\u0026#39;{.items[0].metadata.name}\u0026#39;)\u0026#34; -c ratings -- curl -sS productpage:9080/productpage | grep -o \u0026#34;\u0026lt;title\u0026gt;.*\u0026lt;/title\u0026gt;\u0026#34; export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath=\u0026#39;{.status.loadBalancer.ingress[0].ip}\u0026#39;) export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath=\u0026#39;{.spec.ports[?(@.name==\u0026#34;http2\u0026#34;)].port}\u0026#39;) export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT curl -s \u0026#34;http://${GATEWAY_URL}/productpage\u0026#34; | grep -o \u0026#34;\u0026lt;title\u0026gt;.*\u0026lt;/title\u0026gt;\u0026#34; On my laptop, the value of $GATEWAY_URL is 192.168.64.64:80. Therefore I can browse to site http://192.168.64.64:80/productpage with local browser. We can also check kiali dashboard:\nistioctl dashboard kiali Then we are good to examine ingress and egress. Prep Work The following custom resources are related to traffic management in istio:\nVirtual Service (vs) Destination Rule (dr) Service Entry (se) Gateway (gw, Istio\u0026#8217;s ingress or egress) We will expand on the role of each in a separate article. To execute command from within a Pod, we also use sleep application which is essentially a Pod with a sleeper thread for the purpose allowing user to run troubleshooting command. The sleep application is provided in istio directory:\nkubectl apply -f samples/sleep/sleep.yaml kubectl get po -l app=sleep Find out the Pod name to execute command. Then we use httpbin website to emulate an external service, and use the sleep Pod (suppose the name is sleep-557747455f-q99cz) to access the service:\nkubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/headers As we previously set outboundTrafficPolicy.mode=REGISTRY_ONLY, this step is expected to output nothing. If this is not the case, it means the cluster may have ALLOW_ALL as outbount traffic policy, and you will need to follow this guide to set it to REGISTRY_ONLY. With REGISTRY_ONLY, in order to allow access, we need to register a service entry, by applying the document below:\napiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: httpbin spec: hosts: - httpbin.org ports: - number: 80 name: http protocol: HTTP resolution: DNS Now we should be able to access httpbin service as it is registered:\nkubectl get se httpbin -o yaml Using the exec command above again, we should see output from the curl command.\nIngress Testing We deploy an httpbin service in order to test ingress. The YAML declaration is prepared by Istio installation package:\nkubectl apply -f samples/httpbin/httpbin.yaml Then we deploy the ingress:\napiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: httpbin-gateway spec: selector: istio: ingressgateway servers: - port: number: 80 name: http protocol: HTTP hosts: - \u0026#34;httpbin.digihunch.com\u0026#34; The selector above indicates that the gateway deployed to a Pod labeled with istio: ingressgateway. Then we create a corresponding virtual service for this gateway:\napiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: httpbin spec: hosts: - \u0026#34;httpbin.digihunch.com\u0026#34; gateways: - httpbin-gateway http: - match: - uri: prefix: /status - uri: prefix: /delay route: - destination: port: number: 8000 host: httpbin Note that the hosts setup must match the configuration in gateway. The virtual service defines two services at URIs /status and /delay to expose, and we can confirm the creation of virtual service:\nkubectl get vs httpbin -o yaml We can then test with curl command from MacOS host to verify that the ingress is working. We need to use the IP address of the service (192.168.64.64 with port 80 as previously indicated):\ncurl -I -HHost:httpbin.digihunch.com http://192.168.64.64/status/200 curl -I -HHost:httpbin.digihunch.com http://192.168.64.64/delay/2 We use -HHost to emulate this request going to httpbin.digihunch.com (otherwise we\u0026#8217;d have to fake DNS entry in /etc/hosts). Both command should give their return without error. The second command will give return after a delay of 2 seconds.\nThis validates the success of ingress setup.\nEgress Testing To visit an external service, there are four options:\nJust access external service from Pods, when you have global.outboundTrafficPolicy.mode=ALLOW_ANY Use ServiceEntry (as illustrated above in Prep Work) Bypass envoy proxy (not recommended, because there is no role of service mesh) Configure an Egress Gateway In the Prep Work, we\u0026#8217;ve seen how to use Service Entry. However, there is no control of how Pods access the external service. We are going to configure an Egress Gateway to manage the outgoing traffic flow. This allows you to manage route, and apply observability features to the outgoing traffic, something the Service Entry alone will not provide. The egress gateway have two typical use cases as explained on Istio website.\nFirst, we want to confirm that the egress pod exists, with a quick command:\nkubectl get po -n istio-system -l app=istio-egressgateway \u0026amp;\u0026amp; kubectl get se It should exist as we installed istio as demo version. We then also need a service entry, as we have already configured in Prep Work. At this point, the egress gateway resource has not been created.\nNow, in a separate terminal, we monitor the log of egress pod\nkubectl logs -f istio-egressgateway-7d5d69dcfd-kvwxv -n istio-system With the -f (follow) session on, if we visit external service as we did during Prep Work, there is no new entry pop up at the tail of the log. This is because the external access is still not managed by egress. Now we need to add egress, with the following resources applied:\napiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: istio-egressgateway spec: selector: istio: egressgateway servers: - port: number: 80 name: http protocol: HTTP hosts: - httpbin.org Then we setup the virtual service with related rules:\napiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: vs-for-egressgateway spec: hosts: - \u0026#34;httpbin.org\u0026#34; gateways: - istio-egressgateway - mesh http: - match: - gateways: - mesh port: 80 route: - destination: host: istio-egressgateway.istio-system.svc.cluster.local subset: httpbin port: number: 80 weight: 100 - match: - gateways: - istio-egressgateway port: 80 route: - destination: host: httpbin.org port: number: 80 weight: 100 --- apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: dr-for-egressgateway spec: host: istio-egressgateway.istio-system.svc.cluster.local subsets: - name: httpbin With all applied, we can repeat the access command using curl, and watch for the log tail. kubectl exec -it sleep-557747455f-q99cz -c sleep -- curl http://httpbin.org/ip The log should output a line saying curl command access, which indicates that this outbound access is managed by istio egress gateway. The log line looks like below and in the example the IP address 10.244.2.4 is the Pod IP address.\n[2021-11-22T01:22:47.547Z] \u0026#34;GET /ip HTTP/2\u0026#34; 200 - via_upstream - \u0026#34;-\u0026#34; 0 45 818 816 \u0026#34;10.244.2.9\u0026#34; \u0026#34;curl/7.80.0-DEV\u0026#34; \u0026#34;9ec0a193-77f0-90a5-b132-21c3259307da\u0026#34; \u0026#34;httpbin.org\u0026#34; \u0026#34;3.216.167.140:80\u0026#34; outbound|80||httpbin.org 10.244.2.4:42372 10.244.2.4:8080 10.244.2.9:52710 - - Looking at the virtual service declaration, you might noticed multiple routes in the virtual service. The first route (mesh) tells sidecars to route request to egress gateway. The second route (istio-egressgateway) tells egress gateway to route traffic to external service as destination.\nIn real life, there\u0026#8217;s more to consider. For example, we usually want the host to allow on egress to be a wild card host such as *.api.google.com. While the hosts field in virtual service can be wild card, the route destination in virtual service has to be a list of fully qualified domains. The other situation is the destination website may issue an HTTP redirect to a different host, which also needs to be registered as a destination. These details can add up to administrative effort. Egress example An other real life scenario is the use of authorization policy. Authorization policy rules use SPIFEE identity of a Pod. If a Pod initiates egress connection to destination host in TLS, then the Pod\u0026#8217;s SPIFFEE identity and its identity for external TLS connection are in conflict. With Azure AKS The steps above also works on other cloud platform such as AKS. This is an AKS cluster that I tested the steps. The Terraform code provisions a private cluster, as well as a Bastion Host. From the Bastion host, we can perform the same steps as above. When Istio gets installed, it will created a Service object called istio-ingressgateway in the istio-system namespace. The service is of load balancer type and by default will create a public load balancer in Azure AKS. The test steps will also work for private load balancers. To tell Azure to create private load balancer on a specific subnet, the Service object needs to include annotation. We have to customize Istio installation in order to do that, using the following overlay file (azoverlay.yaml):\napiVersion: install.istio.io/v1alpha2 kind: IstioControlPlane metadata: namespace: istio-operator name: custom-istiocontrolplane spec: profile: demo values: gateways: istio-ingressgateway: serviceAnnotations: service.beta.kubernetes.io/azure-load-balancer-internal: \u0026#34;true\u0026#34; service.beta.kubernetes.io/azure-load-balancer-internal-subnet: \u0026#34;suitable-porpoise-lb-subnet\u0026#34; This is inspired by this post. Then the command to install Istio becomes:\nistioctl install -f azoverlay.yaml -y --verify --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY This concludes the hand-on guide for Istio ingress and egress labs. I will need to write another post explaining the related concepts.\nPrevious PostInfrastructure deployment in Terraform 2/2 Next PostFrom Microservice to Service Mesh ","date":"2021-11-16T21:28:00-04:00","image":"/wp-content/uploads/2025/04/feature-istio-lab-1.webp","permalink":"/2021/11/istio-ingress-egress/","title":"Istio Lab – Ingress and Egress"},{"content":"In a previous post, I introduced Terraform Cloud and covered how to use AWS profiles with Terraform. This time I explored some alternatives to Terraform Cloud, in the context of Azure. I use Scalr as an example of multi-cloud management platform. I will also discuss some issues I\u0026#8217;ve came across while managing permissions and variables for Terraform.\nScalr Scalr is a multi-cloud management platform. I first used it in January but since then it seemed to focus on being a collaboration platform for Terraform. It organizes deployment by environments and workspaces. Accounts in the free tiers is allowed to have one Environment. You will also need to configure (cloud) providers and VCS providers. Once configured, it is important to link a cloud provider with an Environment. Each workspace inside of an Environment can be associated with a VCS provider. In the case of Terraform, this limits a workspace with a single cloud provider.\nPermission with Azure I have a resource group (e.g. named AutomationTest) under a subscription. My account has Contributor role of this resource group. To run Terraform, I could login to Azure as my own account on my environment using AWS CLI. Terraform will pick up the session from Azure CLI and execute as my user. However, it is recommended to run Terraform as a separate own entity. This would allow me to run Terraform template from Scalr, or Terraform Cloud. It is also a good practice for Terraform to use a separate account than a regular user account. There are a number of ways to do this as suggested on the guides for Terraform azurerm provider, including:\nAuthenticating via a Service Principal and a Client Secret Authenticating via a Service Principal and a Client Certificate Authenticating via Managed Identity Authenticating via the Azure CLI, only recommended when running Terraform locally. I chose the first option and followed the instruction, using the following CLI command to create the service principal:\naz ad sp create-for-rbac -n tf-sp --role=\u0026#34;Contributor\u0026#34; --scopes=\u0026#34;/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest\u0026#34; The command returns a few attributes (client ID, tenant ID, secret) that I needed to configure cloud providers in Scalr. The service principal will also be visible under App Registrations in Azure. Once configured I needed to link the provider to an Environment, for Scalr to make an connection to Azure. Otherwise, the Scalr run will return the following Error:\nTerraform error Once provider linking is completed, Scalr automatically populate required environment variables in the workspace. They show up as \u0026#8220;Shell\u0026#8221; variables under VARIABLES tab.\nUnder the Terraform tab are input variables that you wish to put in for Terraform template to pick up. Then you can run the template. This works well until I came across a permission issue when I added azurerm_role_assignment resource in Terraform template. What I was trying to do is something like this:\nresource \u0026#34;azurerm_role_assignment\u0026#34; \u0026#34;admin_assignment\u0026#34; { scope = var.rbac_aks_id role_definition_name = \u0026#34;Azure Kubernetes Service RBAC Admin\u0026#34; principal_id = var.rbac_principal_object_id } And whenever at this line, the following error returned:\nApparently the code 403 indicates Azure doesn\u0026#8217;t think the Terraform Service Principal has the privilege to perform Microsoft.Authorization/roleAssignments action. The reason dates back to the way I created service principle above, where I specified contributor role for resource group. However, contributor as a built-in role does not include the permission to assign roles in Azure RBAC. To address this issue, I needed a custom role, named TerraformContributor, with the following definition:\n{ \u0026#34;assignableScopes\u0026#34;: [ \u0026#34;/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest\u0026#34; ], \u0026#34;description\u0026#34;: \u0026#34;Grants full access to manage all resources, but does not allow you to manage assignments in Azure Blueprints, or share image galleries.\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/providers/Microsoft.Authorization/roleDefinitions/637824aa-52ae-42f6-a24e-26b2a443afdf\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;637824aa-52ae-42f6-a24e-26b2a443afdf\u0026#34;, \u0026#34;permissions\u0026#34;: [ { \u0026#34;actions\u0026#34;: [ \u0026#34;*\u0026#34; ], \u0026#34;dataActions\u0026#34;: [], \u0026#34;notActions\u0026#34;: [ \u0026#34;Microsoft.Blueprint/blueprintAssignments/delete\u0026#34;, \u0026#34;Microsoft.Compute/galleries/share/action\u0026#34;, \u0026#34;Microsoft.Blueprint/blueprintAssignments/write\u0026#34; ], \u0026#34;notDataActions\u0026#34;: [] } ], \u0026#34;roleName\u0026#34;: \u0026#34;TerraformContributor\u0026#34;, \u0026#34;roleType\u0026#34;: \u0026#34;CustomRole\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;Microsoft.Authorization/roleDefinitions\u0026#34; } Compare this with the JSON statement of built-in contributor role, the exclusion of actions for Microsoft.Authorization are removed. The creation can be completed with CLI command \u0026#8220;az role definition create\u0026#8221; in the subscription, or use Azure portal. Once the role is created, create a new service principal using the az ad sp create-for-rbac -n tf-sp --role=\u0026#34;TerraformContributor\u0026#34; --scopes=\u0026#34;/subscriptions/9dd2c898-8111-4322-91d6-a039a00bd513/resourceGroups/AutomationTest\u0026#34; This solution is suggested on this thread. Composite type for Input variable Sometimes we want to define an input variable that describes a hierarchy of attributes on a resource. A good example would be Azure Kubernetes service. We can use either object or map as the variable type. The example below has a variable of each type. variable \u0026#34;cluster_detail\u0026#34; { description = \u0026#34;AKS cluster\u0026#34; type = object({ resource_group = string, cluster_name = string, kubernetes_version = string, node_subnet = object({ subnet_name = string vnet_name = string resource_group = string }), pod_subnet = object({ subnet_name = string vnet_name = string resource_group = string }), lb_subnet = object({ subnet_name = string vnet_name = string resource_group = string }), ad_admin_group_object_ids = list(string) }) } variable \u0026#34;common_tags\u0026#34; { description = \u0026#34;common tags\u0026#34; type = map(any) default = { tagA = \u0026#34;valueA\u0026#34; tagB = \u0026#34;valueB\u0026#34; } } When using object as the type, the default value needs to define all fields.\nIn this example where node pool configuration is exposed, we can see how using variables with hierarchy helps template user customize infrastructure specification.\nParsing Map as Input variable To find out the best multi-cloud management platform, I tested a few of them. I find it inconsistent when I have an input variable of the map type. With Terraform Cloud, I have to specify the variable to be parsed as HCL, and the value has to be:\n{\u0026#34;Environment\u0026#34; = \u0026#34;Dev\u0026#34;, \u0026#34;Owner\u0026#34; = \u0026#34;info@digihunchtest.com\u0026#34;} With Scalr, I also have to specify the variable to be parsed as HCL, and the value can be either the one above, or the one below:\n{\u0026#34;Environment\u0026#34;:\u0026#34;Dev\u0026#34;,\u0026#34;Owner\u0026#34;:\u0026#34;info@digihunchtest.com\u0026#34;} So Scalr is more flexible in parsing maps. Apart from Scalr and Terraform, I also tested env0 but I gave up after an hour. I could not specify to parse a variable as HCL. They need to work harder on this. Neither was I able to figure out the right syntax as plain variable. I did not test Atlantis or SpaceLift.\nEnterprise Deployment When deploying code to enterprises with their own network environment, Scalr supports running a self-hosted agent inside of the Enterprise network. This is also supported by Terraform Cloud (manage in cloud, execution in enterprise network). This is very useful when the execution machine needs to access the resource created in the enterprise environment. A good example, is using Terraform\u0026#8217;s Azure provider to provision an AKS cluster on the corporate network. Then use Terraform\u0026#8217;s kubernetes provider to connect to the newly created cluster and create some Kubernetes object such as service account, as illustrated in this blog post. Previous PostAzure Deets Next PostIstio Lab – Ingress and Egress ","date":"2021-11-05T01:05:00-04:00","image":"/wp-content/uploads/2025/04/feature-terraform-2.webp","permalink":"/2021/11/infrastructure-deployment-in-terraform-2-2/","title":"Infrastructure deployment in Terraform 2/2"},{"content":"Both Azure and AWS are leading players in public cloud. AWS developed a lot of SMB customer in technology. Azure attracted many enterprises from their on-prem customers. In addition to clienteles, their models to manage resources in the cloud are also different in several aspects. While this post is by no means a comprehensive comparison, it serves as a refresher on how Azure is different from AWS in cloud engineering.\nSubscription and Resource Group Both subscription and resource group are means to manage resources in group. Subscription is associated with a credit card and groups resource financially. Resource group groups resources logically. A subscription can have multiple resource groups, as illustrated here in the hierarchy map.\nAzure CLI There are two CLI tools: Azure PowerShell for PowerShell users, and Azure CLI for Linux users. The distinction is not clear, because PowerShell can also run on multiple platforms such as MacOS and Linux. On the other hand, Linux Bash can run on Windows (e.g. using WSL2). The different lies more in the command nomenclature. For example, to list subscriptions, the PowerShell Cmdlets reads:\nGet-AzSubscription The Azure CLI comes much cleaner:\naz account list The command and argument naming in Azure CLI align with Linux commands. I prefer Azure CLI for its succinctness. When looking up VM skus, we can use this command:\naz vm list-skus -l eastus2 The return is a JSON document. In advanced use cases, we can filter the result by using jq utility. Alternatively, we can filter the result and groom the output with jmespath query, just like AWS CLI. For example, in eastus2 region, we look for instances that:\nHas virtualMachines as resource Type Has AcceleratedNetworking enabled Has EncryptionAtHostSupported enabled Has PremiumIO enabled For each of the result, we print out:\nnumber of vCPUs Memory size For this use case we will have to provide a JMESPath query as below, then output the result as a table:\naz vm list-skus -l eastus2 --query \u0026#34;[?resourceType==\u0026#39;virtualMachines\u0026#39; \u0026amp;\u0026amp; capabilities[?name==\u0026#39;AcceleratedNetworkingEnabled\u0026#39; \u0026amp;\u0026amp; value==\u0026#39;True\u0026#39;] \u0026amp;\u0026amp; capabilities[?name==\u0026#39;EncryptionAtHostSupported\u0026#39; \u0026amp;\u0026amp; value==\u0026#39;True\u0026#39;] \u0026amp;\u0026amp; capabilities[?name==\u0026#39;PremiumIO\u0026#39; \u0026amp;\u0026amp; value==\u0026#39;True\u0026#39;]].{Name:name,vCPUs:capabilities[?name==\u0026#39;vCPUs\u0026#39;].value|[0],MemoryGB:capabilities[?name==\u0026#39;MemoryGB\u0026#39;].value|[0]}\u0026#34; --output table The result looks like this:\nI use variations of the command above very often to find out the best instance for AKS nodes. Infrastructure as Code The native infrastructure as code option is ARM (Azure Resource Manager) template in JSON format. It is extremely wordy and perhaps why Azure later developed Bicep as the second generation of IaC tool. Terraform has a provider for Azure as well. For comparison among ARM, Terraform and Bicep, I have written a blog post for Slalom build covering more details.\nNetworking Here is a great post comparing Azure Network with AWS.\nAt a high level, Azure Virtual Network (or VNet) is the equivalent of VPC in Amazon. Likewise, peering can be configured between VNets. As to subnet, Azure is different because there is no conceptual distinction between public subnet and private subnet. In AWS, public subnet is subnet attached with an Internet Gateway via a network route. So \u0026#8220;private\u0026#8221; or \u0026#8220;public\u0026#8221; are in terms of outbound traffic. On the other side, Azure does not distinguish between private or public subnet. Resources connected to a VNet have access out to the Internet by default. As to inbound traffic, you can make a VM available on Internet by giving it a public IP (same as AWS). You can make it available to other VNet, by configuring a service endpoint. Customers typically need custom routes to redirect outbound traffic (e.g. through firewall). In VPC, subnets are mapped to availability zones one-to-one, whereas in Azure, a subnet may traverse multiple availability zones.\nWith Azure, it is also important to understand difference between Azure service endpoint and Azure private endpoint:\nAzure service endpoint: provides connectivity to Azure services over n optimized route over the Azure backbone network. Traffic will leave your VNet. Azure private endpoint: a NIC that uses private IP from your VNet. This NIC connects you privately and securely to a service powered by Azure Private Link. By enabling a private endpoint, you\u0026#8217;re bringing the service into your VNet. On security group, we can associated a network security group with a network interface, or with a subnet. In contrast, in AWS, a security group can only be associated with an instance\u0026#8217;s network interface.\nAzure Bastion and Jump Box There is a managed service called Azure Bastion. It is a SSH/RDP proxy fully managed as PaaS. However, its use case is virtual machines. It cannot be used to access other services. For example, if you create a private AKS cluster, then you need a command terminal to access the API server. This is not what Azure Bastion can do. Instead, you either need a virtual machine in the AKS network as jump box.\nThe alternative is an Azure Cloud Shell, which will require storage but can be configured to be placed inside of a V-Net. However, Azure Cloud Shell is not running inside of a full-fledged Linux operating system. You cannot install commands.\nTo create a bastion host, e.g. without public IP address, use the following CLI command:\naz vm create -n MyBastion -g AutomationTest --image UbuntuLTS --subnet suitable-porpoise-node-subnet --vnet-name suitable-porpoise-vnet --ssh-key-values ~/.ssh/id_rsa.pub --authentication-type ssh If the bastion host is needed with a public Ip, configure the network security group accordingly. The AZ CLI command above will create a VM, with an OS user named after the command line terminal user.\nIAM Azure AD is a managed identity service. Here is the difference between Active Directory and Azure AD.\nAzure RBAC is a mechanism for authorization. Just like IAM policies, Azure RBAC enforces permissions using role assignment, which consists of:\nsecurity principal ( user, group, service principal, or managed idenity) role definition: defines what actions is allowed and what is not allowed scope: the object of the action To enforce RBAC, one needs to create role assignment objects, each specifying principal, role, and scope.\nService Principle and Managed Identity The two concepts may appear confusing. I find this article a great reference to demystify them. The takeaway is: service principle is the equivalent of service account in old Active Directory. Managed identity is a service principle automatically managed by a resource. Managed identity can be user assigned or system assigned.\nManaged Identity is a \u0026#8220;wrapper\u0026#8221; around a service principal. It is automatically created and automatically rotated. Azure DevOps I think of Azure DevOps (ADO) of a managed pipeline implementation, with a repository (just like BitBucket), a board to manage tickets (similar to JIRA), Wiki (just like Confluence), Artifactory. The Pipelines is the part that\u0026#8217;s similar to Jenkins. ADO calls a build pipeline a Pipeline, and a release/deployment pipeline a Release. A pipeline and a release are fundamentally the same but they are used in different ways. A pipeline\u0026#8217;s input is usually the code repository, and the output is artifact. A release\u0026#8217;s input is usually an artifact, and it connects to infrastructure in different environments. ADO has its own ecosystem for plugins, managed under Visual Studio marketplace. Many extensions are open-source. If you are not happy with an extension, you can publish your own extension to market place. Logging and Monitoring Azure Monitor manages metrics, logs and alerts. To further analyze logs, create a log analytics workspaces, where you can run Kusto queries. You can create a workbook and embed Kusto queries into visual objects on the workbook.\nStorage Azure manages storage resources under storage account. The resource classes include Blob (object), File, Queue, Table and Disk (block). They have a few acronyms on redundancy levels.\nRedundancy OptionData CopyAccess levelLRS \u0026#8211; Locally redundantsynchronously copy your data three times within the AZ in the primary region.Write is acknowledged after three synchronous writes.ZRS \u0026#8211; Zone-redundantsynchronously copy your data across three AZs in the primary region.\nyour data is still accessible for both read and write even if one AZ becomes unavailable.Write is acknowledged after three synchronous writes.\nIf an AZ becomes unavailable, Azure undertakes networking updates (e.g. DNS re-pointing). Application may perceive a blip where re-try policies may help.GRS \u0026#8211; Geo-redundantLRS in primary region +\nasynchronously copy your data to a single AZ in the secondary region + LRS in secondary region\nyour data in the secondary region isn\u0026#8217;t available for read or write access unless there is a failover to the secondary region.\nfor read access to the secondary region, configure your storage account to use RA-GRS (read-access geo-redundant storage)\nIf the primary region becomes unavailable, you can choose to fail over to the secondary region. After the failover has completed, the secondary region becomes the primary region, and you can again read and write data.GZRS \u0026#8211; Geo-zone-redundantZRS in primary region +\nasynchronously copy your data to a single AZ in the secondary region + LRS in secondary regionyour data in the secondary region isn\u0026#8217;t available for read or write access unless there is a failover to the secondary region.\nfor read access to the secondary region, configure your storage account to use RA-GZRS (read-access geo-zone-redundant storage)\nIf the primary region becomes unavailable, you can choose to fail over to the secondary region. After the failover has completed, the secondary region becomes the primary region, and you can again read and write data. The disaster recovery and failover happens at storage account level.\nPrevious PostLogging and Monitoring in Kubernetes with PLG stack Next PostInfrastructure deployment in Terraform 2/2 ","date":"2021-10-25T23:13:52-04:00","image":"/wp-content/uploads/2025/04/feature-azure-lesson.webp","permalink":"/2021/10/notes-on-azure/","title":"Azure Deets"},{"content":"We\u0026#8217;ve checked out the the actors in PLG stack (Promtail, Loki, Node Exporter, Prometheus, Grafana) and whipped up a quick pipeline on MacOS. Now I\u0026#8217;m going a little further to implement the same PLG stack (Prometheus Loki and Grafana) in a Kubernetes cluster. This setup is for demo only, therefore no persistent storage is enabled.\nTest Workload I host a deployment of Flog with three pods running on Minikube. Flog is an open-source emulating log generation behaviour of an application. On the Minikube cluster we start the deployment as below:\nminikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2 kubectl create ns obsv kubectl -n obsv create deployment flog --image=mingrammer/flog --replicas=3 -- flog -f rfc3164 -l -d 300ms kubectl -n obsv get po The Pods will come up in a heartbeat. I will use helm to install the objects required for logging and metrics pipelines. There are multiple Helm charts for each components. Some high-level charts (usually with a name suffix of -stack) contain several other resource as sub-charts. They are created as one-stop-shop for multiple components but I found none of them serve my exact purpose. For example, both loki-stack and kube-prometheus-stack include Grafana. But I only need one instance of Grafana. Therefore I stick to the low-level charts.\nLog Shipping Add a helm repo, and install loki and promtail. Note that we need to specify correct loki address when installing Promtail. helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm upgrade --namespace obsv --install loki grafana/loki helm upgrade --namespace obsv --install promtail grafana/promtail --set \u0026#34;config.lokiAddress=http://loki.obsv.svc.cluster.local:3100/loki/api/v1/push\u0026#34; When installing Promtail, a DaemonSet is created on the Node. The default configuration applies appropriate configuration and tagging strategy for Kubernetes Pod and Node. So the only customization I specified is Loki address. We can then check logging with Loki. To do so, first expose port 3100 to host, and then use logcli (e.g. on MacOS) to query for logs:\nkubectl -n obsv port-forward service/loki 3100:3100 logcli labels logcli labels pod logcli query \u0026#39;{pod=\u0026#34;flog-775d5fc5c8-p4rlx\u0026#34;}\u0026#39; Log lines should be pumped to Loki a minute after Loki comes up. The logcli labels command should display the tags. The logcli query command should return the log lines. Metrics I use Premetheus with node exporter. In its architecture, Prometheus contain the server, the pushgateway, and alertmanager. The helm chart for Prometheus contains all of those components. It also has a dependency repo for kube-state-metrics. To install:\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics helm --namespace=obsv install prometheus prometheus-community/prometheus kubectl -n obsv get svc Once installed, a DaemonSet for Prometheus Node Exporter is created. The exporter is already configured by default for Kubernetes monitoring. The Prometheus server is also configured, on port 80 by default. it needs to be forwarded in order to access from Browser:\nkubectl --namespace obsv port-forward service/prometheus-server 9100:80 To verify installation of Prometheus, browse to localhost:9100 to examine the metrics.\nVisualization Last but not least, I will configure Grafana. The repo has been added already so we\u0026#8217;ll just install the chart:\nhelm upgrade --namespace obsv --install grafana grafana/grafana kubectl get secret --namespace obsv grafana -o jsonpath=\u0026#34;{.data.admin-password}\u0026#34; | base64 --decode ; echo The second command retrieves the credential. To access the web portal, we need port forwarding again:\nkubectl port-forward --namespace obsv service/grafana 3000:80 To verify installation, browse to http://localhost:3000 and log in as user admin with the password above. Then add two data sources:\nType: Prometheus, URL: http://prometheus-server.obsv.svc.cluster.local:80 Type: Loki, URL: http://loki.obsv.svc.cluster.local:3100 Then we can explore data using both data sources.\nPLG stack Summary In the last two posts I reviewed the setups for PLG stack in Kubernetes, from a regular environment to k8s cluster. Fluentd, Prometheus are both CNCF projects. The PLG stack seems to be more adopted than EFK but both have their own advantages. Welcome to the PLG vs EFK debate.\nPrevious PostIntro to PLG stack -Prometheus, Loki and Grafana Next PostAzure Deets ","date":"2021-10-13T21:29:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-plg.webp","permalink":"/2021/10/logging-and-monitoring-in-kubernetes-with-plg-stack/","title":"Logging and Monitoring in Kubernetes with PLG stack"},{"content":"Last month we discussed log shipping with EFK. This week I spent sometime checking out its alternative Loki. Having been exposed to the ELK stack extensively, I am also interested in exploring the counterparts in this new stack, such as Prometheus, Loki and Grafana. So I need to address the issues of shipping both metrics and logs.\nLet\u0026#8217;s start by clarifying the terms:\nGrafana is a visualizer. It supports many backends such as Prometheus, Loki, Elasticsearch, CloudWatch and Azure Monitor. It is the flagship product of Grafana Labs. Prometheus is a time-series database and alerting platform. To push metrics to Premethus, you can either integrate your application with client library (in their term, instrumenting), or configure an existing exporters for a third party application such as PostgreSQL. Prometheus collects and stores its metrics as time series data ( i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels) and it comes with basic visualization capability. Premetheus is a CNCF project since 2016 and is maintained by Grafana Labs. Loki is a log aggregation system, also developed by Grafana Labs. Loki does not index the contents of the logs. Instead it groups entries into streams, and indexes a set of labels for each log stream. You can use grafana or logcli to consume the logs. Loki supports clients such as Fluentd, Fluentbit, Logstash and Promtail. Promtail is a log collection agent built for Loki. The roles of the components such as Prometheus, Loki, Grafana and Promtail are similar to the ELK stack. Grafana resembles Kibana. Promtail resembles Filebeat. Premethus exporters resemble Metricbeat. Both Premetheus and Loki resemble Elasticsearch in some aspects. Premetheus keeps metrics and Loki persists log streams. These tools are heavily used in Kubernetes. For a simple start, I\u0026#8217;d like to just configure two minimally working pipelines on my MacBook without any containerization. My example setup is to achieve the followings:\nFake up some log lines and ship them to Loki Ship OS metrics to Prometheus Display the metrics and logs with Grafana Here\u0026#8217;s the diagram of what we want to implement:\nPromtailPromtailNode ExporterNode ExporterLokiLokiPrometheusPrometheusGrafanaGrafanaLog FileLog FileCollectorsCollectorsViewer does not support full SVG 1.1\nThis is simple enough to build a quick and dirty setup\nConfigure Node Exporter We can use home brew to install node_exporter on MacOS. For Linux, use the equivalent package management tool:\nbrew install node_exporter brew services start node_exporter curl http://localhost:9100/metrics | grep \u0026#34;node_\u0026#34; The node_exporter collect metrics and make it available for scrape using port 9100 as shown above. Configure Prometheus Install Prometheus with homebrew and start the service:\nbrew install prometheus brew services start prometheus If curl to port 9090 with GET returns \u0026#8220;Found\u0026#8221;, then Prometheus is successfully installed. We also want to configure it so it scrapes node exporter for metrics. Edit the configuration file, in my case, /usr/local/etc/prometheus.yml, by adding the followings:\n- job_name: node static_configs: - targets: [\u0026#39;localhost:9100\u0026#39;] This tells Prometheus to scrape metrics at specified interval. Restart Prometheus and browse to http://localhost:9090/ for Prometheus UI. Click on Status -\u0026gt; Targets and you can see the node as an export. Click on Graph and Execute a query for example \u0026#8220;node_memory_free_bytes\u0026#8221; and click on Graph. You should see a plot of the metric value.\nWe can later configure to display the chart in Grafana.\nGenerate log lines We use a tool called flog to generate fake logs:\nbrew tap mingrammer/flog brew install flog flog -f rfc3164 -l -d 300ms -t log -o /tmp/test.log -w The command above produces log in RFC3164 format to /tmp/test.log, one line every 300ms. The log does not rotate.\nConfigure Loki Install Loki with homebrew and start the service. We want to configure Loki before Promtail so it is ready to receive logs. You might as well install LogCLI to interact with Loki.\nbrew install loki brew install logcli brew services start loki If nc to port 3100 returns success, then Loki is successfully installed. Configure Promtail Install promtail using homebrew:\nbrew install promtail Note that Promtail is not installed as a homebrew service. Although it can be manually configured as a service, I\u0026#8217;d rather stay focused and use command line just for the time being. A copy of configuration file is located in /usr/local/etc/promtail-local-config.yaml but it needs to be modified first with a job to tell it where to scrap the log lines. The configuration looks like this:\nserver: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://localhost:3100/loki/api/v1/push scrape_configs: - job_name: app static_configs: - targets: - localhost labels: job: applogs __path__: /tmp/test.log Then, use the command as in the documentation to start promtail:\npromtail -config.file /usr/local/etc/promtail-local-config.yaml Port 9080 will be open when Promtail is running. Promtail will push logs to Loki. You can use LogCLI to interact with Loki and see the latest log lines pushed to Loki. logcli labels job logcli query \u0026#39;{job=\u0026#34;applogs\u0026#34;}\u0026#39; However, unlike Prometheus, Loki itself does not have any visualization capabilities. We will need Grafana to display the log nicely on the web.\nConfigure Grafana We use Grafana to visualize both the logs and metrics. To install Grafana on Mac:\nbrew install grafana brew services start grafana If curl to port 3000 with GET returns \u0026#8220;Found\u0026#8221;, then Grafana is successfully installed. Browse to localhost:3000, with default credential admin and admin. From the UI, add two data sources. For the first data source, specify Prometheus as type and localhost:9000 as destination. For the second, specify Loki as the destination http://localhost:3100\nThe steps to see logs in Loki is pretty much the same as on this page of its documentation.\nExploring logs Exploring metrics data is similar. Click on Explore on the side bar, then select Prometheus from the dropdown as data source. Then execute a query such as \u0026#8220;node_memory_free_bytes\u0026#8221;.\nExplore Metrics With this quick and dirty configuration, we established a good understanding of what Prometheus, Loki and Grafana do. Next, we will move all these configurations to K8s cluster and understand some specific points of configurations.\nPrevious PostFile storage vs object storage in the cloud Next PostLogging and Monitoring in Kubernetes with PLG stack ","date":"2021-10-03T12:59:00-04:00","image":"/wp-content/uploads/2025/04/feature-plg-intro.webp","permalink":"/2021/10/intro-to-plg-stack-prometheus-loki-and-grafana/","title":"Intro to PLG stack -Prometheus, Loki and Grafana"},{"content":"File storage (e.g. NFS) used to be prevalent until object storage comes in for competition.\nThe competition Traditionally, enterprise storage product lines are built around three capabilities, as listed in this table below:\nCapabilityTypical ImplementationData servedT1 \u0026#8211; Block stroageDAS (e.g. SAS cable) or SAN (Fibre Cable for FCP protocol, or Ethernet for iSCSI protocol)Mission critical data that are extremely sensitive to latency (e.g. database). Client has block-level access.T2 \u0026#8211; File storageNAS (connect via CIFS or NFS protocols). Storage arrays are typically a mix of HDD and SSD. Storage servers are usually deployed in the same location over low latency network. DR location is usually in the same region.Hot data. Multiple client access at file level. The size of each data request varies from small to medium (e.g. text document)T3 \u0026#8211; Object storageHardware agnostic, connect via layer-7 protocol (e.g. S3). Storage backend can be either on premise, or in the cloud, over WAN connection.Warm and code data. Multiple client access at object level. Traditionally for backup but use cases are expanding. The size of each data request varies significantly, from small to very large (e.g. media content). In the last couple decades, leading players for T2 have been enterprise storage vendors. They each have developed their secret sauces to tackle the challenges. For example, EMC has OneFS, a parallel distributed file system as the foundation of PowerScale (formerly Isilon) product line. NetApp develops ONTAP, featuring proprietary techniques for storage efficiency (deduplication, compaction and compression).\u0026nbsp;\nThe leading players in T3 are mostly public cloud provider, such as Amazon\u0026#8217;s S3. They might work with enterprise storage vendor behind the scene. But the T3 services appear to the end users as provided by the public cloud. Originally, the use case for T3 was archive only for its virtually unlimited capacity. This is not entirely true today. With the drastic improvement in modern network infrastructure, T3 can also brings satisfactory performance to serve hot data. A competition between T2 and T3 arises. After all, both offer storage service over Ethernet, and both support multiple clients. Today when developers architect the storage layer of their applications, they need to weigh between supporting T2 and T3. Since NFS is the typical protocol for T2 storage (sorry Windows guys) and S3 is typical T3 storage. This competition essentially boils down to NFS versus S3.\nFor many, the fancy S3 is a no-brainer. While I have suffered from many NFS drawbacks, and there\u0026#8217;s even a whole article by Linux folks about why NFS sucks, is it sentenced to death today? Does it beat S3 in some cases? Do so many organizations still stick to NFS just out of inertia?\nTo answer these questions, I examine four aspects to explore the differences between file storage via NFS protocol, and object storage in S3. Data request size Storage client can make request by byte range of a file. Therefore, data request size, instead of file size, is what ultimately matters. I pick a few data request sizes (1K, 4K, 16K, 64K, 246K, 1024K and 4096K) in my experiment, and want to see how much network traffic a write operation produces using NFS and using S3.\nTo emulate request size, I created files at each size (using dd command), and copy the entire file to each backend. In the mean time, I use tcpdump to write out traffic across the wire into capture files. The size of capture file gives me an idea of how much network traffic went through the network interface, which is closely related to latency. For NFS, I mounted the target with sync option. This requires NFS client to write out to server synchronously on file copy. I\u0026#8217;ve also set the wsize to be 1M. For S3, I simply use the following CLI command to copy file:\naws s3 cp 1kb.img s3://digihunch5ffafe32ab0fd40f On the network interface, I use tcpdump to filter traffic through specific TCP port (443 for S3, or 2049 for NFS) and record the size of the capture file:\nsudo tcpdump -s0 -pi eth0 dst port 443 or src port 443 -w /tmp/4096kb.cap The key indicator is the payload size (file size) as a percentage of the capture size. I call it payload ratio. The closer it is to 1, the better. I have the following result from my experiment:\nRequestPayloadS3 capture size (byte)NFS capture size (byte)S3 payload ratioNFS payload ratio1K1024935243320.110.244K40961264074040.320.5516K1638425969204720.630.8064K6553679183697460.830.94256K2621442900352714080.900.971024K1048576108536610740760.970.984096K4194304438154742869100.960.98 This result indicates that NFS has a higher ratio in all groups. However, its advantage diminishes as the data request size grows. What it tells us is that if your applications workload issues most request in small chunks of data, such as 1K, 4K, then NFS will require much less traffic over the network, and thus less latency. This essentially explains the use case of NFS against S3: workload with small data requests.\nClient Support NFS is natively supported by Linux operating system kernel. NFS client sits below the virtual file system (VFS) layer, which sits below the system call layer. The NFS client translate system calls into RPC (remote procedure calls). Communication between client and server is completed with RPC, on top of TCP. NFS architecture Because of the native support, in most cases, developer can treat NFS mounts as if they were local. For performance to be sustainable as file system grows, the directory structure on NFS should follow a certain naming conventions so that files are evenly distributed across directories. The client should also use list operation as sparse as it can because that operation is expensive across the network.\nFrom developer\u0026#8217;s perspective, NFS support is brought in by operating system and does not require much effort. On the other hand, S3 client support is not included by default in the operating system. S3 support requires special library, code changes, and integration effort to manage dependency and library version. NFS has an advantage on client supportability. However, as we move applications to containers, and as container storage options mature, we will need an intermediary layer (storage class, storage provisioner, CSI driver, etc), NFS, or in general file storage, does not have this advantage any more.\nClient-side Cache The NFS support behind VFS layer also means it can leverage the I/O caching mechanism on the client side, that comes with operating system. Client operating system with sufficient memory can take advantage of this mechanism to give it a performance boost. Check out this guide for NFS cache tuning.\nIn comparison, S3 does not have a cache mechanism by itself. Either the application needs to implement its own cache mechanism, or a cache architecture needs to be introduced, such as CloudFront. Consistency and concurrency A common consistency problem is whether client can read the changes immediately after it writes the file. S3 and NFS make a tie in this round.\nS3 originally came with eventual consistency model for read after write since 2006. As of Dec 2020 it introduced strong read-after-write consistency. For more information, refer to the guide here.\nNFS has a similar consistency guarantee called close-to-open cache coherency. Any changes made by client are flushed to the server on closing the file, and a cache revalidation occurs when you re-open it. There are more to consider in terms of consistency. For example, multiple clients tries to write the same file/object at the same time. On the S3 side, there is a locking mechanism called S3 object lock at object level (no byte-range lock). Without an object lock, when two PUT requests are simultaneously made to an object, the request with the latest timestamp wins. Refer to the section Concurrent application on this page.\nAs far as NFS goes, managing this kind of consistency problem is not in the scope of the standard. Although there are some tinkers. For example, NFS v4 includes a file locking mechanism. Client can choose to lock the entire file, or a byte range within the file. Locking can be mandatory or advisory.\nThe convergence NFS and S3 each has their respective advantage. Enterprise NAS customers have been looking for ways to expand into the cloud for lower storage cost. To combine the advantages of the two, solution providers started to converge file storage and object storage. There are two types of solutions that reflects this trend of convergence. In the first trend, enterprise NAS deployed on premise now have the ability to scale out into the cloud. In the second trend, public cloud just brought enterprise NAS into their product offerings.\nScale-out NAS NAS is traditionally expensive to scale because it requires physical storage media. The idea of scale-out NAS allows NAS to connect to object storage in the public cloud, making it a hybrid architecture. This essentially makes T3 storage as a backend of T2 and it can be implemented with a virtual storage appliance (VSA). The VSA translate file system activities into API calls for object storage operations. One example is AWS storage gateway. EMC has a similar appliance called ECS and this white paper explains how it proxies file system calls and interact with object backends. NetApp, a vested enterprise NAS provider, also has a counterpart called Cloud Volumes ONTAP (CVO). It works well with NetApp on-premise deployment, but the architecture is similar. Here\u0026#8216;s NetApp\u0026#8217;s take on how CVO is different than AWS Storage Gateway.\nIn the scale-out NAS architecture, the public cloud acts merely as extension to on-premise storage solution, to provide capacity. The NAS on premise serves the storage workload primarily.\nCloud hosted NAS For applications hosted in public cloud, it makes sense for public cloud provider to operate enterprise NAS storage as a service. The underlying storage technology is provided by storage vendor. It is just installed in the data centre managed by the public cloud vendor, instead of customer\u0026#8217;s own data centre. One example is Azure NetApp Files (ANF). ANF is fully managed services, presented to users as storage volumes. The underlying storage technology is NetApp ONTAP. Because it is offered as a fully managed service, the customers are not able to manage the fine details of the storage, as they could with an ONTAP cluster on premise. This takes a lot of flexibility away from the user.\nFSx ONTAP is a managed NetApp storage service by AWS, launched in September 2021. The NetApp arrays are installed in AWS data centre, ready for users to provision from AWS console, or using CLI. The Terraform provider support is not available as of yet. Unlike ANF, FSx ONTAP exposes the ONTAP CLI to users, allowing for advanced storage managed by storage gurus. They can use ONTAP CLI commands to configure custom policy for Snapshot, setup SnapMirror replication, and so forth.\nLikewise, PowerScale landed on GCP as public cloud partner to launch Dell Cloud PowerScale for Google Cloud in 2020. However, it seems to require a purchase agreement before APIs are enabled.\nConclusion Object storage has a great momentum and some sees that as a replacement of file storage in the long run. However file storage has its advantages for small data requests, OS-level cache support, and built-in POSIX compatibility. It will continue to be an option for customers with specific workload. Customer stickiness to file storage is so firm, that public cloud providers now install them in their data centres. From competition to collaboration, it will be interesting to watch what happens next for enterprise storage.\nFollow-up Reading Tom Lyon\u0026#8217;s presentation on why NFS must die.\nPrevious PostLocal multi-node cluster – Minikube, MicroK8s and KinD Next PostIntro to PLG stack -Prometheus, Loki and Grafana ","date":"2021-09-23T22:54:00-04:00","image":"/wp-content/uploads/2025/04/feature-file-obj-storage.webp","permalink":"/2021/09/file-storage-vs-object-storage/","title":"File storage vs object storage in the cloud"},{"content":"In this post we compare Minikube, MicroK8s and KinD as different approaches to build multi-node cluster locally.\nIs Docker desktop bad? In the previous post about docker desktop as a single-node Kubernetes cluster setup, I touched on the deprecation of docker-shim. Now that CRI beats OCI as the standard for container runtime, the docker runtime will no longer be supported by Kubernetes. Also deprecated is docker-shim, the temporary interface that had make Docker runtime work in Kubernetes. This was announced in December 2020, and is coming through in Kubernetes 1.23, expected Oct 2021. However, docker desktop still uses docker runtime in it\u0026#8217;s single-node Kubernetes cluster. This essentially renders itself a non-compliant Kubernetes environment. Docker desktop still has great value for application developers. If your role is development, spending a lot of time coding business logics and need an easy-to-use container runtime on your laptop, Docker desktop is a good choice. The recent moves by the company seems to suggest that this is the business they are targeting now. On the other hand, if your roles are deployment, automation, orchestration, cloud native etc and you are looking for a playground, most likely you do need a runtime compliant to Kubernetes CRI. Docker desktop is not a CNCF-certified project anymore, and it is not your choice. Alternatives There are a number of alternatives, the most well-known ones are Minikube, MicroK8s, KinD and K3s with K3d. This presentation from CNCF in 2020 covers a lot of details about these technologies. I\u0026#8217;ll try to add my opinion.\nK3s is Rancher Lab\u0026#8217;s lightweight Kubernetes distribution that supports multi-node cluster as well as different runtimes (e.g. containerd). It is not straightforward to setup, and k3d is an command-line wrapper to make it easy to install K3s cluster. K3s was accepted as a CNCF project but only at Sandbox maturity level, so it is not my choice. The other three: Minikue, MicroK8s and KinD are all certified CNCF project. I will further discuss how to choose among them. These projects are technologies that takes different approach to address the challenges with deploying multiple nodes in local environment (e.g. my laptop). The challenge with running a Kubernetes cluster with multiple nodes locally is how to manage these nodes. They are separate virtual resources that need to be isolated from computing perspective, and connected as a cluster. This is typically the use case of a Type II hypervisor, or alternatively, it can also be implemented with container technology. This layer of technology (referred to as drivers) makes a big difference.\nMinikube Minikube supports multiple drivers. Depending on your platform (Windows, Linux, or MacOS), the preferred driver is different. Refer to the document here for preferred driver, and this blog post for more instructions. In addition to the documents, here some notes from my personal experience:\nOn MacOS, Minikube lists Docker as preferred driver. I disagree with that. If you have no other reason to install Docker, then I would recommend hyperkit as the the preferred driver. Hyperkit can be installed with a simple Homebrew command. For two reasons I do not recommend Docker as the driver of Minikube. First, it requires a separate installation of Docker Desktop, which includes a built-in instance of hyperkit on its own. This isn\u0026#8217;t neat. Second, I often need Metal LB add-on with Minikube for testing Kubernetes Ingress. With Minikube on Docker, the Ingress ports are not exposed to MacOS\u0026#8217;s. Therefore you cannot directly visit websites spun up on Minikube. This is a known issue for a while due to limitation on docker bridge with Mac. Some reported an ugly workaround with docker-mac-net-connect but I never got it to work. On Windows native environment, the preferred driver is hyper-V. The Minikube cli command have to run from Windows PowerShell. On WSL2, Minikube doesn\u0026#8217;t play well, regardless of driver. The hyperkit driver won\u0026#8217;t work (it is designed for MacOS only). The kvm2 driver would require a KVM2 hypervisor. However, WSL2 itself is a VM on top of hypervisor, as explained here. If KVM2 driver works it would require nested virtualization so I doubt it will ever be supported. As for Docker on WSL2 as driver, Minikube has it as an experimental feature, and requires configuring cgroup to allow setting memory. I am not confident with it. To me, Minikube is the tool for MacOS (I have Intel processor). On MacOS, we first need to install minikube and hyperkit with home brew.\nWe can then start a kubernetes cluster, with minikube in a single command. I noticed a process on my MacBook called dnscrypt-proxy that conflicts with hyperkit DNS server when starting minikube. I had to remove dnscrypt-proxy (part of Cisco Umbrella Roaming Client) in order to get minikube to work, as this thread suggests. You can find out by running:\nsudo lsof -i :53 If dnscrypt-proxy is running, find out the application by PID and remove the application. Otherwise there will be issues. Check out this section on the document. The commands that I use to start multi-node cluster is:\nminikube start --driver=hyperkit --container-runtime=containerd --memory=12288 --cpus=2 --disk-size=150g --nodes 3 kubectl get po -A kubectl describe node minikube|grep Runtime Node administration is simple. To enable dashboard, simply run \u0026#8220;minikube dashboard\u0026#8221;. To SSH to a node, simply do \u0026#8220;minikube ssh -n \u0026lt;node_name\u0026gt;\u0026#8221;. In order to stop the node and delete cluster, run \u0026#8220;minikube stop \u0026amp;\u0026amp; minikube delete\u0026#8221;.\nThere are some addons in minikube, for example, efk, gvisor, istio, metrics-server. To list add-ons, and enable metrics-server, for example, run:\nminikube addons list minikube addons enable metrics-server When creating cluster, instead of specifying the cluster imperatively, the configuration (e.g. driver, container runtime, cpu, memory, number of nodes, etc) can be stored as a profile with -p switch. Like other Minikube configuration information, Minikube profiles are stored in ~/.minikube under the profile directory.\nMinikube also has a page that benchmarks the performance of these technologies, where it presents itself as the most performant.\nMinikube, KinD, k3d and microK8s performance MicroK8s MicroK8s is developed by Canonical. It can use either Multipass or LXD container as driver. Multipass can configure Ubuntu VMs using cloud-init. It supports multiple hypervisor backends as well but hyperkit is the default on MacOS, Hyper-V on Windows, and KVM on Linux.\nMicroK8s supports multi-node configuration across multiple machines. That is, nodes can span across multiple physical machines. This is more powerful than Minikube where multiple nodes are on the same physical machine. It brings MicroK8s additional use cases such as edge and IoT devices.\nWith that capability comes the extra step to configure a MicroK8s cluster. You will need to manually join a node to a cluster because the new node is potentially located on a different machine, and you execute the command from the new machine. On the other hand, with Minikube you simply specify the number of nodes desired in a command or profile.\nSnap is the native package manager to install MicroK8s, making GNU Linux (e.g. Ubuntu) the native platform. It also supports MacOS and Windows. MicroK8s does not rely on Docker (unlike KinD and Minikube with Docker as driver), and uses containerd as runtime.\nMicroK8s comes with its own packaged version of kubectl, and you use that with \u0026#8220;microk8s kubectl\u0026#8221; command, which is not convenient. You can configure your host kubectl to point to the MicroK8s cluster, as an extra step.\nCompared to the other two technologies, MicroK8s is more powerful in the sense that the cluster is build on nodes across multiple machines. However, it takes more step to configure even for a multi-node, single-machine environment. Refer to this post for the steps.\nKinD KinD is similar to Minikube with Docker as driver. It is more restricted than Minikube considering Docker is the only driver it supports. This makes it a requirement to have Docker installed locally.\nAlthough KinD uses Docker to run nodes, it does not use Docker as its container runtime. Therefore it remains as compliant environment.\nAnother advantage of KinD is it supports Docker on WSL2 very well. Simply install KinD on WSL2 and start Docker. This blog post contains the steps required to install KinD vs Minikube on WSL2. There is a comparison table in the conclusion section that highlights the fact that it is much easier to install KinD with WSL2 than to install Minikube.\nHowever, there are currently some known limitations with Docker desktop for Windows (including on WSL2). One is the absence of docker0 bridge. This means on Windows you cannot route traffic to the containers.\nFor cluster specification, KinD can configure a cluster declaratively using YAML file for example, the kind-config.yaml contains the following snippet:\nkind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker - role: worker networking: disableDefaultCNI: true We can bring up a cluster with a command:\nkind create cluster --config=kind-config.yaml The command will also configure the kubectl context so we can check node with kubectl command. The file is in my real-quicK-cluster repo.\nConclusion After reviewing the technologies that back up multi-node kubernetes cluster for my role, I find that Minikube with hyperkit is my favourite for MacOS. On WSL2, I prefer to use KinD. Since I do not use Windows native environment or Ubuntu on my laptop, I cannot make recommendations. However I would start with Minikube (with hypverv or kvm2 as driver). Update July 2022: When the test workload involves persistent storage, KinD is a better choice. When the test workload involves load balancer. Minikube is a better choice.\nAs to storage provisioner, Minikube with storage-provisioner addon uses k8s.io/minikube-hostpath. KinD uses rancher.io/local-path. When I have to test workload with persistent storage (e.g. PostgreSQL with Crunchy pgo), I realized Minikube have permission issues with persistent volume, as discussed here as an issue with multiple nodes. The issue has been open since Aug 2021.\nFor Load Balancer, Minikube has metallb as an addon and I can configure it within a bash script conveniently. With KinD, I\u0026#8217;d have to configure that in a few steps with both kubectl and Docker CLI commands and I was not able to connect to the load balancer by IP even after following the steps. So I tend to just use Minikube to test workload requiring load balancer and service mesh. I find myself switch between Minikube and KinD on my MacBook depending on the test workload.\nPrevious PostLog Shipping in Kubernetes with EFK stack Next PostFile storage vs object storage in the cloud ","date":"2021-09-14T11:18:00-04:00","image":"/wp-content/uploads/2025/04/feature-multi-node-k8s.webp","permalink":"/2021/09/single-node-kubernetes-cluster-minikube/","title":"Local multi-node cluster – Minikube, MicroK8s and KinD"},{"content":"I first worked on log shipping with ELK stack three years ago. In the context of Kubernetes cluster, log shipping has similar challenges. In this post I will discuss the set up of log shipping with Kubernetes cluster using EFK stack\nLogging Architecture As discussed, if the Kubernetes cluster has a runtime in compliant with CRI (e.g. containerd), then the stdout and stderr of the Pod is stored on the node, in the location /var/log/containers/.\nWhen creating log shipping solution, it is important to use a compliant cluster (e.g. minikube) to ensure what you develop will work across environments. The Kubernetes document has a section on logging architecture which is a good start point. It outlines several different patterns. Logging at the node level is turned on by default and does not require special configuration, as explained in the section above. The EFK pattern is close to the diagram under using a node logging agent for cluster-level logging.\nNode logging agent The diagram above is stolen from Kubernetes documentation. In EFK stack, the agent is a daemonset running fluentd Pod. EFK stack While we can use ELK (Elasticsearch, Logstash, Kibana) stack for log shipping, EFK (Elasticsearch, Fluentd, Kibana) is generally recommended in Kubernetes cluster. We compare the two in the following table:\nELKEFKDevelopment languageAll in Java. Logstash and Filebeat both require JVM. Managed as an open-source project by Elastic companyFluentd in Ruby and does not require JVM to run. Fluentd is a CNCF project built to integrate with Kubernetes.Typical patternFilebeat acts as a lightweight collector to monitor the source log. Logstash as aggregator to receive from filebeat, and push to ElasticsearchThe fluentd Pod can be configured to serve as forwarder and aggregator based on configuration. fluentd-forwarder is deployed as daemonset on node, and ship the result to fluentd-aggregator, which may run in a separate cluster. The fluentd-aggregator pushes processed results to Elasticsearch.Metricsuse metric beat for data collectionscrape metrics from prometheus serverEvent Routingtag-basedif-then statementELK and EFK stacks comparison The main problems in log shipping are:\ncolumn mapping : identify column patterns in each log line and map them to appropriate column in Elasticsearch. multi-line processing: identify when a logging entry spread across multiple lines and process accordingly. Suppose we want to congregate the logs from stdout and stderr of PostgreSQL pods. The raw output in /var/log/container on the node, looks like this:\n2021-08-28T15:27:46.75370563Z stdout F server stopped 2021-08-28T15:27:46.757173069Z stderr F postgresql-repmgr 15:27:46.75 INFO ==\u0026gt; Starting PostgreSQL in background... 2021-08-28T15:27:46.883126928Z stderr F postgresql-repmgr 15:27:46.88 INFO ==\u0026gt; Registering Primary... 2021-08-28T15:27:47.017164653Z stderr F postgresql-repmgr 15:27:47.01 INFO ==\u0026gt; Loading custom scripts... 2021-08-28T15:27:47.023334611Z stderr F postgresql-repmgr 15:27:47.02 INFO ==\u0026gt; Loading user\u0026#39;s custom files from /docker-entrypoint-initdb.d ... 2021-08-28T15:27:47.026169813Z stderr F postgresql-repmgr 15:27:47.02 INFO ==\u0026gt; Starting PostgreSQL in background... 2021-08-28T15:27:47.343607487Z stderr F postgresql-repmgr 15:27:47.34 INFO ==\u0026gt; Stopping PostgreSQL... 2021-08-28T15:27:47.448111425Z stdout F waiting for server to shut down.... done 2021-08-28T15:27:47.448172479Z stdout F server stopped 2021-08-28T15:27:47.453722807Z stderr F postgresql-repmgr 15:27:47.45 INFO ==\u0026gt; ** PostgreSQL with Replication Manager setup finished! ** 2021-08-28T15:27:47.453829953Z stdout F 2021-08-28T15:27:47.503516746Z stderr F postgresql-repmgr 15:27:47.50 INFO ==\u0026gt; Starting PostgreSQL in background... 2021-08-28T15:27:47.532558987Z stdout F waiting for server to start....2021-08-28 15:27:47.532 GMT [273] LOG: pgaudit extension initialized 2021-08-28T15:27:47.533307459Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv4 address \u0026#34;0.0.0.0\u0026#34;, port 5432 2021-08-28T15:27:47.533466407Z stdout F 2021-08-28 15:27:47.533 GMT [273] LOG: listening on IPv6 address \u0026#34;::\u0026#34;, port 5432 2021-08-28T15:27:47.537987947Z stdout F 2021-08-28 15:27:47.537 GMT [273] LOG: listening on Unix socket \u0026#34;/tmp/.s.PGSQL.5432\u0026#34; 2021-08-28T15:27:47.547956465Z stdout F 2021-08-28 15:27:47.547 GMT [273] LOG: redirecting log output to logging collector process 2021-08-28T15:27:47.548005463Z stdout F 2021-08-28 15:27:47.547 GMT [273] HINT: Future log output will appear in directory \u0026#34;/opt/bitnami/postgresql/logs\u0026#34;. 2021-08-28T15:27:47.551741571Z stdout F 2021-08-28 15:27:47.551 GMT [275] LOG: database system was shut down at 2021-08-28 15:27:47 GMT 2021-08-28T15:27:47.558012894Z stdout F 2021-08-28 15:27:47.557 GMT [273] LOG: database system is ready to accept connections 2021-08-28T15:27:47.618577092Z stdout F done 2021-08-28T15:27:47.618708978Z stdout F server started 2021-08-28T15:27:47.630065958Z stderr F postgresql-repmgr 15:27:47.62 INFO ==\u0026gt; ** Starting repmgrd ** 2021-08-28T15:27:47.638116348Z stderr F [2021-08-28 15:27:47] [NOTICE] repmgrd (repmgrd 5.2.1) starting up 2021-08-28T15:27:47.65317627Z stderr F INFO: set_repmgrd_pid(): provided pidfile is /opt/bitnami/repmgr/tmp/repmgr.pid 2021-08-28T15:27:47.653232015Z stderr F [2021-08-28 15:27:47] [NOTICE] starting monitoring of node \u0026#34;orthweb-postgresql-ha-postgresql-0\u0026#34; (ID: 1000) 2021-08-28T15:27:47.681683703Z stderr F [2021-08-28 15:27:47] [NOTICE] monitoring cluster primary \u0026#34;orthweb-postgresql-ha-postgresql-0\u0026#34; (ID: 1000) 2021-08-28T15:28:11.742865958Z stderr F [2021-08-28 15:28:11] [NOTICE] new standby \u0026#34;orthweb-postgresql-ha-postgresql-1\u0026#34; (ID: 1001) has connected From this snippet of log, we can see each line in stdout or stderr is appended with a timestamp. There are multi-line log entries but each is still appended with a timestamp. This is just how kubernetes keeps the log file for Pod stdout and stderr. To handle that, we need to first take out the real log line, and then process multi-line.\nWe will go over the installation of EFK stack and the mechanism to address the two challenges above.\nInstall Elasticsearch and Kibana To install Elasticsearch, we use the helm chart provided by the official repository:\nhelm repo add elastic https://helm.elastic.co If we run multiple pods on the same hosts, then we need some customized values in order to get the installation to work. The values.yaml file looks like this:\n--- antiAffinity: \u0026#34;soft\u0026#34; esJavaOpts: \u0026#34;-Xmx128m -Xms128m\u0026#34; # Allocate smaller chunks of memory per pod. resources: requests: cpu: \u0026#34;100m\u0026#34; memory: \u0026#34;512M\u0026#34; limits: cpu: \u0026#34;1000m\u0026#34; memory: \u0026#34;512M\u0026#34; Then we can \u0026#8220;preview\u0026#8221; what values are used for installation, with helm\u0026#8217;s template command:\nhelm template elasticsearch elastic/elasticsearch -f values.yaml The effect of the antiAffinity property allows multiple Elasticsearch Pod to be scheduled on the same node. This is not required in production with multiple nodes. To install Elasticsearch and Kibana, run:\nhelm install elasticsearch elastic/elasticsearch -f values.yaml helm install kibana elastic/kibana The Kibana service is exposed on port 5601 of the cluster. To access the port on cluster, we need to run port-forward command as below:\nkubectl port-forward deployment/kibana-kibana 5601 Install Fluentd There are different ways to configure Fluentd. For example, in the forwarder-aggregator pattern, a forwarder Pod is a Daemonset on each Kubernetes node. The forwarder pushes to the aggregator, at port 24224. This page has some configuration details. This pattern is similar to filebeat -\u0026gt; logstash pattern in ELK stack.\nIn our case, we use a simplified pattern, with a fluentd daemonset acting as collector and then forward to Elasticsearch. Similarly, in ELK stack we can use filebeat to push to Elasticsearch without Logstash. The only reason is the entire architecture isn\u0026#8217;t as complicated as requiring an aggregator. We need to create configmap as below:\nkind: ConfigMap apiVersion: v1 metadata: name: fluentd-cm namespace: default labels: app.kubernetes.io/component: forwarder app.kubernetes.io/instance: fluentd app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: fluentd helm.sh/chart: fluentd-1.3.0 annotations: meta.helm.sh/release-name: fluentd meta.helm.sh/release-namespace: default data: fluentd.conf: | # Ignore fluentd own events \u0026lt;match fluent.**\u0026gt; @type null \u0026lt;/match\u0026gt; # HTTP input for the liveness and readiness probes \u0026lt;source\u0026gt; @type http port 9880 \u0026lt;/source\u0026gt; # Throw the healthcheck to the standard output instead of forwarding it \u0026lt;match fluentd.healthcheck\u0026gt; @type null \u0026lt;/match\u0026gt; # Get the logs from the containers running in the node \u0026lt;source\u0026gt; @type tail read_from_head true tag kubernetes.* path /var/log/containers/orthweb-postgresql-ha-postgresql-**.log pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos \u0026lt;parse\u0026gt; @type regexp expression ^(?\u0026lt;time\u0026gt;\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}.[^Z]*Z)\\s(?\u0026lt;stream\u0026gt;[^\\s]+)\\s(?\u0026lt;character\u0026gt;[^\\s])\\s(?\u0026lt;message\u0026gt;.*)$ \u0026lt;/parse\u0026gt; \u0026lt;/source\u0026gt; # enrich with kubernetes metadata \u0026lt;filter kubernetes.**\u0026gt; @type kubernetes_metadata @id filter_kube_metadata kubernetes_url \u0026#34;#{ENV[\u0026#39;FLUENT_FILTER_KUBERNETES_URL\u0026#39;] || \u0026#39;https://\u0026#39; + ENV.fetch(\u0026#39;KUBERNETES_SERVICE_HOST\u0026#39;) + \u0026#39;:\u0026#39; + ENV.fetch(\u0026#39;KUBERNETES_SERVICE_PORT\u0026#39;) + \u0026#39;/api\u0026#39;}\u0026#34; verify_ssl \u0026#34;#{ENV[\u0026#39;KUBERNETES_VERIFY_SSL\u0026#39;] || true}\u0026#34; ca_file \u0026#34;#{ENV[\u0026#39;KUBERNETES_CA_FILE\u0026#39;]}\u0026#34; skip_labels \u0026#34;#{ENV[\u0026#39;FLUENT_KUBERNETES_METADATA_SKIP_LABELS\u0026#39;] || \u0026#39;false\u0026#39;}\u0026#34; skip_container_metadata \u0026#34;#{ENV[\u0026#39;FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA\u0026#39;] || \u0026#39;false\u0026#39;}\u0026#34; skip_master_url \u0026#34;#{ENV[\u0026#39;FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL\u0026#39;] || \u0026#39;false\u0026#39;}\u0026#34; skip_namespace_metadata \u0026#34;#{ENV[\u0026#39;FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA\u0026#39;] || \u0026#39;false\u0026#39;}\u0026#34; \u0026lt;/filter\u0026gt; \u0026lt;match kubernetes.var.log.containers.orthweb-postgresql-ha-postgresql-**.log\u0026gt; @type elasticsearch include_tag_key true host \u0026#34;elasticsearch-master.default.svc.cluster.local\u0026#34; port \u0026#34;9200\u0026#34; index_name \u0026#34;postgresql-logs\u0026#34; \u0026lt;buffer\u0026gt; @type file path /opt/bitnami/fluentd/logs/buffers/orthanc-logs.buffer flush_thread_count 2 flush_interval 5s \u0026lt;/buffer\u0026gt; \u0026lt;/match\u0026gt; Then we can create the resource, with helm chart pointing to the config map:\nhelm install fluentd bitnami/fluentd --set aggregator.enabled=false --set forwarder.configMap=fluentd-cm We can validate the index creation on Elasticsearch:\nkubectl port-forward service/elasticsearch-master 9200 curl -XGET localhost:9200/_cat/indices From Kibana, we can forward the port as instructed above, and browse to the UI. Once logged on to Kibana, we need to add index pattern first before being able to view the content of index.\nHow about Fluent Bit Fluentd has an even more lightweight brother Fluent Bit, also a CNCF project, designed by the same team, for environments with tighter resource restrictions. The technical differences are outlined on this page outlines the technical differences. In terms of use case, Fluentd is a log aggregator and Fluent Bit is simply a forwarder. In terms of ecosystem, Fluentd has a stronger ecosystem whereas Fluent Bit is more seen in IoT devices. Read this post for more comparison.\nSummary EFK stack (Elasticsearch, Fluentd and Kibana) on Kubernetes is a natural choice for ELK users. Fluentd is a CNCF project created for integration with Kubernetes. It is good alternative to enterprise solution such as Splunk. There are lots of plugins supported and articles on configuration. While developing a solution I had to spend time reading the input plugin documentations. Previous PostCreating X.509 TLS certificate for workload on Kubernetes Next PostLocal multi-node cluster – Minikube, MicroK8s and KinD ","date":"2021-09-04T21:50:00-04:00","image":"/wp-content/uploads/2025/04/feature-elk-logshipping.webp","permalink":"/2021/09/log-shipping-in-kubernetes-with-efk/","title":"Log Shipping in Kubernetes with EFK stack"},{"content":"In deployment automation, I often had to create self-signed X.509 certificate for testing TLS traffic into Kubernetes. Sometimes self-signed, sometimes signed by a CA. This post summarized the approaches I\u0026#8217;ve taken.\nCreate self-signed certificate with OpenSSL Traditionally, this is done in three OpenSSL commands:\nopenssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj \u0026#39;/CN=Health Certificate Authority\u0026#39; openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj \u0026#39;/CN=*.orthweb.com\u0026#39; openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt I have an older post to cover the basics of cryptography in TLS certificate and PKI. In the three commands above, the first produces a private key and self-signed certificate for a CA. The second creates a private key and a CSR for the web site. The third one uses the CA\u0026#8217;s signing private key to sign the CSR from the website. The output is the certificate for the website. Workloads running in Kubernetes typically consume certificates stored in Kubernetes Secret. The cons of this approach is that it usually requires an extra step to import the certificate files into Kubernetes Secret. For example:\nkubectl create -n orthweb secret generic orthweb-cred --from-file=tls.key=server.key --from-file=tls.crt=server.crt --from-file=ca.crt=ca.crt Note, people use the term self-signed certificate loosely. It sometimes means literally a certificate that is self-signed, like the one generated above. Sometimes, I had to self-sign a CA, then use the CA to sign one certificate for the server and one for the client. Because the application being tested requires that the client and server\u0026#8217;s certificates both under the same CA. This would involve a few more commands, for example:\n# Self-sign a CA openssl req -x509 -sha256 -newkey rsa:4096 -days 365 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=issuer.digihunch.com/emailAddress=info@www.digihunch.com -keyout /tmp/ca.key -out /tmp/ca.crt # Generate a CSR for server openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=server.digihunch.com/emailAddress=orthweb@www.digihunch.com -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com -keyout /tmp/server.key -out /tmp/server.csr # Use the self-signed CA to issue a certificate to the server openssl x509 -req -sha256 -days 3650 -in /tmp/server.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/server.crt -extfile \u0026lt;(echo subjectAltName=DNS:orthweb.digihunch.com,DNS:server2.digihunch.com) # Generate a CSR for clietn openssl req -new -newkey rsa:4096 -nodes -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=client.digihunch.com/emailAddress=client@www.digihunch.com -keyout /tmp/client.key -out /tmp/client.csr # Use the self-signed CA to issue a certificate to the client openssl x509 -req -sha256 -days 365 -in /tmp/client.csr -CA /tmp/ca.crt -CAkey /tmp/ca.key -set_serial 01 -out /tmp/client.crt In the example above, it is important to note that even though the server\u0026#8217;s CSR contains subject alternative name (SAN), I still have to specify the SAN again when signing the certificate for the server. Similar to OpenSSL there are other toolkits such as CFSSL that supports specifying configuration files. However, the steps in Shell command are generally not always easy to automate.\nCreate self-signed certificate with Helm Moving to the context of workload deployment in Kubernetes, running openSSL command isn\u0026#8217;t always a viable option. For example, generating a certificate in the middle of deployment using a Helm Chart. In Helm, template functions is for this purpose. In my Korthweb project I used genSignedCert to create self-signed certificate and then store the key, certificate and CA certificate as Kubernetes Secret:\n{{- $dbtlscert := genSignedCert .Values.dbtls.certCommonName nil (list .Values.dbtls.certCommonName) 365 $ca }} apiVersion: v1 kind: Secret metadata: name: {{ .Values.dbtls.certCommonName | quote }} namespace: {{ $.Release.Namespace | quote }} type: kubernetes.io/tls data: tls.crt: {{ $dbtlscert.Cert | b64enc | quote }} tls.key: {{ $dbtlscert.Key | b64enc | quote }} ca.crt: {{ $ca.Cert | b64enc | quote }} {{- end }} The cons of this approach is that the syntax is not straightforward. As indicated in Helm documentation: Helm Chart templates are written in the\u0026nbsp;Go template language, with the addition of 50 or so add-on template functions\u0026nbsp;from the Sprig library\u0026nbsp;and a few other\u0026nbsp;specialized functions. While we talk about the \u0026#8220;Helm template language\u0026#8221; as if it is Helm-specific, it is actually a combination of the Go template language, some extra functions, and a variety of wrappers to expose certain objects to the templates. Many resources on Go templates may be helpful as you learn about templating.\nCreate self-signed certificate with Cert-Manager The Cert Manager project is very popular to produce X.509 certificates directly in Kubernetes secret. We can install cert manager using Helm:\nkubectl create namespace cert-manager helm repo add jetstack https://charts.jetstack.io helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true kubectl get pods -n cert-manager kubectl get crd | grep cert-manager.io Alternatively, FluxCD\u0026#8217;s documentation on Kustomization dependency uses Cert Manager as an example. It is a good way of installing cert-manager if you have GitOps pattern.\nCreating self-signed certificate for website is fairly simple. It starts with bootstrapping a CA issuer. Take the manifest below as an example. When creating the first certificate, make sure to specify isCA=true, so it stores the signing private key along with its own certificate in the ca-secret. Then use the newly created CA as issuer to create the X.509 certificate for the website.\napiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: my-ca namespace: orthweb spec: isCA: true commonName: my-ca secretName: ca-secret privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned-issuer kind: ClusterIssuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: my-ca-issuer namespace: orthweb spec: ca: secretName: ca-secret --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: orthweb-cert namespace: orthweb spec: commonName: orthweb.com secretName: orthweb-secret duration: 2160h renewBefore: 72h subject: organizations: - digihunch dnsNames: - web.orthweb.com - dcm.orthweb.com privateKey: algorithm: ECDSA size: 256 issuerRef: name: my-ca-issuer kind: Issuer group: cert-manager.io The site certificate is directly stored in Kubernetes Secret as specified in the secretName field. To fetch the certificate text, we need to decode the secret entry, for example:\nkubectl -n orthweb get secret orthweb-secret -o jsonpath=\u0026#39;{.data.ca\\.crt}\u0026#39; | base64 -d Note that the example above uses ECDSA algorithm with size 256 for private key and certificate. It requires that the TLS client to support ECDSA algorithm as well. For more supportability, you can use RSA algorithm (2048 or 4096 size).\nIn addition to creating self-signed certificate, Cert Manager supports a number of other issuer types. For example, the support of ACME issuer type enables integration with Let\u0026#8217;s Encrypt. Cert Manager can secure Kubernetes Ingress resources with a sub-component called ingress-shim. It is configured via annotation on the Ingress resource.\nCert Manager Create CA-signed certificate manually For a certificate signed by a CA, there are may paid options, from manual, to self-help, to automated. The classic manual way is using OpenSSL, generating key, CSR. The CA takes CSR to sign a X.509 certificate returned to the website administration.\nMany CA websites charges for a fee and makes it easy. For example, this site currently uses certificate from SSLs.com. Apart from the fee-for-cert option, there is a website named \u0026#8220;SSL for free\u0026#8220;, a CA with free option for 90-day single-domain, non-wildcard certificate and we can request it simply on their website, with proof of domain ownership. The other popular free option is Let\u0026#8217;s Encrypt, which also employs ACME protocol. The protocol requires ACME challenges to be satisfied in order to proof domain ownership. There are a few types of challenges: HTTP-01 challenge DNS-01 challenge TLS-SNI-01 challenge TLS-ALPN-01 challenge I have used the HTTP-01 and DNS-01 challenges. The DNS-01 challenge requires adding TXT records to DNS configuration. The HTTP-01 challenge requires adding a DNS A-record to resolve to the server, then two URIs with pre-defined value.\nWhen I first set up this site I used certbot (the client program for letsencrypt) to create certificate every 90 days from the wordpress server, following this guide, including solving DNS-01 challenges.\nCreate CA-signed certificate automatically with cert manager and letsencrypt With Kubernetes, cert-manager has the ability to integrate with let\u0026#8217;s encrypt for full automation. Here is a good blog post on this. Domain verification is still required but it can be done automatically. We first need to register an A record that resolves host name to the Ingress IP to enable this automation. The domain ownership validation may use the ACME protocol. This should also work on private networks with private DNS and ACME protocol using a private boulder server.\nTake domain name demo1.digihunch.com for example, if ingress exposes a public IP address which the domain name resolves to, then we can configure certificate with the following manifest:\nkind: IngressClass metadata: name: istio spec: controller: istio.io/ingress-controller --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt spec: acme: privateKeySecretRef: name: letsencrypt server: https://acme-staging-v02.api.letsencrypt.org/directory solvers: - http01: ingress: class: istio --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: demo spec: dnsNames: - demo1.digihunch.com issuerRef: kind: ClusterIssuer name: letsencrypt secretName: demo-tls This example uses Istio as ingress controller but the method works regardless of the controller technology behind Ingress. In the ClusterIssuer object, we\u0026#8217;re telling it to use the staging server from letsencrypt. We also specify http01 as challenge type, and that the ingress type is istio. In the Certificate object, we provided dnsName and specified ClusterIssuser. We also tell it to store the credentials to a secret named demo-tls.\nWhen we apply the resources above, the ClusterIssuer connects to letsencrypt server via ACME protocol. Since the DNS name already resolves to the Public IP that the ingress is hosting, the ClusterIssuer configures the required Ingress, Services and Pods accordingly so the token to satisfy the challenge is presented at the designated URI. Instead of a staging server, we can also use production ACME server for production deployment. Note that the production ACME endpoint has a stricter rate limit.\nWhen the ACME validation is in progress, it is important to ensure that port 80 is open and there is no other mechanism (such as routing rule, authorization requirement, mandatory redirect to 443) that blocks access from letsencrypt server.\nBottom line Cert Manager is deployed in Kubernetes, supporting a variety of issuer types. As a Kubernetes-native tool, it is a no-brainer for Kubernetes workload for X.509 certificate. Compared with using template function in Helm, it is not dependent on template function and the syntax is consistent (YAML). Compared with OpenSSL or other binary tools, it is easy to integrate with the platform.\nPrevious PostSingle-node Kubernetes cluster – docker desktop Next PostLog Shipping in Kubernetes with EFK stack ","date":"2021-08-29T23:19:00-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-x509.webp","permalink":"/2021/08/creating-tls-certificate-kubernetes/","title":"Creating X.509 TLS certificate for workload on Kubernetes"},{"content":"While there are many tools to set up single-node Kubernetes cluster (e.g. minikube, MicroK8s, kind, or k3s with the k3d wrapper), docker-desktop has a significant advantage: it comes with Docker installation, on MacOS, or on Windows. It is installed simply by enabling the option \u0026#8220;Enable Kubernetes\u0026#8221;. It can be blown away and reset in a heartbeat (with the button \u0026#8220;Reset Kubernetes Cluster\u0026#8221;). For its versatility, docker-desktop is a great development environment.\nHowever, there are always nuances, which motivates me to write this blog. I wanted to note down what is on earth different about Docker-desktop, because the instructions for applications might differ slightly between single-node cluster on MacOS/Windows and the \u0026#8220;real\u0026#8221; multi-node cluster. I will start with with a deep dive into the docker-desktop architecture, then we\u0026#8217;ll go through the steps to install some common applications with Kubernetes.\nDocker-Desktop on MacOS There are a number of open-source and proprietary projects involved to bring docker-desktop to implementation. Let\u0026#8217;s begin with the following five:\nHypervisor Framework: Apple\u0026#8217;s APIs on MacOS that allows you to interact with virtualization technologies in user space. bhyve: A type-2 hypervisor initially written for FreeBSD (and was contributed to FreeBSD in May 2011). xhyve: A port of bhyve project to MacOS with integration via Apple\u0026#8217;s Hypervisor Framework. The Hypervisor Framework allows xhyve to run entirely in userspace. It is sometimes loosely referred to as xhyve/bhyve hypervisor, and is optimized for lightweight virtual machines and container deployment. HyperKit is an open-source toolkit on macOS based on xhyve. HyperKit is lightweight and therefore allows you to embed hypervisor capabilities in your application. The hypervisor component in HyperKit is based on xhyve/bhyve. HyperKit is designed to be interfaced with higher-level components such as the VPNKit and DataKit. Docker-desktop and MiniKube are built on HyperKit. LinuxKit is a toolkit for building custom minimal, immutable and purpose-build Linux distributions. It supports several well-known hypervisor platforms, such as HyperKit, Hyper-V, qemu and VMware. LinuxKit started as an internal project in Docker Inc and is now managed as a Moby Project. Hyperkit is installed as part of docker desktop. The process can be found with ps command:\nps -Af | grep hyperkit Or in the activity monitor:\nDocker related processes Docker-deskop is essentially a LinuxKit virtual machine (as defined here). It runs containerd process inside of the virtual machine. This is an older article about this architecture. If Kubernetes is enabled, the virtual machine is also installed with kubelet, the agent process running on each Kubernetes node.\nSince MacOS is not the direct host of the containers, there is no way to map MacOS file system to container\u0026#8217;s as you can with a Docker/Kubernetes host. Prior to Docker 20.10, there used to be a trick to indirectly access host volume from MacOS terminal. It has stopped working according to this issue but workarounds are provided here. This post proposes some good alternatives to access the file system of LinuxKit VM. For example, use netcat:\nuser@LinVM # nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock / # cat /etc/os-release cat /etc/os-release PRETTY_NAME=\u0026#34;Docker Desktop\u0026#34; / # cat /etc/kubernetes/current-version cat /etc/kubernetes/current-version kubeadm version: \u0026amp;version.Info{Major:\u0026#34;1\u0026#34;, Minor:\u0026#34;21\u0026#34;, GitVersion:\u0026#34;v1.21.2\u0026#34;, GitCommit:\u0026#34;092fbfbf53427de67cac1e9fa54aaa09a28371d7\u0026#34;, GitTreeState:\u0026#34;archive\u0026#34;, BuildDate:\u0026#34;2021-06-18T05:24:26Z\u0026#34;, GoVersion:\u0026#34;go1.16.5\u0026#34;, Compiler:\u0026#34;gc\u0026#34;, Platform:\u0026#34;linux/amd64\u0026#34;} Typing in this terminal session feels clunky. According to this thread, we can connect to the LinuxKit VM with tty and sane auto completion, using the command below:\nstty -echo -icanon \u0026amp;\u0026amp; nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock \u0026amp;\u0026amp; stty sane There are some other alternatives, using privileged Docker containers:\ndocker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh The following command uses a smaller image:\ndocker run -it --rm --privileged --pid=host justincormack/nsenter1 As with Kubernetes, to access the file system on the node is via a privileged container, you can follow the tips from Azure, identify node name, and debug against the node using a special container:\nkubectl debug node/docker-desktop -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11 Note that the root directory on the host is mounted to container\u0026#8217;s file system as /host. This mapping renders a lot of symbolic link as dangled, even though they are actually not on the host file system. Docker-Desktop on Windows Docker works with Linux kernel. There have been a couple of efforts to run Linux virtual machine on Windows. For example, Hyper-V backend, and Windows Subsystem Linux (WSL) backend.\nTraditionally, Docker on Windows was implemented with Hyper-V as the hypervisor. A LinuxKit distro is running on the Hypver-V VM, provider Linux kernel capabilities. Docker refers to containers running in this architecture as \u0026#8220;Windows Containers\u0026#8221;, which is a misnomer in my opinion.\nThe first release of WSL provides a Linux-compatible kernel interface and runs a GNU user space on top of the interface. Neither the Linux kernel code, or a hypervisor is involved. The user space contains GNU Bash shell, command language, command-line tools and interpreters. The absence of Linux kernel in WSL, makes it useless for Docker setup. At that time The Hypver-V backend was still the only option to host docker container during the first version of WSL. This post has a diagram of Docker on Windows with Hyper-V backend.\nWSL2 comes with a real Linux Kernel (also on top of Hyper-V), making WSL2 a better alternative than the legacy Hyper-V as the backend of Docker on Windows. It can be turned on as the screenshot shows above. The rest of this post assumes WSL2 as backend. In this setup, we run a Bootstrapping distro independent of the WSL2 Linux distro, although both inside of the lightweight Linux Utility VM. Below is the diagram:\nWindowsWindowsHypervisor (Hyper-V)Hypervisor (Hyper-V)Lightweight Linux Utility VMLightweight Linux Utility VMWSL2 Linux KernelWSL2 Linux KernelNT KernelNT KernelWindows UsermodeWindows Usermo\u0026#8230;WSL2-compatible Linux Distroin Usermode (e.g. Ubuntu)WSL2-compatible Linux Distro\u0026#8230;Docker desktop\n(Bootstrapping distro)Docker desktop\u0026#8230;Viewer does not support full SVG 1.1\nThe innovative component is the lightweight Linux Utility VM. It is called a VM, but very different from the traditional sense of VM such as VirtualBox or VMware. Traditional VM is isolated from host OS, slow to boot and has large memory footprint. The lightweight Utility VM on the other hand, is integrated with host OS, super fast to boot (i.e. ~1 second), and comes with small memory footprint. It is not turned on until needed. The VM runs both a WSL2 Linux Kernel and GNU/Linux usermode (known as \u0026#8220;distribution\u0026#8221;, for example, Ubuntu). When an end-user say WSL2, s/he most likely refers to the distribution. Similarly, the so called \u0026#8220;docker-desktop on Windows with WSL2 backend\u0026#8221;, is also managed as two WSL2 distros: the bootstrapping distro (docker-desktop) and the data store distro (docker-desktop-data). The detailed components in the Bootstrapping distro is in the second diagram in this post, which has a detailed discussion. With this architecture, you don\u0026#8217;t even need the WSL2 Linux Distro for Docker desktop to function. You can even run docker CLI command from Windows PowerShell without any Linux distro (although this is implemented only for backward compatibility and not recommended anymore) . In the following session, we first list out the WSL2 distros. Notice that the docker-desktop distro is not the default. We then connect to the distro using -d switch. Last, we run docker info from windows user space.\nPS C:\\WINDOWS\\system32\u0026gt; wsl -l -v NAME STATE VERSION * Ubuntu Running 2 docker-desktop Running 2 docker-desktop-data Running 2 PS C:\\WINDOWS\\system32\u0026gt; wsl -d docker-desktop WINLAPTOP:/mnt/host/c/WINDOWS/system32# cd ~ WINLAPTOP:~# printenv|grep DIST WSL_DISTRO_NAME=docker-desktop WINLAPTOP:~# exit PS C:\\WINDOWS\\system32\u0026gt; docker info Client: Context: default Debug Mode: false Plugins: buildx: Build with BuildKit (Docker Inc., v0.5.1-docker) compose: Docker Compose (Docker Inc., v2.0.0-beta.6) scan: Docker Scan (Docker Inc., v0.8.0) Server: Containers: 93 Running: 80 Paused: 0 Stopped: 13 Images: 28 Server Version: 20.10.7 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 5.10.16.3-microsoft-standard-WSL2 Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 12.32GiB Name: docker-desktop ID: WHDE:PJF3:HVFC:AZJA:EDKH:VUZR:RRUJ:HHXX:TDV5:4UJG:XY4E:PK4F Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false The wsl -d command as illustrated above is a good way to connect to the docker-desktop distro. The alternative to get to the distro is via privileged container (Docker) or helper pod (Kubernetes), which is the same as in Docker-desktop on MacOS. Refer to the section above for specific steps.\nApplication Install on docker desktop The followings are my notes to install commonly used applications in Docker Desktop with Kubernetes. They work on both MacOS or WSL2, requiring Kubernetes enabled.\nMetric server The metric server is provided in the official repository. Releases are publish here, which provides the installation step as follows:\nkubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml However, there is some issues when deploying it on MacOS, the deployment will fail due to certificates not matching the hostname. To fix the issue, it is recommended to download the yaml file (components.yaml), and edit the file by adding \u0026#8211;kubelet-insecure-tls to the args section of the container named metrics-server. This is sufficient to fix the issue. Some people are not comfortable with port 443 being insecure TLS, and would rather change the port to 4443. This is completely unnecessary but if that\u0026#8217;s the case, make sure the named port for https is also updated to 4443.\nOnce metric server has been installed, the following two commands should return meaningful results:\nkubectl top no kubectl top po This should also allow application (Pods) to query for cluster resource usage. When working with single-node cluster on MacOS or WSL2, multiple Pods might come up with a single command and the memory can be easily over-subscribed. The two commands above allows you to check and make adjustment on the node configuration (in Docker preference).\nDashboard Similar to metric server, the dashboard is kept in the official repo, in the the path of aio/deploy/recommended.yaml. Here is the published release, where the instruction says:\nkubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml However, this is not directly applicable either because the login page requires token or kubeconfig. We need to be able to bypass that. To do so, download the yaml file (recommended.yaml), and add parameter \u0026#8211;enable-skip-login to the args section for the container named kubernetes-dashboard.\nTo display the login page properly, we need to start the proxy using this command:\nkubectl proxy The login page will then be available at this URL. The URL reflects the namespace and service name. On the login page, the skip button will be available.\nRancher The official installation guide of Rancher 2.5x recommends RKE Kubernetes. If you prefer not to run a separate cluster on MacOS, you can install it on docker desktop (with Kubernetes enabled). The installation steps require Helm 3 and are completed in three helm commands.\nInstall Nginx Ingress controller using Helm, following the three commands here. Alternatively, you can apply the rendered template as posted here. The controller will later be used by the ingress that Rancher\u0026#8217;s chart creates. Follow the steps on this page to install Rancher, even though the page does not say it applies to docker desktop. If you do not have TLS certificate, the Rancher helm chart can generate one for you, using cert-manager. The installation exposes rancher application on port 443 of the MacBook, and the cert is issued to \u0026#8220;rancher.my.org\u0026#8221; by default. To access it, add \u0026#8220;127.0.0.1 rancher.my.org\u0026#8221; to the host file. To back out from the steps above, just uninstall with helm. For example:\nhelm uninstall rancher -n cattle-system helm uninstall cert-manager -n cert-manager helm uninstall ingress-nginx Jenkins Similar to Rancher, Jenkins instruction assumes minikube cluster instead of docker desktop.\nhelm repo add jenkinsci https://charts.jenkins.io helm repo update kubectl create namespace jenkins helm install jenkins -n jenkins -f https://raw.githubusercontent.com/jenkinsci/helm-charts/main/charts/jenkins/values.yaml jenkinsci/jenkins kubectl --namespace jenkins port-forward svc/jenkins 8080:8080 To find out the default password for admin user:\nkubectl -n jenkins get secrets jenkins -o jsonpath={.data.jenkins-admin-password} | base64 -D To uninstall Jenkins:\nhelm uninstall jenkins -n jenkins Container Runtime Although I seem to be a proponent of docker desktop thus far, this post would be incomplete to not discuss what is missing with docker desktop. One key difference between docker desktop and minikube is the container runtime being used. Docker desktop uses docker as the runtime, and it does not support other runtime as of now. Minikube allows user to choose runtime, including containerd. Because of this difference, Kubernetes nodes with Docker as runtime and with containerd as runtime place pod log files in different locations. To find out the runtime, use the following command:\nkubectl describe node \u0026lt;node_name\u0026gt; | grep Runtime If the runtime is docker, the stdout of container is placed in /var/lib/docker/containers/\u0026lt;sha\u0026gt;/. If the runtime is containerd, the stdout log of pods are stored in /var/log/containers/. This is important to know when you configure log shipping and needs to get stdout from node. The log path used in containerd is the standard path in compliance with Container Runtime Interface (CRI) so you should develop log shipping solution based on that.\nDocker\u0026#8217;s refusal to comply to CRI also caused Kubernetes to stop supporting it as container runtime as of Dec 2020. For more background, refer to this and this. Here is also an article with great diagrams on the removal of docker-shim.\nBottom line Docker-desktop is a great tool for a quick single-node Kubernetes environment. As of docker 20.10, docker-desktop still uses docker as runtime. This limits its use case to development only. If you need a CRI compliant environment, docker-desktop is not a good choice. We will discuss alternatives in the next post.\nPrevious PostInfrastructure deployment in Terraform 1/2 Next PostCreating X.509 TLS certificate for workload on Kubernetes ","date":"2021-08-22T00:24:00-04:00","image":"/wp-content/uploads/2025/04/feature-single-node-k8s.webp","permalink":"/2021/08/docker-desktop-a-single-node-kubernetes-cluster/","title":"Single-node Kubernetes cluster – docker desktop"},{"content":"Terraform is an excellent Infrastructure-as-Code (IaC) tool based on Hashicorp Configuration Language (HCL). Compared to JSON or YAML based declarative templates (e.g. CloudFormation and ARM), HCL is more concise, thanks to the flexibility of HCL. On the other hand, HCL is not as flexible as general purpose languages. For that sake, I see HCL as semi-declarative IaC. This post is my notes about best practices with Terraform development, from the context of AWS, but also applies to other cloud platforms.\nComplex Types There are three primitive types (string, number and bool) that forms collection types and structural types. Here are some common ones:\nlist: element may repeat, and order is maintained: [\u0026#34;orange\u0026#34;, \u0026#34;banana\u0026#34;, \u0026#34;orange\u0026#34;, \u0026#34;apple\u0026#34;] set: elements are unique and unordered [\u0026#34;apple\u0026#34;, \u0026#34;banana\u0026#34;, \u0026#34;orange\u0026#34;] tuple: each element has its own type [\u0026#34;a\u0026#34;, 15, true] object: defined by a schema with named attributes each with its own type { name = \u0026#34;John\u0026#34; age = 52 } list of object [ { alloc_id = \u0026#34;0b7271a3219bc1fc2\u0026#34; subnet_id = \u0026#34;0c02af76c2c3e46fa\u0026#34; }, { alloc_id = \u0026#34;0440c334c48d4247f\u0026#34; subnet_id = \u0026#34;02652e69fa2a71de8\u0026#34; } ] map of string { property = \u0026#34;foo\u0026#34; attribute = \u0026#34;bar\u0026#34; } map of object { objkey1 = { alloc_id = \u0026#34;0b7271a3219bc1fc2\u0026#34; subnet_id = \u0026#34;0c02af76c2c3e46fa\u0026#34; } objkey2 = { alloc_id = \u0026#34;0440c334c48d4247f\u0026#34; subnet_id = \u0026#34;02652e69fa2a71de8\u0026#34; } } Whenever applicable, Terraform converts types implicitly or explicitly. For example, when a list or tuple is converted to set, all elements are converted to string and duplicates are removed. Object and map are very similar. Map of string can be converted to object if the attributes comply with the schema. Additional attributes not in the schema are discarded.\nHCL Types is similar to Python Although being totally different beasts, the complex types between HCL and Python are similar, to the point I suspect the HCL design is influenced by Python. I summarize the similarities as such:\nPython Typeslist []tuple ()set {}dict {}Terraform Typeslist []tuple []set []map {}orderedY\nYou can access item by indexYN you cannot access an item by index or key; however you can loop over all itmesN\nkey-value pair that allows you to access item by keychangeable (mutable)YN\nYou cannot update, add or remove itemsY\nAdd or remove only. no change to existing elementsKeys must remain unique or the values get overwritten;\nValues are mutableallow duplicate elementsYYNKeys must be unique; values don\u0026#8217;t have to In Python, list and tuple allow elements of mixed types but in IaC like Terraform we mostly don\u0026#8217;t need mixed types. In Terraform, an object is a map without a defined type. In most situations, lists and tuples behave identically, as do maps and objects.\nAnother area of similarity is with the comprehension of list and dict/maps. In Python for example, ## Supposed you need to create a list: lst=[] for x in range(10): lst.append(x**2) print(lst) ### that can be simplified as the following to create the list: lst = [x**2 for x in range(10)] ## You can even add contidion even_numbers=[num for num in range(10) if num%2==0] ## You can introduce function calls: words = [\u0026#34;hello\u0026#34;, \u0026#34;world\u0026#34;, \u0026#34;python\u0026#34;, \u0026#34;list\u0026#34;, \u0026#34;comprehension\u0026#34;] lengths = [len(word) for word in words] ## You can even combine two lists lst1=[1,2,3,4] lst2=[\u0026#39;a\u0026#39;,\u0026#39;b\u0026#39;,\u0026#39;c\u0026#39;,\u0026#39;d\u0026#39;] pair=[[i,j] for i in lst1 for j in lst2] print(pair) ## With dict, it\u0026#39;s similar evens={x:x**2 for x in range(10) if x%2==0} print(evens) In Terraform, we use similar techniques:\n[for s in var.list : upper(s)] # build a tuple/list from a list [for k, v in var.map : length(k) + length(v)] # build a list from a map {for s in var.list : s =\u0026gt; upper(s)} # build a map from a list [for s in var.list : upper(s) if s != \u0026#34;\u0026#34;] # build a tuple/list from a list with condition Note that the documentation of Terraform doesn’t explicitly call them out as comprehensions. However, it\u0026#8217;s exactly the same idea as comprehensions in Python. Even the range() function exists both in Python and Terraform.\nModularization Modules allows you to group related resources together. They can also be re-used and called by other modules. It is fairly straightforward to create a module:\nput the resource declarations into a sub-directory define input and output in the directory However, the introduction of module complicates the directory structure and variable referencing, which is important to take into account before starting creating modules. This guideline has further discussion about when to create a module. I re-wrote the terraform templates in Orthweb project to leverage modularization wherever possible, but there is still some stand-alone resource (e.g. random_id) not belonging to any module. To reference resources across modules, you need to import those resources (using data source) from within the module. There are a couple of ways. You may pass the argument of data source as input variable, or you can leverage the filter capability of data source. Let\u0026#8217;s look at one example of each mechanism.\nIn the example below, we import a subnet by subnet id:\ndata \u0026#34;aws_subnet\u0026#34; \u0026#34;private_subnet\u0026#34; { id = var.private_subnet_id } In the example below, we import a subnet by filtering from all subnets in the VPC by tag:\ndata \u0026#34;aws_subnet\u0026#34; \u0026#34;private_subnet\u0026#34; { vpc_id = var.vpc_id filter { name = \u0026#34;tag:Name\u0026#34; values = [\u0026#34;Private\u0026#34;] } } There are pros and cons of each approach. A module with mechanism 1 is more transferrable across different environment, because the ID of subnet is explicitly provided. However, authors needs to manage those explicit variables with code. Mechanism 2 fetches target resources with filter. It depends on a well-implemented tagging policy in the resource farm.\nThe Terraform Registry (since 2017) contains a lot of pre-built modules for each backend platform (e.g. AWS). If you find any module that can be used in your project, the module repo can be referenced directly by Git repository URL. You should be aware of the risk of this practice though. Many platforms are keen to publish modules for their platform. Anyone can publish their own modules to the community as well.\nYou can quickly generate module documentation with terraform-docs.\nLocal Execution Local execution is the basic workflow mode which is mostly seen with very small collaboration team. In this mode, the developer executes terraform binary (Terraform CLI) from their workstation (e.g. Laptop). The Terraform CLI converts code into API calls to interface cloud provider. The most frequently used commands (from terraform directory) are:\nterraform init terraform plan terraform apply The init command initializes the working directory. The plan command figures out the delta between code and infrastructure. It outlines the changes it is about to make. The apply command commits the change. The documentation of Terraform CLI commands is here. Terraform keeps track of the infrastructure it manages in state file. This article explains the purpose of state. State management collaboration difficult with local execution because the state file by default is created in the working directory on user\u0026#8217;s workstation. Although the state file can be configured to be stored in a shared location such as S3, it still requires a mechanism to lock the state in a multi-developer collaboration.\nIn large operations, the same code base in Terraform, is usually used to created several different sets of infrastructures, for example, in different geographic regions. So it is a 1-to-many relationship between the code repo and the infrastructure state. To further complicate things, each state might have been deployed using different revisions of the code. To overcome that challenge, Terraform introduced the concept of workspace, which is essentially an instance of state describing a particular group of infrastructure being managed by the same source code. When there are many workspaces, it becomes tricky to manage them with CLI commands.\nState management is a major challenge that needs to be solved for team collaboration in local execution workflow. Each state must use the same revision of Terraform code. You can use Git in combination as a workaround to that limitation but the point is you cannot tie a workspace to a commit with the workspace commands. In some enterprise environment, the execution is from a VM (e.g. ADO agent on-premise) without Internet access, which poses another challenge. First, we need to pre-load required providers manually. The enterprise needs a proxy solution to safely download packages from Hashicorp website. One good option is Nexus Repository, with both open-source and pro supports. It is a full-function artifactory repo that can host helm repo, apt repo, yum repo, etc. Second, we also need to configure Terraform so it picks up providers locally. Managing plugins without Internet access requires understanding of the order in which Terraform tries to load plugins during initialization. Remote Execution In remote execution, the code is executed in Terraform Enterprise or Terraform Cloud. Both are remote web servers. The difference is that Terraform Enterprise is self-hosted service, requiring IT specialist to install and maintain Terraform Enterprise. Terraform Cloud on the other hand, is a managed SaaS service. The pricing model includes a free plan for small number of users.\nTerraform workspace configuration In Terraform Enterprise or Cloud, the remote execution is organized in workspaces. You need to create an organization, and then create workspace under the organization in order to execute code. With each workflow, you can specify version control system (VCS) and subdirectory, to tell the workspace where to fetch Terraform code from. The workspace also allows you to define secrets and variables specific to the workspace. When you execute a workspace plan, the secrets and variables are passed from workspace to the execution logic.\nTerraform Workspace Variable configuration You will also need to design the Terraform code in a way to work seamlessly with the secrets and variables loaded from the workspace. The variable declaration in code should match the definition in workspace. There are already a number of variables that came in handy. Check out this guide.\nEach execution is referred to as a \u0026#8220;run\u0026#8221;, with its own run id. A workspace involves may runs, which may succeed or fail. Each run pulls a specific commit of the source repository, and goes through stages such as plan, and apply. The UI from each run result list out the status of each result, in a very easy to read format.\nThe state data is persisted in the web server as they were generated. Therefore the collaborator do not need to worry about managing state with CLI tools. If there are files that you do not want picked up by the execution engine, their locations can be added to a file .terraformignore. Refer to this guide.\nAWS profile Local execution still has a lot of use cases in enterprises such as testing with temporary resources. A common challenge is authentication. As discussed, Terraform CLI picks up identity information from AWS CLI and authenticates its way into the backend to run API calls against. So AWS CLI must be configured correctly with the sufficient permission to provision resources. On the other hand, enterprises usually offload IAM to an identity store, such as AzureAD, Okta, etc. Putting those together, the pattern of authentication and authorization usually looks like this:\nUser logs on via SSO (e.g. SAML). The validation response gives a name of an IAM role. Upon successful authentication, user takes the IAM role. The role does not have any capability, except for assuming a second IAM role. The second IAM role (the functional role) grants user the permission to do its business. The steps above, can be carried out in AWS console, or with AWS cli using assume-role command. However, when we put Terraform in the picture, it becomes a little involving because the credential information is updated whenever the functional role is assumed, and the assume-role command takes a pretty long argument.\nTo skip typing the long command every time, there are some handy tools, such as aws-azure-login. An even better tool that works with a variety of identity stores is saml2aws. The tool allows you to configure identity backend, assume the functional role, and update credential information in aws credential file, all with a single command. The AWS CLI configuration reads:\n[default] region = us-east-1 output = json cli_history = enabled cli_pager = role_session_name = functional_operation [profile function_user] source_profile = default role_session_name = functional_operation role_arn = arn:aws:iam::9998887766:role/admin-access region = us-east-1 In Terraform provider, we need to tell it to assume that role as well:\nprovider \u0026#34;aws\u0026#34; { region = \u0026#34;us-east-1\u0026#34; assume_role { role_arn = \u0026#34;arn:aws:iam::9998887766:role/admin-access\u0026#34; session_name = \u0026#34;terraform\u0026#34; } } This will ensure Terraform assumes appropriate role before doing its job.\nAWS EC2 SSH Key Pair RSA key authentication for SSH should be used for Linux Instances. When creating an EC2 instance, we give it our public key so we can then later authenticate through SSH. If the key is already stored in AWS, we just need to tell EC2 the name of the key, in the key_name property. If the code is likely to be executed from several different places by different users, then we can write the code so it picks up public key from user\u0026#8217;s workstation (~/.ssh/id_rsa.pub). Here is an example:\nvariable \u0026#34;local_pubkey_file\u0026#34; { type = string default = \u0026#34;~/.ssh/id_rsa.pub\u0026#34; } data \u0026#34;local_file\u0026#34; \u0026#34;pubkey\u0026#34; { filename = pathexpand(var.local_pubkey_file) } resource \u0026#34;aws_key_pair\u0026#34; \u0026#34;user-pubkey\u0026#34; { key_name = \u0026#34;runner-pubkey\u0026#34; public_key = data.local_file.pubkey.content } resource \u0026#34;aws_instance\u0026#34; \u0026#34;bastion\u0026#34; { instance_type = \u0026#34;t2.micro\u0026#34; key_name = aws_key_pair.user-pubkey.key_name ...... } For remote execution, we can even add an option to pass public key in as variable, to override the key file variable. For an example, check out my orthweb project.\nTo upload files to EC2 instance from Terraform execution environment, we can use the file provisioner with ssh as connection type. Previous PostHelm – Configuration Management for Kubernetes Resources Next PostSingle-node Kubernetes cluster – docker desktop ","date":"2021-08-11T21:44:00-04:00","permalink":"/2021/08/scalable-infrastructure-deployment-in-terraform/","title":"Infrastructure deployment in Terraform 1/2"},{"content":"Developer ships application in Docker container, so it can eventually hosted in Kubernetes cluster. However, there are still some installation steps, before the application can operate online in production. In this post, we use the container image of Orthanc application as a starting point. We first build services in Kubernetes to go through these steps. Then, to automate the steps, we build a helm chart. The code is kept in Korthweb project, in which the manual directory has the files requirement for manual deployment, and the helm directory is the helm chart.\nManual Deployment The manual deployment steps include different kinds of activities, such as creating X.509 certificates, apply config map, create Kubernetes deployment using the YAML declarations, and use helm to install dependency. The steps need to take place in a particular sequence. Some step requires pulling information from secrets created in the previous step. This is why the deployment is not portable. In order to automate the steps, one might think of wrapper script, which is very limited. A configuration management tool is needed in this scenario. Two common options are Kustomize, and Helm. Kustomize is a native tool which can be run by kubectl. It is also driven by declarative statement in YAML, which is simple to grasp. However, in lack of a templating mechanism, Kustomize may require wordy statements. Helm, on the other hand, comes with a templating mechanism which greatly increase reusability, making it more suitable for complex steps required in installation.\nHelm Repo and Chart Helm is known as package manager for applications running on Kubernetes. Helm defines an application as a collection of related Kubernetes resources, and it manages application deployment through a templated approach. An installation workbook is called a chart. Charts are kept in repositories. There are some well-known repositories, such as Bitnami, Helm stable. You need to add a repostory before using the Helm Charts in it. To add a repo, run:\nhelm repo add bitnami https://charts.bitnami.com/bitnami You can host your own repo (public or private) as well. To search for charts across repositories, the best place is artifact hub, which indexes charts from a lot of public repositories. To search for charts from the repositories added, run:\nhelm search repo postgres Template is the soul of Helm chart. A Helm chart consists of a directory of files following specific pattern so Helm can understand how to deploy the application. For example, the chart name is the name of the working directory. Under the directory, the values.yaml and chart.yaml defines variables and constants, both serving as template inputs. The template directory is the most important part of the directory where the installation logics are defined. Helm runs the entire directory hierarchy (except for paths specified in .helmignore file) through a Go template rendering engine. The template result spec out the detailed steps.\nA great example of using Helm chart to simplify installation is the wordpress chart by Bitnami. You can install all the required components in a single command:\nhelm install my-release bitnami/wordpress The helm chart in Korthweb project is also an evolving helm chart I created for installing Orthanc application.\nHelm V3 (released in late 2019) includes an important architectural change \u0026#8211; the removal of tiller. This means Helm can operate on the client-side \u0026#8211; a significant simplification. Helm graduated from CNCF project in 2020. There are also a few changes in V3, as outlined here, including the consolidation of requirements.yaml into Charts.yaml.\nTemplate and Function As discussed, templating is the key towards reusability and flexibility in configuration management. We\u0026#8217;ve worked with Jinja2 template engine in Ansible and Python. Here in Helm, we use Go templates with some enhancement. The syntax is mostly based on Go template, which is somewhat similar to Jinja2. Helm also added all functions from the Sprig library, making it more powerful and flexible than Jinja2. Helm chart developer should be very familiar with these functions, as well as the best practices. For example, the cryptographic and security functions in Sprig library gives us the ability to create self-signed X509 certificates during installation.\nSince template introduces another layer of abstraction, to help troubleshooting we should be able to preview rendered template with the template command:\nhelm template orthanc | less The command above simply renders template without attempting to execute the chart. To go one step further, you can dry-run the installation with:\nhelm install orthweb ./orthanc --debug --dry-run | less Although Jinja2 (using {% \u0026#8230; %} to express template control) and Go (using {{ \u0026#8230; }} to express template control) have different syntaxes, one aspect that is similar between them, is chomping whitespace with minus sign (-). This is pretty common in templating language. The documentation of both Jinja2 and Helm have a section on whitespace control. Not paying attention to this nuance may cause pesky errors. Dependency The Orthanc application relies on Postgres database, which itself is deployed by a separate helm chart. This can be specified in Chart.yaml (Helm V3), like this:\ndependencies: - condition: postgresql-ha.enabled name: postgresql-ha repository: https://charts.bitnami.com/bitnami version: 7.8.x The values of variables of the dependency chart can be specified in values.yaml of the root chart. They can also be imperatively specified as a parameter of helm install command.\nThe section above also requires the dependency chart to be downloaded into the charts sub-directory. This can be done with:\nhelm dependency update Then you will notice a file with tgz extension in the charts sub-directory. Note that when you change the version of the dependency package in Chart.yaml, then you will need to run the command again. Alternatively, this command can be automatically executed before helm install if you specify the switch \u0026#8211;dependency-update with helm install.\nThe main chart (e.g. wordpress) is referred to as parent chart, and the charts it depends on are referred to as sub-chart (e.g. mariadb, memcached). When it comes to managing property values, values from parent chart can override those from sub-chart, as explained here. On the other hand, values from sub-chart can override those from parent chart in two formats: export format (keyword exports) and child-parent format (keyword import-values). This is something to be careful and we can use the aforementioned template command to display the rendered values.\nHooks Helm does a great job in figuring out the dependency relationship between kubernetes objects defined in the chart, and create them in order. So typically you do not need hooks for objects in the chart. However, in certain circumstances, such as cleaning up after uninstallation, we may need hooks. Here is a list of available hooks. It is worth-noting that hook is not tied to an action. Instead it is tied to a kubernetes resource. The resource could be a job, a config map, etc. The resource is tied to a hook simply by resource annotation.\nMoving to GUI Helm is a command-line tool. For a team with varying levels of familiarity with command-line, GUI-based tool is a better option. For that, some enterprises adopt Rancher, a comprehensive Kubernetes cluster management platform. Rancher manages many aspects of Kubernetes cluster through web portal. One aspect is the support of helm chart. Rancher can be install on a cluster of its own. For demo, it can also be installed on docker desktop, a single-node Kubernetes cluster by Docker. In both cases, Nginx ingress controller needs to be configured.\nPrevious PostService and Ingress -Traffic Management in Kubernetes Next PostInfrastructure deployment in Terraform 1/2 ","date":"2021-07-26T19:28:22-04:00","image":"/wp-content/uploads/2025/04/feature-helm.webp","permalink":"/2021/07/helm-configuration-management-for-kubernetes-resources/","title":"Helm – Configuration Management for Kubernetes Resources"},{"content":"Update 2022-08 \u0026#8211; Read my latest article on ingress traffic management. In this post we discuss the traffic management in Kubernetes, specifically on Service and Ingress objects. Let\u0026#8217;s start with a traditional architecture:\nNetwork Load BalancerNetwork Load BalancerClientClientVMVMnginXnginXApp1\nServiceApp1\u0026#8230;App2\nServiceApp2\u0026#8230;VMVMnginXnginXApp1\nServiceApp1\u0026#8230;App2\nServiceApp2\u0026#8230;VMVMnginXnginXApp1\nServiceApp1\u0026#8230;App2\nServiceApp2\u0026#8230;Viewer does not support full SVG 1.1\nIn this traditional architecture, we run application as processes on the operating system on each virtual machine. The application process is bound to a certain ports on the operating system, and is wrapped into services (e.g. systemd). On the same virtual machine, there is also a reverse proxy service (e.g. Nginx). There are several main functional areas as listed below, and how they are fulfilled in traditional architecture:\nRequirementDetailTypically fulfilled byL4 Load balancingTCP/UDP traffic routing, operating at L3 and L4Network Load BalancerTLS terminationTerminate TLS traffic, operating at L4TLS termination is available in many products such as Load Balancer (L4/L7), Nginx, or the application itself.Path-based routingRoute request based on URI path, operating at L7Nginx, modern L7 Load Balancer.AuthenticationIntegrate with external identity store, operating at L7Nginx, modern L7 Load Balancer. These requirements are the problems that Kubernetes needs to solve in its own architecture. They are solved by different abstraction objects in Kubernetes. Before getting to traffic management, we first need to expose an application.\nService During traditional application deployment, we often need to organize a group of homogenous application instances as a single target for batch operation. The Pod object is an abstraction of a single application instance. The Deployment object is an abstraction of a group of homogenous Pods. The purpose of Deployment object is for Pod orchestration only. It is not designed to expose the application. To define how we want to expose an application, we use Service object.\nThe service object does not carry exactly the same functionalities as an operating system service. It connects to the frontend (client), as well as to the backend (server). There are two ways to connect to a backend:\nTo connect to Pods as backend, use selector and label; the target port is Pod\u0026#8217;s port. This is the most common use case. To connect to a custom backend (e.g. external database, services in different namespaces, during workload migration), define an Endpoints object (including address and port), and target the port; On the frontend, there are several ways to expose service to client, as defined in ServiceType property. Each represents a level of exposure:\nClusterIP (default): the service gets an internal IP address in the cluster. This is the lowest level of exposure. The service is only reachable from within the cluster. This is a good choice when the service is for internal assumption, such as database. NodePort: the service is exposed at a static port on each node. The port must be in a range pre-specified during cluster provisioning (default 30000-32767). Each node proxies traffic to that port to the service. Without a load balancer, each node is a point of entry on its own. LoadBalancer: this option works with external load balancer in cloud deployments. The actual creation of the load balancer happens asynchronously, and information about the provisioned balancer is published in the Service\u0026#8217;s\u0026nbsp;.status.loadBalancer\u0026nbsp;field. Some cloud providers allow you to specify the\u0026nbsp;loadBalancerIP. The benefit Load Balancer over NodePort, is it provides a single point of entry (for each service). ExternalName: rare use case with custom endpoint object. Headless service With service type ClusterIP, if you explicitly specify\u0026nbsp;\"None\"\u0026nbsp;for the cluster IP (.spec.clusterIP), the service is considered a headless service. With a headless service, a cluster IP is not allocated, kube-proxy does not handle these services, and there is no load balancing or proxying done by the platform for them. Each connection to the service is forwarded to one randomly selected backing pod. Hence the document points out that you can use a headless Service to interface with other service discovery mechanisms, without being tied to Kubernetes implementation. The behaviour differs slightly based on whether selectors are present, but both resembles DNS routing with multiple A record.\nVirtual IP Kubernetes manages service traffic with virtual IP. When clients connect to virtual IP (VIP), the traffic is automatically transported to an appropriate endpoint. Virtual IP is implemented with kube-proxy. Kube-proxy can work in three modes: userspace, iptables and IPVS. I discussed these terms in this post last year. The takeaway is that IPVS is the recommended mode.\nIngress Ingress in Kubernetes cannot match up with a counterpart in traditional architecture. It is mainly for path-based request routing. Also, do not confuse Ingress object with Ingress rule as a policy type in Network Policy object. Ingress is a high level abstraction and should be considered over Service object when the followings are involved in the routing.\nContent-based or path-based L7 routing Multiple protocols (e.g. gRPC, WebSockets) Authentication Ingress usually work with service object (ClusterIP), as illustrated in Kubernetes documentation:\nAlso note that if you have a service other than HTTP or HTTPS, that you need to expose to the Internet, it is recommended to use a service object of NodePort or LoadBalancer type.\nWe call Ingress a high-level abstraction. Ingress object (aka ingress resource) itself does not expose application. It simply defines a set of routing rules. The implementation is provided by another object (Ingress Controller), who enforces the routing rules by monitoring and manage traffic using its own Service and Pods. You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect. There are a number of Ingress Controllers to choose from. Ingress Resource In an Ingress resource, annotations are used to configure some options, depending on the corresponding Ingress Controller. What annotation can be used depends on the the specific Ingress Controller. The backend can be either a service, or a resource. A common usage for a Resource backend is to ingress data to an object storage backend with static assets. You can define DefaultBackend for an Ingress.\nEach Ingress should specify a class, a reference to an IngressClass resource that contains additional configuration including the name of the controller that should implement the class. Before the IngressClass resource and ingressClassname field were added in Kubernetes 1.8, Ingress classes were specified with a kubernetes.io/ingress.class\u0026nbsp;annotation on the Ingress. This annotation was never formally defined, but was widely supported by Ingress controllers. For example, here is the annotations supported by Nginx Controllers.\nBelow is the yaml output of the ingress from Kubernetes documentation:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minimal-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - http: paths: - path: /testpath pathType: Prefix backend: service: name: test port: number: 80 Ingress Controller Ingress Controller exists in the form of Pods, usually as daemonSet, sometimes as a deployment. The Pods listens for requests to create or modify Ingress within the cluster, and converts the rules in the manifest into configuration directives for a load balancing components. Below is all the components related to Ingress Controller:\n\u0026gt; kubectl -n ingress-nginx get all NAME READY STATUS RESTARTS AGE pod/ingress-nginx-admission-create-s7486 0/1 Completed 0 11d pod/ingress-nginx-admission-patch-sjt2q 0/1 Completed 2 11d pod/ingress-nginx-controller-5b74bc9868-6vmjc 1/1 Running 18 11d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-controller LoadBalancer 10.106.25.194 localhost 80:31774/TCP,443:31576/TCP 11d service/ingress-nginx-controller-admission ClusterIP 10.102.38.191 \u0026lt;none\u0026gt; 443/TCP 11d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-controller 1/1 1 1 11d NAME DESIRED CURRENT READY AGE replicaset.apps/ingress-nginx-controller-5b74bc9868 1 1 1 11d NAME COMPLETIONS DURATION AGE job.batch/ingress-nginx-admission-create 1/1 9s 11d job.batch/ingress-nginx-admission-patch 1/1 25s 11d Ingress Controller can be implemented by load balancer resource from cloud platform, or Nginx. When you have one ingress resource and one controller, the matching is assumed. When you have multiple controllers, you need to use the mechanism from the ingress controller to ensure correct matching.\nNginx is a popular controller and there are a couple of implementations as illustrated here. Let\u0026#8217;s take a look at Nginx Controller as an example. The troubleshooting guide states that, For each Ingress/VirtualServer resource, the Ingress Controller generates a corresponding NGINX configuration file in the\u0026nbsp;/etc/nginx/conf.d\u0026nbsp;folder. Additionally, the Ingress Controller generates the main configuration file\u0026nbsp;/etc/nginx/nginx.conf, which includes all the configurations files from\u0026nbsp;/etc/nginx/conf.d.\u0026nbsp;In the Rancher ingress example above, we can check the nginx configuration with the commands below:\nkubectl exec ingress-nginx-controller-5b74bc9868-6vmjc -n ingress-nginx -- cat /etc/nginx/nginx.conf | less It is important to understand the difference between a load-balancer type service and an ingress. The documentation for ingress states that: An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type\u0026nbsp;Service.Type=NodePort\u0026nbsp;or\u0026nbsp;Service.Type=LoadBalancer. This is because ingress operates at layer 7, so routes connections based on http host header or url path. Load balanced services operate at layer 4 so can load balance arbitrary tcp/udp/sctp services. Ingress should be backed by L7 load balancer, whereas load-balancer service should be backed by L4 load balancer.\nNginx Ingress Controller There are several flavours of Nginx ingress controllers that cause much confusion. It is clarified on a blog post on Nginx website. To recap:\nCommunity version: Found in the kubernetes/ingress-nginx repo, the community Ingress controller is based on Nginx Open Source, with docs on Kuberentes.io. It is maintained by the Kubernetes community with assistance from the F5 Nginx team. Nginx version: Found in the nginxinc/kubernetes-ingress repo, the NGINX Ingress Controller is developed and maintained directly by F5 NGINX team, with docs on docs.nginx.com. It is available in two editions: NGINX Open Source-based NGINX Plus-based There are also a number of other Ingress controller based on NGINX, such as Kong, but their names are easily distinguished. If you\u0026#8217;re not sure which version you\u0026#8217;re using, check the container image, then compare the image name with the repos listed above.\nLoad Balancer Kubernetes by itself does not have an object for Load Balancer. The function of traditional Load Balancer is implemented through Service and Ingress objects in Kubernetes, both of which can be satisfied by a load balancer object from the cloud platform (service-managed load balancer and ingress-managed load balancer). Alternatively, you may stand up a standalone load balancer independent of the Kubernetes cluster, which is not recommended.\nIf your architecture is complex and you have a lot of services (e.g. using microservice), then the overhead of managing everything with Service and Ingress in Kubernetes can be significant. In that case, consider delegating these tasks to a service mesh.\nTroubleshooting There isn\u0026#8217;t a single recipe for troubleshooting service and ingress on Kubernetes. There are some good general guide lines here and here, in addition to the guides (here and here) from official documentation. To run networking command from within the Pod network, you can launch a Pod using nicolaka netshoot image.\nBottom line We compared service and ingress in Kubernetes. In real life, we use both, and oftentimes along with CRDs of service mesh.\nPrevious PostKubernetes Networking Solutions Overview Next PostHelm – Configuration Management for Kubernetes Resources ","date":"2021-07-04T01:30:00-04:00","image":"/wp-content/uploads/2025/04/feature-ingress-service.webp","permalink":"/2021/07/traffic-management-in-kubernetes-service-and-ingress/","title":"Service and Ingress -Traffic Management in Kubernetes"},{"content":"Kubernetes networking involves a lot of details. We discuss some CNI plugins in this post. The most basic mode is kubenet. We use \u0026#8211;network-plugin=kubenet with kubelet process to use it. Kubenet is not a CNI plugin, but it works with bridge, lo and host-local (CNI-compliant implementations). We can directly specify MTU with \u0026#8211;network-plugin-mtu. Kubenet is a basic network plugin, based on bridge plugin, with the addition of port mapping and traffic shaping. It does not offer cross-node networking itself. Today it is typically used with managed clusters by cloud providers, where the cloud provider set up routing rules themselves for inter-node communication.\nWhen a cluster goes multi-node, the main challenge is communication between Pods across different nodes. Pods come and go. The size of cluster could increase or decrease as well. The network solutions come in two network types: overlay network based on encapsulation, or non-overlay networks, most likely using routing techniques. Common backends for for multi-host container networking solutions include VXLAN encapsulation, IPIP encapsulation, host-gw, IPSec. In addition, there are some backends that only used by certain plugins.\nCommon Backends VXLAN: use in-kernel VXLAN to encapsulate the packets. VXLAN is a virtual networking capability in Linux which is also used in virtualization technology. VXLAN is an overlay technology requiring encapsulation of overlay network\u0026#8217;s layer-2 frame into UDP packet at layer 4 of underlay network. When configured, the VxLAN backend creates a Flannel interface on every host. When a container on one node wishes to send traffic to a different node, the packet goes from the container to the bridge interface in the host\u0026#8217;s network namespace. From there the bridge forwards it to the Flannel inteface because the kernel route table designates that this interface is the target for the non-local portion of the overlay network. The Flannel network drive wraps the packet in a UDP packet and sends it to the target host. Once it arrives at its destination, the process flows in reverse, with the Flannel driver on the destination host unwrapping the packet, sending it to the bridge interface, and from there the packet find its way into the overlay network and to the destination Pod.\nhost-gw: the host-gw is a non-overlay solution that maintains route tables on Linux Host to allow Pods to communicate across Nodes. It is only used in Flannel plugin. Suppose we have two hosts, each with two containers as connected below. Initially, container0 is not able to reach container2 because eth0 on node0 does not have an entry that matches container2\u0026#8217;s IP address. The packet is there sent to default route, which isn\u0026#8217;t destined to container2. However, if we build rules to match container IP address, on the route table of each node. The issue would be solved. This is essentially how host-gw works. Specifically, on node 0, we add \u0026#8220;ip route add 192.168.1.0/24 via 10.20.0.2 dev eth0\u0026#8221;, on node 1, we add \u0026#8220;ip route add 192.168.0.0/24 via 10.20.0.1 dev eth0). The host-gw in Flannel will manage rule addition to us. Note that the two hosts must have direct layer 2 connectivity. In other words, there must not be a router between the two nodes. Otherwise, the routing table on the router is out of reach. In fact, all nodes in a Flannel network must have layer 2 connectivity with each other. In other words, all nodes must be in a single LAN. Host-gw provides better performance than VxLAN.\nIPSec uses in-kernel IPSec to encapsulate and encrypt the packets. IPsec is a group of protocols to ensure authentication and encryption per packet between devices. Since it secures traffic at layer 3 and now it has become a major backend technology for VPN. IPsec adds several headers and trailers to datagram containing authentication and encryption information. The two major protocols working in IPSec are AH (Authentication Header) and ESP (Encapsulating Security Payload). AH serves up authentication services only; ESP provides both authentication and encryption abilities. It also uses IKE protocol for key exchange.\nIPSec works in two modes: transport and tunnelling mode. Transport mode creates a secure tunnel between two devices end to end. The payload of each datagram is encrypted, but the original IP header is not. Intermediary routers are thus able to view the final destination of each datagram, unless a separate tunnelling protocol (e.g. GRE) is used. Tunnel mode works between two endpoints, such as two routers, protecting all traffic that goes through the tunnel. The original IP header containing the final destination of the datagram is encrypted, in addition to the payload. To tell intermediary routers where to forward the datagrams, IPsec adds a new IP header. At each end of the tunnel, the routers decrypt the IP headers to deliver the datagram to their destinations. The intermediary routers does not know the final destination, or what transport protocol is used. IPIP (IP over IP) tunnel is typically used to connect two internal IPv4 subnets through public IPv4 internete. It has the lowest overhead but can only transmit IPv4 unicast traffic.\nCNI Plugins Originally, the network functions were developed in-tree. Then the CNI specification came up to allow plugin development out-of-tree to implement cluster networking functions. The Container Network Interface seeks to completely decoupled network management from container runtime. Kubernetes picked CNI over CNM in 2016, as discussed in my virtualization discussion. CNI clearly defines the specification for following activities:\nWhen a Pod comes up, give it a network interface Assign IP to the network interface When a Pod is deleted, delete the associated network interface When we configure a Kubernetes cluster, we must specify \u0026#8211;network-plugin switch, so that the cluster is operational. If we use CNI as network-plugin, we also need to install the plugin, optionally with the help of Rancher.\nOn the worker node, we use \u0026#8211;network-plugin=cni with kubelet process to use CNI plugins. A plugin may consist one or more binaries. The binaries are located in /opt/cni/bin (or otherwise specified by \u0026#8211;cni-bin-dir). The configurations are located in /etc/cni/net.d (or otherwise specified in \u0026#8211;cni-conf-dir). Note that the configuration file may reference different plugin implementations for different network management purpose (e.g. interface creating, address allocation, etc). The container networking repo provided some reference implementations and some of them are used by other plugins. These reference implementations include:\nMain (interface creating): bridge, ipvlan, loopback, ptp, macvlan, etc IPAM (IP address management): host-local, dhcp, static Meta (other plugins): portmap, bandwidth So, a CNI plugin consists of a networking solution for backend, and binaries to cover the aspects outlined above. I discussed some common backends above. Below I will introduce some common plugins and backends only available to each plugin\nFlannel Flannel by CoreOS: supports a range of backends. The advantage of Flannel is it reduces the complexity of doing port mapping. This is a great post that covers the mechanism.\nFlannel with overlay (e.g. VxLAN on UDP encapsulation) It supports VXLAN, host-gw, IPSec, IPIP as well as the followings:\nAmazon VPC: recommended with Amazon VPC. AWS VPC creates IP routes in an AWS route table. The number of records in this table is limited by 50 so you can\u0026#8217;t have more than 50 machines in a cluster. GCE: recommended with Google Compute Engine Network. Instead of using encapsulation, GCE also manipulates IP route to achieve maximum performance. Because of this, a separate flannel interface is not created. UDP: debugging only for old kernels that don\u0026#8217;t support VXLAN Calico Border Gateway Protocol (BGP) is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet.\nCalico operates at layer 3. It prefers BGP without an overlay network for the highest speed and efficiency, but in scenarios where hosts cannot directly communicate with one another, it can utilize an overlay solution (e.g. VxLAN or IP-in-IP). Calico also supports network policies for protecting workloads and nodes from malicious activity or aberrant applications.\nThe Calico networking Pod contains a CNI container to keep track of Pod deployment, and register addresses and routes. It also contains a daemon that announces the IP and route information to the network via the Border Gateway Protocol (BGP). The BGP daemon build a map of the network that enables cross-host communication.\nCalico requires a distributed and fault-tolerant key/value store, and deployments often choose etcd to deliver this component. Calico uses it to store metadata about routes, virtual interfaces, and entwork policy objects. Calico can either use a separate HA deployment of etcd, or the same etcd datastore with the Kubernetes cluster.\nWhen we are unable to use BGP (e.g. with cloud provider, or in an environment where we have no permission to configure router peers. Calico\u0026#8217;s IP-in-IP mode encapsulates packets before sending them to other nodes.\nOnce IP-in-IP is configured, Calico wraps inter-Pod packets in a new packet with headers that indicate the source of the packet is the host with the originating Pod, and the target of the packet is the host with the destination Pod. The Linux kernel performs this encapsulation, and then forwards the packet to the destination host where it is unwrapped and delivered to the destination Pod.\nCanal The followings is quoted from Rancher website:\nCanal seeks to integrate the networking layer provided by Flannel with the networking policy capabilities of Calico. As the contributors worked through the details however, it became apparent that a full integration was not necessarily needed if work was done on both projects to ensure standardization and flexibility. As a result, the official project became somewhat defunct, but the intended ability to deploy the two technology together was achieved. For this reason, it\u0026#8217;s still sometimes easiest to refer to the combination as \u0026#8220;Canal\u0026#8221; even if the project no longer exists. Because Canal is a combination of Flannel and Calico, its benefits are also at the intersection of these two technologies. The networking layer is the simple overlay provided by Flannel that works across many different deployment environments without much additional configuration. The network policy capabilities layered on top supplement the base network with Calico’s powerful networking rule evaluation to provide additional security and control.\nWeave Net Weave Net by Weaveworks offers a different paradigm. Weave creates a mesh overlay network between each of the nodes in the cluster, allowing for flexible routing between participants. Applications use the network just as if the containers were all plugged into the same network switch, with no need to configure port mappings and links.\nFor more good references to determine networking options, check out these posts:\nCalico blog Rancher blog Kubevious blog Previous PostKubernetes Storage Explained – from in-tree plugin to CSI Next PostService and Ingress -Traffic Management in Kubernetes ","date":"2021-06-22T12:14:26-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-networking.webp","permalink":"/2021/06/kubernetes-networking-solutions-overview/","title":"Kubernetes Networking Solutions Overview"},{"content":"To support a variety of storage backend, Kubernetes abstract storage issues with several objects (volume, persistent volume, persistent volume claim, storage class) and adopts container storage interface. Unfortunately, the documents are not very well organized to deliver the idea of these concepts, most likely because features are introduced at very different times. Hence this article. At the bottom of this article, I also go through five examples of using volumes in different ways, taking azure disk (SSD as an example).\nThe first to think about is whether we need just ephemeral storage or persistent storage. Generic volume with ephemeral storage lives and dies with the Pod and we don\u0026#8217;t really care where it is from. With persistent storage, we need to consider where it is from and how to create (provision) the storage. The storage can be created statically or dynamically.\nPersistentVolume (PV) and PersistentVolumeClaim (PVC) Regardlessly of static or dynamic storage provision, we first need to understand two objects before getting to that: Persistent Volume (PV) and Persistent Volume Claim (PVC). We use PV object to represent external storage volume. A single external storage volume can be represented by a single PV. So PV goes with external volumes in 1 to 1 relationship. A 100G volume cannot be represented by two PVs each with 50G, unless the storage administrator divides it into two separate volumes, each with 50G. PVC goes with Pod in 1 to 1 relationship. The Pods needs a PVC in order to claim ownership of a PV. A valid PVC allows a Pod to mount a PV as its volume. Here we call storage volume external in relative to the pods. If the storage volume is mapped to a directory on the host file system, it is still considered an external storage. A single PV can link to multiple PVCs, so long as the total request in PVCs does not exceed PV\u0026#8217;s capacity. So PV and PVC are in 1 to many relationship. How PVC binds to PV is defined by Access Mode, with three options. Note that the options are effective for the entire PV. You cannot have different options for each PVC linked to a PV: RWO (ReadWriteOnce): allowing the PV to be bound to a single PVC (for read write). This mode is typically used in block storage; RWM (ReadWriteMany): allowing the PV to be bound to multiple PVCs (for read write). This mode is only supported by file (e.g. NFS) and object storage; ROM (ReadOnlyMany): allowing the PV bound to multiple PVCs for read only. When a PVC is released, what to do with the PV is defined as persistentVolumeReclaimPolicy, and the two options (effective at PV level) are: Delete Retain Static Provisioning and Dynamic Provisioning With static provisioning, the external storage volume must be pre-created. In this context, a PV object represents a pre-created external storage volume. So PVs must be explicit declared. The K8s literature also refers to such PVs as pre-created PV.\nWith dynamic provisioning, the external storage volume is provisioned dynamically. Therefore, you do not need to explicitly create PVs. By the same token, access mode does not apply. Instead of PV, now we need to explicitly declare storage class, which specifies how to dynamically provision PVs, with the following properties:\nvolumeBindingMode defines when the binding and provisioning of a PersistentVolume occurs, with two options: Immediate (default) WaitForFirstConsumer (recommended): delays until a Pod using the PVC is created reclaimPolicy (the equivalent of persistentVolumeReclaimPolicy for pre-created PV) with two options: Delete (default) Retain provisioners: determines what volume plugin is used for provisioning PVs. There are two categories: Internal provisioner (prefixed with kubernetes.io): common ones are listed here. Note that there isn\u0026#8217;t an internal provisioner for NFS any more. External NFS provisioner is needed. External provisioner: third-party out-of-tree plugins compliant to CSI. For example: Dell XtremIO CSI plugin, Dell Isilon plugin, PureStorage CSI driver, Scality Artesca (launched in Apr 2021), and NetApp Trident CSI drivers, and NFS subdir provisioner in Kubernetes-sigs repo. parameters: each provisioner has its own set of mandatory and optional parameters; allowVolumeExpansion: can be set to true if the underlying storage class supports volume expansion; mountOptions: specify only if the storage class supports it; With the information above, we can simplify the rules as follows:\nIn static provisioning, PV needs to be declared explicitly and SC is not needed In dynamic provisioning, SC is required so we can specify provisioner and the parameters needed by the provisioner. PV doesn\u0026#8217;t need to be explicitly declared, even though it exists in the interaction. In real life however, you might come across the following edge cases which seems to contradict with the two generic rules above:\nLocal volume, currently does not support dynamic provisioning. However a StorageClass should still be created to delay volume binding until Pod scheduling. The volume binding mode WaitForFirstConsumer\u0026nbsp;should be specified. In dynamic provisioning, if a PVC does not explicitly define PVC, the administrator should have specified a default StorageClass in place for the cluster. You might also come across PVC with empty string (\u0026#8220;\u0026#8221;) as storageClassName, which indicates that no storage class will be used (i.e. dynamic provisioning is disabled for the PVC). According to this post, in a PVC: If storageClassName=\u0026#8221;\u0026#8221;, then it is static provisioning If storageClassName is not specified, then the default storage class will be used. If storageClassName is set to a specific value, then the matching storageClassName will be considered. If no corresponding storage class exists, the PVC will fail. The confusing \u0026#8220;Volumes\u0026#8221; We\u0026#8217;ve discussed PersistentVolume, which is a K8s object that represents an external storage volume. When the word Volume stands by itself, it generally refers to the part of storage exposed to the Kubernetes cluster, no matter what type of storage it is or where it comes from. We can distinguish them in the following table:\nGeneric VolumesPersistent VolumesPod assignmentBound to a single pod, declared as part of a Pod.A standalone resource type decoupled from Pod and can be bound to single, or multiple Pods via PVCLifecycleVolume is deleted as the owner Pod dies. Data on the volume may or may not persist.Assuming PVC is gone with Pod, the PV persists. Data on PV may or may not persist depending on ReclaimPolicy.ConfigurationPod creator (e.g. app developer) needs to know the details of storage resource in the cloud environment. (e.g volume ID)Pod creator does not need the details of storage resource in the cloud environment. K8s Cluster administrator can provision PV, either statically or dynamically for Pod creator. If you want to use PeristentVolume to back a Volume in Pod, you\u0026#8217;d have to use PersistentVolumeClaim. This means, some types of volumes (including hostPath) can be both mounted as a persistent volume as well as a regular volume. To compare the two ways of mount volume (direct vs via PVC), we take a look at the Kubernetes configuration examples for Azure Disk. The examples are provided at the bottom of this post. Note that, no matter which method of using the volumes, some types of volumes just work natively, and some requires plugin to operate. The table below summarizes the mechanism behind common volume types.\nVolume TypesMechanismMountable as non-persistent volumemountable as persistent volume (through PVC or SC)emptyDir A native volume type, for temporary data only. Data is wiped along with volume. The storage media is determined by the medium of the filsystem holding the kubelet root dir (typically /var/lib/kubelet). You can even set emptyDir.medium to \u0026#8220;Memory\u0026#8221;YESNO. By definition, emptyDir is not persistent.ConfigMap, SecretNative volume type to store non-sensitive or sensitive configuration data. ConfigMap and Secrets are stored in etcd.YESNO. However, by nature, ConfigMap and Secret are stored persistently. There is no need to mount them as PV.HostPathA native volume type to mount a file or directory from the host node\u0026#8217;s filesystem into the Pod. In addition to path property, you may optionally specify a type for a hostPath volume (e.g. DirectoryOrCreate, Directory, FileOrCreate, etc). Note that there is also a type named empty string (\u0026#8220;\u0026#8221;) which is the default value. It means means that no checks will be performed before mounting the hostPath volume. In addition to the caveat with using hostPath from the documentation, we also need to understand that: 1. HostPath gives Pod the ability to maliciously modify files on the host system, or simply fill up the host file system;\n2. As the document suggests, you may end up with multiple Pods trying to write simultaneously to a host path.YES. Read this.YES. Check out PersistentVolumes typed hostPathLocalIt represents a mounted local storage device such as a disk, partition, or directory. Compared to hostPath volumes, local volumes are used in a durable and portable manner, without manually scheduling pods to nodes. The system is aware of the volume\u0026#8217;s node constraints by looking at the node affinity on the PV. You must set nodeAffinity on the PV when using local volumes. This also means local volumes are subject to the availability of the underlying node. Refer to this post.\nThis is also referred to as Local persistent Volume.NOYES. Static provisioning only. CephFS, NFS, GlusterFS, Ginder, RBD, FC, iSCSI\u0026#8230;\u0026#8230;These volume types are backed by legacy in-tree plugins. They are used to connect to external storage in self-hosted clusters.YESYESawsElasticBlockStore, AzureDisk, AzureFile, GCEPersistentDiskThese volume types are backed by legacy in-tree plugins. They are used to connect to external storage in public cloudYESYES Note that the table above does not list PersistenVolumeClaim as a volume type, because it obviously only support being mounted as persistent volume.\nFrom in-tree plugins to out-of-tree CSI plugins In the table above, the bottom two rows involves in-tree plugins (aka built-in plugins). In-tree means the volume plugins are built in the Kubernetes code repository. They were built, linked, compiled, and shipped with the core Kubernetes binaries. There has been 20+ in-tree plugins. The problems of this plugin development model are:\nThese in-tree plugins introduces risk to the stability of Kubernetes itself; The maintenance and upgrade of plugin is tightly coupled with Kubernetes release The Kubernetes community carries the burden of maintaining plugins for all storage backends. Plugin developers have to open-source all their volume plugin code. The Kubernetes community seeks better alternatives, and has stopped accepting any more in-tree plugins since GA 1.8. The first alternative paradigm for shipping storage plugin, is flexVolume, which existed since version 1.2. However, flexVolume is still not good enough. For example, some packages like Ceph requires dependency package (ceph-common), and the deployment of plugin requires elevated access to the worker node. For that reason, the community later shifted to the Container Storage Interface (CSI) paradigm. A CSI-compliant plugin allows the storage resource to be surfaced as volumes (be it persistent or not) in Kubernetes cluster. More details in this post and here is a list of supported CSI-compliant drivers.\nBack to our azure disk example, this page provides examples for both dynamic and static provisioning.\nCSI-compliant plugin development is more complicate but it offloads it the driver developer. The community hopes users to shift to CSI so the 20+ grandfathered in-tree plugins can eventually be phased out. With that as the goal, there are several types of volumes with the name \u0026#8220;CSI migration\u0026#8221;, allowing users to migrate from in-tree volume plugins to CSI-based plugins.\nAll the CSI-based plugins are fairly recent. As of today, the document outlines three ways to use CSI volume in a Pod:\nthrough a reference to a PersistentVolumeClaim (examples 4 and 5 below) with a generic ephemeral volume (alpha feature) with a CSI ephemeral volume if the driver supports that (beta feature) Examples We\u0026#8217;ll go over five examples, as listed in the able below. Note that out of all the combinations, you cannot mount a csi-based plugin as a volume. No such volume type supported by CSI exist.\nPlug-in mechanismMount methodExample In-tree legacy volume plug-inas volume#1. using azureDisk property of Volume as PV (static)#2. using azureDisk property of PersistentVolume as PV (dynamic)#3. using kubernetes.io/azure-disk as provisioner for SC Out-of-tree CSI volume pluginas volumeThis mode does not exist. Example is not available as PV (static)#4. using disk.csi.azure.com as csi driver of PV as PV (dynamic)#5 using disk.csi.azure.com as provisioner for SC Now, let\u0026#8217;s take a look at the example code snippet. Some examples are from Azure documentation. Some are from the azure-disk-csi-driver repository. I\u0026#8217;ve made minor modifications for conciseness.\nExample 1 uses legacy in-tree plugin, and directly mount the volume. The example code is in Kubernetes repo.\napiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - image: kubernetes/pause name: mypod volumeMounts: - name: azure mountPath: /mnt/azure volumes: - name: azure azureDisk: kind: Managed diskName: myAKSDisk diskURI: /subscriptions/\u0026amp;lt;subscriptionID\u0026gt;/resourceGroups/MC_myAKSCluster_myAKSCluster_eastus/providers/Microsoft.Compute/disks/myAKSDisk Example 2 uses legacy in-tree plugin, and mount the PV statically via PVC. No storage class is used (as indicated by empty string in storage class property)\napiVersion: v1 kind: PersistentVolume metadata: name: azure-disk-pv spec: capacity: storage: 2Gi storageClassName: \u0026#34;\u0026#34; volumeMode: Filesystem accessModes: - ReadWriteOnce azureDisk: kind: Managed diskName: \u0026amp;lt;enter-disk-name\u0026gt; diskURI: \u0026amp;lt;enter-disk-resource-id\u0026gt; --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: azure-disk-pvc spec: storageClassName: \u0026#34;\u0026#34; accessModes: - ReadWriteOnce resources: requests: storage: 2Gi --- apiVersion: apps/v1 kind: Pod metadata: name: logz-deployment spec: containers: - name: pause image: kubernetes/pause volumeMounts: - name: azure-disk-vol mountPath: /mnt/logs volumes: - name: azure-disk-vol persistentVolumeClaim: claimName: azure-disk-pvc Example 3 uses legacy in-tree plugin, and mount the PV dynamically and implicitly via SC. Note that Azure AKS will create several SCs for you by default so use existing ones whenever available.\nallowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-premium parameters: cachingmode: ReadOnly kind: Managed storageaccounttype: Premium_LRS provisioner: kubernetes.io/azure-disk volumeBindingMode: WaitForFirstConsumer --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: azure-managed-disk spec: accessModes: - ReadWriteOnce storageClassName: managed-premium resources: requests: storage: 5Gi --- kind: Pod apiVersion: v1 metadata: name: mypod spec: containers: - name: mypod image: kubernetes/pause volumeMounts: - mountPath: \u0026#34;/mnt/azure\u0026#34; name: volume volumes: - name: volume persistentVolumeClaim: claimName: azure-managed-disk Example 4 uses CSI-based plugin, and mount the PV statically via PVC\n--- apiVersion: v1 kind: PersistentVolume metadata: name: pv-azuredisk spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain csi: driver: disk.csi.azure.com readOnly: false volumeHandle: /subscriptions/{sub-id}/resourcegroups/{group-name}/providers/microsoft.compute/disks/{disk-id} volumeAttributes: fsType: ext4 partition: \u0026#34;1\u0026#34; # optional, remove this if there is no partition --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvc-azuredisk spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi volumeName: pv-azuredisk storageClassName: \u0026#34;\u0026#34; --- kind: Pod apiVersion: v1 metadata: name: nginx-azuredisk spec: nodeSelector: kubernetes.io/os: linux containers: - image: kubernetes/pause name: mypod volumeMounts: - name: azuredisk01 mountPath: \u0026#34;/mnt/azuredisk\u0026#34; volumes: - name: azuredisk01 persistentVolumeClaim: claimName: pvc-azuredisk Example 5 uses CSI-based plugin, and mount the PV dynamically and implicitly via SC\nkind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: azuredisk-csi-waitforfirstconsumer provisioner: disk.csi.azure.com parameters: skuname: StandardSSD_LRS allowVolumeExpansion: true reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-azuredisk spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: managed-csi --- kind: Pod apiVersion: v1 metadata: name: nginx-azuredisk spec: nodeSelector: kubernetes.io/os: linux containers: - image: kubernetes/pause name: mypod volumeMounts: - name: azuredisk01 mountPath: \u0026#34;/mnt/azuredisk\u0026#34; volumes: - name: azuredisk01 persistentVolumeClaim: claimName: pvc-azuredisk Bottomline As of June 2021, the CSI support is still new. Generally, if a CSI-based plugin is available and in GA, you should consider using it. If you have existing legacy volume types using in-tree plugin, you should consider migration, and create a migration plan. Also, try to avoid the use case of mounting as generic volume (without PVC) because it is rare and not supported with CSI drivers. Without PVC, it also cannot take advantage of the volumeClaimTemplates property in StatefulSet object.\nPrevious PostGetting started with GitHub Actions Next PostKubernetes Networking Solutions Overview ","date":"2021-06-12T21:55:46-04:00","image":"/wp-content/uploads/2025/04/feature-k8s-csi.webp","permalink":"/2021/06/kubernetes-storage-explained/","title":"Kubernetes Storage Explained – from in-tree plugin to CSI"},{"content":"In my orthweb project, I had to compile a library on my own. In search for free computing resources I realized that GitHub action can meet all my needs.\nCI/CD pipeline As a development project grows, there are many operational tasks demanding automation. Prior to pipeline technology, developers used to use Makefile to organize command execution locally. Today, its role has declined, but Makefile is a good choice in certain situations. In most cases though, to offload the build command execution to a shared system, automation engines like Jenkins came around. Then Jenkins evolved into pipelines.\nIn strict terms, CI pipeline is the build pipeline; and CD pipeline is release pipeline. The two types of pipelines use pretty much the same pool of building blocks, with different purposes. The build pipeline focuses on producing quality artifact in a consistent manner. The release pipelines focus on system stability while deploying an artifact across different environments. Because release pipelines may connect to different environment, it has to deal with various situations. It is very common to have multiple stages in release pipeline, each stage pointing to a different environment (e.g. DEV, TEST and PROD). At workplace both could be loosely referred to as CI/CD pipeline, or even simply pipeline.\nA lot of projects provide pipeline capability: BitBucket, Bamboo, TeamCity, Jenkins, Azure DevOps, AWS CodePipeline, TravisCI etc. Since late 2018, GitHub also joined the game with GitHub actions. It is openly free for public repositories, and has a free tier for private repositories. It executes task as defined in .github/workflow/action.yaml in the code project. I will take my own project as an example.\nRunners You can run jobs in self-hosted runners or GitHub managed runners, similar to other pipeline solutions (e.g. self-hosted agent vs managed agent from Azure DevOps). The GitHub hosted runners only have three operating systems to support: Windows, Ubuntu and MacOS. The Ubuntu and Windows runners are built from Standard_DS2_v2 VMs in Microsoft Azure. They are pre-installed with a virtual environment with packages required for common build tasks. The same virtual environment is also used in hosted agents by Azure DevOps. While they are free and you can elevate privilege on the runner, you cannot SSH or RDP to it for further troubleshooting. The self-hosted runners require users to manage the instance on their own, including configuring virtual environment, installing GitHub Action Runner, etc.\nWorkflow file Most pipeline declaration uses YAML or JSON, such as Jenkinsfile, AWS CodePipeline. GitHub refers to an automation process as a \u0026#8220;workflow\u0026#8221; and you can program the workflow in YAML (.github/workflow/action.yaml). Here is the reference and an example with environmental variable and versioning: The handling of environment is documented here. There are a lot of custom actions available in GitHub Marketplace. For example, the versioning in the above example, uses an action by Einar Egilsson, which is open source itself.\nExample pipeline My example pipeline consists of two phases: Build Library: spin up a docker container to build source code, and publish the artifact Publish Image: add the artifact to an existing Docker image, and publish the result as my own image. The status of the pipeline is also open, and can be found here. The retention period of artifact is 90 days by default but can be customized. To persist the artifact, I add it to my own Docker image and publish it to DockerHub, hence the second phase.\nWhen building the second phase, I need to generate secret from my DockerHub account and store that encrypted secrets in GitHub settings, so that the secret value can be referenced in workflow file.\nFailures in Actions are displayed in error steps and by default the rest of the steps are skipped. I use the Docker build \u0026amp; push plugin to build and push my own docker image to DockerHub. Apart from DockerHub as my choice, GitHub also has its own artifactory GitHub Packages with a small free tier. It supports NPM, Docker, Maven, Gradle, etc. Triggers of Action Most of the times, GitHub action are triggered upon commit to main branch of the repo. In GitHub, this is known as a workflow_dispatch event. This is not the only event that can trigger GitHub action. All the available events are listed here on its documentation. This makes it very flexible to trigger action at many points in the workflow. One example is to trigger GitHub action during PR review. When a developer opens a PR with a few commits in the proposed branch, the PR can preemptively check linting, style, etc and even build the application. These activities can also be defined in a GitHub action manifest with pull_request as triggering event.\nTroubleshooting In general, it is painful to troubleshoot activities happening inside of runners. I often had to write a few steps for the sake of printing variables, and trigger a run to see what their value is. This requires a lot of time especially when I have to wait for available runners. To help troubleshooting pipeline runs there is an open-source utility called act. You can run GitHub actions locally from a Docker container on your MacBook. You can deliver environment variables and secrets via files. If you ever need to troubleshoot the runner environment, you have the option to connect to the Shell environment inside of the runner container. This tool is extremely helpful.\nClosing remarks GitHub action really makes the CI/CD pipeline capability available to any developers who stores their code on GitHub. GitHub expands from a code repository solution to a full CI/CD solution with a free tier sufficient for personal projects.\nPrevious PostSecure web application deployment Next PostKubernetes Storage Explained – from in-tree plugin to CSI ","date":"2021-05-27T13:52:31-04:00","permalink":"/2021/05/getting-started-with-github-actions/","title":"Getting started with GitHub Actions"},{"content":"In Nov 2020, I created OrthWeb project, a deployment of Orthanc\u0026#8217;s server. Orthanc is a DICOM viewer and repo shipped in Docker container. In the deployment project, I use Terraform to provision infrastructure, including a managed PostgreSQL instance, an EC2 instance for docker runtime, and the init script to bring up the web service. I whipped up the project for a demo, and skipped some security configurations. For example, the password was stored in clear text in Terraform configuration. The web certificate is stored in the repository. I recently had some time to fix that. My effort leads up to the conclusion that this requires a better platform (i.e. managed Kubernetes cluster). So I wanted to note down how I got there.\nSecret store In AWS, both parameter store and secret manager can act as secret store. Secrets manager comes at higher cost but some additional features, such as built-in password generator, secret rotation, and cross-account access. We use Secret Manager but we generate password within Terraform because we need to specify password during database provisioning. Secret store requires certain special characters to be eliminated. Terraform can specify the special characters allowed. For EC2 instance to pull from secret manager, the following entities are needed:\nA secret store A VPC endpoint to expose secret store to subnet via private route. The VPC endpoint needs its own security group The instance profile of the EC2 instance must contain an IAM role to get secret value The security group of EC2 instance needs to allow traffic to secret store The script from EC2 instance uses VPC endpoint This is a common pattern for interaction between computing object and VPC endpoint. The details are in compute.tf, network.tf, secgrp.tf and secret.tf. The secret name needs to be partially randomized to avoid naming conflict with deactivated secrets.\nPassing Secret to container Here is an example CLI command to pull secret:\n$ aws secretsmanager get-secret-value --secret-id DatabaseCreds51c1db4172ae9c54 --query SecretString --output text --endpoint-url https://vpce-0897b168cf1c60df2-khx32o7f.secretsmanager.us-east-1.vpce.amazonaws.com | jq -r .password The connection is made via private network route (whether the instance is in public or private subnet). Traffic is encrypted in TLS. Once in the operating system, the secret is available as standard output and can be stored to file, or saved in environment variable. My first attempted approach is docker\u0026#8217;s secret store and config so that I do not have to store secret in plain text on the file system. I eventually give up this approach due to several hiccups. First, secret and config are part of Docker swarm service. So it requires initializing docker swarm before I could port in the secret, with the following command:\n$ docker swarm init $ echo mdbuser123 | docker secret create db_un - $ echo m1p@ssw0rd | docker secret create db_pw - $ echo 10.2.32.41 | docker config create db_ep - The content of the secret and config are presented as files to the container file system at different locations, as can be verified this way:\n$ docker service create --name=\u0026#34;redis\u0026#34; --secret=db_un --secret=db_pw --config=db_ep redis:alpine $ docker container ls $ docker exec -it c8ed2a278ca8 sh # cat /db_ep # cat /run/secrets/db_un # cat /run/secrets/db_pw This is a great way to pass secret and config to container applications. However, since the values are stored as content of file, the main application must be able to load file content as its own configuration value. In my specific scenario, the application expects explicit value in its configuration file, or environment variable.\nOn the other hand, Docker document states that docker secrets do not set environment variables directly. this was a conscous decision, because env var can unintentionally be leaked between containers. In other word I could present secrets as files but the application cannot use it. There is potentially a workaround here which is great function wise but an additional layer of complexity.\nMoreover, I later discovered that this isn\u0026#8217;t even a viable approach if I use docker compose. This is because I must declare those entries from secret store or config store as external, and external secrets are not even available to containers created by docker-compose. With reluctance, I store the config and secret keys and values to a file, and use the env_file section in docker compose to import them as environment variables. The application can pick up environment variables as configuration values.\nX509 Certificate We use a self-signed X509 certificate, along with the private key. The creation is straightforward. However, when I tested on Mac, the browser does not load the page for this reason. Since macOS 10.15, the certificate requires several extensions: ExtendedKeyUsage, Subject alternative names and DNS name. The native openssl from the operating system is outdated (v 1.0.2) and I had to install openssl11 package and create it as follows:\nopenssl11 req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/private.key -out /tmp/certificate.crt -subj /C=CA/ST=Ontario/L=Waterloo/O=Digihunch/OU=Imaging/CN=digihunch.com/emailAddress=info@3.237.97.93 -addext extendedKeyUsage=serverAuth -addext subjectAltName=DNS:orthweb.digihunch.com,DNS:digihunch.com The Mac uses libreSSL backed openSSL utility and can achieve the same with slightly different command line argument.\nNext Step The limitation with passing secret concerns me and I\u0026#8217;m looking to move to managed Kubernetes platform where secrets can be ported to environment variable of Pods. We can also consider ECS in AWS which allows to inject sensitive data from secret manager to container. Previous PostCertified Kubernetes Administrator (CKA) Exam Next PostGetting started with GitHub Actions ","date":"2021-05-16T15:19:41-04:00","permalink":"/2021/05/secure-web-application-deployment/","title":"Secure web application deployment"},{"content":"The Certified Kubernetes Administrator (CKA) exam is a hands-on session where you need to follow the instructions to configure the system in a bash terminal on the web browser. In my experience, some shortcut keys (such as Alt+F) do not work, which slows me down a little bit. For each question, you need to switch kubectl context as instructed in the question. Some questions share the same context so it is very easy to omit this step. You can verify response with your own command but will not be told whether you scored in each question. During the CKA exam I tried to spin up a terminal session from within Vim editor and the terminal ran out of buffer. I had to reboot the machine with the help of proctor, and my completed work are saved.\nIn general this is an exam I enjoy preparing and writing because it is very hands on. Result is out a day after, and I passed at 96%. I heard about tight timelines but I managed to finish 15 minutes before the end, most likely owing to my dexterity with Linux commands. With that I\u0026#8217;m happy to share my notes in preparation for the CKA exam.\nWorker NodeWorker Nodekube-proxykube-proxykubeletkubeletcontainer runtimecontainer runtimecontainercontainercontainercontainercontainercontainercontainercontainercontainercontainercontainercontainerWorker NodeWorker Nodekube-proxykube-proxykubeletkubeletcontainer runtimecontainer runtimecontainercontainercontainercontainercontainercontainercontainercontainercontainercontainercontainercontainerWorker NodeWorker Nodekube-proxykube-proxykubeletkubeletcontainer runtimecontainer runtimecontainercontainercontainercontainercontainercontainercontainercontainercontainercontainercontainercontainerControl\u0026nbsp; PlaneControl\u0026nbsp; Planekube-controller-managerkube-controller-managerkube-schedulerkube-schedulerkube-controller-managerkube-controller-managerkube-controller-managerkube-controller-managerkube-schedulerkube-schedulerkube-schedulerkube-schedulercloud-controller-managercloud-controller-managercloud-controller-managercloud-controller-managercloud-controller-managercloud-controller-managerkube-api-serverkube-api-serveretcdetcdViewer does not support full SVG 1.1\nFor taking CKA exam, we should be familiar with the diagram above.\nTips for Troubleshooting The CKA exam is hands-on and therefore requires quite a bit of troubleshooting. Here are my notes.\nCheck Node status to start with Check core services on each node: sudo systemctl status kubelet sudo systemctl status docker sudo journalctl -u kubelet sudo journalctl -u docker Check component logs (on hosting VM) /var/log/kube-apiserver.log /var/log/kube-scheduler.log /var/log/kube-controller-manager.log If cluster is built by kubeadm, then some of those services are running in Pods within kube-system namespace. Check those pods: run interactive shell: \u0026gt; kubectl exec podname \u0026#8211;stdin \u0026#8211;tty \u0026#8212; /bin/sh there is an image for lots of useful network tool called nicolaka/netshoot Store pod names to variable. e.g. \u0026gt; POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath=\u0026#8221;{.items[0].metadata.name}\u0026#8221;) With kubectl, you may alias it to k for faster typing \u0026#8211;dry-run: to run imperative command without creating object \u0026#8211;record: record the command that was used to make a change -o: set output format, wide, yaml, or jsonpath=\u0026#8221;expression\u0026#8221;. For example, to get pod name: \u0026gt; kubectl get pods -l run=nginx -o jsonpath=\u0026#8221;{.items[0].metadata.name}\u0026#8221; \u0026#8211;sort-by: use JSONPath expression \u0026#8211;selector: filter results by label Build K8s cluster using kubeadm The CKA exam requires you to know how to build cluster with kubeadm. This involves installing four components (docker-ce, kubeadm, kubectl and kubelet), as outlined below:\nstepcommand 1. Install docker-ce\u003e curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n\u003e sudo add-apt-repository \\\n\"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n$(lsb_release -cs) \\\nstable\"\n\u003e sudo apt-get update\n\u003e sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu\n\u003e sudo apt-mark hold docker-ce\n\u003e sudo systemctl status docker 2. Install kubeadm, kubelet and kubectl\u003e curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\ncat \u003c\u003c EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list\ndeb https://apt.kubernetes.io/ kubernetes-xenial main\nEOF\n\u003e sudo apt-get update\n\u003e sudo apt-get install -y kubelet kubeadm kubectl\n\u003e sudo apt-mark hold kubelet kubeadm kubectl\n3. Form a K8s clusterOn master node:\n\u003e sudo kubeadm init --pod-network-cidr=10.244.0.0/16\nThis command prints out a command for worker nodes to join.\nOn worker node:\nsudo the command generated on master 4. Configure kubectlOn master node:\n\u003e mkdir -p $HOME/.kube\n\u003e sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n\u003e sudo chown $(id -u):$(id -g) $HOME/.kube/config\nOptionally on worker node:\n\u003e mkdir -p $HOME/.kube\nthen scp $HOME/.kube/config from control plane node 5. Set up cluster networking\u003e echo \"net.bridge.bridge-nf-call-iptables=1\" | sudo tee -a /etc/sysctl.conf\n\u003e sudo sysctl -p\nThen from any environment with kubectl, bring up the system pods for cluster networking\n\u003e kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml\nAdd new node to KubeAdm cluster This is fairly simple with the help of kubeadm. The node to join cluster must be able to communicate with master node. Create a token and print join command from master node:\n\u0026gt; kubeadm token create --print-join-command Then from the node to join, run this command as sudo. You will see that it performs the TLS bootstrap for you. Once completed, the standard output will say this node has joined the cluster. You can confirm with command:\n\u0026gt; kubectl get nodes Sometimes one needs to migrate pods to the newly joined node. This can be done by draining the existing nodes.\nNote you can also use kubespray to build K8s cluster as previously discussed, and here is a my IaC project to launch AWS instances and build a K8s cluster with kubespray on top of it. For my learning, I often create a GKE (Google Kubernetes Engine) cluster from GCP\u0026#8217;s cloudshell. There is a guide on how to start a cluster but it comes down to three commands:\n$ gcloud config set compute/zone us-east1-b $ gcloud container clusters create tcluster --num-nodes=3 $ gcloud container clusters get-credentials tcluster The third command above is to configure kubectl on the cloudshell. Follow this guide if you need to SSH to node.\nUpgrade KubeAdm cluster This involves upgrade three components (kubeadm, kubectl and kubelet) on two types of node: master node and worker node. They steps vary slightly for two nodes. But drain and uncordon is needed for both types of nodes. Pick a node and follow the steps below:\nStepCommand 1. drain the node from kubectl client (e.g. master node)\u003e sudo kubectl drain nodename --ignore-daemonsets 2. Determine kubeadm target version\u003e apt-mark showhold\n\u003e sudo apt-mark unhold kubeadm kubectl kubelet\n\u003e apt list --installed | grep kube\n\u003e apt-cache show kubeadm | less\n\u003e sudo apt-get install -y kubeadm=1.20.2-00\n3. update kubeadmOn master node:\n\u003e sudo kubeadm upgrade plan v1.20.2\n\u003e sudo kubeadm upgrade apply v1.20.2\nOn worker node:\n\u003e sudo kubeadm upgrade node\n4. On the node to update, determine target version for kubectl and kubelet, then install\u003e apt-cache show kubectl | less\n\u003e apt-cache show kubelet | less\n\u003e sudo apt-get install -y kubectl=1.20.2-00 kubelet=1.20.2-00\n5. Restart kubelet\u003e sudo systemctl daemon-reload\n\u003e sudo systemctl restart kubelet\n6. Uncordon\u003e kubectl uncordon nodename Backup and restore Etcd Etcd is a distributed key-value store. It uses Raft protocol for distributed consensus. Etcd is the third distributed system I touch on. The previous two are: Cassandra (using Paxos protocol for distributed consensus) and ZooKeeper (using ZAB protocol). Here is a good article that summarizes the protocols. As for the exam we only need to use etcd with the client tool.\nThe etcd itself can run on a cluster of servers, each running etcd as a systemd service as etcd/etcd (user/group). It can be deployed in two ways: stacked etcd: an instance of etcd lives with kube-api-server on the same control plane node external etcd: in a dedicated cluster of etcd Alternatively, etcd can run as a pod, most likely in kube-system namespace. The etcd service listens on port 2379 for client communication and on port 2380 for server (peer-to-peer) communication. When the systemd service was initialized there are a few key environment variables (e.g. cert locations, ETCD_DATA_DIR) privoded as configuration. To see them, run:\n\u0026gt; cat /etc/systemd/system/etcd.service | grep Env These environment variables (prefixed with ETCD_) are for the service only. They can provide current configuration information for us to use later. \u0026nbsp;When it’s running as a pod, check out the directory for static pod for the yaml declaration (e.g. /etc/Kubernetes/manifests/etcd.yaml), where these parameters are passed in as environment variable.\nThe etcdctl utility is a command line client for etcd. The default API version is 3 so no need any more to set ETCDCTL_API=3 before each command. The utility needs three arguments three arguments (\u0026#8211;cacert, \u0026#8211;cert, and \u0026#8211;key) but we can pass the information via environment variables:\n\u0026gt; export ETCDCTL_CACERT= /home/cloud_user/etcd-certs/etcd-ca.pem \u0026gt; export ETCDCTL_CERT= /home/cloud_user/etcd-certs/etcd-server.crt \u0026gt; export ETCDCTL_KEY= /home/cloud_user/etcd-certs/etcd-server.key \u0026gt; export ETCDCTL_ENDPOINTS=https://etcd1:2379 The environment variable names are uppercase of the argument name with prefix ETCDCTL_. Only global options of arguments can be supplied via environment variables. They remain effective throughout the rest of activities. Also note that the CACERT is needed only when client-cert-auth is true. Now to backup, we can simply run:\n\u0026gt; etcdctl snapshot save /home/cloud_user/etcd_backup.db To restore from a file, you want to remove existing etcd data directory first. The directory can be found in ETCD_DATA_DIR variable. Suppose it is /var/lib/etcd, you need root permission to write to it, then correct ownership before starting the service:\n\u0026gt; sudo systemctl stop etcd \u0026amp;\u0026amp; sudo mv /var/lib/etcd/ /tmp/ \u0026gt; sudo etcdctl snapshot restore /home/cloud_user/etcd_backup.db --data-dir /var/lib/etcd \u0026gt; sudo chown -R etcd:etcd /var/lib/etcd \u0026amp;\u0026amp; sudo systemctl start etcd To verify the restore result, simply run:\n\u0026gt; etcdctl get cluster.name Object Management In the CKA exam, we need to interact with many types of built-in Kubernetes objects.\nRBAC objects: A Role defines permissions within namespace. A ClusterRole defines cluster-wide permissions. Both Roles and ClusterRoles are K8s objects that defines a set of permissions RoleBinding and ClusterRoleBinding are objects that connect Roles and ClusterRoles to users. Service Account: an account used by container processes within Pods to authenticate the K8s API. If your Pods need to communicate with the K8s API, you can use service accounts to control their access. RoleBinding\n* roleRef\n* subjectsRoleBinding\u0026#8230;ClusterRoleBinding\n* roleRef\n* subjectsClusterRoleBinding\u0026#8230;ServiceAccountServiceAccountClusterRole:* rules\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026#8211; apiGroups\u0026nbsp; \u0026nbsp;\u0026#8211; resources\u0026nbsp; \u0026nbsp;\u0026#8211; resourceNames\u0026nbsp; \u0026nbsp;\u0026#8211; verbsClusterRole:\u0026#8230;Role:* rules\u0026nbsp;\u0026nbsp; \u0026nbsp;\u0026#8211; apiGroups\u0026nbsp; \u0026nbsp;\u0026#8211; resources\u0026nbsp; \u0026nbsp;\u0026#8211; resourceNames\u0026nbsp; \u0026nbsp;\u0026#8211; verbsRole:\u0026#8230;Viewer does not support full SVG 1.1\nInspect resource usage either with a K8s Metrics Server, or by command: \u0026gt; kubectl top pod --sort-by \u0026lt;JSONPATH\u0026gt; --selector \u0026lt;selector\u0026gt; Here is a good guide to install metrics server and dashboard (e.g. on docker-desktop). Pods and Containers ConfigMaps: store data in key-value map. Secrets: same as ConfigMaps but for sensitive data only Two ways to pass ConfigMap and Secret data to your container: As environment variables in container operating system As files presented on mounted volumes in container file system. Container Resource management: Resource requests: K8s scheduler will use resource requests to avoid scheduling pods on nodes that do not have enough available resources. 1 CPU unit = 1/1000 of one core Resource limits: allow you to limit the amount of resources your containers can use. The container runtime is responsible for enforcement. The enforcement behaviour is different. For example, some terminates container that attempts to use more resource than the limit. Probes Liveness Probe: automatically determine whether or not a container application is in a healthy state. By default K8s does not consider a container to be down until the container process stops. Liveness Probe allow you to customize this detection mechanism and make it more sophisticated. Startup Probes: similar to liveness probes. However, while liveness probes run constantly on a schedule, startup probes run at container startup and stop running once they succeed. Startup probes are used to determine when the application has successfully started up. It is especially useful for legacy applications that can have long startup times. Readiness Probes: determine when a container is ready to accept requests. When you have a service backed by multiple container endpoints, user traffic will not be sent to a particular pod until its containers have all passed the readiness checks defined by their readinesse probes. Use readiness probes to prevent user traffic from being sent to pods that are still in the process of starting up. Restart policy for self-healing pods (default) Always: container will always be restarted if they stop, even if they completed successfully (returned 0). OnFailure: container will be restarted if the container process exists with an error code, or the container is determined to be unhealthy by a liveness probe. Never: let it be Multi-container pods: containers share the same networking namespace and can communicate with one another on any port, even if the port is not exposed to the cluster Container can use volumes to share data in a Pod. Example: a legacy application is hard-coded to write log output to a file on disk. You use a sidecar container to read the log file from shared volume and prints it to the console so the log output will appear in the container log. Init containers: containers that run once during the startup process of a pod. A pod can have any number of init containers, and they will each run once into completion, before the next init container starts. You may use init containers to perform a variety of startup tasks, they can contain and use software and setup scripts that are not needed by your main containers. They are often useful in keeping your main containers lighter and more secure by offloading startup tasks to a separate container. Use case include: cause a pod to wait for another K8s resource to be created before finishing startup perform sensitive startup steps securely outside of app containers populate data into a shared volume at startup communicate with another service at startup Scheduling: Scheduler (a component in control plane) assigns Pods to a suitable Node so kubelets can run them. The factor taken into account: resource request vs available node resources various configurations that affect scheduling using node labels Pod allocation nodeSelector is an attribute of Pod to allow you to limit which Node(s) the Pod can be scheduled on. The selector is based on label. nodeName is an attribute of Pod that allows you to bypass scheduling and assign Pod to a specific Node by name. DaemonSet: automatically runs a copy of a Pod on each node. When a new node is added to the clsuter, DaemonSet will run a new copy of the Pod on it. DaemonSets also respect normal scheduling rules around node labels, taints and tolerations. If a pod would not normally be scheduled on a node, a DaemonSet will not create a copy of the Pod on that node. Static Pod: A Pod that is managed directly by the kubelet on a node, not by the K8s API server. They can run even if there is not K8s API server present. Kubelet automatically creates static Pods from YAML manifest files located in the manifest path on the node. Mirror Pod: Kubelet will create a mirror Pod for each static Pod. Mirror Pods allow you to see the status of the static Pod via the K8s API, but you cannot change or manage them via the API. Taints: applied to nodes to repel a set of pods. A taint specifies key-value and effect. Effect can be NoSchedule or NoEffect. The former prevents pods without matching tolerations to schedule to the tainted node. The latter also evicts pre-existing pods with no matching toleration. Tolerations: applied to pods so they can be scheduled to nodes with matching taints. A toleration consists of key-value pair, effect and operation. The operation can be Equal or Exists. To determine whether a toleration matches a taint. The keys and the effects must be the same. In addition: the operator is Exists (and thus no value should be specified in the toleration); or the operator is Equal, and all the values match those of the taints; Now we have three ways to influence the scheduling behaviour. The first, is simply by specifying nodeSelector on the Pod, with the required the node label. The second, as just discussed, is to use Taints and Tolerations. The third way, is similar to the first, using nodeAffinity attributes on Pods. nodeAffinity is more powerful and flexible than nodeSelector by supporting more complex scheduling rules (e.g. matching rules).\nUse Node Affinity when your scheduling rule is based on direct condition, i.e. schedule a Pod to this Node when XXX. In this case, you have well-known labels on nodes, and specify nodeAffinity on Pods. Use Taints and Tolerations when your scheduling rule is based on inverse statement, i.e. do not schedule a Pod to this Node unless XXX. In this case, you put a taint \u0026#8220;MyCondition:NoSchedule\u0026#8221; on a Node, so that no Pod will ever get scheduled to this Node. The only exception is when a Pod has the Toleration \u0026#8220;MyCondition:NoSchedule\u0026#8221;. Deployments Deployment is an object that defines a desired state for a ReplicaSet (a set of replica Pods). The Deployment Controller seeks to maintain the desired state by creating, deleting, and replacing Pods with new configurations. With Deployments, you can horizontally scale an application up and down by changing the number of replicas. You can perform rolling updates and rollback. Networking The K8s network model defines how Pods communicate with each other, regardless of which Node they are running on. Each Pod has its own unique IP address within the cluster. Any Pod can reach any other Pod using that Pod\u0026#8217;s IP address. This creates a virtual network that allows Pods to easily communicate with each other. One type of K8s network plugin is CNI plugin. It has many flavours such as Calico. Each plugin has its own unique installation process. Kubenetes nodes will remain NotReady until a network plugin is installed. The K8s virtual network uses a DNS (e.g. a Kubeadm cluster uses CoreDNS pod in kube-system namespace) to allow Pods to locate other Pods and Services using domain names. The Pod DNS name follows this format: pod-ip-address.namespace.pod.cluster.local A K8s NetworkPolicy is an object that allows you to control the flow of network communication to and from Pods so you can isolate traffic. NetworkPolicy can apply to Ingress (using from selector), Egress (using to selector) or both. NetworkPolicy has an attribute podSelector to determine to which Pods in the namespace the NetworkPolicy applies, by selecting Pods by with Pod labels. By default, Pods are considered non-isolated and completed open to all communication. If any NetworkPolidy selects a Pod, the Pod is considered isolated and will only be open to traffic allowed by NetworkPolicies. A variety of selector can be used: podSelector, namespaceSelector, ipBlockSelector and port. Services Services provide a way to expose an application running as a set of pods, so clients can access applications in an abstract way without needing to be aware of the application pods. In this model, client make requests to a Service, which routes traffic to its pods in a load-balanced fashion Endpoints are the backend entities to which Services route traffic. If there are multiple Pods behind a service, each Pod will have an endpoint associated with the service. Each service has a type that determines how and where service will expose your application. ClusterIP: expose application inside the cluster network NodePort: expose application outside the cluster network LoadBalancer: expose application outside thecluster network, but use an extermal cloud load balancer from cloud platform. Services are assigned with DNS names. The FQDN follows this format: service.namespace.svc.cluster-domain.example, which is used by pods across namespaces Pods within the same namespace can reference service simply by service name. To manage external access to service, you can also use Ingress object. Ingress object is capable of providing more functionality than a simple NodePort Service, such as SSL termination, advanced load balancing, or name-based virtual hosting. You must install one or more Ingerss controller (many different implementations) to back up the ingress objects. Ingress defines a set of routing rules. Each rule has a set of paths, each with a backend. Requests matching a path will be routed to its associated backend. If a Service uses a named port, an ingress can also use the port\u0026#8217;s name (instead of port number) to choose to which port of a service it will route. Storage Volumes allow you to store data outside the container file system, while allowing the container to access the data at runtime. When Pod is gone, volumes do not persist. Persistent Volumes are a slightly more advanced form of Volume. They allow you to treat storage as an abstract resource and consume it in Pods. PV can be provisioned separately by storage administrator, and they persist regardless of pod lifecycle. PV needs to be claimed by pods. PV uses a set of attributes to describe the underlying storage resource. Both volumes and PVs each have a volume type: NFS, Cloud (AWS, Azure, GCP), ConfigMaps and Secrets, Simple Directory on node Two volume types to distinguish: hostPath: store data in a specified directory on K8s node emptyDir: store data in dynamically created location on the node. The directory exists only as long as the Pod exists on the node. The directory and the data are deleted as Pod is removed. This type is useful for simply sharing data between containers in the same pod. Both volumes and PVs are specified under Pod, and individual containers must include volumeMounts object to map volume name to local mountPath Storage Class object allow K8s admins to specify the types of storage services they offer on their platform. A key property is allowVolumeExpansion. This allows PVC to resize. At storage class level, there are two reclaim policies: Retain and Delete. The default is Delete. PV has an attribute named persistentVolumeReclaimPolicy. This is reclaim policy at PV level. If the attribute is not defined, it is inherited from storage class. The persistentVolumeReclaimPolicy has three options. When PVC is deleted: Retain: keeps all data but requires admin to manually reclaim the volume (i.e. delete PV, clean up data, delete storage asset) Delete (cloud storage only): deletes both PV and the underlying storage resource automatically Recycle: scrub (rm -rf /vol/) all data in the underlying storage resource, and allow the volume to be reused. PVC represents a user\u0026#8217;s request for storage resources. It defines a set of attributes similiar to those of a PV. When a PVC is created, it will look for a PV that is able to meet the requested criteria. If it finds one, it will automatically be bound to the PV. PVC can be mounted to a Pod\u0026#8217;s containers just like any other volume In general the CKA exam experience is quite positive and rewarding. In future posts I will shift focus on Kubernetes not only for the CKA exam, but also for keeping track of my learning.\nGood luck with your CKA exam.\nPrevious PostPublic Key Infrastructure (PKI) – Introduction Next PostSecure web application deployment ","date":"2021-04-30T09:50:00-04:00","permalink":"/2021/04/preparing-certified-kubernetes-administrator-exam/","title":"Certified Kubernetes Administrator (CKA) Exam"},{"content":"A public-key infrastructure (PKI) is a set of roles, policies, hardware, software and procedures needed to create, manage, distribute, use, store and revoke digital certificates and manage public-key encryption. The algorithms are based on Publick-key cryptography. The format of the digital certificate is defined in X.509 standard.\u0026nbsp;\nCertificate Authority \u0026#8211; CA digitally signs and publishes the public key to user. Website requesting certificates start with a key pair. It then converts public key into CSR (certificate signing request), including the identity. Once the identity of requestor is validated, CA will sign the public key of requestor, using its own private key. The output of this is the certificate.\nRegistration Authority \u0026#8211; CA may delegate some roles to registration authority (RA). RA is responsible for accepting requests for certificates and authenticating the entity making the request. However, RAs do not have the signing authority of a CA. Note that Microsoft may have referred to a subordinate CA as an RA, which is incorrect according to X.509 PKI standards.\nValidation types Domain Validation: domain ownerships is usually verified via DNS record. Organization Validation: the organization name and address are verified and put into the certificate. Extended Validation: verifies existence and location of the legal entity, as well as domain ownership. EV cannot be issued as a wildcard certificate. Private Certificate Authorities You can create private CA and use it to sign certificates. Your user need to manually install and trust your private CA so that all certificates issued from the CA will inherit that trust. For revocation, you will also need to maintain an HTTP server for the certificate revocation list, or an OCSP responder.\nCertificate Revocation List (CRL) SSL certificates include information on how to access a certificate revocation list. Client will download and check this list to make sure the certificate has not been revoked. This mechanism has largely been replaced by OCSP responders.\nOnline Certificate Status Protocol (OCSP) The OCSP protocol is a replacement for CRLs, with the benefit of being more real-time and requiring less bandwidth. The general operation is similar: clients are to query to OCSP responder to check if a certificate has been revoked.\nCommercial vs non-profit CA Commercial (e.g. SSLs.com)Non-profit (e.g. Let\u0026#8217;s Encrypt)ValidationDV, OV and EVDV onlyWildcardSupportedSupported (using DNS-01 challenge via ACME v2)CostNot FreeFreeExpiration1-3 years90 days ACME protocol Traditionally, there are several command-line utilities such as openssl, cfssl, or keytool (Java) to manage certificate related tasks. The process are mostly manual. The Internet Security Research Group (ISRG) developed the ACME (Automated Certificate management Environment) protocol.\u0026nbsp;\nIn this protocol, there is a certificate management agent (client) on the given web server. The agent generates a key pair and shares it with the CA at the outset of the validation process. Once validation is finished and the agent is verified as the proven owner of the key pair. It can use its key to digitally sign the CSRs it generates and sends to the CA via HTTPS requests. The CA uses the CSR, along with its associated public key, to issue the certificate and send it back to the agent. The agent downloads and installs it, then notifies the designated contact.\u0026nbsp;The agent can be automated to check in with the CA at given intervals to rotate certificates and keys.\nLet\u0026#8217;s encrypt adopts ACME protocol by using Boulder on the server side, and the most commonly used client is certbot. Smallstep also introduced ACME support in step CA in 2019.\nLets Encrypt I have used let\u0026#8217;s encrypt several times because it is free and easy to manage with certbot, which can be installed using brew on Mac. Here\u0026#8217;s how I quickly generate certificate manually:\nDOMAIN=orthwebdemo.digihunch.com echo $DOMAIN sudo certbot -d $DOMAIN --manual --preferred-challenges dns certonly # get ready to change txt record ls /etc/letsencrypt/live/orthwebdemo.digihunch.com/ The --manual switch starts interactive prompts, which includes configuring TXT record and wait for the update.\nOpen source implementations Here is a list of open-source implementations of PKI management:\nOpenSSL: classic tool for PKI management. The Mac/BSD implementation and GNU implementation are slightly different. Keytool: Java\u0026#8217;s Key and Certificate Management Tool that supports formats used in Java Cfssl: introduced by CloudFlare to simplify the PKI management process. On Ubuntu, the apt package name is golang-cfssl Hashicorp Vault: CA, secret management and encryption. Boulder: implemented in Go based on ACME protocol. Let\u0026#8217;s Encrypt uses Boulder on the server side. EJBCA: a full-featured, enterprise-grade implementation in Java, managed by Swedish company PrimeKey Solutions AB. Managed CA as service AWS Certificate Manager: [Update] as of Sep 2022, the managed CA capability was spun off as a new service called AWS Private Certificate Authority, to distinguish from the certificate management capability. EJBCA Enterprise, as Azure Market place Google Cloud Certificate Authority Service API Previous PostIntro to Data Analytics Platform on Azure Next PostCertified Kubernetes Administrator (CKA) Exam ","date":"2021-04-08T22:07:00-04:00","permalink":"/2021/04/public-key-infrastructure-pki/","title":"Public Key Infrastructure (PKI) – Introduction"},{"content":"Having been in transactional data world for almost the entire career, recently I have to pick up quite a few things to catch up on the analytical workload. The main purpose of data analytics project is to build analysis services models and manage deployed databases. Later in this post I\u0026#8217;ll discuss some useful Azure resources for data analytics.\nData Model Data are typically organized in relational model for better transactional performance, following the normalization forms. The relational model, however, might not be the most appropriate schema for analytics. In this case, it is better to use a separate non-relational repositories that can store information in a format that better aligns with its semantics, and hence more friendly to analytical applications.\nA model consists of: data sources, tables, relationships, measures, KPIs, roles, etc. The model can be deployed to analysis database (e.g. SSAS). When deploying, queries (from source) and calculations are done. Data modelling is the process of determining how your tables are related to each other. This process is done by defining and creating relationships between the tables. From that point, you can enhance the model by defining metrics and adding custom calculations to enrich your data. Creating an effective and proper data model is a critical step in helping organizations understand and gain valuable insights into the data.\u0026nbsp;The model is another critical component that has a direct effect on the performance of your report and overall data analysis. The process of preparing data and modelling data is an iterative process.\u0026nbsp;\nData Warehouse Moreover, organizations have multiple data stores, with varying formats and different structures such as live stream, sensor, etc. They all need to be combined to generate insights. The process of combining all of the local data source is known as data warehousing. The process of analyzing streaming data and data from the Internet is known as Big Data Analytics.\nA data warehouse gathers data from many sources within an organizations. This data is then used as the source for analysis, reporting and OLAP. The focus of a data warehouse is to provide answers to complex queries. A modern data warehouse might contain a mixture of relational and non-relational data, including files, social media streams, IoT sensor data.\nThe diagram above is platform neutral. If you take Azure for example, a typical data warehouse platform involves the following components:\nAzure Data Factory: ingestion of data (integration service)Azure Data Lake Storage: store large quantity of data before analyzingAzure Databricks: other forms of data preparation (transformation, cleaning) by SparkAzure Synapse Analytics: store cleansed data, for Azure Analysis Service to consumeAzure Analysis Service: query Synapse Analytics for detailed analysis and generate insightsPower BI: Generate graphs, charts and reports by using information from Azure Analysis service This is the diagram of those components as seen on Azure documentation:\nModern Data Warehouse Let\u0026#8217;s discuss each components.\nAzure Data Factory (ADF) Big data requires a service that can orchestrate and operationalize process to refine the enormous stores of raw data into actionable business insights. ADF is managed cloud service built for complex hybrid ETL, ELT and data integration projects.\u0026nbsp; ADF retrieves data from one or more data sources, and convert it into a format you can process. The data sources might present data in different ways, and contain noises that need to be discarded. For example, the source data may contain dates with bad format. ADF can unify the data structure. In ADF, you define the work performed as a pipeline of operations. A pipeline can run continuously, or triggered by schedule.\nA linked service provides the information needed for ADF to connect to a source or destination. A pipeline is a logical grouping of activities that together perform a task.\nThe ADF UX (user interface experience) lets you visually author and deploy resources for your data factory without having to write any code. You can drag activities to a pipeline canvas, perform test runs, debug iteratively, and deploy\u0026nbsp; and monitor your pipeline runs.\nAzure Data Lake Storage (ADLS) A data lake is a repository for large quantities of raw data. Because the data is raw and unprocessed, it\u0026#8217;s very fast to load and update, but the data hasn\u0026#8217;t been put into a structure for efficient analysis. You can think of a data lake as a staging point for your ingested data, before it\u0026#8217;s massaged and converted into a format suitable for performing analytics. Note that a data warehouse also stores large quantities of data, but the data in a warehouse has been converted into a format for efficient analysis. Data lake holds raw data, whereas data warehouse holds structured information.\nAzure Data Lake Storage is essentially an extension of Azure Blob storage, organized as a near-infinite file system. It supports POSIX file and directory structure for storage and RBAC. ADLS is also compatible with HDFS, a popular open-source solution to store large quantities of data.\nAzure Databricks Apache Spark is in-memory cluster computing technology, much faster than disk-based applications, and works with multiple programming languages to let you manipulate distributed data sets (DDS). There is no need to structure everything as map and reduce operations. Databricks develops a web-based platform for working with Spark cluster. It provides automated cluster management and IPython-style notebooks.\nAzure Databricks is a managed Apache Spark environment running on Azure to provide big data processing, streaming, and machine learning. Apache Spark is a highly efficient data processing engine, with rich selections of libraries, that can consume and process large amounts of data very quickly. Azure Databricks also supports structured stream processing.\nDelta Lake is an open-source storage layer in Azure Databricks that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs.\nAzure Synapse Analytics Azure Synapse is an end-to-end solution. It unifies data analysis, integration and orchestration, Data Lake, Data Warehouse, ELT/ELT, ML capabilities and visualization. With Synapse, you can process large amounts of data very quickly. You can ingest data from external sources (e.g. flat file, ADLS, other DBMS) and then transform and aggregate the data into a format suitable for analytics processing. You can also use this data as input to further analytical processing using Azure Analysis Services. Azure Synapse is a comprehensive service with the following components:\nSynapse Analytics \u0026#8211;\u0026nbsp; a successor of SQL DW technology. Synapse analytics has inherited its MPP capability.Data Exploration \u0026#8211; Synapse Studio makes data exploration in Data lakes, SQL engine and Spark very easy.\u0026nbsp;Data Integration \u0026#8211; inherited ADF’s data movement and transformation components, which allows building complex ETL pipelines without codeDevelopment \u0026#8211;\u0026nbsp; supports Spark, Python, Scala, Spark notebooks, SQLData visualization \u0026#8211; Synapse Studio allows user to connect to Power BI workspace and get the same report development experience The most critical component, Synapse Analytics is analytics engine, designed to process large amounts of data very quickly. Synapse Analytics leverages a MPP (massively parallel processing) architecture, including a control node and a pool of compute nodes. When you submit a processing request, the control node transforms it into smaller requests and send them to compute nodes. Each compute node runs the queries over the portion of data that they each hold. When each node has finished its processing, the results are sent back to the control node where they\u0026#8217;re combined into an overall result.\nSynapse Analytics supports two computational models: SQL pools and Spark pools. In a SQL pool, each compute node uses an Azure SQL Database and Azure Storage to handle a portion of the data. You can submit queries in the form of T-SQL statement. Synapse Analytics uses a technology named PolyBase to retrieve data from a wide variety of sources (e.g. Blob, ADSL, CSV). You can save the data read in as SQL tables in Synapse Analytics service. In a Spark pool, the nodes are replaced with Spark cluster. You run Spark jobs comprising code written in Notebooks (in Python, Scala, or Spark SQL). The Spark cluster splits the work out into a series of parallel tasks that can be performed concurrently. You can save data generated by your notebooks in Azure Storage or ADLS. To scale Spark pool, you can specify the cluster size, or turn on autoscaling.\nAzure Analysis Service (AAS) AAS is a fully managed PaaS that enables you to build tabular models to support OLAP queries. You can combine data from multiple sources (e.g. Azure SQL Database, ADLS, Cosmos DB, etc). You use those data sources to build models that incorporate your business knowledge. A model is essentially a set of queries and expressions that retrieve data from various data sources and generate results. The results can be cached in-memory for later use, or they can be calculated dynamically, directly from underlying data sources. AAS has significant functional overlap with Synapse Analytics, but it\u0026#8217;s more suited for processing on a smaller scale. The comparison below outlines the difference:\nSynapse AnalyticsAzure Analysis Service (AAS)\u0026#8211; very high volumes of data (multi-terabyte to petabyte sized datasets)\n\u0026#8211; very complex queries and aggregations\n\u0026#8211; data minding, and data exploration\n\u0026#8211; complex ETL operations\n\u0026#8211; low to mid concurrency (127 users or fewer)\u0026#8211; smaller volumes of data (a few terabytes)\n\u0026#8211; multiple resources that can be correlated\n\u0026#8211; high read concurrency\n\u0026#8211; detailed analysis, and drilling into data, using functions in Power BI\n\u0026#8211; rapid dashboard development from tabular data Many scenarios can benefit from using Synapse Analytics and Analysis Services together. If you have large amounts of ingested data that requires preprocessing, you can use Synapse Analytics to read this data and manipulate it into a model that contains business information rather than a large amount of raw data. The scalability of Synapse Analytics gives it the ability to process and reduce many terabytes of data down into a smaller, succinct dataset that summarizes and aggregates much of this data. You can then use AAS to perform detailed interrogation of this information, and visualize the results of these inquiries with Power BI.\nAzure HDInsight Azure HDInsight is a managed analytics service based on Apache Hadoop, a collection of open-source tools and utilities that enable you to run processing tasks over large amounts of data. HDInsight uses a clustered model, similar to that of Synapse Analytics. HDInsight stores data using ADLS. You can use HDInsight to analyze data using frameworks such as Hadoop Map/Reduce, Apache Spark, Apache Hive, Apache Kafka, Apache Storm and more.\nPower BI Microsoft PowerBI is a collection of software services, apps and connectors. It consists of a Microsoft Windows Desktop application Power BI Desktop, an online SaaS service Power BI service, and mobile Power BI apps available on any device. These three elements are designed to let people create, share and consume business insights. A common workflow with Power BI can be outlined as:\nBring data into Power BI Desktop, and create a reportPublish to the Power BI service, where you can create new visualizations or build dashboardsShare dashboard with others, especially people who are on the goView and interact with shared dashboards and reports in Power BI mobile apps. Basic building blocks in Power BI include:\nvisualizations: chart, colour-coded map, etcdatasets;reports: a collection of visualizations that appear together on one or more pages;dashboards: when you\u0026#8217;re ready to share a report, or a collection of visualizations, you create a dashboard, much like the dashboard in a car, a Power BI dashboard is a collection of visuals from a single page that you can share with others. Often, it\u0026#8217;s a selected group of visuals that provide uick insight into the data or story you\u0026#8217;re trying to present. Previous PostGit Branching Strategy Next PostPublic Key Infrastructure (PKI) – Introduction ","date":"2021-03-21T21:57:00-04:00","permalink":"/2021/03/intro-to-data-analytics-platform/","title":"Intro to Data Analytics Platform on Azure"},{"content":"I have been in two discussions about Git branching strategy in different organizations. Too many concepts! So I open this post to jot down the lineage of common branching strategies to help organizations develop their branching policies. In terms of reference, there is a lot from Atlassian documentation. In addition, I also find this one article a good resource.\nCentralized workflow (no branching) In Centralized Workflow, the team uses a central repository to serve as the single-point-of-entry for all changes to the project. The default branch is master, and all changes are committed to this branch. This workflow does not require any other branches beside master. Local changes may conflict with upstream commits, and conflict needs to be resolved.\nThis workflow is usually seen in teams transitioning from SVN, with very basic skill level. This workflow may also be adopted in teams working on configuration management instead of source code. Centralized workflow is great for small teams. The conflict resolution process can form a bottleneck as the team scales in size.\nFeature Branch Workflow Instead of directly committing on their local master branch, developers create a new branch every time they start work on a new feature. Feature branches should have descriptive names (e.g. issue#112). Feature branches are pushed to the central repository so that they can be shared to other developers without touching any official (master) code. To get feedback on the new feature branch, create a pull request in a repository management solution (e.g. Bitbucket Cloud, Bitbucket Server). Before merge, you may have to resolve merge conflicts if others have made changes to the master branch of repo. This is to make sure your local master is synchronized with the upstream master. When your pull request is approved and conflict free, you can merge your branch to master branch.\nfeature branch workflow The Git Feature Branch Workflow is a composable workflow that can be leveraged by other high-level Git workflows. Git Feature Branch Workflow is branching model focused, instead of release focused. The Git Feature Branch Workflow can be incorporated into other workflows. The Gitflow, and Git Forking Workflows traditionally use a Git Feature Branch Workflow in regards to their branching models.\nGitflow Workflow First published in 2010 by Vincent Driessen. Gitflow defines a strict branching model designed around the project release. This provides a robust framework for managing larger projects. In addition to Feature Branch Workflow, Gitflow workflow assigns very specific roles to different branches and defines how and when they should interact.\nInstead of a single master branch, this workflow uses two branches to record the history of the project. The master branch stores the official release history, and the develop branch serves as an integration branch for features. It is also convenient to tag all commits in the master branch with a version number.\nThis workflow is operated in the following ways:\nA develop branch is created from master Feature branches are created from develop. When a feature is complete, with PR reviewed, it is merged into the develop branch. Features branches Once develop has acquired enough features for a release (or a predetermined release date is approaching), we fork a release branch off of develop. Creating this branch starts the next release cycle, so new features can be added to develop after this point. On the release branch itself, only bug fixes, documentation generation, and other release-oriented tasks should go in this branch. Once ready to ship, the release branch gets merged into master and tagged with a version number. In addition, it should also be merged back into develop, which may have progressed since the release was initiated. Maintenance or hotfix branches are used to quicly patch production releases. Hotfix branches are a lot like release branchs and feature branches except they\u0026#8217;re based on master instead of develop. As soon as the fix is complete, it should be merged into both master and develop (or the current release branch), and master should be tagged with an updated version number. Gitflow workflow There is a Git extension named git-flow to provide high-level repository operations for this Workflow, such as start a release, finish a release, start a hotfix, finish a hotfix.\nGitflow is ideally suited for projects that have a scheduled release cycle and for the DevOps best practice of continuous delivery. It ensures that the master branch reflects what is deployed (e.g. in production). However, it is quite complex and have a steep learning curve for organizations. It also runs long-lived branches, which is considered bad from CI/CD perspective. Branches are by definition to isolate and hide changes, whereas continuous integration is about exposing changes early on and frequently. In that sense, the Gitflow branching model and CI/CD are mutually exclusive ideas. In 2020, Vincent Driessen added a note at the beginning of his article on Gitflow:\nIf your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team.\nIf, however, you are building software that is explicitly versioned, or if you need to support multiple versions of your software in the wild, then git-flow may still be as good of a fit to your team as it has been to people in the last 10 years. Driessen\u0026#8217;s notes also points to some simple alternatives. On the other hand, the Atlassian tutorial on Gitflow has described Gitflow workflow as a legacy (since Aug 2021 based on web archive). It points out at the beginning that Gitflow has fallen in popularity in favor of\u0026nbsp;trunk-based workflows.\nGitHub flow, Trunk-based development (TBD) and GitLab flow It is operationally expensive to manage multiple mainlines in Git flow workflow, with both source control and release in the picture. Some simple alternatives have been evolved, with single mainline, for example, GitHub flow and trunk-based development. They differ in where the release is performed from. In the GitHub flow, release is performed from branch before being merged back to master (trunk).\nGitHub flow In trunk-based development, release is not performed until the feature branch has been merged to the trunk (master). In trunk-based development, feature branches are supposed to be short-lived. It is a common practice among DevOps teams, since it streamlines merging and integration phases.\nTrunk-based development Trunk-based development has gained some momentum in recent years, due to its DevOps friendliness. This is a website that advocates it and here\u0026#8217;s a DZone article about it.\nGitlab Flow In response to the shortcomings of GitHub flow and Gitflow, Gitlab introduced its own proposal of branching strategy, known as Gitlab flow. The most distinctive aspect is the environment branches. In Gitlab flow, you run multiple long-lived branches, each of them representing an environment. The typical steps are as follows:\nYou create short-lived feature branches, and merge them often to the master. Every developer starts from master and targets master. Other branches are merged from previous lower environment branches. You can deploy a new version to production, by merging master into the production branch. If you need to know what code is in production, you can check out the production branch to see. Gitlab flow You only need to work with release branches, if you need to release software code to the outside world. Here are some best practices in GitLab flow.\nForking Workflow Forking workflow is fundamentally different. The key steps are as follows:\nA developer \u0026#8216;forks\u0026#8217; an \u0026#8216;official\u0026#8217; server-side repository. This creates their own server-side copy. This is their personal public repository, and no other developer are allowed to push to it. The new server-side copy is cloned to their local system. This forms an environment dedicated to this developer. With the local clone, developer needs to create the upstream remote manually using \u0026#8220;git remote add upstream\u0026#8221; command. This allows the developer keep the local repository up-to-date as the official project progresses. A new local feature branch is created. Developer commits to the new local branch, and pushes to their own copy of repository on server. Developer files a pull request from the new branch (in own copy of repository) to the \u0026#8216;official\u0026#8217; repository. The project maintainer knows that an update is ready to be integrated. The PR also serves as a discussion thread. The PR gets approved for merge and is merged into the original server-side repository. This workflow has other names, such as fork-and-branch workflow, and is commonly used in GitHub for managing open-source projects. However, this should not be confused with the aforementioned GitHub flow.\nConclusion In order to fully support distributed source control, Git abstract version control problems into concepts such as commit, branch, etc. This makes discussion about Git workflow and branching strategy difficult due to the conceptual hurdles and organization differences. We covered choices of Git branching strategy in this post. As Vincent Driessen commented in his original Gitflow posting, panaceas don\u0026#8217;t exist. We should consider the context (e.g. team size, Git skill level, etc) to determine the best Git branching strategy.\nPrevious PostCensus Data from Statistics Canada Next PostIntro to Data Analytics Platform on Azure ","date":"2021-03-07T19:20:42-04:00","permalink":"/2021/03/git-branching-strategy/","title":"Git Branching Strategy"},{"content":"Statistics Canada carries census every 5 years, with 2016 being the last run. The census data by Statistics Canada provides a wealth of insights but are published in raw format. Post-processing work is needed to extrapolate information, such as median income of a neighbourhood, age distribution of a city, etc. For someone like myself without any background in geographical informatics, it took a bit of learning to see how these work together. The following information are typically included in the Census data:\nPopulation Population density Age Structural type of dewellings Family size Marital status Language Income Place of birth Level of education Occupation We will start with level of Geographics. The level of geographics may change slightly between census programs in different years. The most recent 2016 census uses the following diagram to depict levels of geographics:\nGeographic Levels This diagram reflects a number of different hierarchies of geographic units. The best resource to understand each block, is the illustrated glossary and the chapter Census Geography in comprehensive Guide to the Census Population. For example, the chain on the far left of the diagram runs across these levels:\nCanadaCanadaGeographical Region of CanadaGeographical Region\u0026#8230;Province or TerritoryProvince or TerritoryForward Sortation AreaForward Sortation Ar\u0026#8230;Postal CodePostal CodeViewer does not support full SVG 1.1\nIn this hierarchy, the level of Geographical Region of Canada is standardized in Standard Geographic Classification (SGC), in which the provinces and territories are also encoded. Note that each Census include a dictionary where all sorts of codes are kept. The dictionary also includes definition of the rest two levels: FSA (forward sortation area as the first three digits of postal code) and postal code (all six digits). Note that postal code is a mark of Canada Post Corporation, and you may translate postal code into other levels in standard geographic areas, such as CD. This is not straightforward though. You will need a product called Postal Code Conversion File (PCCF) for the conversion. Statistics Canada does not directly distribute this product. It works with its Data Liberation Initiative (DLI) partners to deliver this product.\nOn the diagram there are also other path to run down the hierarchy. For example, from Canada down to federal electoral district (aka ridings). However, the census is not carried out by either election ridings or postal code. Instead, it is carried out by its own collection of levels dedicated for census purpose. When using census data, we need to be familiar with these units.\nCensus metropolitan area (CMA) and census agglomeration (CA): formed by one or more adjacent municipalities centred on a population centre (known as the core), such as Chatham-Kent CA, Kitchener-Cambridge-Waterloo CMA. Note that CMA and CA can expand across provincial borders, such as Ottawa \u0026#8211; Gatineau CMA. So CMA or CA is not a unit under province or territory.\nCensus Division (CD, essentially a region or county): general term for provincially legislated areas (such as county, municipalité régionale de comté and regional district) or their equivalents.\nCensus Subdivision (CSD, essentially a city): the general term for municipalities or areas treated as municipal equivalents for statistical purposes.\nCensus Tract (CT): small, relatively stable geographic areas that usually have a population of less than 10,000 persons, based on data from the previous Census of Population Program.\nDissemination Area (DA): \u0026nbsp;is a small, relatively stable geographic unit composed of one or more adjacent dissemination blocks with an average population of\u0026nbsp;400 to 700\u0026nbsp;persons based on data from the previous Census of Population Program. It is the smallest standard geographic area for which all census data are disseminated.\nDissemination Block (DB): an area bounded on all sides by roads and/or boundaries of standard geographic areas. The dissemination block is the smallest geographic area for which population and dwelling counts are disseminated.\nWith these in mind, we can build two hierarchies closely related to census data:\nCanadaCanadaCMA/CACMA/CACensus TractCensus TractDissemination AreaDissemination AreaDissemination BlockDissemination BlockCanadaCanadaGeo. RegionGeo. RegionCensus DivisionCensus DivisionCensus SubdivisionCensus SubdivisionDissemination AreaDissemination AreaDissemination BlockDissemination BlockViewer does not support full SVG 1.1\nNow we download census profile data from Statistics Canada. In the dropdown you can pick from the many of the aforementioned geographic levels.\nIf you pick Census tracts (CT), there is one data file. The CSV file is about 160M. Also note that under geographic level column, it indicates two levels: CA/CMA and CT, which is important to keep in mind as we go through the data. In the content of the CSV, under the GEO_LEVEL column, value 1 stands for CA/CMA and value 2 stands for CT. Therefore, when GEO_LEVEL=1, the GEO_CODE value is a CA/CMA code based on Statistical Area Classification; when GEO_LEVEL=2, the GEO_CODE value is a CT numerical name (preceded by CMA/CA code). What CT numerical name represents what geographic area, is all defined in Census Tract Reference Map. There is no textual name for each census tract.\nTo take another example, select Dissemination areas (DAs) from the dropdown. Now the size of the CSV becomes 1.6G, but smaller data files are provided by province and territories. Select the data file for Ontario only.\nNote that there are five geographic levels as indicated: Canada, provinces/territories, CDs, CSDs and DAs. This suggests we will see 5 different values under the GEO_LEVEL column in the data file:\n0 \u0026#8211; Canada 1 \u0026#8211; Provinces and Territories 2 \u0026#8211; CDs 3 \u0026#8211; CSDs 4 \u0026#8211; DAs Read the SGC documentation to understand the code from level Canada to level CSD. DA is similar to CT because the code is defined in reference map here. Apart from DA and CT, there are other levels (such as ridings) with reference maps, as outlined in the Census geography page.\nWith all the above information, we can parse the data programmatically. Of course, the schema and coding information applies to Canada. Outside of Canada, pretty much all states have a counterpart government agency that manages census and statistics, just with different formats to understand from ground up. A lot of census geography concepts applies to other countries as well. For example:\nCensus Bureau of United States Australian Bureau of Statistics Office for National Statistics (UK) Eurostat (European Union) Welcome to the world of data.\nPrevious PostBasic Resource Object in Kubernetes 2 of 2 Next PostGit Branching Strategy ","date":"2021-02-25T21:18:09-04:00","permalink":"/2021/02/interpret-census-data-from-statistics-canada/","title":"Census Data from Statistics Canada"},{"content":"We continued from previous posting about resource object, starting from storage related ones. Volume In Kubernetes, we use the term volume to refer to a section storage device. There are many plugins, compliant to Container Storage Interface (CSI), to allow heterogeneous storage resources to be surfaced as volumes in Kubernetes. CSI allows storage driver to operate in parallel to the main Kubernetes code tree. Any driver that complies with CSI would work with any orchestration platform that requires CSI, such as Docker Swarm, Kubernetes. Three main resources in the storage system are: PV (persistent volumes), PVC (persistent volume claims), and SC (storage classes).\nPersistent Volume Persistent Volumes (PV) allows you to map external storage onto the Kubernetes cluster. It is a representation of the external storage on the cluster. A single external storage volume can only be represented by a single PV. For example, you cannot have a 50GB external volume that has two 25GB PVs each representing half of it.\nPV can be mounted in three options:\nRWO (ReadWriteOnce): allows single PVC to mount. This is common for block device.RWM (ReadWriteMany): allows multiple PVCs to bind as read and write. This is common for file and object level access.ROM (ReadOnlyMany): allows multiple PVCs to bind as read only. Think of it along the lines of ISO media. Note that a PV can only be opened in one of the modes above. All connecting PVC (if multiple are allowed) will use that mode.\nPersistent Volume Claim Persistent Volume Claims (PVC) act like tickets that authorize applications (Pods) to use a PV. Once a Pod has the PVC, it can bind the respective PV as a volume. You need to specify PV name when declaring a PVC to associate them. Pods do not act directly on PVs, they always act on the PVC object that is bound to the PV. When a PVC is released, two actions can be configured in the policy: Delete and Retain. The delete policy will delete the PV as well as associated storage resource on the external storage system. The retain policy will keep the associated PV object on the cluster as well as any data stored on the associated external assets.\nThe spec section of PVC object declaration must match the fields in the corresponding PV it binds to. For example access modes, capacity and storage class name.\nStorage Class Storage classes allow you to define different classes (or tiers) of storage using an external provisioner such as aws-ebs. This works well with cloud storage provider. As long as the plugin for storage backend is available, you can configure as many StorageClass object as you need, and even specify to encrypt them. Storage classes create PV dynamically, so you will need to create PVC object that reference the newly created storage class, in order to use cloud storage.\nThe whole purpose of storage class is to create PVs dynamically, for various storage backend/plugin. You just create the StorageClass object and use a plugin to tie it to a particular type of storage on a particular storage back-end. When matching PVCs appear, the StorageClass dynamically creates the required volume on the back-end storage system.\nIf a cluster has a default storage class, you can deploy a Pod using just PVC with PodSpec, without explicitly declare storage class per Pod. However, this is not recommended in production.\nConfigMaps With modern application it is a good practice to decouple configurations from application execution environment. They are stored separately but brought together at runtime. ConfigMap (CM) allows you to store configuration data outside of a Pod, and dynamically inject the configuration data into a Pod at runtime. ConfigMaps are essentially key/value pairs, and each key/value pair is called an entry.\nOnce data is stored in a ConfigMap, it can be injected into containers at run-time via one of the three methods:\nenvironment variables: updates to ConfigMap is not updated arguments to the container\u0026#8217;s startup command (very limited)files in a volume (most flexible): requires creating a ConfigMap volume in the Pod template and mounting. Eateries in the ConfigMap will appear in the container as individual files. You can make changes to entries after a container is deployed, and the change is seen in the file. The application is unaware that the data originally came from a ConfigMap. Also note that ConfigMap is not to store sensitive data.\nSecret Kubernetes Secrets let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Storing confidential information in a Secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image.\nThe name of a Secret object must be a valid DNS subdomain name. A Secret can be used with a Pod in three ways:\nAs files in a volume mounted on one or more of its containers.As container environment variable.By the kubelet when pulling images for the Pod. Ingress Ingress manages manages external access to the services in a cluster, typically HTTP. It may provide load balancing, SSL termination and name-based virtual hosting. Also, you must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.You can choose from a number of Ingress controllers. Nginx is a common flavour.\nBy default, containers run with unbounded compute resources on a Kubernetes cluster. With resource quotas, cluster administrators can restrict resource consumption and creation on a namespace basis. Within a namespace, a Pod or Container can consume as much CPU and memory as defined by the namespace\u0026#8217;s resource quota. There is a concern that one Pod or Container could monopolize all available resources. A LimitRange is a policy to constrain resource allocations (to Pods or Containers) in a namespace.\nA LimitRange provides constraints that can:\nEnforce minimum and maximum compute resources usage per Pod or Container in a namespace.Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace.Enforce a ratio between request and limit for a resource in a namespace.\nSet default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime. Resource Quotas When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources. Resource quotas are a tool for administrators to address this concern.\nA resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.\nPrevious PostA shallow dive into Artificial Intelligence Next PostCensus Data from Statistics Canada ","date":"2021-02-08T21:02:16-04:00","permalink":"/2021/02/basic-resource-object-in-kubernetes-2-of-2/","title":"Basic Resource Object in Kubernetes 2 of 2"},{"content":"This is what I have learned after writing the Azure AI fundamentals exam. Overview Artificial intelligence is the software that imitates human behaviours and capabilities. AI encompasses a very broad range of areas. In Azure\u0026#8217;s product offering, it breaks it down into four application areas: Machine Learning, Computer Vision, Natural language processing and conversational AI. Note that the media, sometimes including tech companies, tend to use the terms AI and ML interchangeably, which is incorrect. ML did not really surface as a key\u0026nbsp; driver of AI commercially, until the last 10 \u0026#8211; 15 years. However, other areas of AI, such as computer vision and natural language processing had been around for a quite a while.\nNow we know the distinction between AI and ML: ML is just one of the many areas of AI but it has recently become the most attention-grabbing and cutting-edge area. We will introduce ML the last.\nComputer Vision Computer Vision is the ability of software to interpret the world visually through cameras, video and images. It has the following application scenarios:\nImage Classification: training ML model to classify images based on contents.Object Detection: training ML model to classify individual objects within an image, and identify their location with bounding box.Semantic Segmentation: An advanced ML technique in which individual pixels in the image are classified according to the object to which they belong. This forms mask layerImage Analysis: extract information from imagesFace detection, analysis, and recognition: specialized form of object detection that locates human face in image. This can be combined with classification and facial geometry analysis techniques to infer details such as gender, age, and emotional state. Face detection is impaired by extreme angles.OCR: detect and read text in images. The model training process is an iterative process in which the Custom Vision service repeatedly trains the model using some of the data, but holds some back to evaluate the model. The evaluation metrics include:\nprecision: what percentage of the class predictions made by the model were correct. E.g. model predicts 10 images are oranges. 8 actually are. Precision = 0.8recall: what percentage of class predictions did the model correctly identify. E.g. 10 images of apples, the model find 7. recall = 0.7AP (average precision): an overall metric that takes into account both precision and recall. Azure-specific: in Azure, computer vision services include:\nComputer Vision: analyze images and video, and extract descriptions, tags, objects and text;Custom Vision: train custom image classification (two special form: celebrity and landscape) and object detection models using your own image;Face: build face detection and facial recognition solutionsForm recognizer: extract information from scanned forms and invoices Azure-specific: difference between Computer Vision and Cognitive Service\nComputer Vision: A specific resource for the computer vision services. Use this type of resource if you don\u0026#8217;t intend to use any other cognitive services. Or if you want to track utilization and costs for your computer vision resource separatelyCognitive Service: A general cognitive service resource that include Computer Vision along with many other cognitive services, such as Text Analytics, Translator Text, and others. Use this resource type if you plan to use multiple cognitive services and want to simplify administration and development. Natural Language Processing NLP is the ability of computer to interpret written or spoken language, and respond in kind.\nAnalyze textRecognize (speech-to-text api) and synthesize speech (text-to-speech api to generate spoken output)Translate text and speechLanguage understanding Models that you use to accomplish speech recognition:\nacoustic model \u0026#8211; converts audio signal into phonemes (representations of specific sounds)language model \u0026#8211; maps phonemes to words, usually using a statistical algorithm that predicts the most probable sequence of words based on the phonemes Core concepts in language understanding\nutterance \u0026#8211; an example of something a user might say, and your application must interpret. Eg. Switch the fan on. Turn on the light.entities \u0026#8211; an item to which an utterance refers. e.g. fan, light. four types of entities: machine-learned, list, regex, pattern.anyintents \u0026#8211; represents the purpose, or goal, expressed in user\u0026#8217;s utterance.\u0026nbsp; E.g. Turn onNone intent Azure-specific: To create a language understanding application:\nFirst you must define entities, intents, and utterances with which to train the language model \u0026#8212; referred to as authoring the modelThen you must publish the model so that client applications can use it for intent and entity prediction based on user input Azure-specific: in Azure, NLP services include\nText Analytics: analyze text documents and extract key phrases, detect entities (places, people, dates), and evaluate sentiment (positive, negative). mixed language or ambiguous content will produce \u0026#8220;NaN\u0026#8221; in the result.Translator Text: translate text between languagesSpeech: recognize and synthesize speech, and translate spoken languageLanguage Understanding Intelligent Service (LUIS): train a language model that can understand spoken or text-based commands. Conversational AI This is the capability of a software agent to participate in a conversation.\nAzure-specific: Azure Bot service is a platform for creating, publishing and managing bots. Developers can use the Bot Framework to create a bot and manage it with Azure Bot service \u0026#8211; integrating back-end services like QnA maker and LUIS, and connecting to channels for web chat. QnA Maker enables you to quickly build a knowledge base of questions and answers that can form the basis of a dialog between a human and an AI agent.\nMachine Learning The reason I put machine learning the last, is because it\u0026#8217;s most important, and it involves some brain-burning mathematical details. Machine learning is a technique that uses mathematics and statistics to create a model that can predict unknown values. Machine learning is based on huge volumes of data. Data scientist can use all of that data to train machine learning models that can make predictions and inferences based on the relationships they find in the data.\nMachine learning algorithms There are so many sub-areas of machine learning and Google has an entire crash course for it. As far as application is concerned, we need to first match a new problem with an existing problem, and from there pick an appropriate algorithm. There is a cheat sheet for such purpose for Azure services. For learning purpose, we should focus on three problems. Before getting to that, we need to first distinguish supervised learning and unsupervised learning:\nSupervised learning: you train the machine using data which is well \u0026#8220;labelled\u0026#8221;. So some data is already tagged with correct answer. A supervised learning algorithm learns from labelled training data, and helps you predict outcomes for unforeseen data. Two typical types of supervised learning techniques are classification, and regression.Unsupervised learning: you do not need to supervise the model. Instead, you need to allow the model to work on its own to discover information by dealing with unlabeled data. Typical unsupervised learning technique is clustering. It mainly deals with finding a structure or pattern in a collection of uncategorized data. So the three problem we are going to focus on are:\nRegression Model (supervised): Use historic data to train the model to predict the numerical valueClassification Model (supervised): fit the features into the model and predict the classification of the labelClustering Model (unsupervised):\u0026nbsp; you don\u0026#8217;t have a label to predict. you only have features. You have to group similar items into clusters based on features. Now we will review how to assess the learning model in each technique:\nPerformance Metrics for Regression Model In all of the equations below, pi denotes predicted value, ai denotes actual value, and ā denotes the mean of actual values.\nMean Absolute Error (MAE): It has the same unit with original data so it only can be used to compare models whose errors are measured in the same unit.It has similar magnitude as RMSE (as will discuss below), but smaller in valueThe lower this value is, the better the model is predicting. 2. Mean Square Error (MSE):\n3. Root Mean Square Error (RMSE):\nIt measures the error rate of a regression modelIt can only be compared between models whos errors are measured in the same unit.RMSE and SD (standard deviation) have similar (not same) formula yet different purposes. SD measures the spread of data around the mean. RMSE measures the error of prediction (predicted vs true). The two formula produce the same result only if you use the mean as prediction. 4. Relative Square Error (RSE): A relative metric between 0 and 1. It has no units so can be used to compare models whose errors are measured in different units.The closer to 0 this metric is, the better the model is performing 5. Relative Absolute Error (RAE):\nA relative metric between 0 and 1.\u0026nbsp; It has no units so can be used to compare models whose errors are measured in different units.The closer to 0 this metric is, the better the model is performing 6. Coefficient of determination (R2):\nAlso known as r-squared. It summarizes the explanatory power of the regression model. In other words, how much of the variance between predicted and actual values is explained by the model.It is computed from the sums-of-squares terms, including Sum of Squares Total (SST), Sum of Squares Regression (SSR), and Sum of Squares Error (SSE), as illustrated above.R2 describes the proportion of variance of the dependent variable explained by the regression modelThe closer to 1 this value is, the better the model is performing. If the regression model is perfect, SSE = 0, R2 = 1If the regression is a total failure, SSE=SST, no variance is explained by regression, and R2 = 0 Performance Metrics for Classification Model Now we review the metrics for classification model. Credit to this positing. Let\u0026#8217;s go start with some classification result, more famously known as confusion matrix:\nclassified as negativeclassified as positiveactually negativeTN=9000FP=7009700 are actually negative (TN+FP)actually positiveFN=200TP=100300 are actually positive (FN+TP)9100 classified correctly (TN+TP)900 classified incorrectly (FN+FP)Suppose threshold=0.5 Note that what a classification model predicts is the probability for each possible class. In the case of binary classification model, we can set a threshold (e.g. 0.5), such that predictions greater than 0.5 indicates positive, otherwise negative. So for each classification result, a changing threshold would change each value in the quadrant. Accuracy: The ratio of correct predictions (true) to the total number of predictions.Indicates out of all the predictions, how much are identified correctly by the modelThis metric is intuitive but not very useful (e.g. 3% of population is diabetic, then a model that always predicts false would be 97% accurate\u0026#8230;) so data scientists use other metrics like precision and recall to assess classification model performance 2. Precision: The fraction of positive cases correctly identified.Indicates out of all the positive predictions, how much are actually true case.In other words, in your catch, what percent are actually a problem.This is much more useful than accuracy. Example: out of all the cases identified as diabetics, the rate of correct identifications. 3. Recall: The fraction of the cases classified as positive that are actually positiveIndicates out of all the positive cases, how much are identified by the modelAlso known as true positive rate (TPR); and is also much more useful than accuracy. Example: out of all the real diabetics cases, the rate of the ones correctly identified by model.In other words, what percent of the problem did the model catch. Going over the example data in the confusion matrix, Accuracy=0.91 , Precision=0.125, Recall=0.333 and now you see how useless accuracy is. The more uneven the class distribution is, the less useful accuracy is.\n4. F1 score:\nF1 score combines Recall and Precision to one performance metrics with weighted average. So it takes both false positives (the problems the model caught wrong) and false negatives (the problems the model failed to catch) into account.F1 is useful because you always have to use both Recall and Precision. 5. In addition, there is a metric called FPR (false positive rate) in compliment to TPR (recall):\nIt indicates what percent in the catch did the model get wrong. We learned that both recall and precision needs to be looked at when assessing a classification model. Unfortunately, Precision and Recall are often in tension: improving one typically reduces the other. We\u0026#8217;ve also learned that, those performance metrics are different as threshold changes.\n6. ROC curve: to summarize performance over all possible thresholds, we introduce the ROC curve. The name ROC (Receiver Operating Characteristics) stems historically from communications theory. The ROC curve is created by plotting the TRP against the FPR, at different thresholds. It indicates how well your classification model can separate positive and negative examples and to identify the best threshold for separating them.\nROC curve plotting for each T (threshold) value Below is the result of the ROC curve\n7. AUC (Area Under the Curve)\nThe model performance is determined by looking at the area under the ROC curve (aka AUC), which can range from 0 to 1. The larger the AUC, the better the model is performing. An excellent model has AUC near 1.0, indicating a great ability to separate positive from negative, as opposed to random guessing (coin flipping):\nThat\u0026#8217;s it for classification\u0026#8230;\nPerformance Metrics for Clustering Model Evaluating a clustering model is difficult by the fact that there are no previously known true values for the cluster assignments. A successful clustering model is defined as one that achieves a good level of separation between the items in each cluster, so we need metrics to help us measure that separation.\nA common clustering algorithm is K-Means Clustering. Below are some measurements:\nAverage Distance to Other Centre: indicates how close, on average, each point in the cluster is to the centroids of all other clusters.Average Distance to Cluster Centre: indicates how close, on average, each point in the cluster is to the centroid of the cluster.Number of Points: the number of points assigned to the cluster.Maximal Distance to Cluster Centre: the maximum of the distances between each point and the centroid of that point’s cluster. If this number is high, the cluster may be widely dispersed. This statistic in combination with the Average Distance to Cluster Center helps you determine the cluster’s spread. Azure-specific: Azure Machine Learning service provides cloud-based platform for creating, managing and publishing machine learning models. including:\nautomated machine learningAzure machine learning designer (no-code development environment)Data and computer managementPipelines: to orchestrate model training, deployment and manage tasks. Here\u0026#8217;s what pipelines typically look like:\nRegression pipeline classification pipeline Clustering model Machine Learning studio (ml.azure.com) provides a more focused UI for managing workspace resources. The following kinds of compute resources can be used to train models:\ncompute instances: development workstation that data scientist can use to work with data and modelscompute clusters: scalable cluster of VMs for on-demand processing of experiment codeinference clusters: deployment targets for predictive services that use your trained modelsattached computer: links to existing azure compute resources, such as VMs or data-bricks clusters Previous PostBlockchain and DeFi Next PostBasic Resource Object in Kubernetes 2 of 2 ","date":"2021-01-30T19:10:49-04:00","permalink":"/2021/01/a-shallow-dive-into-artificial-intelligence/","title":"A shallow dive into Artificial Intelligence"},{"content":"Background This is a reading note for book \u0026#8220;How to DeFi\u0026#8220;.\nWhen Tom hand in $20 to Jack in exchange of goods. Both Tom and Jack has to agree that the greenback with $20 sign actually is worth the value of the goods. Since the paper money is signed by some big shot from the central banker, which they both trust, they can therefore reach consensus on the value of that paper. The central bank acts as a centralized body of trust. Now you\u0026#8217;d ask what if the central banker cheats on us? As long as we centralize our trust to a single body, we have to worry about the centralized trust deteriorate. This is what decentralized finance aspires to address.\nThe other background problem is the centralized payment and clearance system. When you need to send money from one country to another, there are not only high fees involved, but also days holding for clearance. This is another opportunity for decentralized finance.\nDecentralized Finance (DeFi) The technologies in DeFi falls under three categories based on the level of decentralization:\nCentralized: custodial, uses centralized price feeds, centrally determined interest ratesSemi-Decentralized: non-custodial, decentralized price feedsCompletely Decentralized: every component is decentralized. Most DeFi dapps are sitting in the semi-decentralized category. There is no DeFi protocol that is completely decentralized yet. DeFi involves protocols that covers financial sectors such as Lending \u0026amp; Borrowing, Exchanges, Derivatives, Lottery, Payments, Insurance, etc. This all sounds futurism, but there are a few protocols already at play.\nEthereum The majority of the DeFi Dapps are built on the Ethereum blockchain, a global, open-source platform for decentralized applications. You can think of it as a world computer that cannot be shutdown. Developers can also deploy smart contracts to the Ethereum network, where it will run 24/7. Smart Contract is a programmable contract that allows two counterparties to set conditions of a transaction without needing to trust another third party for the execution. Whenever a certain condition is fulfilled, the smart contract will carry out the operation as programmed, and the process is transparent to all involved parties, bypassing the need for a trusted third party intermediary.\nEther is the native currency of the Ethereum blockchain so Ether is similar to Bitcoin. Ether is also used to pay for the fee that allows smart contracts and Dapps to run on the Ethereum network. Ether is also evolving to become its own unique reserve currency and store of value.\nDapps (decentralized applications) are interfaces that interact with the blockchain through the use of smart contracts. On Ethereum, all transactions and smart contract executions require a small fee to be paid. The fee is called Gas. In technical terms, Gas refers to the unit of measure on the amount of computational effort required to execute an operation or a smart contract. Gas fees are paied entirely in ETH. The price of gas can fluctuate from time to time depending on the network demand.\nEthereum can also be used for two other functions: creating DAO (Decentralized Autonomous Organization), or issuing other cryptocurrencies. A DAO is a fully autonomous organization which is not governed by a single person but is instead governed through code. This code is based on smart contracts and enables DAOs to replace how traditional organizations are typically run. As it runs on code, it would be protected from human intervention and will operate transparently. Governance decisions or rulings would be decided via DAO token voting. There are currently two popular protocols for tokens on the Ethereum Network: ERC-20 and ERC-721\nA wallet is a user-friendly interface to the blockchain network. It manages your private keys, which are basically keys to the lock on your cryptocurrencies\u0026#8217; vault. Wallets allow you to receive, store and send cryptocurrencies.\nCustodial wallets are wallets where third-parties keep and maintain control over your cryptocurrencies on your behalf. By using a custodial wallet, you trust an external party to store your coins safely. However, by trusting a third party with your cryptocurrencies, you open yourself up to the risk of the custodian losing your cryptocurrencies through mismanagement or hacks (Mt. Gox)\nNon-custodial wallets are wallets where you take full control and ownership of your cryptocurrencies. By using a non-custodial wallet, you trust no external party and only yourself to ensure safe storage. However, you pass the burden of security to yourself and you have to be fully equipped to store your private keys safely. If you lose your private keys, you will lose access to your cryptocurrencies too. Example of non-custodial wallet: Argent\nStablecoins Stablecoins are pegged to other stable assets such as the USD. The top 5 cryptocurrency stablecoins as of Feb 2020 are:\nTether (USDT)USD Coin (USDC)Paxos Standard (PAX)True USD(TUSD)Dai (DAI) Not all stablecoins are the same as they employ different mechanisms to keep their peg against USD. There are two types of pegs: fiat-collateralized (e.g. USDT)and crypto-collateralized (e.g. DAI). Most stablecoins are the former.\nUSDT pegs itself to $1 by maintaining reserves of $1 per Tether token minted. While Tether is the largest and most widely used USD stablecoin, Thether reserves are kept in financial institutions and users will have to trust Tether as an entity to actually have the reserved amounts that they claim. Tether is therefore a centralized, fiat-collateralized stablecoin.\nDai (DAI) on the other hand, is collateralized using cryptocurrencies such as Ethereum (ETH). Its value is pegged to $1 through protocols voted on by a decentralized autonomous organization and smart contracts. At any given time, the collateral to generate DAI can be easily validated by users. DAI is a decentralized, crypto-collateralized stablecoin.\nDAI has a smaller market capitalization but is increasing tremendously. DAI is the native stablecoin used most widely in the DeFi ecosystem. It is the preferred USD stablecoin used in DeFi trading, lending and more. DAI operates on Maker, a smart-contract platform that runs on the Ethereum blockchain and has three tokens: Sai \u0026#8211; aka Single Collateral Dai, backed only by Ether(ETH) as collateral. It is legacy Dai, and will be phased out.Dai \u0026#8211; aka multi-collateral Dai. Currently backed by Ether (ETH) and Basic Attention Token (BAT) as collaterals with plans to add other assetsMaker (MKR): is Maker\u0026#8217;s governance token and users can use it to vote for improvements on the Maker platform via the Maker Improvement Proposals. Maker is a type of DAO. MKR holders have voting rights proportional to the amount of MKR tokens they own in the DAO and can vote on parameters governing the Maker Protocol. The parameters that MKR holder vote on are vital in keeping the ecosystem healthy, which in turn helps ensure that Dai remains pegged to $1.\nThe amount of Dai that can be minted is dependent on the collateral ratio (150% worth of ETH or BAT to mint Dai). There is a stability fee and Dai Saving Rate (DSR).\nLending and Borrowing Compound Fiannce is an Ethereum-based open-source money market protocol where anyone can supply or borrow cryptocurrencies frictionlessly. Many tokens (BAT, ETH, USDC, DAI, and more) can be supplied or used as collateral on the Compound platform.\nCompound operates as a liquidity pool that is built on the Ethereum blockchain. Suppliers supply asset to the pool and earn interest, while borrowers take a loan from the pool and pay interest on their debt. In essense, Compound bridges the gaps between the lenders who wish to accrue interest from idle funds and borrowers who wish to borrow funds for productive or investment use. Suppliers and Borrowers interact directly with the protocol for interest rate.\nAnyone with a supported cryptocurrency wallet such as Argent can start using Compound immediately. To earn interest, you have to supply assets to the protocol. Once you have deposited your asset into Compound, you will immediately begin to earn interest on the assets you have put in. Upon deposit, you will receive corresponding amounts of cTokens. If you supply DAI, you will receive cDAI. If you supply Ether, you will receive cETH. Interest is not immediately distributed to you, but rather accrues on the cTokens which you now hold and are redeemable for the underlying asset and interest it represents. cTokens represents your balance in the protocol. cTokens become convertible into an increasing amount of the underlying asset it represents over time.\nIf you want to borrow, you have to first supply assets into the system as collateral for your loan. Borrowed assets are sent directly to your Ethereum wallet and from there you can use them. Do not that borrowing incurs a small fee of 0.025% to avoid spams and misuse of the Compound protocol\nDEX (Decentralized Exchanges) Uniswap Exchange is a decentralized token exchange protocol built on Ethereum that allows direct swapping of tokens without the need to use a centralized exchange. On Uniswap, you can simply swap your tokens directly from your wallet without having to go through centralized exchange.\ndYdX is a decentralized exchange protocol for lending, borrowing and margein/leveraged trading. It supports ETH, USDC, and DAI. You can enter either short or long positions with leverages up to 5x.\nOther Use Cases Derivatives: A derivative is a contract whose value is derived from another underlying asset such as stocks, commodities, currencies, indexes, bonds, or interest rates. There are several types of derivatives such as futures, options and swaps, each serving a different purpose. In DeFi, the biggest derivative protocol is Synthetix\nFund Management: In DeFi, fund management is conducted in a manner where it removes the investment manager and lets you choose the asset management strategy that best suits your financial need. TokenSets is a platform that allows crypto users to buy Strategy Enabled Tokens\nPayments: Lighting Network, Request Network, xDai and Sablier\nInsurance: Nexus Mutual is a decentralized insurance protocol built on Ethereum that currently offers cover on any smart contract on the Ethereum blockchain.\nDashboard: a dashboard is a simple platform that aggregates all your DeFi activities in one place.\nPrevious PostBasic Resource Object in Kubernetes 1 of 2 Next PostA shallow dive into Artificial Intelligence ","date":"2021-01-23T11:30:00-04:00","permalink":"/2021/01/blockchain-and-di-fi/","title":"Blockchain and DeFi"},{"content":"For someone from a system administration background, it would be amazing to discover that Kubernetes provides a solution to every pain point in the traditional software deployment landscape. On the contrary, it also brings about a lot of complexity due to the types of resource objects introduced. Pod A Pod is a shared execution environment for one or more containers. The containers running in a Pod share resources such as memory, volumes, network namespace (e.g. IP address, port range, hostname, routing table), UTS namespace (e.g. hostname) and IPC namespace (Unix domain sockets). Every Pod has its own IP address that is routable on the Pod network. All Pods connect to the same flat network called the Pod network.\nA pod most commonly only contains a single container, which is considered a good practice, unless there is good reasons to put two containers in a single pod (sharing resource). One such good reason is to co-schedule tightly-coupled workloads (such as logging, sharing volume, etc). Within the Pod, the containers communicate with each other via localhost interface of the Pod. In service mesh model, there is also a proxy container in each application Pod. The proxy container handles all network traffic entering and leaving the Pod. Also, within the Pod, to avoid competing for resources, individual containers can have their own cgroup limits, which actively police resource usage.\nPods are mortal (composable). They come and go (with dynamic IPs), so application should not store state in Pods. Deploying a Pod is an atomic (all or nothing) operation. When a Pod is scheduled to a node, it enters the pending state while the container runtime on the node downloads images and starts any containers. Once\u0026#8217;s everything is ready, the Pod enters the running state.\nWe typically deploy Pods via higher-level controllers such as Deployments (to offer scalability and rolling updates), DaemonSets (to run one instance of a service on every node in the cluster), StatefulSets (for stateful application components), and CronJobs (for short-lived tasks that need to run at set times just like a Linux cronjob).\nDeployments Deployment manages multiple replicas of the same Pod (via ReplicaSets). To follow best practice, you interact with Deployments instead of ReplicaSets, and use YAML file (declarative model). You can perform rolling update or rollback.\nReplicaSets ReplicaSets provide self-healing and scaling capabilities to Pods. If a Pod fails, it will be replaced. If load increases, then the ReplicaSets creates new Pod. This is all implemented with a background reconciliation loop that is constantly checking whether the right number of Pod replicas are present on the cluster. If not, Kubernetes declares a red-alert condition, orders the control plan to bring up more replicas. The best practice however, is that you should not manage ReplicaSets directly. Instead, you should perform all actions against the Deployment object and leave the Deployment to manage ReplicaSets.\nService Pods themselves are mortal (IP churn) so it\u0026#8217;s a bad idea to talk directly to individual Pods. Service object provides stable and reliable networking for a set of dynamic Pods. Service gets its own stable IP address, stable port and stable DNS name. It can also load-balance request across the Pods.\nServices are loosely coupled with Pods via labels and label selectors. You specify label selector for Service and labels on Pods when creating them. All the labels in label selector are used to select target Pods. Service acts as front-end, consisting of stable IP, DNS name and port, with Pods acting as backend, consisting of constantly changing Pods. Labels are simple yet extremely powerful. During blue-green update, you may use version label as a technique to control what backend pool is used behind Service object. For example, start with version=1, deploy version 2, remove version from label selector, and eventually add version=2 back to label selector, before phasing out the old Deployment.\nServices learn Pod status via Endpoint object, more details to follow.\nThere are several types of Service, the default being ClusterIP. A ClusterIP Service has a stable IP address and port that is only accessible from inside the cluster. The ClusterIP gets registered against the name of the Service on the cluster\u0026#8217;s internal DNS service (implemented via coreDNS with Control plane Pods). This means that the ClusterIP only works within the cluster, not outside. The other type of Service is called a NodePort, which is built on top of ClusterIP, but also enables access from outside of the cluster. The Service object has a reliable NodePort mapped to every node in the cluster. The NodePort value is the same on every cluster. Traffic from outside of the cluster can hit any node in the cluster on the NodePort and get through the the Pods.\nOther types of Services include LoadBalancer and ExternalName. LoadBalancer Services integrate with load-balancers from cloud provider. They build on top of NodePort Services and allow clients on the internet to reach your Pods via the load balancer of cloud vendor. ExternalName Services route traffic to systems outside of your K8s cluster.\nFor service discovery within the cluster, Kubelet program every container with the knowledge of the internal DNS (/etc/resolv.conf). The internal DNS service watches constantly the API server for new Services and automatically register them in the DNS. The other means of service discovery is through environment variables. However, in this method the Pods have no way of learning about new Services added to the cluster after the Pod itself is created.\nEndpoints Endpoints object is a dynamic list of all the healthy Pods on the cluster that match the Service\u0026#8217;s label selector. Each Service gets its own Endpoints objects for an up-to-date list of matching Pods. Kubernetes is constantly evaluating the Service\u0026#8217;s label selector against the currently list of healthy Pods on the cluster. Any new Pods that match the selector get added to the Endpoints object, and any Pods that disappear get removed.\nWhen sending traffic to Pods, via a Service, an application will query the cluster\u0026#8217;s internal DNS for the IP address of a Service, then sends the traffic to this stable IP address. Service then forwards it on to a Pod. Kubernetes-native application however, has the ability to query the Endpoints API directly, bypassing the DNS lookup and use of the Service\u0026#8217;s IP.\nIt requires a thorough understanding of Services, Endpoints and the service discovery mechanism to perform effective troubleshooting in Kubernetes.\nThe aforementioned internal DNS service (we usually call it the \u0026#8220;cluster DNS\u0026#8221;) is implemented in the kube-system Namespace as a set of Pods managed by a Deployment called coredns. These Pods are fronted by a Service called kube-dns. The cluster DNS is constantly looking for new Services and automatically register their details (metadata.name). We might need to check the logs for each of the coredns Pods during troubleshooting. The kubelet process on every node is watching the API Server for new Endpoints objects, when it sees them, it creates local networking rules that redirect ClusterIP traffic to Pod IPs, using IPVS technology on Linux to manage these rules.\nDaemonSet A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.\nSome typical uses of a DaemonSet are: cluster storage daemon on every node, logs collection daemon on every node, a node monitoring daemon on every node.\nHorizontal Pod Autoscaler The Horizontal Pod Autoscaler automatically scales the number of Pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can\u0026#8217;t be scaled, for example, DaemonSets.\nThe Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behaviour of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment to match the observed average CPU utilization to the target specified by user.\nThere are more details about HPA here and here.\nStatefulSets are designed for stateful application, which creates and saves valuable data. The three properties that form the state of a Pod are:\nPod names (\u0026lt;StatefulSetName\u0026gt;-\u0026lt;Integer\u0026gt;)DNS hostnamesvolume bindings They are sometimes referred to as the Pods sticky ID. StatefulSets ensures that these are all predictable and persistent. For example, failed Pods managed by a StatefulSet will be replaced by new Pods with the exact same Pod name, the exact same DNS hostname, and the exact same volumes, even if the replacement Pod is started on a different cluster Node.\nNote that StatefulSets create one Pod at a time, and always wait for previous Pods to be running and ready before creating the next. Scaling operations are also governed by the same ordered startup rules. This is different from Deployments that use a ReplicaSet controller to start all Pods at the same time, causing potential race conditions. The way StatefulSet controllers do their own self-healing and scaling is architecturally different to Deployments which use a separate ReplicaSet controller for these operations. The reason it is a game changer to know the order in which Pods will be scaled down, as well as that Pods will not be terminated in parallel, is because clustered apps that store data are usually at high risk of losing data if multiple replicas go down at the same time.\nDeleting a StatefulSet does not terminate Pods in order. So you may want to scale a StatefulSet to 0 replicas before deleting it. You might also set 10 seconds grace period before terminating to allow applications a chance to flush local buffers and safely commit any writes still in flight.\nIn Kubernetes, Volumes are decoupled from Pods via PersistentVolumes and PersistentVolumeClaims. So volumes have separate lifecycles to Pods and can survive Pod failures and termination operations. When a StatefulSet Pod is created, any volumes it needs are created at the same time and named in a way to connect them to the right Pod. Any time a StatefulSet Pod fails or is terminated, the associated volumes are unaffected. This allows replacement Pods to attach to the same storage as the Pods they\u0026#8217;re replacing, even if the replacement Pod is scheduled to a different cluster Node. Similarly, if a StatefulSet Pod is detected as part of a scale-down operation, subsequent scale-up operations will attach new Pods to the existing volumes that match their names.\nSince each StatefulSet Pod needs its own unique storage, hence its own PVC, this can be done by volumeClaimTemplate, which dynamically creates a PVC each time a new Pod replica is dynamically created. This eliminates the hassle to have to pre-create a unique PVC for every potential StatefulSet Pod.\nNamespaces Namespaces allows you to partition resource objects. For example, you may create a Namespace called prod and dev. Object names must be unique within Namespaces but not across Namespaces.\nPrevious PostAWS CDK example in Typescript – provision an AWX server Next PostBlockchain and DeFi ","date":"2021-01-16T22:13:00-04:00","permalink":"/2021/01/basic-kubernetes-resource-object-1-of-2/","title":"Basic Resource Object in Kubernetes 1 of 2"},{"content":"This post provides an example of using AWS CDK in Typescript.\nAnsible Tower and AWX We have used open-source Ansible extensively in the past. While the automation is convenient, the lack of UI makes it not as suitable as a team collaboration tool. One way to allow team collaboration with open-source Ansible, is to use Jenkins to glue the components together, as discussed in the Automated Deployment Pipeline series. In this setup, the open-source Ansible remains command-line driven, with Jenkins building up the command, rather than a human user. There are many upsides in this configuration, but it is not built specifically for Ansible. Ansible is agent-less, and can be run from any host. This sounds appealing and can work well in smaller server fleet. However, since it requires some configuration on the controlling host for Ansible to function properly, it become unnecessary to configure Ansible environment on every single host (e.g. production). A typically environment only has Ansible environment configured on the bastion host. This brings the need for a dedicated controller server to drive all Ansible tasks.\nAnsible Tower is Red Hat\u0026#8217;s commercial enhancement to the open source Ansible, providing web-based console, REST API and other services such as Role-based Access Control (RBAC). Managing Ansible via REST API is still somewhat involving but this also enables other open-source contributions to simplify the use of API. For example Tower CLI allows you to use Ansible Tower with simplified command. Ansible Tower has an open-source upstream project called AWX, maintained by Red Hat. AWX is essentially a preview release of Ansible Tower without commercial support. AWX can serve as an engine for all Ansible related task. AWX server is essentially an Ansible control server. AWX, or Ansible Tower, also brings several concepts on top of Ansible:\nJob template: defines how an Ansible playbook should be executed, including details such as machine credential, project, inventory, and playbook file. Job: the actual execution of job template Project: connects Ansible Tower to source control such as BitBucket. It is tied to a Git repository and a branch within that repository To deploy AWX on EC2 instances, there is a reference deployment by AWS. However, it is provided as CloudFormation template and appears to be outdated (from 2018). In our project (late 2020, named ansible tower lab, or dubbed as \u0026#8220;atlab\u0026#8221;), we provide the infrastructure in AWS CDK (written in typescript), to provision the AWX environment. The goal is that once the configuration is completed, you can run ansible ping against a target EC2 instance. The steps are as automated as possible. However, a number of key steps are purposefully left manual for learning purpose, such as the installation of AWX on EC2 instance.\nInfrastructure as Code In previous posting, I created infrastructure as code in AWS CDK with Python, so I decided to change to typescript in this project, with the assumption it is just a matter of syntax mapping. However, I underestimated the transition to a new language I never learned before. A fuzzy understanding of little details such as when to use let a=4 vs this.a=4, may produce elusive errors that takes hours to troubleshoot. I would therefore strongly recommend reading the basic syntax guide for typescript, before getting started. In typescript, this example project provides an implementation of configuring autoscaling groups, including cloud init, user data, etc on AWS. Other than the language, everything else is very similar to the project in this post, which was developed in Python. Also, note that the project directory structure varies slightly based on the language being used.\nIf you are absolutely new to AWS cdk, start with this app. It is beyond the scope of this post, to cover extensively the installation and environment configuration of AWS CDK.\nIn the provision process for Bastion host, the cloudformation init script pulls a specific version from AWX repository, then makes slight modification. User will need to install it manually. Note that AWX can be installed on three types of platforms:\nOpenShift Kubernetes Docker Compose All are documented in their README file. For simplicity in this project, the installation is on standalone docker compose. This is the default mode so there is no need to modify the inventory file.\nThe code repo The repository is version controlled here. To run the project, you need to have aws cli environment, then install the required packages including node js, and npm packages such as aws cdk. Once configured, validate with command:\ncdk ls This should display the stacks available. Use cdk deploy to deploy each stack.\nWhen BastionStack is deployed, dependent packages should be installed with user data and cloud init. You will just need to SSH on to the server to manually install AWX, as explained in the instruction, to manually install AWX:\nansible-playbook -i inventory install.yml Then you can browse to the server (at port 80 by default). Before the log-in page for the first time, the AWX will upgrade itself, with the following screen presented:\nNow log on with default credential (in README.md), you will have the UI for AWX:\nAWX Web Console From here, you can edit inventory by adding the host. Or use the helper script (~/awxcompose-helper.sh) from bastion host to create a new inventory (named Private Instance Inventory), and populate it with the hosts in the stack. The helper script does so by querying aws resource, and isssue rest API calls to AWX. After executing the script, you can see a new inventory, and the Private instance inventory should contain all hosts in the stack:\nAutomatically populated inventory You can then run Ansible ping against the host to validate connectivity. Note that during inventory creation, the ansible_user is already set to ec2-user (by the helper script):\nPing result Some technical details The initialization process on Bastion host creates an RSA key pair, stores the public key to AWS, for the upcoming private instances to uses. It keeps the private key locally in order to make outgoing SSH connection to the private instances. To ensure connectivity between AWX and private instances, there are a couple of (bash) helper scripts involved. Both reflects some technical details that I had to work through.\nawxcompose-helper.sh: the initialization process pulls AWX installation file from git repo. The installation process will build a docker-compose file in ~/.awx/awxcompose, based on a template (~/awx-*/installer/roles/local_docker/templates/docker-compose.yml.j2). When user tells AWX to connect to private instance, the connection was made out of a docker container (instead of from the OS of bastion host), we need this script to map SSH key file from host to container, by modifying the template file. Without this helper, outgoing SSH connection will fail with error (Permission denied (publickey,gssapi-keyex,gssapi-with-mic)). This script is invoked in the cloud init process without requiring manual execution. awxinvt-helper.sh: once the private stack is up and the installation has completed, we need to add the hosts to AWX inventory. This script gets the instance ID and IP addresses of the private instances, and uses Rest API calls to create inventory and populate it with hosts. Ansible has multiple ways of authentication. This script uses the non-stateful basic authentication with each curl command requiring credential. Ansible Rest API guide is provided here and be wary of the convention where URI must end with a slash. This project is just a start of AWX on AWS CDK project using Typescript. In real life scenarios, there are some work to do to make this even more automated. For example, use cfn-hup service to monitor changes of private stack, and therefore update inventories accordingly. Previous PostHigh Performance Computing Next PostBasic Resource Object in Kubernetes 1 of 2 ","date":"2020-12-19T17:14:00-04:00","permalink":"/2020/12/ansible-tower-lab-environment-on-aws/","title":"AWS CDK example in Typescript – provision an AWX server"},{"content":"Overview High Performance Computing (HPC) has recently been commoditized with the advent of commodity server hardware (x86 server), virtualization technology and cloud delivery model. It is common in specialized industries where intensive computing tasks are required, for example:\nHCL (healthcare and life science): drug discovery, computer aided diagnosis (CAD), genome engineering; CAD, CAE, CAM (computer aided design, engineering, and manufacturing): 3D modeling, computational fluid dynamics (CFD), finite element analysis (FEA), structural mechanical design, etc Finance: portfolio management, automated trading, risk analysis Geoscience and geo-engineering: oil and gas exploration, geographic data, weather forecasting; Scientific computation Computing performance is measured in FLOPS (floating point operations per second) and is usually delivered in a cluster to aggregate the computing power from a number of networked nodes. This is referred to as an HPC cluster. Hardware stack An HPC cluster features the following components:\nHead node (aka master node or login node): a gateway and coordinator; head node may be broken into several nodes Compute node (worker node): the executor of jobs; the compute node can either be homogenous or heterogeneous, for different purposes. the number of compute nodes can be quite large There are four common form factors for server: tower, rack-mount, blade, mainframe. Traditionally, the nodes are rack-mount 1U \u0026#8220;pizza box\u0026#8221; servers. Bladed systems started to replace due to the increased node density, thanks to the shared/redundant power and cooling management. In the past, the HPC cluster is operated in data centres, which is an expensive operation item. In the last decades, many organizations extends their compute workload to the cloud, forming a hybrid model.\nHPC typically has specialized storage system because HPC applications notoriously create large amounts of data. NFS traditionally does not scale well as number of node increases. Some proprietary storage system such as Isilon provides good performance via NFS protocol. There are also open-source parallel file system such as Lustre and HDFS. HPC networking handles three types of traffic:\ncomputation traffic between compute nodes (if the compute nodes interact with each other) file system traffic: for compute nodes to read and write on file system (e.g. NFS) administrative traffic: fairly light compared to the two above For that, many HPC runs two networks, a private (backend) network and a public (frontend) network. Backend network must be high speed and low latency, typically in the form of 10Gig Ethernet, or InfiniBand.\nSoftware stack On the software layer, the core functionality is \u0026nbsp;Message Passing Interface (MPI), a specification for the developers and users of message passing libraries. MPI constitutes a standardized and portable message-passing system which consists of a library and a protocol to support parallel computing. MPI enables passing information between various nodes of a HPC cluster or between particular clusters, and has different implementations that provide the libraries to run HPC applications in a distributed manner across different physical nodes.\nIn the operation, user submits a job through head node in order to request the resource. User needs to specify the resources for the job (e.g. how many CPU cores, how much memory, etc). The head node runs a scheduler to allocate computing resource based on pre-defined policies, based on priority of jobs, availability of resources, distribution of load, etc. Depending on the nature of the computing jobs, the nodes participating in the task may or may not communicate with one another. If they do need to talk to each other, the program must support it. Such program can be called a cluster program, and the MPI (message passing interface) library greatly facilitates the development of such program. The sub-jobs communicating with each other also creates a considerable amount of network traffic within the cluster.\nCluster software ties all nodes in the cluster together. It turns raw hardware into a functioning cluster by provisioning (installing and configuring) the head nodes. Compute nodes can usually be added or removed dynamically therefore the head nodes should be able to provision compute nodes, and administer cluster, leaving the programming as the job for the user to complete. As mentioned, in parallel programming, the most important HPC tool is MPI (Message Passing Interface), which allows programs to talk to one another over cluster networks. There are both open (e.g. Open MPI) and commercial MPI (e.g. Microsoft MPI) versions. Cluster software should also provide compilers, debuggers, and profilers in addition to MPI.\nThere are cluster software in both Linux and Windows operating systems: Rocks Clusters, Oscar (Open Source Clusters Application Resources), Red Hat HPC solution, Microsoft HPC pack and AWS Parallel Cluster.\nImplementation Here is an example of setting up HPC cluster with CentOS. Despite of the well documented steps, note that the author of the document refers to HPC cluster simply as cluster, which is ambiguous. There are three basic motivators for creating a cluster: high performance computing (HPC), network traffic load balancing, and service resilience in the form of high availability (HA). The author should be specific in the document about the HPC cluster. If RDMA (Infiniband) network is involved, a configuration guide is provided in RedHat literature.\nHere is an example of deploying HPC cluster in AWS. Here is the guide to deploy HPC pack in Microsoft technologies.\nHPC and Big Data HPC and Big Data are two distinctive computing paradigmes. Although there is some signs of convergence and blurred boundaries, it is still a long way before one can treat HPC and Big Data interchangeably. This paper does a phenomenal job in comparing the two paradigms. The fundamental difference lies in the respective problems they intend to address. HPC focuses on the large computational loads, whereas Big Data targets applications that need to handle very large and complex data sets (usually in the order of multi-terabytes or exabytes). Many scientific data analytics applications are becoming I/O bound in modern systems, such as seismic algorithms, Big Data applications are thus very demanding in terms of storage, to accommodate such a masive amount of data, while HPC is usualy thought more in inters of sheer computational needs. The open-source projects in Big Data also aims to run on conventional hardware to make it easier and less expensive to scale. This is not the main focus of HPC.\nSo, you can run Big Data (e.g. Hadoop) analytics jobs on HPC gear. On the other hand, you can\u0026#8217;t run HPC jobs on commodity hardware as commonly seen in the Big Data stack. Both HPC and Hadoop analytics use parallel processing of data. In a Hadoop/analytics environment, data is stored on commodity hardware and distributed across multiple nodes of hardware. In HPC, where the size of data file is much greater, data storage in centralized. Also, because of the sheer volume of its files, HPC also requires more expensive networking communications such as Infiniband, because the size of the file it processes require high throughput and low latency.\nIn BigData job, each query in Hadoop reads data from disk and runs as a separate MapReduce job. Spark enables in-memory iterative processing (through the RDD abstraction), allowing the user to query repeatedly on a dataset without having to perform intermediate disk operations. RDD are exposed in the Spark API where each dataset is represented as a read-only object, and transformations are invoked using methods on these objects. For an example project, check out this post.\nThe underlying software stacks for HPC and Big Data are fundamentally different, mainly due to the differences represent in their target class of applications, as outlined in the diagram below:\nsoftware stack difference between HPC and Big Data As to which one is for me, the over-simplified advice is: if you can avoid HPC and just use Hadoop for your analytics, do it. It is cheaper, easier, and more cloud friendly. However, bear in mind that an all-Hadoop shop is not possible for many industries such as life sciences, weather, pharmaceutical, and academic applications.\nPrevious PostAWS CDK example in Python – provision Kubernetes Nodes Next PostAWS CDK example in Typescript – provision an AWX server ","date":"2020-12-11T23:42:00-04:00","permalink":"/2020/12/high-performance-computing-cluster/","title":"High Performance Computing"},{"content":"There are two mechanisms to initialize instances in AWS. Cloud init and CloudFormation Init. Both are widely used and we discuss each of them in this posting. Then we will give an example of using AWS CDK in Python.\nCloud-Init Cloud-Init is a service originally built for Ubuntu, as a bootstrapping utility to customize a Linux VM as it boots for the first time. It has evolved to be an industry standard multi-distribution method for cross-platform (public or private) cloud instance initialization, or even bare-metal installation. In cloud-init you can install packages and write files, or configure users and security. Because cloud-init is called during the initial boot process, there are no additional steps or required agents to apply your configuration. Cloud-Init uses UserData, which is part of instance metadata. With AWS, you can pass two types of user data to Amazon EC2: shell scripts and cloud-init directives.\nThe cloud-config files are text files encoded in base64, with more details covered in the documentation here. cloud-init also works across distributions. For example, you don\u0026#8217;t use apt-get install or yum install to install a package. Instead you can define a list of packages to install. cloud-init automatically uses the native package management tool for the distro you select.\nCloudFormation Init The cloudformation init mechanism does not only initialize instance, it also provides a mechanism for the resource being created to communicate with other resources. It allows an instance to emit signal to a different resource (via cfn-signal). It can also monitor changes to external resource and invoke local action (using cfn-hup with hooks). CloudFormation Init requires several components to work together:\nThe cloudformation resource should have metadata. The metadata must have a key AWS::CloudFormation::Init in which configsets are declared. The UserData must use helper script (cfn-init) to invoke configuration jobs The UserData can use helper script (cfn-signal) to signal with a CreationPolicy or WaitCondition (of the same or different resource), so you can synchronize other resources in the stack when the prerequisite resource or application is ready. The cfn-hup service on the instance can be configured, to check for updates to metadata and execute custom hooks when changes are detected. Comparison While there are overlaps between the functionalities of Cloud Init and CloudFormation Init, the major difference is the latter support extended features (signal, update, etc); whereas the former is vendor neutral. The table below summarized some the differences:\nCloud InitCloudFormation InitWorks onLinux OS distributionCloudFormation resource, in combination with cfn helper scriptsUsecaseInitialization onlyBoth initialization and resource updateTriggercloud-init systemd serviceInitial: from UserData\nUpdate: by cfn hookAdoptionMultiple cloud vendors and bare-metal systemAWS cloud instancesAction Playbook/var/lib/cloud/\nInstance Metadata -\u0026gt; User Data, encoded in base 64CloudFormation Resource -\u0026gt; Metadata section -\u0026gt; AWS::CloudFormation::Init -\u0026gt; configSets and configsLog file and stdout/var/log/cloud-init.log\n/var/log/cloud-init-output.log/var/log/cfn-init.log\n/var/log/cfn-init-cmd.logComparison between cloud-init and cfn-init AWS Cloud Development Toolkit (CDK) Traditionally, AWS CloudFormation uses template in YAML or JSON for resource declaration. As the size of system grows, the amount of resource involved grows quickly and the size of such declaration file may grow beyond manageable. Nested stacks and export of output are mechanisms designed to combat the template sprawling, but to a very limited extent. Two reasons it is hard to control template size are:\nIn declarative statements, each line carries very small piece of information. Without flow controls such as if-else, loops, object oriented structure, the level of code reusability is very low; Some auxiliary resources (such as AWS::EC2::VPCGatewayAttachment) must be declared explicitly, even though they are insignificant to the stack functionality To address these challenges, AWS introduced AWS CDK (cloud development tookkit), which supports multiple languages (JavaScript, TypeScript, Python, Java, and C#). The CDK was natively developed in TypeScript, which is supposed to be the preferred development language. A tutorial is provided here, with detailed API documentation here.\nTo install aws cdk and create a hello world project, follow this example.\nAn Example in Python I have created an example for AWS CDK in Python. The purpose is to create some EC2 instance to complete a lab for Kubernetes (without using managed EKS service). The example provisions the followings:\nVPC, a public and private subnets, Internet and NAT gateways; Relevant security groups and permissions Bastion host, public instances in public subnet Private instances in private subnet, with public route through NAT gateway The private instances forms a cluster for Kubernetes lab. We will use kubespray to initialize these instances. During the bootstraping, we download kubespray, install ansible, etc.\nHere is the code repo for this example. With CloudFormation only, the single template could go well beyond 1000 lines. With CDK, the code are organized into several different python files, each representing a stack. The stacks can be stood up with command:\ncdk deploy vpc-stack cdk deploy security-stack cdk deploy bastion-stack cdk deploy private-stack Although the documentation in Python is available, there are generally not a lot of examples built out on the Internet. The pypi site provides some Python specific examples for each module (e.g. core and aws-ec2). Given these libraries are available for only 2 years (since 2018), many advocates TypeScript as the language. However, I have implemented some CloudFormation init, used helper script, and UserData in this example, without running into any language specific issues.It should be noted that the EC2 instance by default will call cfn-init. So there is no need to explicitly run cfn-signal or cfn-init from user data in python code (example). This can be verified in file /var/lib/cloud/instances/\u0026lt;instance-id\u0026gt;/user-data.txt which automatically includes the following lines:\n# fingerprint: e1b32ead13878deb ( set +e /opt/aws/bin/cfn-init -v --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 -c config_set_1,config_set_2 /opt/aws/bin/cfn-signal -e $? --region us-east-1 --stack bastion-stack --resource bastionhost5F466975da9934ba490de456 cat /var/log/cfn-init.log \u0026gt;\u0026amp;2 ) In addition to Python, AWS CDK also supports other languages. In the next post, we will discuss use of CDK in Typescript.\nPrevious PostIPVS, iptables and kube-proxy Next PostHigh Performance Computing ","date":"2020-12-03T21:14:00-04:00","permalink":"/2020/12/instance-initialization-with-aws-cdk-in-python/","title":"AWS CDK example in Python – provision Kubernetes Nodes"},{"content":"This is an overview of the underlying technologies that drives load balancing. It covers LVS, Netfilter, iptables, IPVS and eventually kube-proxy.\nLVS (Linux Virtual Server) One of the ways to implement software load balancing is via LVS (Linux Virtual Server), as previously discussed. The diagram below shows the LVS framework, with IPVS as the fundamental technology:\nThe major work of the LVS project is to develop advanced IP load balancing software (IPVS), application-level load balancing software (KTCPVS), cluster management components. KTCPVS implements application-level load balancing inside the Linux kernel (still under development). IPVS is an advanced IP load balancing software implemented inside the Linux kernel. The IPVS code was already included into the standard Linux kernel 2.4 and 2.6.\nNetfilter Both IPVS and iptables (the technology behind Linux firewall, discussed here) are based on netfilter, a packet-filtering framework provided by the Linux kernel. In this section, we will discuss them all together, starting with Netfilter and then discuss how iptables and IPVS uses netfilter. Netfilter allows various networking-related operations to be implemented in the form of customized handlers, by offers various functions and operations for packet filtering, network address translation, and port translation, which provide the functionality required for directing packets through a network and prohibiting packets from reaching sensitive locations within a network. Netfilter represents a set of hooks inside the Linux kernel, allowing specific kernel modules to register callback functions with the kernel\u0026#8217;s networking stack. Those functions, usually applied to the traffic in the form of filtering and modification rules, are called for every packet that traverses the respective hook within the networking stack.\nIptables The kernel modules named ip_tables, ip6_tables, arp_tables (the underscore is part of the name), and ebtables comprise the legacy packet filtering portion of the Netfilter hook system. They provide a table-based system for defining firewall rules that can filter or transform packets. The tables can be administered through the user-space tools iptables, ip6tables, arptables, and ebtables. Notice that although both the kernel modules and userspace utilities have similar names, each of them is a different entity with different functionality.\nWhen a network packet is received on a network device, it first passes through the Prerouting hook. This is where the routing decision takes place. The kernel decides whether the packet is destined for a local process (e.g., a listening socket on a server in this system) or whether to forward it (system operates as a router). In the first case, the packet passes the Input hook and is then handed over to the local process. If the packet is destined to be forwarded, it traverses the Forward hook and then a final Postrouting hook before being sent out on a network device. For packets that are generated locally (e.g., by a client or server process that likes sending things out), they must first pass the Output hook and then the Postrouting hook before being sent out on a network device.\nThe aforementioned hooks \u0026nbsp;exist independently for the IPv4 and IPv6 protocols. Thus, IPv4 and IPv6 packets each traverse their own hooks. There are also other hooks for ARP packets and for Bridging. And all the \u0026nbsp;hooks exist independently within each network namespace. Additionally, there is an\u0026nbsp;ingress\u0026nbsp;hook for each network device. The list goes on… More explanations are from here and here.\nIPVS In LVS, IPVS is also based on netfilter framework, but works only on INPUT chain, by registering ip_vs_in hook function, to process request. IPVS (aka layer-4 switching) runs on a host at the front of a cluster of real servers. It directs requests for TCP/UDP based servers to the real server, while ensuring the resonse from (one or several) real server appears to the client as if they were all from a virtual service on a sigle IP address. It is based on in-kernel hash tables. The userspace utility is ipvsadm.\nWhen the client request reaches the kernel space of load balancer, it arrives at PREROUTING chain. Route will determine whether the request packet is for the local host or not, based on the destination address of the packet. The packet is sent to INPUT chain if it is. The ip_vs_in function is hooked to LOCAL_IN and will examine the packet. If it finds a matching IPVS rule, it will (bypass INPUT chain) directly trigger POSTROUTING chain, skipping iptables rules.vThis is discussed in detail here. IPVS supports 8 load balancing algorithms (round robin, weighted round robin, least-connection, weighted least connection, locality-based least-connection, locality-based least-connection with replication, destination-hashing, and source-hashing) and 3 packet-forwarding methods (NAT, tunneling and direct routing).\nThe main difference between iptables and IPVS, is iptables includes a number of tables, each with a number of chains, each further involves a number of rules. The total number of rules is large. The packet is assessed against many of such rules. For the same reason, the order of the rule matters. IPVS on the other hand, leverages hash table, with a complexity of O(1), or O(n) in the worst case scenarios. They vary significantly in the efficiency of packet filtering and forwarding, especially when the rules gets complicated. Iptable also presents more latency when adding or removing rules as more rules are involved. This presentation includes some quantitative comparison.\nKubeProxy In Kubernetes architecture, KubeProxy takes care of load balancing. Kube-proxy can run in three modes: userspace, iptables and IPVS. userspace proxy mode The userspace mode is old and inefficient. The packet is compared against iptables rule and then forwarded to a pod named kube-Proxy, which operates as an application to forward packet to backend pods.\niptables proxy mode The iptables mode is better since it uses the kernel feature of iptables, which is fairly mature. kube-proxy manages iptables rule based on the service yaml of Kubernetes.\nIPVS proxy mode With the comparison between iptables and IPVS earlier, we can expect that iptables operations slow down dramatically in large scale cluster. Therefore IPVS based kubeproxy was brought up. This presentation illustrated the differences.\nIn this post we discussed load balancing technologies from ipvs to iptables and then to kube-proxy, which is used in Kubernetes nodes.\nPrevious PostHow imaging devices talk to each other (in DICOM) Next PostAWS CDK example in Python – provision Kubernetes Nodes ","date":"2020-11-24T13:17:00-04:00","permalink":"/2020/11/ipvs-iptables-and-kube-proxy/","title":"IPVS, iptables and kube-proxy"},{"content":"Overview In the previous post I briefly touched on DICOM as the crucial standard in medical imaging for both data exchanging and data storage. It is important to understand that DICOM is such a massive standard that, beyond data exchanging and storage, has expanded into many different areas around imaging, that no device (or information system) can ever implement every single aspect of the standard. A device or information system complies to (and implements) a subset of the DICOM standard. The manufacturer must provide a document (DICOM conformance statement) to spec out which parts of the standard are implemented. Care providers (e.g. hospitals) are supposed to review these specs as part of the procurement process to ensure interoperability with existing information system.\nMedical Imaging Informatics When it comes to diagnosis, historical examinations provide baseline reference for radiologist. They are sometimes even more revealing than current imaging data acquired from a patient. Compared to the current exams, historical ones are referred to as priors. Priors are useful only if they are relevant to the current exams in terms of body part, modality, and exam procedure (the particular problem being studied). The effort to find out and pre-load relevant priors so they are ready to display along with current exams, is called \u0026#8220;prefetch\u0026#8221;. With huge demand in exchanging imaging data, \u0026#8220;prefetch\u0026#8221; has developed into its own sub-market in imaging informatics industry. Trust me, this is a difficult undertaking (and why I was in the industry). First, patient identities in each healthcare organization are usually different for lack of universal medical record number; patient\u0026#8217;s name can change (marriage, divorce, or just for fun); or it can be unavailable, if patient is simply not in a condition to provide identify (e.g. trauma). Second, even if you get \u0026#8220;who\u0026#8217;s who\u0026#8221; right, you\u0026#8217;d have to dig into all his history for useful information from several different systems. The definition of relevant prior can be different depending on the specific medical specialty. Then, the old data are typically stored in a slow part of the storage from their source system, yet the patient might be bleeding and dying on the table, waiting for prior retrieval like pulling teeth. Last, but not least, the priors being retrieve might be from a modality of previous generation from 1990s; good luck with current display application. Sorry that sounds a lot but in real life, there are even more challenges. In General, there are four categories of applications that need to support DICOM:\nAcquisition devices: Modalities need to store newly acquired studies persistently; Routing applications: usually by the name of some routers, gateways or bridges that receive imaging studies, decorate the metadata (because a lot of legacy devices can\u0026#8217;t do it), and send to one or multiple defined destination; Archives (e.g. PACS, VNA): They usually use dedicated database (metadata) and storage systems (pixel data). They are the repository of imaging data and must provide full support of transfer capability; Peripheral applications that uses DICOM data, such as 3D post-processing or DICOM testing (grassroot dicom, dcmtk) No matter what the devices are, they must follow certain protocols in order to communicate with each other. Data structure and encoding DICOM PS3.5 defines data types in VR (value representation). Each VR has its own purpose, allowed characters, and length limit. For example:\nVRDefinitionAllowed CharactersLength LimitSH\nShort StringA string of characters16 maximumLO\nLong StringA string of characters64 maximumAE\nApplication EntityA string of characters that identifies a DICOM application running on a compliant device16 maximumCS\nCode StringA string to represent code16 maximumPN\nPerson NamePerson\u0026#8217;s name, with caret (^) as delimiter64 maximumUI\nUnique IdentifierAn ID that uniquely identify an item, such as 1.2.840.100008.1.10-9 and period (.)64 maximumDA\nDateA string to represent date YYYYMMDD0-98US\nUnsigned ShortUnsigned binary integer, 16 bits long2SQ\nSequence of other itemsSequence of other itemsUN\nUnknownA string of bytes where the encoding of contents is unknownValue Representatives DICOM metadata is a dataset comprised of a set of data elements. Each element includes tag, (optional) VR, length of value, and the actual value, as shown below: DICOM data elements VR is optional because it can be implicitly determined based on DICOM data dictionary defined in PS3.6. The most common tags are:\nLevelTagMeaningVRPatient0010,0010Patient\u0026#8217;s NamePNPatient0010,0020Patient IDLOPatient0010,0021Issuer of Patient IDLOPatient0010,0024Issuer of Patient ID Qualifier SequenceSQPatient0010,0030Patient\u0026#8217;s Birth DateDAStudy0008,0020Study DateDAStudy0008,0050Accession NumberSHStudy0008,0061Modalities In StudyCSStudy0008,1030Study DescriptionLOStudy0020,000DStudy Instance UIDUIStudy0020,0010Study IDSHSeries0008,103ESeries DescriptionLOSeries0008,0015Body Part ExaminedCSSeries0008,0060ModalityCSSeries0020,0011Series NumberISSeries0020,000ESeries Instance UIDUISeries0020,0060LateralityCSSOP inst0008,0016SOP Class UIDUISOP inst0008,0018SOP Instance UIDUISOP inst0012,0010Transfer Syntax UIDUISOP inst0020,0013Instance NumberISCommon DICOM tags When VR is not explicitly spelled out, the data encoding is known as implicit VR. The opposite is explicit VR, where each data element spells out the VR type. When storing numeric value, such as tags or numeric values for tags, the predominant format stores lower byte before higher bytes, known as little endian. Rarely seen in DICOM is big endian, the opposite order of storing numeric values.\nThe tag include a group number (e.g. 0020) and element number (e.g. 0013). If group number is odd, it is a private tag not defined in PS3.6\nThe actual length of tag value shall always be even number. Odd-sized value should add an additional character (e.g. trailing space), known as even-length padding to meet this requirement.\nAbove is the basic rules for DICOM data structure and encoding. For more information about encoding, including transfer syntax, refer to this previous post.\nTransactions (DIMSEs and SOP classes) DICOM has its own information model of real world. It requires some clinical knowledge to come to full understanding. For technical people, we just need to understand the patient-study-series-image hierarchy.\nOne patient may have multiple studies Each study may include one or more image series Each series has one or more images Images are also referred to as SOP instance, a general terms that include not only images, but also reports, and other types of objects.\nA business transaction in DICOM is termed DIMSE, for example:\nNameGroupTypeDescriptionC-STOREDIMSE-COperationA stores an image to BC-MOVEDIMSE-COperationA tells B to store an image to CC-FINDDIMSE-COperationQuery for patient, study, series, imagesC-ECHODIMSE-COperationDICOM level \u0026#8220;ping\u0026#8221;N-EVENT-REPORTDIMSE-NNotificationReport an eventN-ACTIONDIMSE-NOperationDIMSE Some DIMSEs involves multiple SOP classes based on the type of image being processed. For example, SOP Class UID (10.2.840.10008.5.1.4.1.1.1) represents storage for CR image. A c-store transaction includes:\nRequestor (C-STORE SCU) sends a request (C-STORE-RQ), to store specified SOP class in certain transfer syntax The request is followed by the actual data (PDU) Once completed, the Response (C-STORE SCP) respond with C-STORE-RP Handshake (Association) In the previous C-STORE example, proper syntax that are mutually supported must be used in order for the requestor and receiver to process the transaction. Both parties learn each other\u0026#8217;s supported transfer syntax through an upfront handshake process known as DICOM association.\nIn DICOM association, the initiating party presents a list of supported pairs of SOP class and transfer syntax. Each pair is called a presentation context, and the responding party must respond to each presentation context in the proposed list, with either yes or no.\nThis negotiation is similar to the cipher negotiation in TLS handshake.\nFor more detailed information, please refer to Pianykh\u0026#8217;s book \u0026#8220;DICOM, a practical introduction and survival guide\u0026#8220;.\nPrevious PostAutomatic deployment of Orthanc on AWS Next PostIPVS, iptables and kube-proxy ","date":"2020-11-15T18:40:00-04:00","permalink":"/2020/11/how-imaging-devices-talk-to-each-other-tip-in-dicom/","title":"How imaging devices talk to each other (in DICOM)"},{"content":"[Update] I changed reverse proxy from Nginx to Envoy. Here\u0026#8216;s the detail.\n[Update] Some security improvement was introduced in may 2021. Here\u0026#8216;s detail.\n[Update] Here\u0026#8217;s the link to the orthweb repository.\nIn this project we introduce a medical imaging web service based on Orthanc, an open-source project of DICOM server, and a pipeline to deploy such server automatically and consistently. We deploy Orthanc on AWS automatically. This little project involves a number of technical deets in DevOps, to deliver a web application prototype with an automated deployment pipeline.\nA brief on imaging In medical imaging, scanning devices are the data collectors. It consists of various categories of scanners, such as Computed Tomography (CT), and Ultrasound (US). They are collectively referred to as modality, but vary significantly in terms of image generation and hardware manufacturing. The challenges to exchange data between these heterogeneous scanning devices and centralized computers came around as early as the 1980s, which brought about ACR-NEMA standard in 1985, under the initiative between American College Radiology (ACR) and National Electrical Manufacturers Association (NEMA). The standard lately evolved into DICOM (Digital Imaging Communication in Medicine), a comprehensive set of standard in the ISO framework that governs modern imaging data storage and exchange across several disciplines (radiology, cardiology, pathology, etc) that operate around images in medicine.\nIn addition to defining a file format to store imaging data, DICOM also includes an upper layer protocol that dictates how two compliant devices (referred as application entity, each identified by AE title) can negotiate a common syntax to transfer objects (e.g. an image, a report or a discovery). Upper layer refers to layer 5-7 in OSI model, or application layer in TCP/IP model.\nImaging server Once scanner acquires images from patient, they stores the exams to imaging server for persistent storage. The functionalities of such server expands overtime since 1990s and hence go by different names in different eras, such as PACS (Picture Archive and Communication Systems), VNA (Vendor Neutral Archive) and EI (enterprise imaging) archive. Regardless of naming, they can be generally seen as a highly specialized variation of enterprise content management system. They are usually hosted with a centralized database to index clinical information at patient, exam and image levels. The other key component is the persistent storage devices, usually in the form of a NAS. Orthanc is an open-source initiative for such imaging servers. It provides a DICOM endpoint, allowing scanning devices to store medical images. It also provides a web viewer allowing users to see the images stored. It is released for many platforms, including Docker images.\nInfrastructure as code We use Amazon Web Service (AWS) for infrastructure as service, and Terraform as the tool to provision resources off AWS, in a reliable and consistent mechanism, known as Infrastructure-as-Code. Terraform is an alternative to CloudFormation, AWS\u0026#8217;s proprietary infrastructure-as-code technology. Terraform is developed by Hashicorp as an open-source project, and therefore is vendor neutral. It supports multiple public cloud vendor through different providers. Each provider accesses the vendor specific SDK. For example, the AWS provider integrates with AWS SDK. As a result, the code used in one vendor cannot just be applied to a different vendor without a major overhaul. Terraform\u0026#8217;s current version is 0.13 as of Oct 2020, and has gone through some syntax changes since version 0.11. Terraform also produces files for state management locally in the working directory. When executing, Terraform combines all files in the working directory to assess variables, and create required resources. It is compatible with the most of AWS resources. For example, you can specify user data with templates when creating EC2 instances. You can also create managed service instance as long as it is supported by the provider.\nArchitecture Orthanc web server stores data in sqlite by default, but also has a plugin to support PostgreSQL, an open-source relational database. AWS has managed service (RDS) based on PostgreSQL. In this project, we create an RDS instance that span across two availability zones for minimum high availability. Orthanc also supports storing imaging data including pixels in PostgreSQL, which obviates the need for a dedicated file storage system.\nWe deploy the application in Docker\u0026#8217;s containers for compatibility and portability. The Orthanc server is shipped in Docker images, available in Docker hub registry. The docker environment is configured as part of EC2 instance bootstrapping, including installing packages with YUM, initializing and customizing environment variables. The docker-compose file, and the auxiliary configuration files are provided in the repo. The bootstrapping script installs git and pulls required files from this GitHub repo. This demo project does not include load balancing, DNS management, or container orchestration.\nSecurity Orthanc\u0026#8217;s web browser natively supports HTTPS. However, the DICOM port does not support TLS natively, as their development has made clear in the FAQ. This leaves a severe security vulnerability because all patient data (protected health information in HIPPA context) would be sent across the Internet in the clear, visible to every network interface along the route. To address this issue we brought in Nginx as a reverse proxy to work at TCP layer to terminate encrypted traffic for Orthanc\u0026#8217;s DICOM end point. DICOM upper layer works on top of TCP layer. InternetInternetEncrypted DICOM trafficEncrypted DICOM trafficDICOM ArchiveDICOM\u0026#8230;NginxNginxUnencrypted DICOM trafficUnencrypted DICOM trafficDICOM Device supporting TLSDICOM Device support\u0026#8230;Unencrypted DICOM trafficUnencrypted DICOM trafficEncrypted DICOM trafficEncrypted DICOM trafficcorporate firewallcorporate firewallcorporate firewallcorporate firewallViewer does not support full SVG 1.1\nIn Nginx literature, this use case is referred to as SSL Termination for TCP Upstream Servers. Note that Nginx is providing layer 4 capability in this use case so the certificate and key configuration should not be placed under http section of the configuration file. This layer 4 capability in fact enables security configurations of all protocol that operates in upper layers and can be used in a broad range of situations. It is also noteworthy that Nginx can re-encrypt the traffic on the way out to upstream, for even tighter security control measure as outlined in this use case.\nuser nginx; worker_processes 1; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } stream { upstream dicom_backend { server orthanc-backend:4242; } server { listen 11112 ssl; proxy_pass dicom_backend; ssl_certificate conf.d/site.pem; ssl_certificate_key conf.d/site.pem; ssl_protocols SSLv3 TLSv1 TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5:ECDH+AESGCM; ssl_session_cache shared:SSL:20m; ssl_session_timeout 4h; ssl_handshake_timeout 30s; } } It is also helpful to use Nginx to terminate HTTPS traffic, using a pair of certificate and key. When testing with self-signed certificate I realized that Chrome browser has specific requirement on self-signed certificate, or it won\u0026#8217;t load the page. So the certificate has to be created as instructed here.\nFor better security, it is advisable that the RDS instance is provisioned in private subnet, with its data encrypted both in-transit and at-rest. Docker service should also manage sensitive information as secrets.\nSummary The deliverable is stored in this Github repo. The docker part of it can be executed on MacBook with PostgreSQL. The entire hardware stack represented by terraform code, can be executed against AWS to create required resources. Checkout README for further instruction. To emulate a modality, one will need a TLS supported DICOM application entity, Horos is a great project on MacOS to serve this purpose, both as DICOM-compliant sender and a viewer. Alternatively, consider some command-line based DICOM toolkit such as dcmtk, or grassroot dicom.\nPrevious PostDocker storage Next PostHow imaging devices talk to each other (in DICOM) ","date":"2020-11-08T00:54:06-04:00","permalink":"/2020/11/medical-imaging-web-server-deployment-pipeline/","title":"Automatic deployment of Orthanc on AWS"},{"content":"Microservices are all about stateless and ephemeral workloads, and containers are great microservices. This may suggest that that Docker is all about ephemeral storage. In fact, Docker supports both non-persistent and persistent storage, such as database, kafka, etc. Non-persistent storage is automatically created, alongside the container and is tied to the lifecycle of the container. On Linux system, it is /var/lib/docker/ as part of container. This is referred to as local storage.\nDocker has a concept of volume, which is essentially a file or a directory. Volumes are for persistent data. they are de-coupled from containers and are not tied to the lifecycle of any container. Volume allows process in docker container to bypass the default uionFS, and stores file or directory on host machine. It also allows different containers to share data. You may mount a volume to a container. even if container is deleted, volume persists.\nBy default, Docker creates new volumes with the built-in local driver. Local volumes are only available to containers on the node they\u0026#8217;re created on. There are also third-party drivers as plugins that provides advanced options to integrate external storage system with Docker. (NAS, SAN, etc)\nThere are more than 25 volume plugins that you can specify with -d switch, to cover all three categories of storage\nBlock storage tends to be high performance and good for small-block random access workloads. File storage is high performance, shared amongs multiple containers with NFS or SMB protocols. Object storage is good for long term storage of large data blobs that do not change frequently. It is often content addressable and relatively low performance. Note that if you share volume with multiple containers, the application needs to worry about data collision.\nYou may use docker volume create command to create volume. Note that there is no quota management within docker so the partition needs to be managed at operating system level.\nImplementation of Volume Remember that Docker image is built on multi-layer file system. When we run a container, Docker places a read-write layer on top of the image, such that the active files in running container are all placed in this read-write layer. When container is deleted, so are the files. The file system in Docker is a pseudo file system implemented in unionFS. Volumes bypasses the uionFS and directly accesses the host file system. When we create a Docker volume, Docker places the volume data to /var/lib/docker/volumes and under each directory named after volume, creates a directory _data, which is attached to the corresponding container.\nYou can even mount an NFS volume to container. Reference here.\nWe mentioned UnionFS a couple times so far. UnionFS is a light-weight, layered file system. It can mount the contents of multiple directories to the same directory, to form a single file system. User can use unionFS like a directory. It is the foundation of Docker image and container and enables saving of spaces.\nThere are three common types of union FS: AUFS, DeviceMapper, and OverlayFS.\nAUFS file system AUFS is the earliest driver that Docker uses for file system, most common in Ubuntu and Debian. To check if the system support AUFS, check out the documentation here.\nAUFS is recommended in Ubuntu or Debian. For CentOS and Redhat, it needs to be installed and make sure the command above returns aufs. To configure AUFS, create file /etc/docker/daemon.json and add:\n{ \u0026#34;storage-driver\u0026#34;:\u0026#34;aufs\u0026#34; } Then restart docker service. Run \u0026#8220;docker info\u0026#8221; and examine the Storage Driver section, as documented here. AUFS layers multiple directories on a single Linux host and presents them as a single directory. These directories are called branches in AUFS terminology, and layers in Docker terminology. The unification process is referred to as a union mount.\nLayers of a Ubuntu container This section describes how the layers work and this section describes how it reads and writes files (Copy-on-Write (CoW) strategy to maximize storage efficiency and minimize overhead). CoW characterized AUFS.\nAUFS has not been adopted in the Linux kernel mainline for lack of maintainability. So for CentOS, the recommended file system driver is devicemapper.\nDevicemapper file system Devicemapper is a technical framework to map physical block device to virtual block device, introduced since kernel 2.6.9. So it\u0026#8217;s essentially different from AUFS. The Logical Volume Manager (LVM) in Linux is also implemented based on devicemapper.\nThe three critical components in devicemapper are:\nmapped device: a virtual device that devicemapper provides to client target device: the underlying physical device or a section of it. map table: keeps track of the offset, range, etc between mapped and target devices. Devicemapper uses target driver to block, filter, and forward I/O requests (e.g. Raid, encryption, think provisioning, etc). In thin provisioning, storage driver only assigns spaces that are needed. Docker uses snapshot technology in thin provisioning. This part of the documentation provides further details as to how device mapper works.\nUbuntu and busybox image layers Devicemapper has to modes:\nloop-lvm: in dev and test environment direct-lvm: recommended in production Here is the performance best practice. To configure devicemapper, create /etc/docker/daemon.json file and add:\n{ \u0026#34;storage-driver\u0026#34;:\u0026#34;devicemapper\u0026#34; \u0026#34;storage-opts\u0026#34;:[ \u0026#34;dm.directlvm_device=/dev/xdf\u0026#34;, \u0026#34;dm.thinp_percent=95\u0026#34;, \u0026#34;dm.thinp_metapercent=1\u0026#34;, \u0026#34;dm.thinp_autoextend_threshold=80\u0026#34;, \u0026#34;dm.thinp_autoextend_percent=20\u0026#34;, \u0026#34;dm.directlvm_device_force=false\u0026#34; ] } Then restart docker service. Run \u0026#8220;docker info\u0026#8221; and examine the Storage Driver section to ensure direct-lvm mode is on. Since devicemapper uses block device to store files, it is faster than directly operate on file system. It is adopted as default driver as unionFS for a long time, ensuring stable performance under Red Hat and CentOS.\nOverlayFS file system Earlier versions of OverlayFS (known as overlay driver) is not stable. Later version is known as overlay2, which is very stable and recommended in overlay2. It requires:\nDocker version higher than 17.06.02; Kernel version higher than 3.10.0-514 for CentOS and RHEL; or higher than 4.0 for other distributions of Linux; Using with xfs file system with d_type turned on In production environment, it is recommended to moutn /var/lib/docker to separate disk or partition, to prevent the directory getting full from impacting the host OS. The option pquota is recommended for mounting options in /etc/fstab.\nTo configure storage driver, create file /etc/docker/daemon.json, with the following content:\n{ \u0026#34;storage-driver\u0026#34;:\u0026#34;overlay2\u0026#34;, \u0026#34;storage-opts\u0026#34;:[ \u0026#34;overlay2.size=20G\u0026#34;, \u0026#34;overlay2.override_kernel_check=true\u0026#34; ] } Then restart docker service. Run \u0026#8220;docker info\u0026#8221; and examine the Storage Driver section to ensure storage driver is overlay2 and d_type is true.\nThe way overlay2 works is similar to AUFS, involving union mount process, with lowerdir, upperdir and merged. More details are here, including how overlay2 works with file read and file write (e.g. CopyOnWrite).\nToday, overlay2 driver is officially recommended by Docker for its stability and performance, it should be used if all the conditions are met.\nPrevious PostDocker components Next PostAutomatic deployment of Orthanc on AWS ","date":"2020-11-03T20:22:00-04:00","permalink":"/2020/11/docker-storage/","title":"Docker storage"},{"content":"The previous post about virtualization and containerization brought up some underlying technologies which Docker build containers on, including:\nnamespaces \u0026#8211; a Linux kernel mechanism to isolate resources. It allows a process to run within an isolated environment (mnt, pid, net, ipt, uts, user, cgroup) cgroups \u0026#8211; a Linux kernel mechanism to limit resource usage of a process or process group unionFS (this will be further discussed under Docker storage) In this post we further discuss the components in Docker, the dominant and popular player in container technology, as shown in the diagram below:\nThe component names can be seen under docker install directory. It consists of three groups:\nDocker related: docker, dockerd, docker-init and docker-proxy Containerd related: containerd, containerd-shim and ctr Container runtime: runc Now we discuss each group:\nDocker-related components docker is just an implementation of docker client, it supports commands to achieve all functions between client and server. Alternatively, user may use REST API, or Docker SDK to communicate with Docker server.\ndockerd is the server process, to receive requests from docker (client), SDK library or REST API caller. It executes the request and returns status to client. There are three ways for docker (client) to communicate with dockerd.\nBy Unix Socket (unix://socket_path). The default socket path used by dockerd is /var/run/docker.sock, which is why only root can use docker after installation. TCP request (tcp://host:port). It is recommended to configure TLS communication in production environment. By file descriptor (fd://) used in systemd service. Unix socket is the default communication method. To allow remote access to dockerd, use -H to specify HOST and PORT when starting dockerd.\ndocker-init is used by Docker as PID 1 process for containers, in case it needs to recycle zombie containers. To use this, specify \u0026#8211;init when running container.\ndocker-proxy is used for port mapping. When you use -p switch with docker run, this docker-proxy is the service that maps the container port to host port. It does so by modifying the iptables nat rule.\nContainerd related components containerd component was separated from dockerd since Docker 1.11, in compliance with OCI standard. It is responsible for life cycle management of containers, it also manages images (e.g. pulling from repo), request from dockerd to call runc, storage and network resources.\ndockerd uses UNIX socket to send request to containerd. The default socket path for containerd is /run/containerd/containerd.sock. containerd execute the task and return status to dockerd. You may also directly use containerd to manage containers.\nctr (containderd-ctr) is the client of containerd, mostly used only in development and testing. If the environment does not have dockerd, then you can use ctr as client, to send request directly to containerd.\ncontainerd-shim is used to decouple containerd from the containers. containerd-shim is the parent process of containers. This is so that restarting containerd does not impact the running containers.\nContainer runtime runc is a standard implementation of OCI container runtime. It is a command-line tool to create and run containers.\nPrevious PostHost legacy application in Docker 2 of 2 Next PostDocker storage ","date":"2020-10-28T20:23:00-04:00","permalink":"/2020/10/docker-under-the-hood/","title":"Docker components"},{"content":"My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months\u0026#8230;\nUse Case I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:\nApplication serviceTCP port to bindBusiness service A8030Business service B8040Business service C8050\u0026#8230;\u0026#8230;\u0026#8230;TCP port requirement The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:\nEach instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances. The OS needs to host 6 processes of the same application all running at the same time; The OS does not allow multiple processes to bind to a single TCP port (duh!); It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward. From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.\nOSContainer IDApplication Servicecontainer portpublished port Host\nCentOSContainer 1\n(Instance #1)Business Service A80309301 Business Service B80409401 Business Service C80509501 Container 2\n(Instance #2)Business Service A80309302 Business Service B80409402 Business Service C80509502 Container 3\n(Instance #3)Business Service A80309601 Business Service B80409602 Business Service C80509603 This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker\u0026#8217;s ability to map ports for publishing.\nBelow is an example of the docker compose file:\nversion: \u0026#39;3.6\u0026#39; services: dapp1: image: docker.digihunch.com/dapp:${DAPP_VER} container_name: dapp1 entrypoint: [\u0026#34;/opt/docker-entrypoint.sh\u0026#34;,\u0026#34;dapp\u0026#34;] ports: - 9301:8030 # BUSINESS SERVICE A - 9401:8040 # BUSINESS SERVICE B - 9501:8050 # BUSINESS SERVICE C mac_address: 2c:1f:4e:c5:9e:cf environment: - INSTANCE_TAG=dapp1 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M} networks: - vcnet volumes: - /opt/dapp/etc:/opt/dapp/etc:ro - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro deploy: resources: limits: cpus: \u0026#39;0.5\u0026#39; memory: ${DAPP_MEM:-4096M} reservations: memory: ${DAPP_MEM:-4096M} tty: true dapp2: image: docker.digihunch.com/dapp:${DAPP_VER} container_name: dapp2 entrypoint: [\u0026#34;/opt/docker-entrypoint.sh\u0026#34;,\u0026#34;dapp\u0026#34;] ports: - 9302:8030 # BUSINESS SERVICE A - 9402:8040 # BUSINESS SERVICE B - 9502:8050 # BUSINESS SERVICE C mac_address: 2c:1f:4e:c5:9e:d0 environment: - INSTANCE_TAG=dapp2 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M} networks: - vcnet volumes: - /opt/dapp/etc:/opt/dapp/etc:ro - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro deploy: resources: limits: cpus: \u0026#39;0.5\u0026#39; memory: ${DAPP_MEM:-4096M} reservations: memory: ${DAPP_MEM:-4096M} tty: true dapp3: image: docker.digihunch.com/dapp:${DAPP_VER} container_name: dapp3 entrypoint: [\u0026#34;/opt/docker-entrypoint.sh\u0026#34;,\u0026#34;dapp\u0026#34;] ports: - 9601:8030 # BUSINESS SERVICE A - 9602:8040 # BUSINESS SERVICE B - 9603:8050 # BUSINESS SERVICE C mac_address: 2c:1f:4e:c5:9e:d1 environment: - INSTANCE_TAG=dapp3 - MAX_JVM_HEAP=${DAPP_HEAP:-3892M} networks: - vcnet volumes: - /opt/dapp/etc:/opt/dapp/etc:ro - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro deploy: resources: limits: cpus: \u0026#39;0.5\u0026#39; memory: ${DAPP_MEM:-4096M} reservations: memory: ${DAPP_MEM:-4096M} tty: true networks: vcnet: driver: bridge driver_opts: com.docker.network.enable_ipv6: \u0026#34;false\u0026#34; In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}). Helper scripts The docker commands are fairly long so I had to organize them into several helper scripts. For example:\ndocker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for: Initialization work that cannot be done in Dockerfile, such as setting environment variable Launch the application, including pointing log file to stdout Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux build_image.sh: this script makes the image build process smoother check if image to build already exist, and ask permission to delete the existing image if so; build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post. stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered. Permission The container uses a non-root user to run application (e.g. with su dhunch -c \u0026#8220;command\u0026#8221; from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it\u0026#8217;s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.\nIn the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission \u0026#8211; owner alignment. For example, we have the following mount statement under volumes:\n\u0026#8211; /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive\nWe also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:\nDockerfile instruction for containerPermission issue during mount by docker-composeBad configRUN mkdir -p /var/lib/dapp \u0026amp;\u0026amp; chown -R dhunch:dhunch /var/lib/dapp\nThe directory \u0026#8220;dcontainer\u0026#8221; was not created until mount time and it is created implicitly with root as owner (since there is no user section in docker-compose, so root as default is used). The application running as dhunch user in container will have permission issue going into dcontainer directory after mount.Good configRUN mkdir -p /var/lib/dapp/dcontainer/archive \u0026amp;\u0026amp; chown -R dhunch:dhunch /var/lib/dappThe directory \u0026#8220;dcontainer\u0026#8221; was already created with proper permission prior to mount and the main application process running as dhunch user will not have permission issue. For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:\nsu dhunch -c \u0026#34;ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log\u0026#34; However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it\u0026#8217;s needed on every container):\nusermod -a -G tty dhunch For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver. Java application For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.\nIt is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container\u0026#8217;s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.\nPrevious PostAutomated Deployment Pipeline 3 of 3 Next PostDocker components ","date":"2020-10-22T17:54:00-04:00","permalink":"/2020/10/host-legacy-application-in-docker-2-of-2/","title":"Host legacy application in Docker 2 of 2"},{"content":"Background We have previously covered a pipeline example with Jenkins calling Ansible to leverage OpenSSH configuration and Ansible inventory. We also discussed a use case with declarative pipeline.\nIn this posting, I will provide another advanced example, built on declarative pipeline. The pipeline file will be pulled from Git repository. Also, the script is executed on a remote agent, instead of the Jenkins master server. The reason this example is important, is that:\nJenkinsfile is now version controlled (pipeline as code); Service script (e.g. python) is also version controlled from a central repository; Computing resource is provided by a remote agent. Since the script is pulled before running, the agent is still fungible; The result from service script execution is archived (similar to the way build artifact is stored in Jenkins) in master. The architecture of this pipeline now becomes the followings:\nVendorVendorCustomer BCustomer BCustomer ACustomer ASSH ProxySSH ProxyJenkins (Master)\nAnsible\nOpenSSHJenkins (Master\u0026#8230;VPN\nGatewayVPN\u0026#8230;VPN\nGatewayVPN\u0026#8230;VPN\nGatewayVPN\u0026#8230;Internet\nInternet\u0026#8230;agent.jar\nAnsible\nOpenSSH\nGit\nPythonagent.jar\u0026#8230;agent.jar\nAnsible\nOpenSSH\nGit\nPythonagent.jar\u0026#8230;Customer InventoryCustomer InventoryCustomer InventoryCustomer InventoryGit Repository\nJenkisfile\nscriptsGit\u0026#8230;Viewer does not support full SVG 1.1\nConfigure Agent Jenkins has a plugin called SSH Build Agent, that allows you to configure a Linux agent, communicating with Jenkins master in SSH. For Window agent, it uses JNLP to communicate with master, which is outside of our scope of discussion. As I touched on in previous post, Jenkins uses its own implementation of SSH protocol to achieve this. This means that it cannot re-use the configurations in ~/.ssh/config and thus the ability to do SSH chaining is eliminated. This is incompatible with our architecture so I have to register Jenkins agent using a different launch method \u0026#8220;Launch agent via execution of command on the master\u0026#8221;. The execution of command on master can still leverage OpenSSH config file. In order to do so, we must copy the agent.jar file to the remote agent first (URL is ${JENKINS_URL}/jnlpJars/agent.jar). Then use SSH command to call the jar file from agent (aka slave) machine. You may add some java argument for troubleshooting. Below is an example configuration for the node.\nI shall also note that if the remote agent is a different operating system where the path of bash might be different, then you need to include the directory of bash executable in PATH environment variable. That can be done as in the screenshot above (PATH=${PATH:/usr/bin}). If this is incorrect, you might run into issues when running sh step in Jenkins pipeline. Here is an article about this. To translate that page, the symptom of this issue includes:\nError from pipeline execution that says the following, which is very generic: process apparently never started in /home/dhunch/jenkins/workspace/site-remote-job@tmp/durable-b997d26c (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) Job status shows: hudson.AbortException: script returned exit code -2 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:659) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:605) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:549) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Threaddump (only available during execution) shows: Thread #6 at DSL.sh(awaiting process completion in /home/dhunch/jenkins/workspace/site-remote-job/durable-b997d26c; recurrence period: 9543ms; check task scheduled; cancelled? false done? false) at WorkflowScript.run(WorkflowScript:9) at DSL.script(Native Method) This issue is tricky because none of the symptoms above make you think about the environment variable defined for the agent, and that the issue does not occur until you actually execute a Jenkins pipeline, usually well after the node agent is registered, and only impacts shell step (sh). It is recommended to test this with sh steps in pipeline.\nConfigure Repository Our example job executes service script pulled from SCM, on a remote agent. Then the master pulls the result file to itself. The git repository can be set in Jenkins job, where you specify that the script named Jenkinsfile (from the repo) is the pipeline file that needs to be executed.\nThis is fairly simple, but what makes the situation more complex is the following few requirements:\nit\u0026#8217;s the agent node that needs git (installed) and clone to repo; the master does not need (and should not attempt to) clone from repo; the master needs to do its job without having to pull SCM; We need to be able to specify whether each step needs to pull from SCM. The following pipeline syntax shows how this is done:\ndef agent_dir = \u0026#39;initial_value\u0026#39; pipeline { agent none options { skipDefaultCheckout(true) } stages { stage(\u0026#39;Execute Job\u0026#39;) { agent { label \u0026#39;remote-agent-customer1\u0026#39; } options { skipDefaultCheckout(false) } steps { echo \u0026#39;Executing job on node\u0026#39; sh \u0026#39;whoami \u0026amp;\u0026amp; pwd\u0026#39; script { agent_dir = sh(returnStdout: true, script: \u0026#39;echo -n ${WORKSPACE}\u0026#39;) // the variable should not include carriage return } sh \u0026#34;echo ${agent_dir}\u0026#34; } } stage(\u0026#39;Pull Result\u0026#39;) { agent { label \u0026#39;master\u0026#39; } options { skipDefaultCheckout(true) } // no need to pull scm to this agent steps { echo \u0026#39;Pulling job below\u0026#39; sh \u0026#34;echo ${agent_dir}\u0026#34; sh \u0026#34;scp dhunch@site1:\\\u0026#34;${agent_dir}\\\u0026#34;/result.csv ./\u0026#34; } } } } In this example pipeline file, we declare agent none for the pipeline, then an agent for each specific stage. we also specify the option skipDefaultCheckout as true at the step where pulling from SCM is not needed. This allows us to finish job with multiple agent, and only pull from SCM as needed. This snippet also exemplifies how to declare a variable, assign it from stdout from one agent, and persist the value across ensuing stages.\nStore Result The reason we run pipeline jobs on this remote agent is because it is sitting in customer network and has local direct access to data. So this is perfect for situation such as data analytical jobs, which access to database on local network and store result. We need to pull the result file back to agent and make it available on Jenkins. This is so similar to \u0026#8220;archive artifact\u0026#8221; task (commonly seen in CI process) that we can simply use its plugin to achieve what we need. Before archive artifact, we need to pull it to local (master), as shown in the example code above. After that, we need another stage to archive the result.\nstage(\u0026#39;Archive Result\u0026#39;) { agent { label \u0026#39;master\u0026#39; } options { skipDefaultCheckout(true) } // no need to pull scm to this agent steps { archiveArtifacts artifacts: \u0026#39;*.csv\u0026#39;,onlyIfSuccessful: true,fingerprint: true } } I believe there is plugins to compress artifacts as well.\nConclusion At the end I\u0026#8217;d like to reiterate my perception about Jenkins. It is a very generic and adaptive automation platform that originally evolved from use cases in build automation. Due to this original root, many components in Jenkins are named around Continuous Integration use cases, such as the \u0026#8220;build\u0026#8221; button, and the \u0026#8220;archiveArtifacts\u0026#8221; step. These misnomers underplays what Jenkins can potentially do in continuous deployment or other automation scenarios. It is important for automation engineers to understand Jenkins components and plugins, through their functionalities and not by the name, and therefore make creative use of Jenkins as automation engine in all scenarios. An alternative to this proposed pipeline would be Ansible Tower, a commercial project based on open-source Ansible, but with nice UI support. Ansible Tower is Ansible oriented, and it does not have everything that Jenkins can do. It should still be a decent alternative given the proposes pipeline uses Ansible a lot.\nPrevious PostAutomated Deployment Pipeline 2 of 3 Next PostHost legacy application in Docker 2 of 2 ","date":"2020-10-14T17:27:00-04:00","permalink":"/2020/10/automated-deployment-pipeline-3-of-3/","title":"Automated Deployment Pipeline 3 of 3"},{"content":"In this posting, we continue to discuss Jenkins\u0026#8217; ability to automate deployment routines. Jenkins supports freestyle project out of the box, as well as Pipeline with several plugins. Freestyle project allows user to specify multiple steps on UI. This does not scale well when your entire process involves many steps. As explained on Jenkins\u0026#8217; website:\nWhile standard Jenkins “freestyle” jobs support simple continuous integration by allowing you to define sequential tasks in an application lifecycle, they do not create a persistent record of execution, enable one script to address all the steps in a complex workflow, or confer the other advantages of pipelines.\nIn contrast to freestyle jobs, pipelines enable you to define the whole application lifecycle. Pipeline functionality helps Jenkins to support continuous delivery (CD). The Pipeline plugin was built with requirements for a flexible, extensible, and script-based CD workflow capability in mind.\nSo although freestyle projects are easy to set up, and can do technically everything that Jenkins pipeline can do, the major advantage of Jenkins Pipeline is the ability to manage multiple-step as code, and version control the pipeline-as-code. Here is some more information.\nWe will start with a freestyle project to understand Jenkins\u0026#8217; ability and then advance to building pipelines.\nFreestyle projects As mentioned, Jenkins calls a task a \u0026#8220;build\u0026#8221;, and the build can be triggered in a variety of ways:\nTime schedule (with or without parameter) Remotely via API On completion of other projects Poll SCM for changes Commit to SCM There are also several ways to execute a job:\nExecute Shell command, batch command or groovy script Invoke Ansible adhoc command, playbook or vault Conditional on specified boolean value, file existence, etc After the job one can specify post-build jobs, for example:\nstart other build projects notification of various means publishing result file In deployment, it is common task to execute a task over SSH, using SSH command or invoke Ansible command with Ansible plugin. The former fits simple command line tasks. For example:\nThe Ansible plugin is good for more steps and more complicated inventory hierarchies. This post includes an example of an inventory involving multiple layers. The other limitation with SSH command is lack of a straightforward configuration to escalate privilege and run remote command. On the other hand, Ansible addressed this with become method. Below is a screenshot In order to use Ansible command, you will also need to specify where the Ansible binary in Global Tool Configuration. You will also need to store vault credential in Jenkins credential store so it\u0026#8217;s not being prompted.\nJenkins Pipeline Jenkins pipeline allows one to describe actions in a pipeline in groovy Domain Specific Language (DSL). There are two styles of pipelines:\nDeclarative pipeline is identified by a block named \u0026#8216;pipeline\u0026#8217;, it is relatively new and supports the pipeline-as-code concept. It can be stored as Jenkinsfile in code repository or edited in Jenkins\u0026#8217; UI. Scripted pipeline is identified by a block named \u0026#8216;node\u0026#8217;, it is the conventional format and can only be edited in Jenkins\u0026#8217; UI. To fully understand the two styles of pipelines you also need to know declarative programming and imperative programming. Read this instruction for a better explanation. Here is another good one that focus on their differences.\nJenkins\u0026#8217; plugin provides UI components for user to input information (e.g. Invoke Ansible Command), now with Jenkins pipelines, plugins can provide step functions in order to help Jenkins user. This page from Jenkins lists the most common ones, many of which requires plugin installation. One example of step function is SSH Pipeline Steps. It allows one to issue SSH connection from Jenkinsfile. However, in our deployment scenarios, it has some limitations:\nUnable to match a group of host by specific pattern Cannot use host name aliases SSH tunneling is not supported No means of privilege escalation Out of these 1 through 3 are due to the fact that SSH step function does not use the OpenSSH configuration on the machine. The\u0026nbsp;jenkins-ssh-slaves plugin\u0026nbsp;uses\u0026nbsp;trilead SSH2 implementation\u0026nbsp;written in Java. Only OpenSSH implementation uses ~/.ssh/config file. This creates problem whenever SSH tunneling is needed. For example, to register a remote slave node via SSH tunneling, we need to select \u0026#8220;Launch agent via execution of command on the master\u0026#8221; instead of \u0026#8220;Launch agent via SSH\u0026#8221; as launch method. Here is an instruction and below is what it looks like:\nSimilarly, if we use ssh-agent plugin as tool for deployment, we cannot use any configuration made by OpenSSH. Therefore Ansible in Jenkins Pipeline is a better tool for deployment because it can use OpenSSH.\nPipelines with Ansible In this section I demonstrate the use of Ansible playbook and adhoc command in pipeline through two examples. In example 1, the job pulls authorized key file from SCM, and then use a playbook from SCM, to push the key file to all servers in the specified inventory. Here is the playbook file for example 1:\n--- # Example: # ansible-playbook -l all -i ~/ansible/inventories/bh.yml push-key.yml --ask-vault-pass - name: push key to target hosts: all tasks: - name: sync key to host copy: src: \u0026#34;{{authorized_keys_src}}\u0026#34; dest: \u0026#34;/home/dhunch/.ssh/authorized_keys\u0026#34; force: yes mode: 0600 Here is the pipeline script (note ampersand is mistakenly displayed as \u0026amp;amp; in the box below):\nimport java.net.URLEncoder; pipeline { agent any options { skipDefaultCheckout(true) } environment { BITBUCKET_CREDS = credentials(\u0026#39;bitbucket\u0026#39;) //BITBUCKET_CREDS_USR and BITBUCKET_CREDS_PSW are set BITBUCKET_CREDS_USR = \u0026#34;${BITBUCKET_CREDS_USR}\u0026#34; BITBUCKET_CREDS_PSW = URLEncoder.encode(\u0026#34;${BITBUCKET_CREDS_PSW}\u0026#34;, \u0026#34;UTF-8\u0026#34;) // if password contains special character we need to url encode it. e.g. @-\u0026gt;%40 } stages { stage(\u0026#39;Start\u0026#39;) { steps { echo \u0026#34;Starting pipeline ...\u0026#34; deleteDir() } } stage(\u0026#39;Prep\u0026#39;) { steps{ wrap([$class: \u0026#39;MaskPasswordsBuildWrapper\u0026#39;, varPasswordPairs: [[password: \u0026#34;${BITBUCKET_CREDS_PSW}\u0026#34;, var: \u0026#39;RANDOM\u0026#39;]]]) { // MaskPasswordsBuildWrapper requires Mask Passwords Plugin and is to mask specific string in console output. // Otherwise BITBUCKET_CREDS_PSW will display in the clear sh \u0026#34;git init \u0026amp;\u0026amp; git config core.sparsecheckout true\u0026#34; sh \u0026#34;git remote add origin https://$BITBUCKET_CREDS_USR:${BITBUCKET_CREDS_PSW}@bitbucket.org/vendorcompoany/configmanagerepo.git\u0026#34; sh \u0026#34;echo \u0026#39;public_keys/*\u0026#39; \u0026gt;\u0026gt; .git/info/sparse-checkout\u0026#34; echo \u0026#34;Downloading key file\u0026#34; sh \u0026#34;git pull --depth=1 origin master\u0026#34; } } } stage(\u0026#39;Deploy to Site 1\u0026#39;) { steps { echo \u0026#39;\u0026gt; Deploying to Site 1 ...\u0026#39; ansiblePlaybook ( installation: \u0026#39;Ansible on Mac\u0026#39;, playbook: \u0026#39;${WORKSPACE}/public_keys/push-key.yml\u0026#39;, inventory: \u0026#39;~/ansible/inventories/site1.yml\u0026#39;, vaultCredentialsId: \u0026#39;ansible-vault-pass\u0026#39;, extraVars: [authorized_keys_src: \u0026#34;$WORKSPACE/public_keys/authorized_keys\u0026#34;,] ) echo \u0026#39;\u0026gt; Deployed to Site 1 ...\u0026#39; } } stage(\u0026#39;Deploy to Site 2\u0026#39;) { steps { echo \u0026#39;\u0026gt; Deploying to Site 2 ...\u0026#39; ansiblePlaybook ( installation: \u0026#39;Ansible on Mac\u0026#39;, playbook: \u0026#39;${WORKSPACE}/public_keys/push-key.yml\u0026#39;, inventory: \u0026#39;~/ansible/inventories/site2.yml\u0026#39;, vaultCredentialsId: \u0026#39;ansible-vault-pass\u0026#39;, extraVars: [authorized_keys_src: \u0026#34;$WORKSPACE/public_keys/authorized_keys\u0026#34;,] ) echo \u0026#39;\u0026gt; Deployed to Site 2 ...\u0026#39; } } stage(\u0026#39;Deploy to Site 3\u0026#39;) { steps { echo \u0026#39;\u0026gt; Deploying to Site 3 ...\u0026#39; ansiblePlaybook ( installation: \u0026#39;Ansible on Mac\u0026#39;, playbook: \u0026#39;${WORKSPACE}/public_keys/push-key.yml\u0026#39;, inventory: \u0026#39;~/ansible/inventories/site3.yml\u0026#39;, vaultCredentialsId: \u0026#39;ansible-vault-pass\u0026#39;, extraVars: [authorized_keys_src: \u0026#34;$WORKSPACE/public_keys/authorized_keys\u0026#34;,] ) echo \u0026#39;\u0026gt; Deployed to Site 3 ...\u0026#39; } } } post { always { cleanWs() } } } In this example, we pull a sub-directory from git repo. We use two tricks to minimize amount of traffic. First, we use git pull with depth=1 so only the required recent commits are pulled, not the entire history. Second we use sparse checkout to get result from a sub-directory, not the entire repo.\nWhen connecting to repo, password is required. We pull the credential from Jenkins\u0026#8217; credential store and they are masked by default. However, the password must be converted to URL string when used in git remote add origin. Otherwise if the password contains special character the URL will not work. This modification to password brings about another challenge, password masking during the job execution. We will have to build our own wrapper function using class MaskPasswordsBuildWrapper in order to mask any variable.\nIn example 2, we wrap an Ansible step function to call adhoc command to check version on all hosts. Here is the pipeline script:\ndef HunchVersionQuery(pattern, siteinventory){ step([ $class: \u0026#39;AnsibleAdHocCommandBuilder\u0026#39;, ansibleName: \u0026#39;Ansible on Mac\u0026#39;, inventory: [$class: \u0026#39;InventoryPath\u0026#39;,path: siteinventory], hostPattern: pattern, module: \u0026#39;shell\u0026#39;, command: \u0026#39;cat /etc/*release\u0026#39;, forks: 1, vaultCredentialsId: \u0026#39;ansible-vault-pass\u0026#39; ]); } pipeline { agent any stages { stage(\u0026#39;Start\u0026#39;) { steps { echo \u0026#39;Staring Pipeline\u0026#39; } } stage(\u0026#39;Query Site1\u0026#39;) { steps { HunchVersionQuery(\u0026#39;*app\u0026#39;,\u0026#39;~/ansible/inventories/site1.yml\u0026#39;) } } stage(\u0026#39;Query Site2\u0026#39;) { steps { HunchVersionQuery(\u0026#39;*app\u0026#39;,\u0026#39;~/ansible/inventories/site2.yml\u0026#39;) } } stage(\u0026#39;Query Site3\u0026#39;) { steps { HunchVersionQuery(\u0026#39;*app\u0026#39;,\u0026#39;~/ansible/inventories/site3.yml\u0026#39;) } } } } Because Ansible plugin does not provide a warpper function for adhoc command step, we will have to build our own wrapper function in the above code.\nPrevious PostAutomated Deployment Pipeline 1 of 3 Next PostAutomated Deployment Pipeline 3 of 3 ","date":"2020-10-06T22:05:00-04:00","permalink":"/2020/10/automated-deployment-pipeline-2-of-2/","title":"Automated Deployment Pipeline 2 of 3"},{"content":"The business case You launched a software application. You installed it on two customer sites. You support the application mostly by SSH to customer server and run Bash commands, or slightly better, Bash scripts. The product is a hit to the market. You hired 20 support specialists in a customer service department. The dream client came through: an enterprise giving you a fleet of 100 servers to deploy your application on.\nMore staff, more business, more installations, more incidents, but the same old command driven steps. Problems:\nNon-standard support procedures. Every one takes notes and everyone\u0026#8217;s notes are slightly different.Information sharing among team members are ad hoc, and at high level.Post-mortem discussion is driven by memory and command fragments, instead of evidence end-to-end If that looks like your organization, chances are you also suffer from some secondary damages over the long term, such as:\nDowntime resolutions rely on the knowledgeable fewDocumentation helps. But it never catches up to the latest version of applicationLack of auditing of commands during support I propose an automation scheme to existing support and deployment practice. This automation scheme combines a suite of common technologies, such as Bash, Python, Ansible, OpenSSH and Jenkins. The automation allows the department to, either fully or partially, operationalize the steps in support and deployment, and eventually shift towards agile practice. Bash, Python and Ansible Bash script is based on shell command, perfect for running critical system tasks such as volume management. When it turns into a script, it can be cumbersome, especially with complex data structure. Python, as a tool for system administration, is a good complement to that.\nPython 2 comes with most Linux distributions, and is also a dependency of other built-in tools such as YUM. Python3 can be installed easily from default YUM repositories. Both Python2 and Python3 can exist on the same operating system, although new module development are now shifted to Python3. Python\u0026#8217;s syntax is very simple and offers object-oriented programming ability. Moreover, there is an entire open-source community behind Python, which offers modules in every aspect of IT (for example, Datastax has a driver module for connecting to Cassandra). Those modules are installed with PIP tool, or PIP3 for python3.\nBoth bash and Python executes on local machine. To run them on remote servers over SSH. You want to have a list of target hosts, and specify which one to execute the script against. This is where Ansible comes in handy. Ansible is superior in the following aspects:\nFree and open-source, with commercial alternative (Towers);Inventory management (inventory);Desire state engine (roles) Ansible is built on Python and is agent-less. Connectivity to remote host is done via secure shell so it can take advantage of existing SSH configurations. Job execution on the target machine is done through Python. With Python you can also develop custom module in Ansible. For some use cases in customer support with Ansible, refer to my two previous postings about Ansible at scale.\nJenkins The tools above forms a package for automation. The issue is that all of them are command-line based. Any task that requires Ansible requires the IT professional craft up long command, such as running playbook, executing a role, or ad-hoc command. This is inconvenient when a task needs to be done during an incident. Such tasks also require trained professional with the relevant skills.\nThese tasks can be stored in, or initiated by Jenkins. Although Jenkins is well known for build automation in continuous integration, it is automation engine for any command-line based IT tasks. The button to start such tasks in Jenkins UI is called \u0026#8220;Build\u0026#8221;, which is also a misnomer that underplays Jenkins\u0026#8217; versatility: building application from source code is just one of the many IT tasks that involves multiple long running commands. In this and next article we introduce Jenkins as an engine for deployment automation. VendorVendorCustomer BCustomer BCustomer ACustomer ASSH ProxySSH ProxyJenkins\nAnsible\nOpenSSHJenkins\u0026#8230;VPN\nGatewayVPN\u0026#8230;VPN\nGatewayVPN\u0026#8230;VPN\nGatewayVPN\u0026#8230;InternetInternetServer Fleet Managed by VendorServer Fleet Managed\u0026#8230;Server Fleet Managed by VendorServer Fleet Managed\u0026#8230;Viewer does not support full SVG 1.1\nThe infrastructure architecture is diagramed as above, and with the connection across Internet, the target hosts must be hardened properly in the following aspects:\nConnectivity to remote host is via SSH chaining, through an SSH proxy;Root login must be disabled for remote session or by password;Service user may be shared, but must be authenticated by individual RSA key pair;Service user connected remotely needs to escalate privilege by su if needed; I want to make a theoretical distinction between our topic here and continuous deployment. We simply focus on the technical side of deployment automation. Essentially automating a few bash scripts. On the other hand, a continuous deployment process is an extension to an existing continuous integration pipeline, with the vision to streamline the process end-to-end from code commit to production rollout. Implementing CI/CD pipelines should be approached as an organizational program rather than an individual technical initiative. Here is a good technical overview on CI/CD pipeline with Jenkins and Ansible.\nSecurity The security mechanism of this system is based on OpenSSH because the connectivity between servers are through SSH chaining. RSA key authentication must be used in order to encrypt traffic with password-less login. Connection to an SSH host can be done through a proxy server. Below is an example of SSH configuration:\nInclude customer1.config Include customer2.config Host * IdentityFile ~/.ssh/id_rsa ServerAliveInterval 60 ServerAliveCountMax 3 Compression yes ControlPersist 3h ControlPath ~/.ssh/sockets/%r@%h-%p Host gateway Hostname support.digihunch.com User jdoe Port 2223 Host customer-server-0 Hostname 192.168.201.12 User support ProxyCommand ssh -W %h:%p gateway Open SSH configuration file (~/.ssh/config) needs to be configured properly with useful host names and aliases. To prevent the config files from growing too long, include statement can be used to reference other configuration file (available with OpenSSH v7.3 sp1 and up). The host names (as well as aliases) listed in SSH configuration can be directly referenced in Ansible inventory, allowing Ansible (and Jenkins) to reference site by alias and connect to target host through proxy.\nPlugins Jenkins has a community that develops a variety of plugins, which makes Jenkins the most powerful automation platform. Here are some examples of useful plugins:\nAudit Trail: output job execution history to file or Elasticsearch;Credentials: stores credentials in Jenkins;Pipeline: build declarative (new) or scripted (old) pipeline for Jenkins jobs;Simple Theme: just a theme but allows console output to be dark (using CSS);Job Configuration History: job configuration audit;Mask password: mask variables (including password) from console output Ansible: invokes ad-hoc commands and playbooksSSH agent, SSH pipeline steps, SSH credentials: features related in SSH in Jenkins pipelines.Purge Job History: purge all of build history, or purge by time and number of old builds.Parameterized Scheduler: schedule to run a job and provide parameterWorkspace cleanup: clean up workspace when invoked. In the next article, we will go over some common job configurations.\nPrevious PostLog file navigator (lnav) Next PostAutomated Deployment Pipeline 2 of 3 ","date":"2020-09-30T22:04:00-04:00","permalink":"/2020/09/automated-deployment-pipeline-1-2/","title":"Automated Deployment Pipeline 1 of 3"},{"content":"I\u0026#8217;ve used a number of log viewers in command terminal, on MacOS and Linux server. I read system logs, log4j formats, as well as json formats. Unfortunately, I have not found an ideal (free) log viewer, either on UI or in command terminal.\nOut of those I tried lnav is one of the better ones. It\u0026#8217;s been around for more than 10 years and is configurable for a variety of formats. It is available in EPEL-repository for Linux and home brew for Mac.\nFor example, if our log (produced by log4j) looks like this:\n2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): start ARTIM 5000ms 2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): Client closed connecti on without sending data 2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550) enter state: Sta1 2020-10-06 20:59:25,471,INFO,org.dcm4che2.net.Association - [platform-dicomServer-44104-574917] Association(552550): close Socket[addr=/10.100.101.10,port=24976,localport=44104] 2020-10-06 20:59:25,471,DEBUG,org.dcm4che2.net.AssociationReaper - [platform-dicomServer-44104-574917] Stop check for idle Association(552550) We can introduce custom formatting, for example:\n{ \u0026#34;dapp\u0026#34; : { \u0026#34;title\u0026#34; : \u0026#34;dapp log4j format\u0026#34;, \u0026#34;description\u0026#34; : \u0026#34;dapp log4j format\u0026#34;, \u0026#34;regex\u0026#34; : { \u0026#34;dapp\u0026#34; : { \u0026#34;pattern\u0026#34;: \u0026#34;^(?\u0026lt;timestamp\u0026gt;\\\\d{4}-\\\\d{2}-\\\\d{2} \\\\d{2}:\\\\d{2}:\\\\d{2},\\\\d{3}),(?\u0026lt;level\u0026gt;\\\\w+),(?\u0026lt;component\u0026gt;[\\\\w-.]+) - \\\\[(?\u0026lt;thread\u0026gt;[^ ]+)\\\\] (?\u0026lt;body\u0026gt;.*)$\u0026#34; } }, \u0026#34;level-field\u0026#34; : \u0026#34;level\u0026#34;, \u0026#34;level\u0026#34; : { \u0026#34;error\u0026#34; : \u0026#34;ERROR\u0026#34;, \u0026#34;warning\u0026#34; : \u0026#34;WARN\u0026#34;, \u0026#34;info\u0026#34; : \u0026#34;INFO\u0026#34;, \u0026#34;debug\u0026#34; : \u0026#34;DEBUG\u0026#34; }, \u0026#34;value\u0026#34; : { \u0026#34;level\u0026#34; : { \u0026#34;kind\u0026#34; : \u0026#34;string\u0026#34;, \u0026#34;identifier\u0026#34; : true }, \u0026#34;component\u0026#34; : { \u0026#34;kind\u0026#34; : \u0026#34;string\u0026#34;, \u0026#34;identifier\u0026#34; : true }, \u0026#34;thread\u0026#34; : { \u0026#34;kind\u0026#34; : \u0026#34;string\u0026#34;, \u0026#34;identifier\u0026#34; : true }, \u0026#34;body\u0026#34; : { \u0026#34;kind\u0026#34; : \u0026#34;string\u0026#34; } }, \u0026#34;highlights\u0026#34; : { \u0026#34;DIMSE\u0026#34; : { \u0026#34;pattern\u0026#34; : \u0026#34;A-(ASSOCIATE-(RQ|AC)|RELEASE-(RQ|RP)|ABORT)|C-(STORE|MOVE|FIND|ECHO)-(RQ|RSP)\u0026#34;, \u0026#34;color\u0026#34; : \u0026#34;Red\u0026#34; } }, \u0026#34;sample\u0026#34; : [ { \u0026#34;line\u0026#34; : \u0026#34;2020-10-06 12:00:28,500,INFO,dicom.dicom-main - [main] Start listening on port 44104\u0026#34; } ] } } Save the content above as ~/.lnav/formats/installed/dapp.json, then load the log file with lnav, lnav will display the log by presenting columns in different colours.\nOne of the default behaviours is highlighting the IPv4 address, as shown above. This behaviour is however, not optional and currently cannot be turned off, which is reported here.\nPrevious PostSpark, Cassandra and Python Next PostAutomated Deployment Pipeline 1 of 3 ","date":"2020-09-23T21:03:00-04:00","permalink":"/2020/09/log-file-navigator-lnav/","title":"Log file navigator (lnav)"},{"content":"In this post we touch briefly on Apache Spark as a cluster computing framework that supports a number of drivers to pipe data in, and that its stunning performance thanks much to resilient distributed dataset (RDD) as its architectural foundation. In this hands-on guide, we expand on how to configure Spark, and use Python to connect to Cassandra data source. Spark supports Sala, Java and Python shells. I\u0026#8217;m not familiar with Scala but I have had Python background and know it\u0026#8217;s importance in big data processing. One key data structure with big data processing in Python is Pandas data frame. Spark has the ability to map its own data frame to Pandas data frame.\nSpark also needs a third party connector to connect to Cassandra. This connector is provided by Datastax in this open-source project called spark-cassandra-connector. The Github page includes a README with compatibility matrix, which is very important to understand before any configuration works. However, the Github is only the source code repository for anyone to build the project themselves. An alternative source of the dependency is this page from Maven repository. When running Spark we can simply reference that page URL as dependency.\nSuppose we install spark onto CentOS, we download and unzip this package to somewhere such as user directory (~). Assuming we already have Open JDK 1.8 installed, when we run spark binary, it places cache and jar files in ~/.ivy2, potentially we need to manually move the following dependencies to ~/.ivy2/jars:\norg.codehaus.groovy_groovy-json-2.5.7.jarcom.github.jnr_jffi-1.2.19.jarorg.codehaus.groovy_groovy-2.5.7.jar These jar files are available for download from Maven\u0026#8217;s repository as well if you wish provide them as package dependencies. We have two flavours of interactive shells to connect to Spark: the Scala shell (spark-shell) and python shell (PySpark)\nScala Shell\nWe can enter the default scala shell by $ ./bin/spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 --conf spark.cassandra.connection.host=10.10.10.151 --verbose During the start, note a stdout line that says:\nSpark context Web UI available at http://spark-host:4040 Then we can open that tcp port on iptables and view that job in browser. From within scala shell we can test connectivity to Cassandra with the following commands:\n\u0026gt;\u0026gt;\u0026gt; val new_exam = spark.read.format(\u0026#34;org.apache.spark.sql.cassandra\u0026#34;).options(Map(\u0026#34;table\u0026#34; -\u0026gt; \u0026#34;new_exam\u0026#34;,\u0026#34;keyspace\u0026#34; -\u0026gt; \u0026#34;examarchive\u0026#34;)).load() Python Shell\nPython Shell (aka PySpark) brings Python shell which is known to many engineers from system admin or development background. By default, python 2 will be used. To specify python version, set some environment variables before we start pyspark with cassandra connector package specified:\n$ export PYSPARK_PYTHON=python3 $ export PYSPARK_DRIVER_PYTHON=python3 $ export SPARK_HOME=/home/dhunch/spark-2.4.6-bin-hadoop2.7 $ export PATH=$SPARK_HOME/bin:$PATH $ ./bin/pyspark --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 --conf spark.cassandra.connection.host=10.10.10.151 Once you\u0026#8217;re in the interactive shell, you can start with loading required python libraries, and test your connectivity:\n\u0026gt;\u0026gt;\u0026gt; from pyspark import SparkContext, SparkConf \u0026gt;\u0026gt;\u0026gt; from pyspark.sql import SQLContext \u0026gt;\u0026gt;\u0026gt; load_options = { \u0026#34;table\u0026#34;: \u0026#34;new_exam\u0026#34;, \u0026#34;keyspace\u0026#34;: \u0026#34;examarchive\u0026#34;} \u0026gt;\u0026gt;\u0026gt; df=spark.read.format(\u0026#34;org.apache.spark.sql.cassandra\u0026#34;).options(**load_options).load() \u0026gt;\u0026gt;\u0026gt; df.show() \u0026gt;\u0026gt;\u0026gt; df.write.csv(\u0026#39;/tmp/mycsv.csv\u0026#39;) \u0026gt;\u0026gt;\u0026gt; #df.registerTempTable(\u0026#34;ne\u0026#34;) \u0026gt;\u0026gt;\u0026gt; df.createTempView(\u0026#34;ne\u0026#34;) \u0026gt;\u0026gt;\u0026gt; tw1=sqlContext.sql(\u0026#34;select count(*) from ne\u0026#34;) \u0026gt;\u0026gt;\u0026gt; tw1.show() \u0026gt;\u0026gt;\u0026gt; qrdf2=sqlContext.sql(\u0026#34;select study_key, image_count from ne where current_exam_version=exam_version\u0026#34;) \u0026gt;\u0026gt;\u0026gt; qrdf2.write.csv(\u0026#39;/tmp/tw2\u0026#39;) Note that the load method returns type pyspark.sql.dataframe.DataFrame, which is already a distributed data structure. So there is no need to parallelize it with parallelize() method. As of Spark 2.0, we are supposed to use createTempView() method instead of the old registerTempTables() method. Read this for further information.\nPython Application\nWith interactive shell you run one or several commands at a time. We can build a python script and submit the whole script as an application. This is an example command:\n./bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 sample.py Note that the sample.py script name must be provided after \u0026#8211;packages switch. Otherwise, you will get an error saying missing dependency (Failed to find data source: org.apache.spark.sql.cassandra). In the script, we can manipulate the data from Cassandra with greater flexibility. For example, we can map one field to several fields. For example, if one of the fields stores an XML document, the script can drill down the XML tree structure parse out values at different levels of child nodes, into separate data base columns. Here is an example of python script where we register a custom UDF declared in python and apply it to some existing columns to build new columns:\n#! /usr/bin/python3 # To submit this script as an application to spark: # ./bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 examstat.py # Note that the script name must be placed after --packages import sys,datetime,re import xml.etree.ElementTree as ET from pyspark import SparkContext, SparkConf from pyspark.sql import SQLContext, SparkSession from pyspark.sql.functions import udf from pyspark.sql.types import StringType,StructType,StructField cluster_seeds=[\u0026#39;dest_cass_host\u0026#39;] def pTrimExamCode(raw_code): return \u0026#39;NULL\u0026#39; if raw_code is None or raw_code==\u0026#39;None\u0026#39; else str(raw_code).replace(\u0026#39;,\u0026#39;,\u0026#39;\u0026#39;).rstrip(\u0026#39;\\r\\n\u0026#39;) def is_valid_date(date_str): isValidDate=bool(re.match(\u0026#34;^(19|20)\\d\\d(0[1-9]|1[012])(0[1-9]|[12][0-9]|3[01])$\u0026#34;,date_str)) if isValidDate: try: datetime.datetime(int(date_str[:4]),int(date_str[4:6]),int(date_str[6:8])) except ValueError: isValidDate=False return isValidDate def pPullTags(study_key,raw_xml_field): ns={\u0026#34;vc\u0026#34;:\u0026#34;http://medical.nema.org/mint\u0026#34;} StudyDateTag=\u0026#39;None\u0026#39; StudyDescriptionTag=\u0026#39;None\u0026#39; try: if raw_xml_field is not None: summary_tree=ET.fromstring(str(raw_xml_field)) # str function outputs \u0026#39;None\u0026#39; or null object xml_find_res=summary_tree.find(\u0026#34;vc:attributes/vc:attr[@tag=\u0026#39;00080020\u0026#39;]\u0026#34;,ns) if xml_find_res is not None: StudyDateTag=str(xml_find_res.attrib.get(\u0026#39;val\u0026#39;)) xml_find_res=summary_tree.find(\u0026#34;vc:attributes/vc:attr[@tag=\u0026#39;00081030\u0026#39;]\u0026#34;,ns) if xml_find_res is not None: StudyDescriptionTag=str(xml_find_res.attrib.get(\u0026#39;val\u0026#39;)) except: print(\u0026#34;-----------------------\u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt; examstat: error parsing metadta for study_key \u0026#34;+study_key) return (StudyDateTag,StudyDescriptionTag) # custom StructType for the output tuple XMLExtractType=StructType([ StructField(\u0026#34;StudyDate\u0026#34;,StringType(),False), StructField(\u0026#34;StudyDescription\u0026#34;,StringType(),False)]) if __name__ == \u0026#34;__main__\u0026#34;: sparkSession=SparkSession.builder \\ .appName(\u0026#39;examstat\u0026#39;) \\ .config(\u0026#39;spark.cassandra.connection.host\u0026#39;,\u0026#39;,\u0026#39;.join(cluster_seeds)) \\ .master(\u0026#39;local[*]\u0026#39;) \\ .getOrCreate() load_options = {\u0026#34;table\u0026#34;: \u0026#34;new_exam\u0026#34;, \u0026#34;keyspace\u0026#34;: \u0026#34;examarchive\u0026#34;} sqlContext=SQLContext(sparkSession) # pyspark.sql.dataframe.DataFrame is already a distributed data structure. No need to parallelize it. df0=sqlContext.read.format(\u0026#39;org.apache.spark.sql.cassandra\u0026#39;).options(**load_options).load() df0.createTempView(\u0026#34;new_exam\u0026#34;) # pyspark.sql.functions.udf(python function,output type) sparkSession.udf.register(\u0026#34;uTrimExamCode\u0026#34;,udf(pTrimExamCode,StringType())) sparkSession.udf.register(\u0026#34;uPullTags\u0026#34;,udf(pPullTags,XMLExtractType)) # use custom UDFs uTrimExamCode and uPullTags to calculate new columns and remove dups and deleted studies df1=sqlContext.sql(\u0026#34;select study_key as StudyKey,uTrimExamCode(exam_id) as ExamCode,image_count as ImgCnt,Total_pixel_data_size as PixelSize, uPullTags(study_key,metadata_summary) as XMLExtract, metadata_summary from new_exam where exam_version=current_exam_version and is_deleted=False\u0026#34;) df1.createTempView(\u0026#34;uniq_study\u0026#34;) # map the four fields in XMLExtract to separate columns. we take this as separate step as we don\u0026#39;t want uPullTags to execute multiple times in previous step df2=sqlContext.sql(\u0026#34;select StudyKey,ExamCode,ImgCnt,PixelSize,XMLExtract.StudyDate as StudyDate,XMLExtract.StudyDescription as StudyDescription from uniq_study\u0026#34;) df2.createTempView(\u0026#34;uniq_study_stat\u0026#34;) # Run analytical query df3=sqlContext.sql(\u0026#34;SELECT ExamCode, round(avg(PixelSize)/1024/1024) as avg_size_mb, round(sum(PixelSize)/1024/1024/1024,2) as total_size_gb,count(StudyKey) as study_count FROM uniq_study_stat GROUP BY ExamCode order by study_count desc\u0026#34;) #data frames are lazily loaded and processing not started until the following call df3.write.csv(\u0026#39;/tmp/examstat_\u0026#39;+datetime.datetime.now().strftime(\u0026#34;%m%d%H%M%S\u0026#34;)) It is important to understand the concept of lazy evaluation in Spark RDD here. The execution of function to RDD does not start until an action is triggered (eg. show method, or write method). Spark maintains the record of which operation is being called through DAG (directed acyclic graph). Such record is referred to as a transformation. We need to understand whether each RDD method is a transformation, or an action so we know whether it will be lazily evaluated (here\u0026#8217;s more information).\nThis is a major difference between Apache Spark and Hadoop MapReduce. With MapReduce, developer spend a lot of time in minimizing the number of MapReduce passes. It happens by clubbing the operations together. Previous PostIntro to Big Data Projects Next PostLog file navigator (lnav) ","date":"2020-09-15T16:24:09-04:00","permalink":"/2020/09/spark-cassandra-and-python/","title":"Spark, Cassandra and Python"},{"content":"Modern applications produce super large datasets beyond what traditional data-processing application can handle. Big data is a discipline that specialize in processing such data. For example, analysis, information extraction etc. The scale of large dataset grows well beyond the capacity of a single computer, which calls for computing power delivered by multi-node clustered systems. Intensive computing tasks are completed in a distributed system consisting multiple nodes each performing some tasks, known as High-Performance Computing Cluster (HPCC).\nCluster computing inherit the challenges of distributed system. Moreover, two main challenges to solve are: distributed storage, and distributed computation. In Apache Hadoop projects, HDFS and MapReduce address these two challenges respectively. Now the Hadoop ecosystem has evolved to include several core projects:\nHDFS A distributed file system for reliably storing huge amount of unstructured, semi-structured or structured data in the form of files. Parts of a single large file can be stored on different nodes across the cluster. HDFS works in master-slave mode:\nNameNode (master): holds file system namespace, controls access, keep track of DataNodes and replication factor DataNode (slave): stores user data HDFS is Java-based so is portable across all platforms. User interact with HDFS using a command-line interface called \u0026#8220;FS shell\u0026#8221;. There is also an interface called FUSE (filesystem in userspace) to mount HDFS to Linux OS. Since HDFS supports commodity hardware it is great for storing data for further processing. However, HDFS is not suitable for storing data related to applications requiring low latency access, nor is it good for simultaneous writes to the same file. Also HDFS is not suitable for large number of small files because the metadata for each file needs to be stored on the NameNode and is held in memory. Here is the architecture guide for HDFS, and this page expands further on the read and write operations in HDFS.\nCompared to NAS(e.g. NFS), HDFS is distributed by design. The data blocks are distributed across different nodes. NFS storage may or may not be distributed depending on the implementation. HDFS is designed to work with MapReduce paradigm, where computation is moved to the data. In NAS, data is stored separately from the computations. Lastly, NAS is usually made up of enterprise grade hard drive but HDFS works with commodity hardware.\nMapReduce Hadoop MapRecude is a distributed algorithm framework that allows parallel processing of huge amounts of data. It breaks a large chunk into smaller ones to be processed separately on different data nodes and automatically gather the results across the multiple nodes to return a single result. If the duration of linear data processing can be done during night hours, it makes sense to choose Hadoop MapReduce. MapReduce runs on Hadoop cluster but also supports other database formats like Cassandra and HBase. MapReduce includes:\nJob: a unit of work to be performed as requested by the client.Task: Jobs are divided into sub-jobs known as tasks. The tasks can be run independent of each other on different nodes. There are two types of tasks: Map task is performed by map() function to process one or more chunks of data and produce the output resultsReduce task is performed by reduce() function to consolidate the results produced by each of the map taskJobTracker: like the storage (HDFS), the computation (MapReduce) also works in master-slave fashion. A JobTracker node acts as the master to schedule task on appropriate nodes, coordinate execution of tasks, get the result back after execution of each task, re-execute failed tasks, and monitor overall progress. There is only one JobTracker node per Hadoop Cluster.TaskTracker: a TaskTracker node acts as teh slave and is responsible for executing a task assigned to it by the JobTracker. There are usually a number of JobTracker nodes in a Hadoop Cluster. They execute the heavy lifting tasks.Data Locality: if MapReduce cannot place the data and the compute on the same node, data locality put the compute on the node nearest to the respective data node(s) which contains the data to be processed. The MapReduce programming model includes these steps: input-\u0026gt;split-\u0026gt;map-\u0026gt;combine-\u0026gt;shuffle\u0026amp;sort-\u0026gt;reduce-\u0026gt;output.\nMapReduce programming model YARN YARN (yet another resource negotiator) is a system to schedule applications and services on an HDFS cluster and manage the cluster resources like memory and CPU. The two components are:\nResourceManager: receives the processing requests, and then passes the parts of requests to corresponding NodeManager accordingly based on the needs. ResourceManager is a central authority.NodeManager: installed on every DataNode, is responsible for execution of the task on every single DataNode, monitoring the resource usage and reporting to the ResourceManager. HBase A key-value pair NoSQL database based on HDFS storage, with column family data representation, and mater-slave replication. HBase is based on Google\u0026#8217;s BigTable concept (similar to Cassandra). It runs on a cluster of commodity hardware and scales linearly. Compared with Cassandra, HBase doesn\u0026#8217;t have a query language of its own. You will have to work with JRuby-based shell, or Apache Hive. HBase is also a master-slave architecture and it uses Zookeeper as a status manager. In that sense, Cassandra is a \u0026#8220;self-sufficient\u0026#8221; database technology whereas HBase relies on other components in Hadoop. This article also compares the data model difference between the two.\nHive Hive is a SQL interface over MapReduce for developers and analysts who prefer SQL interface over native Java MapReduce programming to query and manage large datasets residing in HDFS. With Hive you can map a tabular structure on to data stored in distributed storage. The Hive queries are written in SQL-like language known as HiveQL, executed via MapReduce. When a HiveQL query is issued, it triggers a Map and/or Reduce job(s) to perform the operation defined in the query.\nPig A scripting interface over MapReduce for developers who prefer scripting interface over the native Java MapReduce programming. It is a runtime environment with a shell (named Grunt Shell) for execution of MapReduce jobs via a high-level scripting language called Pig Latin. Pig is an abstraction (high-level programming language) on top of a Hadoop cluster. The Pig Latin query/command are complied into one or more MapReduce jobs and then executed on Hadoop cluster. The most common commands in Pig are:\nDUMP: displays the results to screenSTORE: stores the results to HDFS Hadoop Ecosystem There are some other Apache projects, which are sometimes considered as in the Hadoop ecosystem as well:\nOozie: worflow scheduling system to manage Hadoop jobs. In Oozie, a workflow is defined as a collection of control flow nodes and action nodes in a directed acyclic graph. Control flow nodes define the beginning and the end of a workflow, as well as a mechanism to control the workflow execution path. Action nodes are the mechanism by which a workkflow triggers the execution of a computation/processing task, such as MapReduce, Pig, etc.Sqoop (SQL-to-Hadoop): a command-line interpreter tool for importing data from database (e.g. MySQL, data warehouse, etc) into the Hadoop environment (e.g. HDFS, Hive). It can also export the data back.Flume: data ingestion for streaming logs into Hadoop environment. Flume is a distributed and reliable service for collecting and aggregating huge amounts of log data.ZooKeeper: distributed service coordinator, as previously discussed. It is based on a Paxos algorithm variant called ZAB protocol.Ambari: a framework for provisioning, managing and monitoring Hadoop clusters. Hortonworks sandbox provide a VM image that have some Hadoop services pre-installed for beginners to get a taste of how it works all together.\nSpark Hadoop is used in the industry owing to a simple programming model (MapReduce) but the speed and waiting time (between queries and running the program). Spark is introduced to speed up the computing process. Spark uses Hadoop for storage (HDFS) and processing. It extends the MapReduce model to efficiently use more types of computations which includes interactive queries and stream processing. Spark started as a sub-project of Hadoop in 2009 but since 2014 Apache has run it as a top-level project. It is a lightning-fast in-memory cluster computing technology. The features are:\nSpeed: in-memory computing makes super fast processing;Built-in APIs supports multiple languages: Scala, Python and Java;Advanced analytics \u0026#8211; apart from map and reduce, Spark also has libraries that supports SQL query, near real-time stream processing, Graph algorithms and machine learning. Spark can run in standalone mode, on Mesos, or with YARN cluster manager. The document also provides guide on deployment on EC2 and Kubernetes. Spark contains these components:\nSpark Core: the underlying general execution engine for spakr platform that all other functionality is built upon. It provides in-memory computing and referencing datasets in external storage systems.SparkSQL: a components on top of Spark Core that introduces a new data abstraction called SchemaRDD, which supports both structured and semi-structured data.Spark Streaming: perform streaming analytics on top of Spark Core. It ingests data in mini-batches and performs RDD (Resilient Distributed Datasets) transformation on the fly.MLib: a distributed machine learning framework GraphX: a distributed graph-processing framework The speed of Spark is owing to its fundamental data structure \u0026#8211; Resilient Distributed Datasets (RDD), an immutable distributed collection of objects. Each dataset in RDD (object collection) is divided into logical partitions, which can be computed on different nodes of the cluster. The object can be any type of Python, Java or Scala object, including user-defined classes. There are two ways to create RDDS:\nParallelizing an existing collection in your driver programReferencing a dataset from external storage system (e.g. HDFS, HBase) or data source offering a Hadoop Input Format You can also create RDD based on other existing RDDs. This page explains further how RDD speeds up computing compared to MapReduce.\nPrevious PostHost legacy application in Docker 1 of 2 Next PostSpark, Cassandra and Python ","date":"2020-09-10T21:33:00-04:00","permalink":"/2020/09/intro-to-big-data-projects/","title":"Intro to Big Data Projects"},{"content":"This is my notes from containerizing a legacy application with Docker compose. We have to run multiple instances of our application because we\u0026#8217;re unable to secure additional VMs for this single-VM education environment. The application is target of containerization, because it requires mass reconfiguration (around TCP port) to run multiple instances of the application. We want to use the same application configuration file for multiple containers, and map the TCP port to different groups of ports on the host, leveraging port mapping in Docker. On the other hand, the auxiliary services are not being containerized, such as Cassandra database and ElasticSearch because they can be shared for multiple application instances. In other words, we use Docker to isolate processes of the same application.\nPrepare environment The CentOS server needs to have docker-ce (through YUM) as well as docker-compose (direct download). They can be installed this way:\n$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo $ sudo yum install docker-ce docker-ce-cli containerd.io $ curl -L \u0026#34;https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)\u0026#34; -o /usr/local/bin/docker-compose $ sudo chmod +x /usr/local/bin/docker-compose $ sudo systemctl start docker Our Docker registry is not publicly available. So we need to port the Docker image we need to remote server and load it into the local registry. We first examine the registry locally:\n$ curl -XGET https://admin:password@docker.digihunch.com/v2/dhunch/tags/list | python -m json.tool Once we identify the image, we export it to a tar file:\n$ docker save docker.digihunch.com/dhunch \u0026gt; dhunch_image.tar SCP the file to remote server and load it locally:\n$ docker load -i /home/dhunch/dhunch_image.tar $ docker image ls We need to distinguish these commands:\ndocker save: saves an (non-running) image with all layers to file docker export: saves a running or paused container to file docker import: import the contents from a tarball to create a filesystem image, most used with docker export docker load: load an image from a tar archive or STDIN, most used with docker save Build docker-compose file I need to cater to the customer environment with a newly create docker-compose file. The customer environment includes specific storage and networking configurations. Docker compose\u0026#8217;s official documentation is here. We repeat the following commands for our troubleshooting:\n$ docker-compose up -d $ docker-compose exec -it dhunch1 bash $ docker container ls Once we start the container, the status might go unhealthy after it starts. The documentation explains two reasons you\u0026#8217;re seeing an unhealthy container:\na single run of the command takes longer than the specified timeout health check fails; the health check command will retry a number of times before it declares the container as unhealthy. In our case,\u0026nbsp; It is most likely because it does not pass a built-in health check mechanism. We need to understand where the health check was defined. There are four ways to enable health check:\nDockerfile instruction when building the image Docker run command Docker-compose or docker stack yaml file Docker service With #1, unfortunately, you can\u0026#8217;t reverse engineer an image and view the Dockerfile that were used to built it and review the health check statement. What you can do is check docker events, or inspect the container, and go to the log files as specified under logPath section in the inspection result and look for HealthCheck section. We determined it is the case, then we can disable, or override the built-in healthcheck command from image, with a statement in docker compose.\nFor network interface, docker compose also\u0026nbsp;allows us to specify MAC address for each container with mac_address keyword (for license key). MAC address generator are available on the internet. EntryPoint vs CMD The difference between EntryPoint and CMD is very important when launching container. Some literature also mentions RUN, which is only used when building a new layer of images so it is not relevant here (in the context of launching a container from image). EntryPoint and CMD has similar functionalities both allowing you to specify a command to run. The difference is whether they can be overwritten by command line arguments that user provide to docker-compose or docker run in an ad-hoc manner. As their names suggests, EntryPOINT means what is specified under it must be executed as it launches into the container, regardless of any adhoc commands. On the other hand, CMD is just an entry to save users from typing in a command every time they run docker compose or docker run. Should user prefer a different command, it can be provided as an explicit argument and it will be respected overwriting the pre-defined CMDentry in Dockerfile or command entry in docker-compose.yml.\nBoth CMD and EntryPoint supports shell and exec forms. More details here.\nChoice of Networking With single-host deployment, the containerized application needs to communicate with other existing, non-containerized service on host, such as database or elastic search. If docker uses host network, the container shares interface with the host and it does not have its own IP address. Host network removes isolation between container and host. This allows container to run the application that was licensed to the host based on MAC address. There is also no port mapping from container to host network. Container simply uses port on host, and is subject to the availability of TCP/UDP port on host.\nWe will have to use bridge network here. We can force MAC address the app container, and pre-generate license. For container to communicate with a service on host, through bridge network, there are two problems to address:\nContainer knows the IP of the host (layer-3 connectivity, ping); Making host service available to container (layer-4 connectivity, telnet); Docker creates its own interface for bridge network. If it\u0026#8217;s an unnamed network, i.e. not explicitly declared under networks section in docker compose, then interface docker0 is used. If it\u0026#8217;s a named network, then an interface name starting with br- is used.\nThe first problem is easier to address, we simply needs to IP address of the host on the interface. We can validate by pinging from container to host. Docker can also use host.docker.internal to reference the host. Unfortunately, this stopped working for linux since 18.09.3.\nIt is reportedly to be fixed in 20.04 and until it is available, we may add it to manual dns. The following command outputs the entry to add to /etc/hosts in container.\n# ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po \u0026#39;inet \\K[\\d.]+\u0026#39; | awk \u0026#39;{print $1 \u0026#34; host.docker.internal\u0026#34;}\u0026#39; To do this automatically in docker compose, we need some tricks:\nStore the Host IP in host environment variable ( use an export command) Use compose to pass host environment variable to container environment variable Have the container write its environment variable to /etc/hosts The compose file will contain a line like this:\nservices: myenv1: image: alpine command: \u0026gt; sh -c \u0026#34;apk update \u0026amp;\u0026amp; echo $$HostDNSLine \u0026gt;\u0026gt; /etc/hosts \u0026amp;\u0026amp; bash\u0026#34; #network_mode: bridge environment: - HostDNSLine=${HOSTDNSREC} Note ampersand might be mistakenly displayed as \u0026amp;amp; in the above. Then we run it with the following:\n# export HOSTDNSREC=$(echo 1.2.3.4 host.docker.internal) \u0026amp;\u0026amp; docker-compose up The second problem is harder to address because the service on host may not bind to docker\u0026#8217;s interface. Some services such as ssh bind to all interfaces on host and you can telnet to port 22 with any IP address the host is associated with. This is however not the case for most other services, such as Cassandra or Elastic Search. They typically only bind to main interface, such as ens192, or eth0, and not to the docker interface. In order to make the service available to container, we either need to bind these services to the docker interface, or use iptables rules as an alternative.\nSuppose it is a named network and Docker\u0026#8217;s interface name is br-90ae024d5324, and the service on host listens to port 9042, we will need\u0026nbsp; the following two commands from host:\n# sysctl -w net.ipv4.conf.br-90ae024d5324.route_localnet=1 # iptables -t nat -A PREROUTING -p tcp -i br-90ae024d5324 --dport 9042 -j DNAT --to-destination 127.0.0.1:9042 Note that docker compose can configure to run sysctl in container but not from host. If there are multiple ports, we can turn this into a shell script:\n#!/bin/bash tcp_port_list=\u0026#34;9200 9042 8302 8303 8304 8305 8306\u0026#34; if_name=$(basename -a /sys/class/net/* | grep ^br- | head -1) echo enable route localnet on interface $if_name sysctl -w net.ipv4.conf.$if_name.route_localnet=1 for tcp_port in $tcp_port_list; do echo open host tcp port $tcp_port to interface $if_name iptables -t nat -A PREROUTING -p tcp -i $if_name --dport $tcp_port -j DNAT --to-destination 127.0.0.1:$tcp_port done echo $(ip -4 addr show $(basename -a /sys/class/net/* | grep ^br-) | grep -Po \u0026#39;inet \\K[\\d.]+\u0026#39; | awk \u0026#39;{print $1 \u0026#34; host.docker.internal\u0026#34;}\u0026#39;) On the other hand, binding service to multiple interfaces usually require some re-configuration on the service itself. For example, if it is Elastic Search, we need to update [network.host] entry in elasticsearch.yml to include multiple IP addresses. For Cassandra, we need to update rpc_address to 0.0.0.0 or set rpc_interface in cassandra.yml.\nIntegration with storage The application in the container need to store files to storage available to host, whether it is an NFS share or a block disk. We can use volume mapping with Docker compose, to map a path in container to a path presented to host as persistent volume. At this step, we might run into permission issues. By default, containers initializes as root (uid=1) within the container, and the entrypoint script launches application as root. When application writes to persistent volume, files are written as root user. In the legacy non-container setup, we expect the application to write file as dhunch user. Moreover, NFS volume will not allow writing files as root (if the server has root squash configured). To address this, there are two approaches:\nlaunch container as a regular user launch container as root user, then have the entrypoint script launch application as regular user (dhunch) For approach 1, we need to tell Docker to launch container as a regular user by specify the uid and gid for container to run application. We can specify the following envrionment variable in the compose yaml:\nuser: ${CURRENT_UID} Then we assign the environment variable before running docker-compose:\n# export CURRENT_UID=$(id -u dhunch):$(id -g dhunch) \u0026amp;amp;\u0026amp;amp; docker-compose up This allows container to initialize as the regular user. However, if the entry point script needs to perform activities that requires root permission within the container, it will fail. For example, a regular user in container will not be able to update /etc/hosts;\nWith approach 2, we do not specify user in docker compose so container initializes as root. Then the entry point script launches application as regular user. For example, use su command before launch Java:\nsu dhunch -c \u0026#34; exec java \\ -Xms512M -Xmx8192M \\ -Djava.io.tmpdir=$APP_HOME/var/tmp \\ -server \\ -XX:CompileCommandFile=$APP_HOME/etc/hotspot_compiler \\ -jar $APP_HOME/lib/jar/jruby-complete-*.jar \\ --1.9 \\ $APP_HOME/lib/rubybin/runapp.rb \u0026#34; Before doing this, we need to first create user dhunch within container, and the uid and gid must match those of the host. So that when container picks up dhunch user, it converts it to the correct uid.\ngroupadd -g 1011 dhunch useradd -m -c \u0026#39;regular user\u0026#39; -u 1011 -g 1011 dhunch To further understand how uid and gid work, here are two posts with more information.\nThis user ownership setup will also work for NFS. To configure NFS, we need some extra client-side configurations in the container, as well as a special volume driver for NFS. Refer to this post.\nPrevious PostZookeeper Summary Next PostIntro to Big Data Projects ","date":"2020-09-04T16:24:00-04:00","permalink":"/2020/09/host-legacy-application-with-docker-compose/","title":"Host legacy application in Docker 1 of 2"},{"content":"Distributed systems Distributed system involves independent computing entities linked together by network. The components communicate and coordinate with each other to achieve a common goal. In early days, designers and developers often had made some assumptions (aka. fallacies) of distributed computing:\nThe network is reliable Latency is zero Bandwidth is infinite Network is secure Topology doesn\u0026#8217;t change: in reality, components to a network get removed/added over time. the system should tolerate such changes. There is one administrator: for distributed systems to function, they interact with external system beyond administrative control. Transport cost is zero:\u0026nbsp; cost is involved everywhere, in the form of CPU cycles spent, to actual dollars paid to service provider. Network is homogenous These fallacies make coordinating distributed computing entities a huge challenge and Zookeeper is introduced to address these challenges. Zookeeper implements common tasks for distributed coordination, such as:\nConfiguration Management (propagate configuration changes to all worker nodes dynamically) Naming service\u0026nbsp; Distributed synchronization (locks and barriers) Cluster membership operations (e.g. detection of node leave/join) ZooKeeper is a centralized coordination service for the distributed application. ZooKeeper itself is distributed as well. It runs on its own cluster of servers called a ZooKeeper ensemble, separate from application\u0026#8217;s cluster. Distributed consensus, group management, presence protocols, and leader election are implemented by the service so that the application developers do not need to reinvent the wheel by implementing them on their own.\nDevelopers will have to use APIs through ZooKeeper\u0026#8217;s client library, which has language bindings for almost all popular programming languages. The client library is responsible for the interactions of an application with the ZooKeeper service. For testing with API access one can alternatively use its Java-based command-line shell (zkCli.sh)\n$ zkCli.sh -server zknode:2181 How Zookeeper works Data Model ZooKeeper allows distributed process to coordinate with each other through a shared hierarchical namespace of data registers (znodes). The hierarchy start with root node which has child znode(s). Each znode can have their children, as well as store its own data (hence the name data register). The data in a znode is stored in byte format for a maximum of 1MB (ZooKeeper by design is just a coordinator service of host application, so its own data set size is fairly small).\nZookeeper data model Znodes have two types (set at time of creation) persistent znode: for storing persistent data, such as configuration. The znodes and their data will exist even if the creator client dies. ephemeral znode: deleted by ZooKeeper service when the creating client\u0026#8217;s session ends (due to disconnection or explicit termination). It can also be explicitly deleted by creator client through delete API call. They cannot have children. Their visibility is controlled by ACL policy ZooKeeper can assign an incremental sequence number as part of znode name during its creation. This makes a sequential node. Both persistent znode and ephemeral znode can be either sequential or not.\nIn typical client-server architecture, server is passively open and do not initiate communication to client. Client pulls information from server. This is however an anti-pattern for large scale distributed system. ZooKeeper implements a Watch mechanism where clients can get notifications from ZooKeeper service, instead of having to poll for events. Clients can register with the ZooKeeper service (by setting a watch on znode) for any changes associated with a znode. A watch will only trigger notification once, and needs to be re-registered (by client) for trigger the next notification. A watch is triggered upon:\nAny changes to the data of a znode; any changes to the children of a znode; Creation of deletion of a znode ZooKeeper guarantees that notifications are delivered in the order of event occurrence. When a client disconnects from ZooKeeper server, it doesn\u0026#8217;t receive any watches until the connection is re-established. API Operations The ZooKeeper operations are:\nOperationDescriptioncreateCreates a znode in the specified pathdeleteDeletes a znodes from the specified path. Not allowed if the znode has children. version number requiredexistsCheck if a znode at the specified path exists, and get version number; support watchgetChildrenGet a list of children of a znode; support watchgetDataget the data associated with a znode; support watchsetDatawrites data into the data field of a znode. Version number required.getACLget the ACL of a znodesetACLset the ACL in a znodesyncsynchronizes a client\u0026#8217;s view of a znode The write operations (setData, create, delete) are atomic, durable and eventually consistent. Every znode has a stat structure including cZxid, mZxid an dpZxid that keeps track of the ID of the transactions that created, last modified this znode, or pertains to adding or removing its children.\nProduction znode ensemble with more than one node is running in quorum mode. Updates to ZooKeeper tree by clients must be persistently stored in this quorum of nodes for a transaction to be completed successfully. Odd number of node is recommended to avoid split-brain where network partition causes two subsets of servers in the ensemble function independently, and different clients get different results for the same requests, depending upon the server they are connected to.\nAll ZooKeeper nodes are listed in the configuration for client application to randomly pick from and try to connect and establish a session. The session is associated with every operation the client executes in a ZooKeeper service. The session also has a timeout period specified by the application client during session establishment. If the connection remains idle for more than the timeout period, the server expires the session. Appropriate session timeout should be set based on network condition. Sessions are kept alive by client sending heartbeat to ZooKeeper service. Application developer needs to handle connection-loss scenarios properly.\nLeader Election and Atomic Broadcast ZooKeeper ensemble contains a leader nodes, follower nodes and observer nodes.\nThe leader node is elected by the cluster. It handles all write requests. The follower nodes are leader candidates that are not elected. They are backup to the leader nodes. They handle read request, and receive the updates proposed by the leader, and through a majority consensus mechanism, a consistent state is maintained across the ensemble. The observer nodes are ineligible as leader candidates. They have otherwise the same function as followers. The service relies on the replication mechanism to ensure that all updates are persistent in all servers that constitute the ensemble. This is the core mechanism in ZooKeeper, implemented as a special atomic messaging protocol called ZooKeeper Atomic Broadcast (ZAB). ZAB (a variant of Paxos algorithm) ensures the election of new leader in the event of old leader crash, and ensures integrity of data. It defines three states (looking, following and leading) of a node, and goes through four phases (election, discovery, sync, broadcast) in its operation.\nAll read requests (exists, getData, getChildren) are process locally by the ZooKeeper node where the client is connected to. This makes read operation fast. All write requests (create, delete, and setData) are forwarded to the leader in the ensemble, which carries out the client request as a transaction. A transaction is identified by zxid and is idempotent. Transaction also satisfies the property of isolation (no transaction is interfered with by any other transaction). Only after a majority of the followers acknowledge that they have persisted the change does the leader commit the update.\nTransaction processing involves two steps in ZooKeeper: leader election and atomic broadcast. This resembles a two-phase commit protocol (which also includes a leader election and an atomic broadcast)\nZooKeeper use local storage to persist transactions. The transactions are logged to transaction logs, in sync\u0026#8217;ed write, requiring a dedicated block device separated from boot device of server. The local storage also keep point-in-time copies (snapshots) of the ZooKeeper tree.\nZooKeeper Recipes The ZooKeeper recipes defines high-level implementation (construct) of some common distributed coordination mechanism:\nBarrier: any thread/process must stop at this point and cannot proceed until all other threads/processes reach this barrier. Queue: allow FIFO in distributed system Lock: Fully distributed locks that are globally synchronous, meaning at any snapshot in time no two clients think they hold the same lock. Leader Election: designate a single process as the organizer of some task distributed among several nodes. Group membership: node may join or leave a group, which needs to be made available to clients. An alternative to ZooKeeper to manage group membership is gossip protocol. Service discovery: help client to determine IP and port for a service that are hosted by multiple servers. Two-phase commit: a mechanism for atomic commitment in two steps: first a commit request phase involving a voting by participants; and second, either a commit action, or an abort action, based on the voting result. Zookeeper Administration The official documentation includes all we need to know about administration. In addition, we need to configure log4j for proper logging. As best practices, we also should turn off swapping on ZooKeeper. We should clean up the data directory periodically if auto purge is not enabled. For optimal performance, ZooKeeper transaction log should be configured in a dedicated device.\nFor monitoring, ZooKeeper responds to a small sets of four-letter commands issued through telnet or nc to server\u0026#8217;s client port. This allows the admin to check health of server or diagnose any problems. This requires the following property in zoo keeper config:\n4lw.commands.whitelist=stat, ruok, conf, isro, wchc The value can be set to asterick to allow all four-letter keyword. Once enabled, we can check server status\n$ echo ruok | nc localhost 2181 imok More four-letter commands are listed here. Apart from the four-letter commands, ZooKeeper can also be managed through Java Management Extensions (JMX).\nConclusion Apache ZooKeeper is a coordination service for distributed application. It has become the solution for high availability for many other projects. Some of Apache\u0026#8217;s well known open-source distributed services include:\nApache Hadoop (an umbrella of projects including many components for BigData processing such as Hadoop Common, Hadoop Distributed File System (HDFS), Hadoop YARN (yet another resource negotiator) and Hadoop MapReduce) Apache HBase: non-relational database on top of HDFS Apache Hive: data warehouse with SQL-like interface Apache Kafka: stream processing Apache Nifi: automated data flow processing. Some of them, such as Nifi, has an embedded implementation of ZooKeeper ensemble if there isn\u0026#8217;t a separate ensemble. There is some limitation with embedded Zookeeper ensemble. First, we cannot start ZooKeeper without starting Nifi service on the same server. Second, we need to orchestrate the configuration so that the ZooKeeper ensemble does not grow too large. We need to keep in mind that the ZooKeeper ensemble is a separate cluster of its own, and the it is not recommended to have more than 7 nodes on ZooKeeper.\nPrevious PostVirtualization 4 of 4 – Networking Next PostHost legacy application in Docker 1 of 2 ","date":"2020-08-26T23:10:00-04:00","permalink":"/2020/08/zookeeper/","title":"Zookeeper Summary"},{"content":"Virtual LAN (VLAN) Although VLAN emerged before virtualization and is technically not part of virtualization topic. I\u0026#8217;d just like to start from here as a refresher. Suppose we have computers from finance department and computers from sales department all connected to a single layer-2 switch. There are at least three problems: 1) too many devices on the same broadcast domain causes traffic congestion; 2) security can be compromised and 3) each department might have several physical locations. We introduce multi-layer switch to address these with two main features: 1) the VLAN feature can map ports to logical networks, so that all hosts are physically connected to a single switch, but logically to their own network (VLAN) 2) the SVI (switch virtual interface) feature allows inter-VLAN routing at layer 3. Such multi-layer switch is sometimes referred to as layer-3 switch. VLAN is local to a switch and a tag is required in ethernet frame in order to pass VLAN info across switches. This link between switches is called a trunk. IEEE 802.1q (aka dot1q) is the networking standard for VLAN, which standardizes the tagging traffic between switches to tell which traffic belongs to which VLAN. The dot1q trunk (aka dot1q link) provides VLAN IDs fro frames traversing between switches. A trunk can be configured between two switches, or between a switch and a router. Trunking is the process of traversing different VLAN traffic over the trunk. The ports on each switch need to be configured to enable trunking. While Cisco calls such ports trunk port, others call them tagged port. Their function is to add the VLAN tag to ethernet frame. In contrast, regular ports that send and receive frames without VLAN tag are called access port or untagged port. Trunk port carries traffic for multiple VLANS whereas access port carries traffic for a single VLAN. A network device connected to access port has no idea about its VLAN belonging. VLAN creation and management are the responsibility of the switch. Common trunking protocols include VTP (VLAN trunking protocol) and DTP (dynamic trunking protocol)\nThis video and this video have good explanations on VLAN.\nVirtual Extensible LAN (VXLAN) VXLAN is an overlay protocol. Remember that in the standard TCP/IP stack, you normally encapsulate layer-3 IP datagram into a layer-2 ethernet frame. With the VXLAN encapsulation technique however, layer-2 frames can be encapsulated within layer-4 UDP packet.\nVXLAN allows you to stretch layer 2 connection over an intervening layer 3 network. VXLAN tunnel endpoints (VTEPs) are the endpoint device that terminate VXLAN tunnels and it can be either virtual or physical switch ports. It encapsulate VXLAN traffic and de-encapsulate the traffic when it leaves the VXLAN tunnel.\nThe VXLAN encapsulation includes the followings:\nOuter Ethernet Header (source and dest MAC for underlay VTEPs) Outer IP header (source and dest IP on underlay network) Outer UDP header (including source and dest ports, 4789 default) VXLAN Header (including VNI) Inner Ethernet Frame (with source and dest MAC for overlay interfaces) The VNI (VXLAN network identifier, aka VNID) included in the VXLAN header is 24-bit long. It is conceptually similiar to VLAN ID in VLAN but only with 12-bit length.\nVXLAN Enapsulation The VXLAN protocol is documented in RFC7348. Its specification was originally created by VMware, Cisco and Arista. As it became more common in network virtualization (with data centre virtualization, and application containerization) several other players joined the list of contributors and they manufacture switches that support VXLAN. This is a section on VXLAN from the document of Huawei Cloud Engine 5800 switch.\nOpen vSwitch is an example of a software-based virtual network switch that supports VXLAN overlay networks.\nIn summary, the main benefits of VXLAN over VLAN are:\nVXLAN scales up to 16 million logical networks, thanks to the 24-bit length of VNI VXLAN supports layer 2 adjacency across IP networks. A VM belonging to existing layer 2 domain can be created in different data centre (where more computing resources are available), without being constrained by layer 2 boundaries, or being forced to create geographically stretched layer 2 domains (stretched VLAN). Virtual Machine Networking For VM to connect to each other, within or across hosts, we need not only vNIC on VM, but also vSwitch to connect vNICs. A vSwitch (aka bridge) is a logically defined layer-2 device that passes frames between vNICs. On the same host, vNICs are directly connected to vSwitch, which is then connected to the physical NIC. Each vSwtich connects a broadcast domain. When we setup vNIC there are three modes:\nBridged networking: VM connect to outside network using host\u0026#8217;s physical NIC, which acts as a bridge between vNIC and outside network. The VM is a full participant in the network as if it were a physical computer on the network. i.e. it obtain IP addressing information from a DHCP server on the outside (physical) network. The VM\u0026#8217;s IP address is also visible and directly accessible by other computers on the network. bridge networking is common for servers as VMs.\nNAT networking: The VM relies on the host to act as NAT device to make outgoing network connection. The IP address of VM is assigned by virtual DHCP server on host. The guest VMs form a private network and computers on the outside network are external. The host translates private IP address into the host\u0026#8217;s IP address on the way out, and listens for returning traffic. Outside network sees traffic from VM guest as if it were from the host. This network mode is common when the VMs are mainly used as a client workstation.\nHost-only networking: creates a network that is completely contained within the host computer. The vSwtich is the hub of the private network and the physical NIC on the host is not involved. The VM will not have access to the outside network. This mode is useful when the VMs needs to be isolated from outside network, and only need to communicate with peers on the same host.\nThe difference between NAT networking and host-only networking is the exposure of VM guest to external network. All of these networking modes are available on VMWare fusion, for example. Advanced virtualization platform such as vSphere usually support multi-hosting. Multiple host can also be configured to form a distributed vSwitch, such as vSphere Distributed Switch, in addition to standard switches.\nDocker Networking I had a brief on Docker network covering three modes. Out of the three modes, single-host bridge network is the equivalent of host-only networking. MacVLAN driver is similar to bridged networking, in the sense that container may connect to external network, using host NIC as a bridge. However, the external network is still bound by physical location. This is when overlay network comes in handy.\nCNM and CNI In the Docker networking, container needs to map its own port to host, of which the port resource is implemented by IP tables, which limits the scale and performance of the solution. Also, those networking modes do not address the problem of multi-host networking. As multi-host networking became a real need for containers, the industry started looking into different solutions. Container project favour a model where networking is decoupled from the container runtime. This also greatly improves application mobility. In this model, networking is handled by a \u0026#8216;plugin\u0026#8217; or \u0026#8216;driver\u0026#8217; that manages the network interface, and how the containers are connected to the network. The plugin also assigns the IP address to the container\u0026#8217;s network interfaces. In order for this model to succeed, there needs to be a well-defined interface or API between the container runtime and the network plugins.\nDocker, the company behind the Docker container runtime, came up with the Container Network Model (CNM). Around the same time, CoreOS, the company responsible for creating the rkt container runtime, came up with the Container Network Interface (CNI). Kubernetes originally seeks to use CNM for its plugins, but they eventually decided to go with CNI. The primary reason was that CNM was still seen as something designed with Docker container runtime in mind and was hard to decouple from it. After this decision, several other open source project also turned to CNI for their container runtimes.\nThis article expands further into the difference between CNM and CNI.\nPrevious PostVirtualization 3 of 4 – Containers Next PostZookeeper Summary ","date":"2020-08-21T21:53:32-04:00","permalink":"/2020/08/virtualization-4-of-4-networking/","title":"Virtualization 4 of 4 – Networking"},{"content":"In broad terms, virtualization of computing resource is about isolation of resources at different levels. We have covered hypervisor-based virtualization in the other post. In this article, we continue to dive into OS level virtualization.\nRemember again that the gist of virtualization is isolation of resource. To support OS level virtualization, the OS must have its own capability to isolate computing resource. There are many implementations of OS level virtualization.\nLinux Kernel provides low-level mechanisms some two kernel features(namespaces, cgroups and chroot) for building various lightweight tools that can virtualize the system environment. Docker is such framework that builds on chroot namespaces and cgroups.\nChroot Traditionally, root directory (/) is the top directory shared amongst all processes in the OS. There was a chroot() system call that allows each process to have its own idea of root directory. A chroot is an operation that changes the apparent root directory(/) for the current running process and their children. A program that is run in such a modified environment cannot access files and commands outside that environmental directory tree. This modified environment is called a chroot jail. By separating a process using chroot() we ensure security by restricting the process from accessing outside its environment (breaking the jail). This short video is a great lab.\nAlthough chroot() has a basic idea of isolation, it simply modifies pathname lookups for a process and its children (by prepending the new root path to any name starting with /). Relative paths can still refer any locations outside of the new root. So chroot() does not intend to defend against intentional tampering by privileged users.\nNamespace Isolation Namespaces are fundamentally the mechanisms to abstract, isolate, and limit the visibility that a group of processes has over various system entities such as process trees, network interfaces, user IDs and file system mounts. So there are several categories of namespaces:\nMount namespaces \u0026#8211; traditionally, there is one global mount namespace seen by all processes. The mount namespaces confine the set of filesystem mount points visible within a process namespace, enabling one process group in a mount namespace to have an exclusive view of the filesystem list, compared to another process.UTS namespaces \u0026#8211; allows isolation of hostname per namespace. Each namespace can have its own hostname on the networkUser namespaces \u0026#8211; allow a process to use unique user and group IDsCgroup namespaces \u0026#8211; processes inside a cgroup namespace are only able to view paths relative to their namespace root.IPC namespaces \u0026#8211; isolates the System V inter-process communication between namespaces, as well as POSIX message queues within each namespace. POSIX message queue allow process to exchange data in the form of messsages.PID namespaces \u0026#8211; traditionally, *nix kernels spawn the init process with PID 1 during system boot, which in turn starts other user-mode process and is considered the root of the process tree (all the other processes start below this process in the tree). The PID namespace allows a process to spin off a new tree of processes under it with its own root process (PID=1). PID namespaces isolate process ID numbers, and allow duplication of PID numbers across different PID namespaces. The process IDs only needs to be unique within a PID namespace, and are assigned sequentially starting with PID 1. PID namespaces are used in containers.Network namespaces \u0026#8211; traditionally, all processes in the entire OS share a single set of network interfaces and routing table entries. The routing table entries can be modified at operating system level. With network namespace, this assumption is no longer valid. Network namespace provides abstraction and virtualization of network protocol and interfaces. Each network namespace will have its own network device instances that can be configured with individual network addresses. Other network services, such as routing table, port number, are isolated as well. Namespaces are created with the \u0026#8220;unshare\u0026#8221; command or syscall, or as new flags in a clone() syscall. The flags are listed here in the man page for namespace. Note that the clone() syscall is a more generic implementation of fork() syscall.\nCgroup cgroups is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc) of a collection of processes (not to be confused with process group, which has its own meaning). Cgroup has two versions. The control groups functionality (version 1) was merged into Linux kernel mainline in version 2.6.24, released in 2008, and version 2 in kernel 4.5 (March 2016), with significant changes to the interface and internal functionality.\nUsing cgroups, you can allocate resources such as CPU time, network and memory. Similiar to the process model in Linux, where each process is a child to a parent and relatively descends from the init process thus forming a single-tree like structure, cgroups are hierarchical, where child cgroups inherit the attributes of the parent, but what makes it different is that multiple cgroup hierarchies can exist within a single system, with each having distinct resource prerogatives.\nApplying cgroups on namespaces results in isolation of processes into containers within a system, where resources are managed distinctly. Each container is a lightweight virtual machine, all of which run as individual entities and are oblivious of other entities within the same system.\nContainer Implementation Above we covered some kernel features that enables container technology. There are many ways to use these technologies to implement the isolation. We call them container runtime. LXC is a user space interface for those Linux kernel containment features. It allows for running isolated containers on a control host using a single kernel. Users can launch a system init for each containers, also referred to as virtual environment (as opposed to virtual machines). The author of this article regard LXC as a suprcharged chroot on Linux. LXC has rest API tool called LXD. LXC was targeting sysadmin\u0026#8217;s use cases (not developer) to isolate users\u0026#8217; own private workloads from one another. In early days Docker was built on LXC. Docker\u0026#8217;s target market is developers, and it moved beyond LXC with its own execution environment called libcontainer. With the initial success of Docker, a large community (Docker, CoreOS, Google, etc) emerged around the idea of using containers as the standard unit of software delivery. They started the Open Container Initiative (OCI) to define industry standards around container runtime (runtime spec) and image format (image spec). Docker donated the libcontainer codebase to run independently under OCI, as runc. Docker implements isolation using the following technologies:\nNamespace: to isolate process ID, networking, mount points, IPC, host and domain name;Cgroups: to isolate the usage of CPU and memory between containersUnionFS: isolate file system Another container runtime technology is OpenVZ, which includes an extension of the Linux kernel. It uses container for entire operating systems (not just application and processes). All OpenVZ containers have to share the same Linux kernel version as host. The adoption of OpenVZ is not high.\nFrameworkRuntime implementationManagement toolLXClibvert\nLXCLXD (rest API)OCIDocker\u0026#8217;s runc\nCoreOS\u0026#8217;s rtkdocker engine (daemon and cli)\nrtk clicontainer runtimes Docker is now widely adopted for application hosting in production environment. Container and Cloud Public cloud vendors also has managed services around Docker. Here are some examples:\nManaged ContainerImage RegistryManaged OrchestrationAWSElastic Container ServiceElastic Container RegistryElastic Kubernetes ServicesAzureContainer InstancesContainer RegistryAzure Kubernetes ServiceGCPCloudRunContainer RegistryGoogle Kubernetes EngineDigital OceanN/AContainer RegistryKubernetesContainer services from public cloud Cloud service was originally developed with VM as a unit of computing resource to service. OS level virtualization allows container to be a unit of computing resource. All these new technologies breed the serverless architecture and cloud-native deployment model. This has significant impact on the creation and delivery of software services. The cloud native landscape page illustrates more tools around containers.\nPrevious PostCloud storage overview Next PostVirtualization 4 of 4 – Networking ","date":"2020-08-18T20:44:35-04:00","permalink":"/2020/08/virtualization-3-of-3-containers/","title":"Virtualization 3 of 4 – Containers"},{"content":"In a narrow sense, cloud storage refers to object storage. In a broader sense, it refers to any storage service (block, file or object level) provided by cloud vendors, in a cloud business model. The underlying technology of storage, is the same be it in the cloud or on-premise. Block storageFile storageObjectInteraction with OSOS has direct byte-level access to disk blocks.OS manages storage by file, or byte range of file. Files are organized in POSIX hierarchy.OS reads and writes the entire object, or a byte range, via rest API calls.MetadataN/AStored in file system, for directory or filecustomizable metadataCommon protocolN/ANFSS3ImplementationSAN (bock device is typically dedicated to a single VM) or DASNAS, file storage is usually shared amongst multiple VMs. Locking mechanism is usually in place to keep access in order.S3Workloaddatabase storage, scratch data, etcpersistent data, content management, etcarchive data, media streaming, data analytics, static asset serving, etc Below is a list of common storage services provided by public cloud vendors to day.\nBlock StorageFile StorageObject StorageOther managed storage serviceAWSElastic Block Store (EBS)Elastic File System (EFS)\nFSx for Windows\nFSx for LustreSimple Storage Service (S3)Storage Gateway Snow Family\nDataSyncAzureAzure Managed DisksAzure FilesAzure BlobsAzure Table Azure QueuesGCPPersistent Disk\nlocal SSDFilestoreCloud StorageCloud Storage for Firebase\nData TransferDigital OceanVolumes Block storage\nlocal SSDN/ASpace object storage (S3 compatible)Content Delivery NetworkStorage Products from common public cloud vendor Since AWS is the first vendor that provides a full suite of storage service, this post will focus on the storage product lines, as a refresher of AWS cloud storage options: Simple Storage Service, Elastic File Storage and Elastic Block Storage). There will be some overlap with the AWS storage service whitepaper.\nBefore getting further to details, here\u0026#8217;s a reminder of two types of policies in AWS:\nIAM policyResource-based policyPrincipalMust be attached to individual user, group, or role to take effectNeeds to be explicitly specified, can be ARN under other AWS accountElementAction/NotAction\nResource/NotResource\nEffect (Allow/Deny)\nConditionPrincipal/NotPrincipal\nAction/NotAction\nResource/NotResource\nEffect (Allow/Deny)\nConditionExampleManaged policy, custom policyFile system policy, S3 bucket policy, access point policy, etcTwo types of policies Although the resource is usually assumed in a resource-based policy, the policy usually target a sub-section of a resource (e.g. object with certain prefix), so resource section is still required in resource-based policy. In storage services, we may use S3 bucket policy, access point policy, or file system policy for EFS.\nBelow we go over the three families of storage service in AWS.\nEBS (Elastic Block Storage) EBS is a distributed system. Each volume is a logical volume, made up of multiple physical devices. EBS data is persistent, and access is dedicated to a single EC2 instance at a time. If EC2 instance failed, the attached EBS volume can be detached, and then re-attached to other instance, in the same Availability Zone. There are two types of EBS:\nEC2 Instance store: ephemeral, block-level storage for EC2 instance, no replication by default, no snapshot support. Used as buffers, caches, scratch data, temporary content. EBS volume (persistent) : used for database, dev/test, enterprise application, etc. There are two sub-categories: SSD-backed volumes: Optimized for transnational workloads that requires very low latency Dominant performance attribute is IOPS For frequent, read/write with small size and random I/O Typical use case include relational database (PostgresQL, MySQL) and NoSQL (Cassandra, Mongo) gp2 (general purpose) and io1 (provisioned IOPS) HDD-backed volumes: Optimized for large streaming workloads demanding throughput Dominant performance attribute is throughput For workloads with lots of sequential I/O Typical use case icnlude big data, analytics (Kafka, Splunk, Hadoop, data warehousing), file/media server st1 (throughput optimized0 and sc1 (cold HDD) The four types of EBS are compared here:\nNote that the volume can be modified (change type, increase size) after creation. However, you cannot decrease size. If you increase the size, the file system must be extended after the increase.\nAnother way to deliver better performance is to use EBS-optimized instances. These instances have dedicated network bandwidth for its I/O traffic to and from EBS. Without EBS-optimized instance, the traffic between EBS volume and EC2 instance uses shared network link with EC2, which is subject to latency during heavy traffic. This distinction is similiar to the difference between iSCSI SAN and FC SAN. Also, you may increase read-ahead buffer in OS for better EBS performance.\nOn EBS, users can create snapshot, a point-in-time incremental backup. When snapshot is restored to a volume, data is loaded lazily in the background, so that volume is available immediately. This also means that initial read of data that is not yet loaded will be subject to latency, known as first read penalty. To achieve target performance, user may run an initialization on the volume, by reading all blocks with data upfront.\nIn a newly created snapshot, only the data blocks modified since the previous snapshot are stored as is. The rest are pointers to unchanged data blocks in the original snapshot. When a previous snapshot is deleted, AWS ensures changes are reconciled into the newer snapshot so there is no loss of data. Creation of snapshots on many volumes can be automated with Data Lifecycle Manager (DLM).\nAs far as encryption goes, the best practice is to create your own master key. KMS uses envelop encryption, where the data key encrypts the data, and the master key encrypts the data key. The encryption key is stored in EC2 instance memory only and never written to disk, for security and performance considerations.\nEFS (Elastic File Storage) EFS is a managed implementation of file storage that supports NFS 4.0 and 4.1, with strong data consistency and file locking. An EFS includes a single mount target in (one subnet of) each availability zone. EC2 instance, or on-premise client via Direct Connect, can mount EFS volumes using amazon-efs-utils yum package. EC2 instance can also be configured to automatic mount EFS volume in launch wizard. EFS also has a lifecycle management policy, and a storage class for infrequent access.\nPerformance wise, EFS has two performance modes and two throughput modes. The two performance modes are:\nGeneral Purpose: for latency-sensitive applications and general-purpose workloads. limit of 7k ops/sec, best choice for most workloads Max I/O: for large-scale and data-heavy applications, with virtually unlimited ability to scale out throughput/IOPS, but with slightly higher latencies. consider this for large scale-out workloads The two throughput modes are:\nBursting throughput: recommended for the majority of workload. Since file system workload is typically spiky, aws use credit system to determine when the file system throughput can burst. credit accumates idle time, and consumed in retrieval Provisioned throughput: recommended for higher throughput to storage ratio workload, can increase the provisioned throughput afterwards. but it incurs separate throughput charge Other ways to achieve higher performance, include parallelization of file operation (e.g. multiple threads, more instances); and increase I/O size for better throughput.\nIn terms of security, EFS encryption at rest must be selected at the time of file system creation. There is an TLS mount option to encrypt traffic in transit. EFS involves its own resource-based policy called file system policy to manage file-level POSIX permissions. IAM policy is used to manage NFS administration access and client access. EFS access points is also a means to enforce the use of a specific operating system user, and group to access EFS.\nS3 (Simple Storage Service) S3 is one of the earliest and maturest AWS services for object storage. It is very cheap and easy to use, and supports user-defined metadata on objects as well as many peripheral features. There is no limit to the number of objects in a bucket. As the object in bucket increases, S3 scales to request rate by automatically creating more partitions to meet the target number of request per partition. There used to be a performance trick, that requires client to make object key naming pattern distribute across multiple prefixes. It is not required any more as of July 2018.\nVersioning can be enabled at bucket level, and suspended afterwards. New version of object is created on every upload, without performance penalty. S3 integrate well with other event-driven AWS services, such as SNS, SQS, Lambda, etc. Event can fire on request such as PUT, POST, COPY. Object tags (not to be confused with object metadata) can help categorize storage. It also facilitates access control (i.e. by being referenced in bucket policy or IAM policy), lifecycle policy, analysis and CloudWatch configurations.\nS3 select is a way to retrieve only a subset of data from an object based on a SQL expression, to reduce amount of data and help with performance. The input can be json or CSV and output will be in CSV.\nS3 Inventory is a tool to audit object replication status and encryption status. It generates CSV report with all objects in the given bucket name, including: key name, version id, islatest, size, last modified date, etag, storage class, multipart upload flag, delete marker, replication status, encryption status. For storage-class analysis, S3 inventory is much faster than list-object API call which parses through all objects.\nS3 also has access point, similar to EFS, with unique hostnames that customers create to enforce distinct permissions and network controls for any request made through the access point.\nS3 transfer acceleration take advantage of edge locations (at additional charge) to speed up transfer of large object over long distance, by providing a separate end point. It is also helpful for faster uploads over long distances. Apart from transfer acceleration, for faster uploads for large object, user may also consider multi-part upload API when the object reaches 100MB. Orphaned uploaded parts can be cleaned up in lifecycle configuration. For better download performance, take advantage of CloudFront and byte range request.\nPrevious PostJava Garbage Collection Next PostVirtualization 3 of 4 – Containers ","date":"2020-08-12T22:19:00-04:00","permalink":"/2020/08/cloud-storage-overview/","title":"Cloud storage overview"},{"content":"Tuning the garbage collector is the most important thing that can be done to improve the performance of a Java application. GC is typically caused when the JVM decides GC is necessary, specifically when:\na minor GC will be triggered when the new generation is full; a full GC will be triggered when the old generation is full; a concurrent GC (if applicable) will be triggered when the heap starts to fill up OpenJDK has three collectors suitable for production, with different performance characteristics. In order to study the GC behaviours in application, it is important to turn on GC logging. The detailed step is different in JDK 8 and JDK 11 (read about java version here).\nJava developers don\u0026#8217;t need to manage life cycle of objects explicitly as the JVM automatically fress the object. In order to track objects that are still in use, it is insufficient to count references to objects. Instead, the JVM must periodically search the heap for unused objects. Once it finds unused objects, the JVM frees the memory occupied by those objects. It also needs to compact the memory to prevent memory fragmentation. The performance of GC is dominated by these basic operations (finding unused objects; freeing up their memory; compacting the heap), no matter which collector is used. Some algorithms delay compaction until absolutely necessary, some compact entire sections of the heap at a time, and some compact the heap by relocating small amounts of memory at a time. These different approaches are why different algorithms have different performance characteristics.\nJava programs are typically heavily multithreaded, and the garbage collector itself often runs multiple threads too. We refer to the application logic threads as mutator threads, since they are mutating objects as part of the application logic. When GC threads track object references or move objects around in memory, they must make sure application threads are not using those objects on the move. This introduces a pause when all application threads are stopped (known as stop-the-world pauses), which generally has the greatest impact on the performance of an application. Minimizing those pauses is one important consideration when tuning GC.\nGarbage collectors are generational Most garbage collectors work by splitting the heap into generations. These are called the old (or tenured) generation, and the young generation, which is further divided into sections known as eden and survivor spaces, with eden taking up the vast majority of the young generation. The rationale for having separate generations is that many objects are used for a very short period of time in the real life of application programming. Garbage collector is designed to take advantage of this. Objects are first allocated in the young generation, which is a subset of the entire heap. When the young generation fills up, the garbage collector will stop all the application threads and empty out the young generation. Objects that are no longer in use are discarded, and objects that are still in use are moved elsewhere. This operation is called a minor GC or a young GC. Common GC algorithms have stop-the-world pauses during collection of the young generation.\nHeap Generation This design has two performance advantages. First, Cleaning up young generation as a only a portion of the entire heap causes shorter pause than cleaning up the entire heap. Second, by moving used objects to survivor spaces or old generation, and discarding unused objects, compatction is achieved.\nWith used objects moved to the old generation, eventually it woo will fill up, and the JVM will needt o find any objects within the old generation that are no longer in use to discard. This is where GC algorithms have their biggest differences. The simpler alghorithms stop all application threads, find the unused objects, free their memory, and then compact the heap. This process is called a full GC, and it generally causes a relatively long pause for the application threads.\nOn the other hand, sophisticated alghrithms are able to find unused objects while application threads are running. These algorithms are called concurrent collectors, or low-pause collectors. A concurrent collector typically allows an application to experience fewer and shorter pauses. The biggest trade-off here is the overall CPU required by the sophisticated algorithms.\nThe three main algorithms Serial GC is the simplest and the default for single core host (e.g. client-class machine, single-processor VM or Docker container). The serial collector uses a single thread to process the heap. It will stop all application threads as the heap is processed (for either a minor or full GC). During a full GC, it will fully compact the old generation. The serial collector is enabled by using the -XX:+UseSerialGC flag.\nThe throughput collector (aka parallel collector) is the default collector for any 64-bit machine with two or more CPUs. The throughput collector uses multiple threads to collect the young generation, which makes minor GCs much faster than when the serial collector is used. This uses multiple threads to process the old generation as well. The throughput collector stops all application threads during both minor and full GCs, and it fully compacts the old generation during a full GC. Since it is the default in most situations where it would be used, it needen\u0026#8217;t be expliticly enabled. To enable it where necessary, use the flag -XX:+UseParallelGC\nThe G1 GC (or garbage first garbage collector) uses a concurrent collection strategy to collect the heap with minimal pauses. It is the default collector in JDK 11 and later for 64-bit JVMs on machines with two or more CPUs. G1 GC divides the heap into regions, but it still considers the heap to have two generations. Some of those regions make up the young generation, and the young generation is still collected by stopping all application threads and moving all objects that are alive into the old generation or the survisor spaces, using multiple threads. In G1 GC, the old generation is processed by background threads that don\u0026#8217;t need to stop the application threads to perform most of their work.\nIn G1 GC, the old generation is processed by background threads that don\u0026#8217;t need to stop the application threads to perform most of their work. Because the old generation is divided into regions, G1 GC can clean up objects from the old generation by copying from one region into another, which means that it compacts the heap during normal processing. This helps keep G1 GC heaps from becoming fragmented.\nThe trade-off for avoiding the full GC cycles is CPU time; the multiple background threads G1 GC uses to process the old generation requires CPU cycles available at the same time the application threads are running. G1 GC is enabled by specifying the flag -XX:+UseG1GC. It is the default in JDK 11, and functional in JDK 8 as well, with some performance feature missing.\nIn all cases, GC is caused when the JVM decices GC is necessary; a minor GC will be triggered when the new generation is full; a full GC will be triggered when the old generation is full, or a concurrent GC (if applicable) will be triggered when the heap starts to fill up. Java also provides a mechanism for applications to force a GC to occur: the System.gc() method, although it is always a bad idea to call that method explicitly because it triggers a full GC which hangs the application threads. This method can be disabled by including -XX:+DisableExplicitGC in the JVM arguments.\nAs to choosing GC algorithm, the rule of thumb is that G1 GC is the better choice. However, in JDK 8, the ability of G1 GC to avoid a full GC is also a key consideration. In this case we may need to choose betwen serial collectors and throughput collectors, based on the number of CPUs on the machine.\nThe serial collector makes sense when running CPU-bound applications on a machine with a single CPU, even if that single CPU is hyper-threaded. The throughput collector makes sens on multi-CPU machines running jobs that are CPU bound. Even for jobs that are not CPU bound, the throughput collector can be the better choice if it does relatively few full GCs or if the old generation is generally full.\nBasic GC tuning Sizing the heap If the heap is too small, the program will spend too much time performing GC and not enough time performing application logic. On the contrary, a very large heap will increase the time spent in GC pauses, even thought the pauses occur less frequently. It is also potentially dangerous due to interaction with memory swap. If a Java program with a 12 GB heap is running on a system where swap is enabled, the OS may handle it by keeping 8GB of the heap in RAM and 4GB on disk. The JVM does not know about this because swapping is handled by the OS. The JVM will happily fill up all 12GB of heap it has been told to use. This can cause a sever performance penalty when OS swaps data from disk to RAM. Worse, the one time this swapping is guaranteed to occur is during a full GC, when the JVM must access the entire heap. Swapping during full GC makes the pause an order of magnitude longer.\nSo heap size (total for all JVMs) should never exceed the amount of physical memory on the machine. Size of heap is controled by two values (Xms as initial value and Xmx as maximum value). Having an initial and maximum size for the heap allows the JVM to tune its behaviour depending on the workload. If the JVM sees that it is doing too much GC, with the initial heap size, it will continually increase the heap until the JVM is doing the \u0026#8220;correct\u0026#8221; amount of GC, or until the heap hits its maximum size.\nA good rule of thumb is to size the heap so that it is 30% occupied after a full GC. To calculate this, start your application and push it to high load. Then connect to the application with jconsole, force a full GC, and observe how much memory is used when the full GC completes.\nSizing the generations The JVM must also decide how much of the heap to allocate respectively to they young generation and old generation. THe JVM usually does this automatically and usually does a good job in determining the optimal ratio. In some cases you might hand-tune these values.\nIn general, if there is a relatively larger young generation, young GC pause times will increase, but the young generation will be collected less often, and fewer objects will be promoted into the old genration. But on the other hand, older generation will be relatively smaller and fill up more frequently and do more full GCs. The command-line flag to tune the generation sizes are:\n-XX:NewRatio=N\n-XX:NewSize=N\n-XX:MaxNewSize=N\n-Xmn N\nThe size of initial young generation is determined by initial heap size and new ratio:\nInitial Young Gen Size = Initial Heap Size / (1 + NewRatio)\nThe young generation will grow in tandem with the overall heap size, but it can also fluctuate as a percentage of the total heap (based on the initial and maximum size of the young generation). Adaptive sizing controls how the JVM alters the ratio of young genration to old gneeration within the heap. It should be kept enabled in general. For finely tuned heaps, adaptive sizing can be disabled for a small performance boost.\nSizing the metaspace When the JVM loads classes, it must keep track of certain metadata about those classes. This occupies a separate heap space called the metaspace. In older JVMs this was handled by a different implementation called permgen. To end users, the metaspace is opaque. It does not hold the actual instance of the class. The objects are held in the regular heap. Information in the metaspace is used only by the compiler and JVM runtime, and the data it holds is referred to as class metadata.\nTuning the metaspace is fairly rare these days because the default values for the size of metaspace are very generous. It is sized dynamically based on an initial size (-XX:MetaspaceSize=N) and will increase as needed to a maximum size (-XX:MaxMetaspaceSize=N).\nResizing the metaspace requires a full GC, so it is an expensive operation. If there are a lot of full GCs during the startup of a program (as it it loading classes), it is often because permgen or metaspace is being resized, so increasing the initial size is a good idea to improve startup in that case.\nControlling Parallelism All GC alghorithms except the serial collector use multiple threads. The number of these threads is controlled by the -XX:ParallelGCThreads=N flag. Bacuase these GC operations stop all application threads from executing, the JVM attempts to use as many CPU resources as it can in order to minimize the pause time. By default, that means the JVM will run one thread for each CPU on a machine, up to eight. Once that threashold has reached, the JVM adds new thread for only every 1.6 CPus. Sometimes this number is too large relative to the heap size and hand tuning is needed.\nReference: Java Performance by Scott Oaks\nThis post also contains some helpful information, where the original Oracle white paper about GC was cited. Further than GC, this website from Oracle describes more about JVM.\nPrevious PostVirtualization 2 of 4 – Graphics Computing Next PostCloud storage overview ","date":"2020-08-07T23:19:17-04:00","permalink":"/2020/08/java-garbage-collection/","title":"Java Garbage Collection"},{"content":"We covered hypervisor in previous post. In this article we focus on the virtualization of graphics computing resource.\nGPU vs CPU GPU is a specialized type of microprocessor primarily designed for quick image rendering. GPU appeared as a response to graphically intense applications that put a burden on the CPU and degrated computer performance. They became a way to offload those tasks from CPUs, but modern graphics processors are powerful enough to perform rapid mathematical calculations for many other purposes apart from rendering.\nCPU consists of a few cores (up to 23) optimized for sequential serial processing, which is designed to maximize the performance of a single task within a job. GPU uses thousands of smaller and more efficient cores for massively parallel architecture aimed at handling multiple functions at the same time. Typical uses cases for GPUs, in addition to graphics display, includes Games, 3D visualization, Image processing, big data and deep machine learning.\nMoving to virtualization world, the most primitive mechanism for graphics acceleration is Soft 3D, which is commonly used in virtual desktops, or DaaS (desktop as a service). The Software 3D renderer (Soft 3D) uses the Soft 3D graphics driver to provide support for software-accelerated 3D graphics without any physical GPUs being installed in the ESXi host. With respect to GPU in virtualized environment, VMware developed a few technologies.\nvSGA (Virtual Shared Graphics Acceleration) The physical GPUs in the server are virtualized and shared across multiple guest VMs. This option involves installing an Nvidia driver into the hypervisor itself, and each guest VM uses a proprietary VMware SVGA 3D driver that communicates with the Nvidia driver in ESX. The biggest limitation here is that these drivers only work with DirectX up to 9.0c, and OpenGL up to 2.1. This technology was introduced in early 2013 and is used in light workload for knowledge worker, such as PowerPoint, Visio and web browsing.\nvSGA vDGA (Virtual Dedicated Graphics Acceleration) vDGA, also known as \u0026#8220;GPU passthrough\u0026#8221;. It provides each VM with unrestricted, fully dedicated access to one of the host\u0026#8217;s GPUs. The hypervisor is drilling a direct hole in itself between the GPU and the guest. This technology allows you to present an internal PCI GPU directly to a VM guest. The device acts as if it were directly driven by the VM guest, and the guest detects the PCI device as if it were physically connected, using the \u0026#8220;real\u0026#8221; driver. There is no special drivers in the hypervisor. vDGA offers the highest level of performance for users with the most intensive graphics computing needs.\nGPU passthrough The main advantage to vDGA is that since the GPU is passed through to the guest and the guest uses regular Nvidia drivers, it fully supports everything the Nvidia driver can do natively. This enables all versions of DirectX, OpenGL and even CUDA. The downside is that vDGA is expensive, since you need one GPU per user. There is also a lack of vMotion support. VMware added support for vDGA in late 2013. The target market is high-end users with intensive graphical applications (oil\u0026amp;gas, scientific simulations, CAD/CAM, etc\nvGPU (Virtual GPU) vGPU is also known as Virtual Shared Pass-Through Graphics Acceleration. This technology sites somewhere in between the two previously introduced, as an option to strike a balance between cost-effectiveness and resource-sharing. It is essentially vDGA but with multiple users per GPU, instead of one-to-one mapping. Like vDGA, with vGPU you install the real Nvidia driver in guest VMs, and the hypervisor passes the graphics commands directly to the hypervisor without any translation.\nvGPU gives you all that plus the ability to share a GPU across up to 8 VMs. The idea of vGPU is that you get better performance than vSGA option, with a portion of cost when compared to vDGA. The use case for vGPU will be the higher-end knowledge workers who need real \u0026#8220;GPU\u0026#8221; access but don\u0026#8217;t need full-on multi-thousand dollar graphics workstations.\nVMware partners with Nvidia on vGPU development. Below is the use-case chart from previous VMware white paper:\nThe diagram below illustrates the architecture of virtual GPU (NVIDIA Grid):\nhigh-level architecture of GRID vGPU The best white paper about the three technologies and their use cases is on VMware website.\nIdentify Graphics driver On Linux VM, we can simply use lspci to identify graphics driver.\n[root@ghrender ~]# lspci | grep VGA 03:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04) 3b:00.0 VGA compatible controller: NVIDIA Corporation GP104GL [Quadro P5000] (rev a1)In the result, the far left column is specified domain, e.g. 3b:00.0 To display details on graphics card by specified domain (3b:00.0 for example) with memory information:\n[root@ghrender ~]# lspci -v -s 3b:00.0 3b:00.0 VGA compatible controller: NVIDIA Corporation GP104GL [Quadro P5000] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 11b2 Flags: bus master, fast devsel, latency 0, IRQ 190, NUMA node 0 Memory at ab000000 (32-bit, non-prefetchable) [size=16M] Memory at 382fe0000000 (64-bit, prefetchable) [size=256M] Memory at 382ff0000000 (64-bit, prefetchable) [size=32M] I/O ports at 6000 [size=128] [virtual] Expansion ROM at ac080000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [128] Power Budgeting \u0026lt;?\u0026gt; Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 \u0026lt;?\u0026gt; Capabilities: [900] #19 Kernel driver in use: nvidia Kernel modules: nouveau, nvidia_drm, nvidia The lshw command can also identify onboard Intel/AMD or Nvidia dedicated GPU:\n[root@ghrender ~]# lshw -C display *-display description: VGA compatible controller product: Integrated Matrox G200eW3 Graphics Controller vendor: Matrox Electronics Systems Ltd. physical id: 0 bus info: pci@0000:03:00.0 version: 04 width: 32 bits clock: 66MHz capabilities: pm vga_controller bus_master cap_list rom configuration: driver=mgag200 latency=64 maxlatency=32 mingnt=16 resources: irq:16 memory:91000000-91ffffff memory:92808000-9280bfff memory:92000000-927fffff *-display description: VGA compatible controller product: GP104GL [Quadro P5000] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:3b:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:382f0-382ef iomemory:382f0-382ef irq:190 memory:ab000000-abffffff memory:382fe0000000-382fefffffff memory:382ff0000000-382ff1ffffff ioport:6000(size=128) memory:ac080000-ac0fffff Previous PostVirtualization 1 of 4 – Hypervisor Next PostJava Garbage Collection ","date":"2020-08-01T18:24:00-04:00","permalink":"/2020/08/virtualization-of-graphics-computing-resource/","title":"Virtualization 2 of 4 – Graphics Computing"},{"content":"In broad terms, virtualization of computing resource is about isolation of resources, at different levels. There are five levels of virtualization:\nApplication level, such as JVM, .NET CLR Library (user-level API) level Operating system level, such as LXC, Docker, OpenVZ Hardware abstraction layer (HAL) level, such as VMware, Xen, etc Instruction set architecture (ISA) level In my context I deal mostly with OS level and HAL (hardware abstraction layer) level of virtualization. In loose terms, the word containerization refers to OS level virtualization, while the word virtualization is exclusively reserved for HAL level virtualization, also referred to as hypervisor-based virtualization. This post will just focus on this family of technology and loosely refers to it as virtualization.\nVirtualization technology evolved from on-premise data centre environment and now is the backbone of cloud computing. The challenges of IT operation in the era of virtualization involves managing VM sprawling, investigating performance issues, planning capacity and addressing storage I/O block. The idea of virtualization is sharing (thus isolating) resources for better utilization, leading to better return on investment. This posting is to cover only the very basics of virtualization.\nHypervisor Hypervisor is the software layer which provides the capability to run multiple virtual machines on the same physical host. It is broken down into two categories:\nType I hypervisor (aka bare metal hypervisor): directly run on physical hardware. They control the hardware as well as manage the virtual machines. For example, Linux KVM, VMware ESXi, Xen and Microsoft Hyper-V Type II hypervisor: runs as an application or service on top of the host operating system, which is installed on the bare metal. Guest operating system calls need to traverse via the host operating system stack to reach hardware resource. For example, Oracle Virtual Box, VMware Fusion and Linux Containers (LXC) Hypervisor Types Virtualization Techniques The most primitive form of technology that can be arguably categorized under virtualization is hardware emulation, where a piece of (more accessible) hardware imitates another (less accessible). The architecture limits itself in functional testing only, and is not built for performance or production at all.\nThe original virtualization technology deals with CPU and memory virtualization. In this well-written whitepaper fromVMware, there are three CPU virtualization techniques introduced for x86 architecture.\nThe x86 architecture offers four levels of privilege known as Ring 0,1,2 and 3 to operating system and applications to manage access to the computer hardware. User-level applications typically run in Ring 3, the OS must execute its privileged instructions in Ring 0 since it needs to have direct access to memory and hardware. The two main challenges with virtualizing x86 architecture are:\nA virtualization layer between hardware operating system who expects Ring 0 privilege; Some instructions with different semantics when not executed in Ring 0 cannot be virtualized effectively. They need to be translated at runtime. These challenges makes true virtualization of x86 architecture impossible and thus VMware developed three alternative technologies.\nFull virtualization (using binary translation): virtual machine presents a complete simulation of the actual hardware environment so that an unmodified guest OS can run in isolation. The Guest OS is not aware that the underlying environment it is running on is virtualized, and issues hardware calls to communicate with (what it thinks as) hardware. The virtual processors have to understand guest CPU instruction, and reproduce the equivalent CPU instructions of the host machine. VMware\u0026#8217;s technology to address this is called Binary Translation. This overhead makes true full virtualization difficult to achieve. In real life, a virtual environment that provides \u0026#8220;enough representation of the underlying hardware\u0026#8221; can be considered to provide full virtualization as long as it allows guest OS to run without modification. Full virtualization comes with a performance penalty. Paravirtualization (aka OS assisted virtualization): refers to communication between the guest OS and the hypervisor to improve performance and efficiency. In this technology, guest OS is modified with an interface to host hardware to be able to communicate and operate seamlessly. Since the guest OS is modified, the VM does not need to be a complete simulation of the hardware. The modified guest OS knows it is running on a virtualized environment, and (vm driver) makes API calls (known as \u0026#8216;hyper calls\u0026#8217;) to the hypervisor. This allows para-virtualization technology to achieve performance closer to non-virtualized environment. However, since paravirtualization cannot support unmodified operating systems, its compatibility and portability is poor. Hardware-Assisted Virtualization: hardware vendors such as Intel and AMD both have developed extensions (new features) to simplify virtualization techniques, for example, the introduction of privileged instructions with new CPU execution mode feature to allow hypervisor to run in a new root mode below ring 0. This removed the need for full virtualization and paravirtualization. With VMware originally as a promoter of full virtualization and Xen for paravirtualization, most virtualization technologies today utilizes hardware-assisted virtualization feature, for example, Linux KVM, VMware workstation, VMware fusion, Xen, VirtualBox, etc. Intel\u0026#8217;s virtualization extension is VT-x. AMD\u0026#8217;s counterpart is AMD-V technology.\nTo virtualize memory, another level of memory virtualization is required (similar to the virtual memory support in Linux). Hypervisor is responsible for mapping guest physical memory to the actual machine memory, and it uses shadow page tables to accelerate the mappings, usually at a performance cost.\nPopular hypervisors On the market there are a few popular hypervisor technologies. They are all type 1 hypervisors:\nXen is an open-source hypervisor project originally developed in Cambridge University, licensed under GPLv2. . Based on that, Citrix developed its commercial product XenServer, a bare-metal virtualization platform with enterprise-grade features for x86 and AMD environments. Oracle VM is another commercial implementation of Xen. The Xen project also supports many cloud platforms such as Openstack, Cloudstac, etc. Xen project supports paravirtualization (Xen-PV) as well as hardware-assisted virtualization (Xen-HVM) for virtualization of X86, IA64, ARM and other CPU architectures. The earlier versions does not support memory overcommit (aka \u0026#8220;dynamic memory optimization\u0026#8221;, \u0026#8220;memory ballooning\u0026#8220;, or as Citrix calls it \u0026#8220;dynamic memory control, DMC\u0026#8221;). This delivers better performance but also has higher budgetary requirement on hardware since there isn\u0026#8217;t room for over-subscription. Hyper-V is a Microsoft product. It executes in high CPU privilege (Microsoft calls it ring -1 which is equivalent to root mode as Intel calls it). On the guest VM, OS kernel and drivers run in ring 0, application rin in ring 3. This eliminates the need for binary translation. Hyper-V does not support memory overcommit either. Hyper-V is well integrated with Windows platform. It supports Linux as well although with some performance penalty.\nLinux KVM (Kernel-based Virtual Machine) is a full open-source virtualization solution for GNU/Linux. What makes KVM a special hypervisor is that it uses a loadable kernel module kvm.ko that turns itself into a hypervisor and provides VMs with direct access to the hardware. So it is a type 1 hypervisor despite of the presence of Linux OS. KVM also contains a processor specific module, kvm-intel.ko or kvm-amd.ko. KVM leverages qemu to access devices. Because KVM runs as a process inside of Linux OS, KVM can use many existing feature in Linux kernel. Redhat has an enterprise solution based on KVM.\nXen vs KVM VMware ESXi is VMware\u0026#8217;s premium hypervisor product (not open-source) and is available for free download, although the advanced features are not free. (Update no free download link anymore.) VMware vSphere is virtualization platform built on top of ESXi, including a whole family of virtualization products.\nMarket segments and players Virtualization involves many market segments such as virtual desktop infrastructure (VDI, for desktop virtualization), server virtualization is the predominant domain in the virtualization of data centre environment. This effort led to Hyper-Converged Infrastructure (HCI) where almost all the traditional hardware resources are software-defined through the virtualization layer. The management of infrastructure is abstracted away from the physical hardware management. The three most fundamental areas in HCI are:\nServer (compute) virtualization: the previous section covers the virtualization of memory and x86 CPU, which are the main focus on computing resource virtualization. Additionally, graphics computing resources can be virtualized today. Example products include: VMware vShpere (compute virtualization based on ESXi hypervisor).\nStorage Virtualization: the technology to abstract physical data storage resource to make them appear as if they were a centralized resource. Storage virtualization takes place at three levels depending on the use case: block-level, file-level and object level. Example products include: VMWare vSAN (vSphere-native storage), HPE 3PAR (Tier-1 storage), EMC VxRail, PureStorage Flash Array (Tier 1), etc. Storage Virtualization enables Software-Defined Storage (SDS), the provisioning and management of data storage independent of the underlying hardware.\u0026nbsp;\nNetwork Virtualization: the technology to abstract network resources that were traditionally delivered in hardware to software. Network virtualization decouples network services from the underlying hardware management and allows virtual provisioning of an entire network. VLAN is a classic example of network virtualization. There are also various overlay technologies such as VXLAN, which provides an industry framework for overlaying virtualized layer 2 network over layer 3 network (used in Docker network) using an encapsulation mechanism and a control plane. Example products include: VMware NSX Data Center (L2-L7 network and security virtualization platform), Cisco ACI, Palo Alto Panorama. Network Virtualization enables Software-Defined Network (SDN), an approach to network management that enables dynamic, programmatically efficient network configuration in order to improve network performance and monitoring, making it more like cloud computing than traditional network management.\nDelivery model Virtualization allows managed service providers (MSPs) to deliver IT service in the following three models:\nIaas (Infrastructure as a Service): MSP delivers VM to customers. PaaS (Platform as a Service): MSP delivers environments to customers (e.g. Database as a Service, managed RabbitMQ service, etc). SaaS (Software as a Service): MSP delivers entire application for the customer. IT service delivery models enabled by virtualization technology Since virtualization is the backbone of cloud computing. This model is also referred to as cloud computing delivery model.\nVirtualization and Containerization These two concepts are similar and could be confusing to beginners. Both provide a mechanism to isolate computing resource for different applications, for the purpose of higher utilization of resource. The difference lies in how and where the isolation is made. Virtualization requires a guest operating system per VM (OS level isolation), whereas the container technology isolates application processes along with its runtime into a container (dependency level isolation), using some new Linux kernel features such as namespaces and cgroups. All containers make their system calls to the container engine on the host operating system. So they share a kernel on the same host. In this sense, container engine running on OS could be considered as type 2 hypervisor.\nFrom VMs to containers VMware is a major player in enterprise data centre virtualization, which is facing fierce competition from public and private cloud vendors. VMware also has its own private cloud services. Docker is the most popular container technology that conforms to the specifications of Open Container Initiative (OCI), a governance structure for industry standards around container formats and runtimes.\nVirtualization and Cloud Among public cloud vendors, AWS EC2 used Xen PV and Xen HVM in its earlier implementations. It has transitioned to AWS bare metal. The history is well summarized here. Microsoft Azure runs Azure Hypervisor as the native hypervisor in Azure Cloud Services platform. It is a customized version of Microsoft Hyper-V specifically for Azure platform. With GCP, Google Compute Engine (GCE) instance runs VMs on KVM as hypervisor. It can also enable nested virtualization.\nThe scope of cloud computing is evolving overtime. It originally only refers to a business model of offering IT services (in one of the three delivery models outlined above) based on virtualization technology. Therefore I cannot make comparison between a technology and a business model. Today, with public cloud vendor extending their offerings (with various managed services and platforms) and people\u0026#8217;s misuse of the terms, the buzz-word \u0026#8220;cloud\u0026#8221; seems to suggest anything that is offered in public cloud service. The essence still remain the same where managed services and managed platforms are built on top of virtualized compute unit under the hood, which are driven by virtualization technologies.\nPrevious PostKafka high-level Overview Next PostVirtualization 2 of 4 – Graphics Computing ","date":"2020-07-27T22:52:00-04:00","permalink":"/2020/07/overview-of-virtualization/","title":"Virtualization 1 of 4 – Hypervisor"},{"content":"Zookeeper General definition of distributed system: a software system that is composed of independent computing entities linked together by a computer network whose components communicate and coordinate with each other to achieve a common computational goal. Implementing coordination among components of a distributed system is hard. For example, designated master node becomes single point of failure; cluster needs to detect availability of new nodes as it joins cluster.\nZookeeper is designed to simplify cluster coordination. Zookeeper implements key aspects in cluster coordination, such as distributed consensus, group management, presence protocols and leader election. In order to coordinate a cluster, zookeeper itself also runs in its own cluster, called ensemble. Zookeeper exposes a simple but powerful interface of primitives. Applications can be designed on these primitives implemented through ZooKeeper APIs to solve the problems of distributed synchronization, cluster configuration management, group membership, etc.\nZookeeper Ensemble Clients can connect to a Zookeeper service by connecting to any member of the ensemble. The members of the ensemble are aware of each other\u0026#8217;s state. As long as a majority of the nodes are available, the service will be available. Zookeeper cli (zkCli.sh) can be used to connect to Zookeeper server. they can be downloaded from here.\nZookeeper is integrated with many other services apart from Kafka, such as Nifi and Hadoop.\nKafka Kafka is a messaging system that is horizontally scalable, fault tolerant. It can also serve as queue storage system and stream processing system. It is distributed and use Zookeeper for cluster coordination. Each node is called a broker.\nTopics in Kafka (think of table in database) is a category or feed name to which messages (records) are published. Topic is broken up into ordered commit logs called partitions. Each partition has an ID. Each message in a partition is assigned an offset. Topics that are created in Kafka are distributed across brokers based on the partition, replication, and other factors. Each partition is replicated across several brokers depending on replication factor. For each partition, Kafka elect one replica as the leader of partition.\nWrites to a partition is generally sequential. Reading messages can either be from the beginning, or rewind or skip to any port in partition given an offset value. Data in a topic is retained for a configurable period of time. A message is a unit of data in Kafka, in the format of key-value pair. A key is used to control the message that is to be written to partitions. Messages with the same keys are always written to the same partition (hash map)\nA producer publishes new message to a topic. Producers do not care which partition the message is written to and will balance messages over every partition of a topic evenly. Directing messages to a partition is done using the message key and a partitioner, this will generate a hash of the key and map it to a partition.\nA consumer is subscribed to one or more topics and read messages sequentially. The consumer keeps track of messages it has consumed by keeping track on the offset of the message. The offset is a bit of metadata (an integer value that continually increases) that kafka adds to each message. Each partition has a unique offset which is stored with the offset of the last consumed message. A consumer can stop and start without losing its current state.\nA Kafka broker is designed to operate as part of a cluster. One broker in the cluster also function as the cluster\u0026#8217;s controller, which is responsible for administrative operations such as: assigning partitions to brokers; monitoring for broker failures in cluster. A particular partition is owned by a broker and that broker is called the leader of the partition.\nAll consumers and producers operating on that partition must connect to the leader.\nKafka cluster may replicate across cluster using MirrorMaker.\nReference: Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale\nPrevious PostHow RPC and NFS work Next PostVirtualization 1 of 4 – Hypervisor ","date":"2020-07-21T23:19:00-04:00","permalink":"/2020/07/zookeeper-and-kafka-overview/","title":"Kafka high-level Overview"},{"content":"I touched on NFS in several previous postings, and here is a deeper dive of this particular protocol. NFS is built on top of Remote Procedure Call (RPC) and therefore it is important to understand RPC first. In fact NFS is one of the most prominent user of RPC and the best example for learning RPC.\nRPC overview According to Wikipedia, an RPC is when a computer program causes a procedure to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote. This is a form of client–server interaction (caller is client, executor is server), typically implemented via a request–response message-passing system. In the object-oriented programming paradigm, RPCs are represented by remote method invocation (RMI), such as Java RMI API.\nRPCs are a form of inter-process communication (IPC), in that other processes have a different address spaces: if on the same host machine, they have distinct virtual address spaces, even though the physical address space is the same; while if they are on different hosts, the physical address space is different. RPC is a\u0026nbsp;request–response\u0026nbsp;protocol, and therefore synchronous. An RPC is initiated by the\u0026nbsp;client, which sends a request message to a known remote\u0026nbsp;server\u0026nbsp;to execute a specified procedure with supplied parameters. The remote server sends a response to the client, and the application continues its process. While the server is processing the call, the client is blocked (it waits until the server has finished processing before resuming execution), unless the client sends an asynchronous request to the server. There are many variations and subtleties in various implementations, resulting in a variety of different (incompatible) RPC protocols.\nNFS overview NFS is defined as a set of RPCs, including their arguments, results and effects. RPC makes NFS protocol transparent. RPC is also stateless so the server does not keep the state of RPCs once the request has been served. Each RPC contains the necessary information to complete the call. In the event of server failure, client will need to resubmit requests. NFS has several versions, with v3 and v4 most popular. We will focus on v3 in this posting and brief on v4.\nPortmap RPC makes a remote call appears to client application as a local call, with the help of portmap. The utility for portmap is rpcbind. In RedHat/CentOS 5 or prior, it was even simply called portmap but they are essentially the same service for RPC port mapper. The rpcbind service is required on both NFS client and NFS server. On the client, it talks to client application, as well as its counterpart on the server. Its main function is query its counterpart on the server, providing a RPC program number, and expects a port number in return. On the server, the rpcbind utility listens at port 111, waiting for request with RPC program number(service), and return the TCP or UDP port number on which the requested service is hosted. RPC program number is reserved numeric indicator of services as outlined in RFC5531. For example, 100005 for mountd, 100021 for nlockmgr, and 100003 for nfs. The port that rpcbind service itself listens on is always at port 111, which is known to both client and server. This is also referred to as portmapper daemon. Other than this fixed port, each NFS-related service (with their respect reserved program number) may be hosted on different ports on the server. The client rpcbind service must first look up for the port for the requested program number, then it directs the client to initiate connection to the specified port for the specific service. With the following command rpcinfo command you may look up the program to port number mapping on destination server isilon.company.com\n# rpcinfo -p isilon.company.com The following command is commonly used for displaying mountpoint and troubleshooting mount. Under the hood it is using the information from rpcinfo.\nshowmount -e isilon.dcb.digitalhunch.com On the NFS server side, rpcbind service must start before nfs service start. Otherwise nfs service cannot register ports to rpcbind. If you restart rpcbind, every service that had registered to rpcbind must restart to register themselves again. By default, NFS server can dynamically assign a port for mountd, nlockmgr within a port range. This makes firewall setting a pain. These dynamically assigned port can be fixed via configuration files. File handle NFS uses file handles (or fhandle) to represent files. It is a better mechanism to reference a file object than pathname for three reasons: 1. file handle has fixed length (32bytes); 2. If the file is renamed, the file handle reference remain the same; 3. If a file is deleted, and then a new file is created with the same path, a new file handle will be created. A file handle has three parts:\nvolume ID: to identify the mounted file systeminode #: to identify the file within the mounted file systemgeneration #: to detect when file handle refers to an older version of inode. Traditional Unix filesystems may reuse inode and thus NFS client could mistakenly use an old file handle and access a new file. The file handle information is only meaningful to the server. New file handles are returned to client by certain procedures, such as LOOKUP, CREATE, and MKDIR. The file handle for the root of the file system, is obtained by the client when it mounts the file system, as permission allows.\nPermission and Locking When accessing a file on server, the client passes uid/gid info in RPCs, and the server performs permission checks as if the user was performing the operation locally. So users and groups are represented as integers. There are two security problems:\nThe mapping from uid/gid to user must be the same on all clients. This is not practical in large deployment, although can be solved via Network Information Service (NIS);Whether the root user on the client has root access to files on the server, is a server policy configuration. This can be addressed by enabling \u0026#8220;root squashing\u0026#8221; on server, so that client\u0026#8217;s uid 0 (root) is mapped to 65534 (nobody). Unix has two locking mechanisms (fcntl and flock). NFS protocol supports fcntl but not flock. The flock function is managed by a separate service (nfslock) to allow NFS to lock files. The nfslock daemon provides the ability to lock regions of NFS files. NFS service itself is still completely stateless with locking managed separately. This is changed in NFSv4.\nProcedures used in NFS service NFS service defines a list of procedures. Here is a list with brief summary of activities. The bottom five RPCs are introduced in v3.\nProcedureActivityGETATTR(fh)Returns the attributes of a file, similar to stat syscall.SETATTR(fh, attr)Sets the attributes of a file (mode, uid, gid, size, atime, mtime); setting the size to 0 truncates the fileSTATFS(fh)Returns the status of a filesystem, such as block size, number of free blocks. e.g. df command.LOOKUP (dirfh, name)Returns fhandle and attributes for the named file in the directory specified by dirfhREAD (fh, offset, count)Reads from a file, with offset and count specified. In v2, the length is up to 8192 bytes; v3 support more.WRITE (fh, offset, count, data)Writes to a file, with offset and count specified, as well as a separate field called data. Returns the new attributes of the file after the write.CREATE (dirfh, name, attr)Creates a file with the name, in directory, returns new fhandle and attributesREMOVE (dirfh, name)Deletes the named file in from directory dirfh and returns status.RENAME (dirfh, name, tofh, toname)Renames name in directory dirfh, to toname in directory tofh.LINK (dirfh, name, tofh, toname)Creates a hard link toname, in directory tofh, that points to name, indirectory dirfh.SYMLINK (dirfh, name, string)Creates a symbolic link name, in the directory dirfh, with value string.READLINK (fh)Reads a symbolic link and get file name of the target.MKDIR (dirfh, name, attr)Creates a directory name in the directory dirfh, and returns the new fh and attributes.RMDIR(dirfh, name)Removes a directory with the name, from parent directory dirfh.READDIR (dirfh, cookie, count)Reads a directory and returns up to count bytes of directory entries from the directory dirfh. The cookie is used in subsequent readdir calls to start reading at a specific entry in the directory. Cookie of zero get the server to start with the first entry in the directory.NULLNo activity. Used for testing only.ACCESSHelps with client caching.MKNODMakes a device special file.FSINFOReturns information about the server\u0026#8217;s capabilities. READDIRPLUSReturns both file handle and attributes to eliminate LOOKUP calls when scanning a directoryCOMMITIn NFSv3, the server can reply to WRITE RPCs immediately without syncing to disk. When client wants to ensure that the data is on stable storage, it sends a COMMIT RPC. This is used in asynchronous writes for better performance, which is an option negotiated at mount time. The addition of COMMIT procedure in v3 offers the option to improve write performance in place of synchronous write. However, asynchronous write requires more coordination to ensure data integrity during transmission, in the event of server crash. NFSv3 uses write verifier for this purpose. A write verifier is an 8-bye value that the server must change if it crashes. After an asynchronous write, the reply from WRITE RPC includes a write verifier, the client must keep it for later use;The client then sends a COMMIT RPC and the reply contains another write verifier;The client compares the verifiers from the two returns for crash detection. If the verifiers don\u0026#8217;t match, the client must rewrite all uncommitted data.The client must keep all uncommitted data in case of a server crash. Additional daemon processes In addition to the three essential services, (nfs, rpcbind and nfslock), there are several auxiliary processes that facilitates NFS services. Their functions are listed here:\nProcessDescriptionrpc.mountdUsed by NFS server to process MOUNT requests from NFSv3 client. It checks that the requested NFS share is currently exported by the NFS server, and that the client is allowed to access it. If the mount request is allowed, the rpc.mountd server replies with a Success status and provides the File-Handle for this NFS share back to the NFS client.rpc.nfsdAllows explicit NFS versions and protocols the server advertises to be defined. It works with the Linux kernel to meet the dynamic demands of NFS clients, such as providing server threads each time an NFS client connects. This process corresponds to the nfs service.rpc.lockdA kernel thread which runs on both clients and servers. It implements the Network Lock Manager (NLM) protocol, which allows NFSv3 clients to lock files on the server, using procedures such as NLM_NULL, NLM_TEST, NLM_LOCK, NLM_GRANTED, NLM_UNLOCK, NLM_FREE. The service is started automatically whenever the NFS server is run and whenever an NFS file system is mounted.rpc.statdThis process implements the Network Status Monitor (NSM) RPC protocol, which notifies NFS clients when an NFS server is restarted without being gracefully brought down. rpc.statd is started automatically by the nfslock service, and does not require user configuration. This is not used with NFSv4.rpc.rquotadThis process provides user quota information for remote users. rpc.rquotad is started automatically by the nfs service and does not require user configuration.rpc.idmapdprovides NFSv4 client and server upcalls, which map between on-the-wire NFSv4 names (strings in the form of user@domain) and local UIDs and GIDs. For idmapd to function with NFSv4, the /etc/idmapd.conf file must be configured. At a minimum, the \u0026#8220;Domain\u0026#8221; parameter should be specified, which defines the NFSv4 mapping domain. If the NFSv4 mapping domain is the same as the DNS domain name, this parameter can be skipped. The client and server must agree on the NFSv4 mapping domain for ID mapping to function properly. NFSv4 Even NFSv4 was introduced in 20 years ago, it improves access and performance of NFS on the Internet. It should be the default option for any new deployment.\nNFSv4 is TCP only protocol and it is stateful. NFSv4 combines mount and lock protocols into NFS so only one port is being used. Users and groups are identified with strings (user@domain, or group@domain where domain represents a registered DNS domain or sub-domain), instead of integers. The access control policies are compatible with both Unix and Windows.NFSv4 mandates strong RPC security built on cryptography, with negotiation at the time of mountNFSv4 adopted a framework for authentication, integrity and privacy at RPC levelIntroduced new RPC COMPOUND, which allows for several operations in one go. At the server, operations are evaluated in order, and each has a return value. NFSv4.1 was release in 2010, and 4.2 in 2016. Both AWS EFS and Azure File storage supports 4.1.\nPrevious PostEMC Isilon storage product Next PostKafka high-level Overview ","date":"2020-07-15T10:45:00-04:00","permalink":"/2020/07/nfs-network-file-system-and-rpc-remote-procedure-call/","title":"How RPC and NFS work"},{"content":"EMC has several product lines for different use cases in enterprise data storage. Like may other IT solutions, the website is clouded with marketing terms and slogans, and is purposefully not technical. This makes it difficult for technical staff to grasp the advantage of its product in a glimpse. I personally have to know their product (mostly with Isilon and ECS) well in order to make integration decisions. So I\u0026#8217;m putting together this note (updated as of July 2020), with lots of details from their technical white paper.\nOverview of EMC storage At the highest level, the EMC enterprise data storage product lines are categorized into two groups: primary storage (along the lines of block-level storage) and unstructured storage (mostly file and object storage). The primary storage includes the following product:\nPowerMax for OLTP database (Oracle, MicrosoftSQL and SAP) PowerFlex: for Software defined storage, Oracle RAC, Elastic Stack, Kubernetes, Splunk XtremIO for VMware, VDI, SAP PowerStore for Database, VMware PowerVault for Entry-level SAN and DAS environment This post only expands on the unstructured storage product line, which mainly consists of PowerScale and ECS. ECS (elastic cloud storage) is EMC\u0026#8217;s object storage. PowerScale (aka Isilon) is scale-out NAS platform for high-volume storage (up to 50 PB in a single file system), backup and archiving of unstructured data. For the rest of this post, I will still refer to PowerScale as Isilon. Dell\u0026#8217;s official support website is the most resourceful place to get information. For example, when I want to read about Isilon. I start with Dell support, then click on knowledgebase at the top, then go to \u0026#8220;servers, storage and networking\u0026#8220;, then \u0026#8220;storage technical documents and videos\u0026#8220;. There I can select a productline such as Isilon.\nOverview of Isilon Family Isilon is a clustered storage system consisting of three or more nodes. A node is a server with OneFS as its operating system. Based on FreeBSD, OneFS is EMC\u0026#8217;s proprietary operating system to unify a cluster of nodes into a single shared resource. So OneFS is for Isilon only. It is the basis of Isilon. Isilon has three series of products:\nF series: F200, F600, F800 and F810. H series: typical models are H400, H500 and H600, which seeks to balance performance and capacity A series: typical models are A200 and A2000 for active and deep archive storage In June 2020, Dell decoupled OneFS software (with 9.0 released) from server hardware (referred to as PowerScale). Going forward EMC will refer to Isilon as PowerScale for OneFS version newer than 9.0 in spec sheets and white papers.\nF200 is the cost-effective choice with SSD for remote office, small hospital, retail outlets, IOT or factory floor. F600 uses NVMe drives instead, and has more ECC memory and faster ethernet backend network. and is higher than F200 in its use case. Both F200 and F600 provide inline data compression and deduplication capabilities. F800 and F810 both use SSD and they are similar. F800 comes with InfiniBand backend network and F810 provides inline data compression and deduplication capabilities. H series tries to strike a balance between performance and capacity so they are pretty much everything in betwee. On the other end, A200 and A2000 are almost the same except for capacity difference.\nIsilon\u0026#8217;s advantage Isilon has lots of intelligence built into its solution compared to a traditional NAS. Here are some aspects from its product white paper:\nAspects of DesignIsilon OneFS Scale-Out NASTraditional NASNetworkSeparation of front-end and back-end network to isolate node-to-node communication to a private low-latency network. Front-end traffic load balanced with SmartConnectSingle network for both external and internal trafficFile system structure and NameSpaceThe storage is completely virtualized to users as a truly single file system with one namespace. There is no partitioning or volumes. The single file tree can grow organically without requiring planning or oversight about how the tree grows. SmartPool handles tiering of files to appropriate disk, without disrupting the single file tree.An appearance of single namespace is typically achieved through namespace aggregation, where files are still managed in separate volumes, and a simple \u0026#8220;veneer\u0026#8221; layer glues individual directories to a \u0026#8220;top-level\u0026#8221; tree via symbolic links. LUNs and volumes, as well as volume limits are still present. Files have to be manually moved from volume-to-volume to load-balance.Data LayoutOneFS controls the placement of file directly, down to the sector-level on any drive anywhere in the cluster. The addressing scheme for data and metadata is indexed at physical level by a tuple of {node, drive, offset}Data are sent through RAID and volume management layers, introducing inefficiencies in data layout and providing non-optimized block access. Redundancy ControlOneFS can flexibly control the type of striping as well as the redundancy level of the storage system at the system, directory and even file-levels.The entire RAID volume is dedicated to a particular performance type and protection setting. Isilon terms The Isilon technology re-implemented the read and write path during file storage and introduced several terms along with its technology.\nSmartPools \u0026#8211; Job that runs and moves data between the tiers of nodes within the same cluster. Also executes the CloudPools functionality if licensed and configured. FilePolicy is changelist-based SmartPools file pool policy job. SmartPoolsTree enforces SmartPools file policies on a subtree. Storage Pools \u0026#8211; Storage pools provide the ability to define subsets of hardware within a single cluster, allowing file layout to be aligned with specific sets of nodes through the configuration of storage pool policies. The notion of Storage pools is an abstraction that encompasses disk pools, node pools, and tiers.\nDisk Pools \u0026#8211; Disk pools are the smallest unit within the storage pools hierarchy. OneFS provisioning works on the premise of dividing similar nodes’ drives into sets, or disk pools, with each pool representing a separate failure domain. Disk pools are laid out across all five sleds in each node.\nNode Pools \u0026#8211; groups of disk pools, spread across similar storage nodes (or equivalent classes). Multiple groups of different node types can work together in a single, heterogeneous cluster. For example, one node pool of all-flash F-Series anodes, one node pool of H-series, and one node pool of A-series. Each node pool only contains disk pools from the same type of storage nodes.\nTiers \u0026#8211; groups of nodepools combined into a logical superset to optimize data storage, according to OneFS platform type. this allows customers who consistently purchase highest capacity nodes available to consolidate a variety of node styles within a single tier, and manage them as one logical group. SmartPools users typically deploy 2 to 4 tiers. different node pools under a tier needs to be compatible.\nGlobal Namespace Acceleration (GNA)\u0026#8217;s principal goal is to help accelerate metadata read operations by keeping a copy of a cluster\u0026#8217;s metadata on high performance, low latency SSD media.\nSmartConnect is a load balancer that works at the front-end Ethernet layer to evenly distribute client connections across the cluster. SmartConnect supports dynamic NFS failover and failback to ensure that when a node failure occurs, or preventative maintenance is performed, all in-flight reads and writes are handed off to another node in the cluster to finish its operation without any user or application interruption.\nAuto Balance reallocates and rebalances data and make storage space more usable and efficient.\nSmartQuotas is directory-level quota management. Note: there is no partitioning, and no need for volume creation in OneFS.\nSmartRead creates a data \u0026#8220;pipeline\u0026#8221; from L2 cache, prefetching into a local \u0026#8220;L1\u0026#8221; cache, on the captain node, in order to greatly improve sequential-read performance. For high-sequential cases, SmartRead can very aggressively prefetch ahead. SmartRead can control how aggresive the pre-fetching is, and how long data stays in the cache, and optimizes where data is cached.\nIn-line Data Reduction \u0026#8211; the write path involves zero block removal, in-line deduplication, and in-line compression. This is supported in some models only.\nSmart Dedupe \u0026#8211; post-process, asynchronous deduplication. Smart Dedupe scans the on-disk data for identical blcoks and then eliminate the duplicates. After duplicate blocks are discovered, SmartDedupe movees a single copy of those blocks to a special set of files known as shadow stored. With post-process deduplication, new data is first stored on the storage device and then a subsequent process analyzes the data looking for commonality. This means that initial file write or modify performance is not impacted, since no additional computation is required in the write path, as opposed to in-line deduplication. This is supported on some models only.\nOneFS SSD strategy \u0026#8211; How OneFS leverage the SSD for performance. It has these options:\nL3 cache (implemented at nodepool level) metadata read metadata read/write Global Namespace Acceleration (GNA) Data on SSD L3 cache consumes all the SSD in node pool. L3 cannot coexist with other SSD strategies, with the exception of GNA just because L3 cache node pool SSD cannot participate in GNA.\nIsilon\u0026#8217;s High Availability The OneFS is distributed across all nodes in the cluster and is accessible by clients connecting to any node in the cluster. Metadata and locking tasks are managed by all nodes collectively and equally in a peer-to-peer architecture. This symmetry is key to the simplicity and resiliency of the architecture. There is no single metadata server, lock manager or gateway node.\nThe entire cluster forms a single file system with a single namespace that runs across every node equally. No one node controls or \u0026#8220;masters\u0026#8221; the cluster; all nodes are true peers.\nDuring failover, clients are evenly redistributed across all remaining nodes in the cluster, ensuring minimal performance impact. If a node is brought down for any reason, including a failure, the virtual IP addresses on that node is seamlessly migrated to another node in the cluster. When the offline node is brought back online, SmartConnect automatically rebalances the NFS and SMB3 clients across the entire cluster to ensure maximum storage and performance utilization. This functionality allows for per-node rolling upgrades affording full-availability throughout the duration of the maintenance window.\nThere are two logical roles in processing an I/O request from client:\nThe initiator: the node that the client connects to with front-end protocol. The initiator acts as the \u0026#8216;captain\u0026#8217; for the entire I/O operation. The participant: Every node in the cluster is a participant for a particular I/O operation. File Write in Isilon OneFS employs a patented transaction system during write to eliminate single point of failure. In a write operation, the initiator \u0026#8220;captains\u0026#8221; or orchestrates the layout of data and metadata, the creation of erasure codes, and the normal operations of lock management and permission control.\nWhen a client connects to a node to write a file, it is connecting to the Initiator. OneFS breaks the file down into atomic units. An atomic unit is a smaller logical chunk of data, also called stripe, or protection groups in the context of data protection. The size of each file chunk is referred to as the stripe unit size. After this division, OneFS then write the stripe individually to the Participant (with disks). This design ensures that data is protected at the specified level as soon as it is being written. Redundancy is built into protection groups, such that if every protection group of a file is safe, then the entire file is safe. In terms of protection mechanism, OneFS can use either Reed-Solomon erasure coding system, or simply mirroring for data protection. Erasure coding is the predominant mechanism with very high performance without sacrificing on-disk efficiency.\nThe initiator node uses a modified two-phase commit transaction to safely distribute writes to multiple NVRAMs across the cluster. As client initiates write to OneFS cluster, instead of immediately writing to disk, OneFS temporarily writes the data to an NVRAM-based journal cache on the initiator node and acknowledge the write the client. As outlined above, these writes are also mirrored to participant nodes\u0026#8217; NVRANM journals to satisfy the file\u0026#8217;s protection requirement. Later, at a more convenient time, OneFS then flush these cached writes to disks asynchronously.\nSince NVRADM journals all the transactions that are occurring across every node in the storage cluster. If a node fails mid-transaction, and then re-joins the cluster, the uncommitted cached writes are fully protected, and the only required actions for the node, are to replay its journal from NVRAM, and occasionally for AutoBalance to rebalance files that were involved in the transaction. Writes are never blocked due to a failure. There is no \u0026#8216;fsck\u0026#8217; or \u0026#8216;disk-check\u0026#8217; process.\nOneFS file system block size is 8KB. A file smaller than 8KB will use a full 8KB block. For larger files, OneFS can maximize sequential performance by taking advantage of a stripe unit consisting of 16 contiguous blocks, for a total of 128KB per stripe unit.\nCache in Isilon OneFS aggregates the cache present on each node in a cluster into one globally accessible pool of memory by using a messaging system similar to NUMA (non-uniform memory access). This allows all the nodes\u0026#8217; memory cache to be available to each and every node in the cluster. Remote memory is access over internal network with much lower latency than accessing hard disk drives. The internal network as distributed system bus, is a redundant, under-subscribed flat Ethernet up to 40Gb. The oneFS caching subsystem is coherent across the cluster, due to the use of MESI protocol to maintain cache coherency. If the same content exists in the private caches of multiple nodes, this cached data is consistent across all instances.\nOneFS uses up to three levels of read cache, plus an NVRAM-backed write cache, or coalescer.\nOneFS Caching Hierarchy L1 cache \u0026#8211; prefetches data from remote nodes. Data is prefetched per file, and this is optimized in order to reduce the latency associated with the nodes’ back-end network. The L1 cache refers to memory on the same node as the initiator. It is only accessible to the local node, and typically the cache is not the master copy of the data.\nL1 is also known as remote cache because it contains data retrieved from other nodes in the cluster. It is coherent across the cluster but is used only by the node on which it resides and is not accessible by other nodes. Data in L1 cache on storage nodes is aggressively discarded after it is used. L1 cache uses file-based addressing, in which data is accessed via an offset into a file object.\nOneFS also uses a dedicated inode cache in which recently requested inodes are kept. The inode cache frequently has a large impact on performance, because clients often cache data, and many network I/O activities are primarily requests for file attributes and metadata, which can be quickly returned from the cached inode.\nL2 cache (backend cache) refers to local memory on the node on which a particular block of data is stored. L2 cache is globally accessible from any node in the cluster and is used to reduce the latency of a read operation by not requiring a seek directly from the disk drives.\nL2 cache is also known as local cache because it contains data retrieved from disk drives located on that node and then made available for requests from remote nodes. Data in L2 cache is evicted according to a Least Recently Used (LRU) algorithm.\nL3 cache, or Smart Flash, is configurable on nodes that contain solid state drives. Smart Flash (L3) is an eviction cache that is populated by L2 cache blocks as they are aged out from memory.\nDuring I/O request, clients talk to L1 cache and write coalescer; L1 cache talks to L2 cache on all cluster nodes. L2 cache buffers to and from disks. L3 cache is optionally enabled per node pool, as an extension from L2. L3 and L2 communicate in backend network.\nNameMediumDescriptionL1 Cache (aka front-end cache or remote cache)RAM (volatile)holds clean, cluster coherent copies of file system data and metadata blocks requested by clients via front-end networkL2 Cache (aka back-end cache or local cache)RAM (volatile)contains clean copies of file system data and metadata on a local nodeSmartCache (Write Coalescer)Battery-backed NVRAM (Persistent)a persistent journal cache that buffers any pending writes to front-end files that have not been committed to diskSmartFlash or L3 CacheSSD (persistent)contains file data and metadata blocks evicted from L2 cache, effectively increasing L2 cache capacity File Read in Isilon The high-level steps for fulfilling a read request with cache interaction involves:\nStep 1 \u0026#8211; on local node, determine whether part of the requested data is in the local L1 cache:\nif so, return to client if not, request data from remote nodes Step 2 \u0026#8211; on remote nodes, determine whether requested data is in the local L2 or L3 cache:\nif so, return to the requesting node if not, read from disk and return to requesting node During a read operation, the “captain” node gathers all of the data from the various nodes in the cluster and presents it in a cohesive way to the requestor. The cluster provides a high ratio of cache to disk (multiple GB per node) that is dynamically allocated for read and write operations as needed. This RAM-based cache is unified and coherent across all nodes in the cluster, allowing a client read request on one node to benefit from I/O already transacted on another node. As the cluster grows larger, the cache benefit increases. For this reason, the amount of I/O to disk on a cluster is generally substantially lower than it is on traditional platforms.\nFor files marked with an access pattern of concurrent or streaming, OneFS can take advantage of pre-fetching of data based on heuristics used by the SmartRead component\nConclusion This post provided a high level introduction to EMC storage product line and expanded into some technical details in the read write operation in OneFS/Isilon. Some of the features can be seen in OneFS simulator which is a free tool from EMC.\nPrevious PostDocker network in different modes Next PostHow RPC and NFS work ","date":"2020-07-08T20:04:00-04:00","permalink":"/2020/07/emc-productlines/","title":"EMC Isilon storage product"},{"content":"Reading notes of \u0026#8220;Docker DeepDive\u0026#8221;. Docker networking is backed by libnetwork, which is an implementation of Container Network Model (CNM), an open-source pluggable architecture designed to provide networking to containers. Libnetwork also provides native service discovery and basic container load balancing solution. Docker networking also involves some drivers that extend the CNM model with specific network topology implementation.\nSandbox \u0026#8211; an isolated network stack, including Ethernet interfaces, ports, routing tables, and DNS config, usually implemented through Linux namespace. Endpoints \u0026#8211; behave like regular network adapters, and can only be connected to a single network at a time. It connects sandbox to network. Endpoint is implemented in veth pair in Linux. Networks \u0026#8211; software implementation of an 802.1 bridge (aka switch). They group together, and isolate, a collection of endpoints that need to communicate. Docker company separates network project out from its container project, as a plugin called libnetwork, which is developed in Golang and compliant to CNM. Libnetwork is the official implementation of CNM.\nLibnetwork supports the following network modes:\nnetwork modemechanismuse casenullno network is provided to containersquarantined environment for securitybridgecontainers communicate with each other through bridgecontainers needs to communicate with each other or with host servicehostprocess in container has access to host network stack and use host portcontainer needs to use host network stack (e.g. licence by mac address)containerplace containers in a single net namespace so they can communicate with each other as localhostproxy, kubernetes Linux veth comes in pairs to connect virtual network devices. For example, connect two net namespaces to allow intercommunication. Linux bridge is a virtual device, to connect two net namespaces.\nDockers ships with several built-in drivers, known as native drivers or local drivers, such as bridge, overlay and macvlan on Linux. There are also 3rd-party network drivers for docker (aka remote drivers).\nHost network In this mode libnetwork will not create network and net namespace for container. Container process shares the network configuration of the host, and therefore uses the ports on host. Other than network sharing, other aspects (e.g. process, file system, hostname, etc) are separated from host.\nBridge networks This type of network only exist on a single Docker host and can only connect containers that are on the same host. The word bridge refers to 802.1d bridge (layer 2 switch), which is used to connect multiple network interfaces.\nEvery Docker host gets a default single-host network, called bridge on Linux. This is the network that all new containers will attach to by default.\nDocker networks built with the bridge driver on Linux hosts are based on the linux bridge technology that has existed in the Linux kernel for a while. They\u0026#8217;re high performance and extremely stable. Linux brctl tool can inspect the linux bridge.\nBridge networks allows container on the same host to communicate with each other. Port mapping allows network connectivity between container and host. Traffic hitting host port will be redirected to container port.\nMulti-host overlays Cross-host networking usually uses an overlay network, which builds a mesh between host and employs a large block of IP addresses within that mesh. A mesh network is a local network topology in which the infrastructure nodes connect directly, dynamically and non-hierarchically to as many other nodes as possible and cooperate with one another to efficiently route data from/to clients.\nYou can attach a service to overlay network, which spans across multiple Docker hosts so that containers on different hosts can communicate at layer 2. They are much better alternatives than bridge network for container-to-container communication. Overlay networking is very common due to its scalability. The trick is basically the layer 2 frame of the overlay network is encapsulated into layer 3 datagram transmitted across underlay network, at layer 3. This is achieved through VXLAN tunnels, which allows you to create a virtual Layer 2 network on top of an existing Layer 3 infrastructure. VXLAN is an encapsulation technology that existing routers and network infrastructure just see as regular IP/UDP packets without issue.\nTo create the virtual Layer 2 overlay network, a VXLAN tunnel is created through the underlying Layer 3 IP infrastructure (aka underlay network). Each end of the VXLAN tunnel is terminated by a VXLAN Tunnel Endpoint (VTEP). It\u0026#8217;s this VTEP that performs the encapsulation/de-encapsulation.\nVXLAN networking To accomplish overlay network across multiple hosts, a new network sandbox was created on each host. A sandbox is like a container, but instead of running an application, it runs an isolated network stack \u0026#8211; one that\u0026#8217;s sandboxed from the network stack of the host itself.\nA virtual switch (aka virtual bridge) called Br0 is created inside the sandbox. A VTEP is also created with one end plumbed into the Br0 virtual switch, and the other end plumbed into the host network stack (VTEP). The end in the host network gets an IP address on the underlay network the host is connected to and is bound to a UDP socket on port 4789. The two VTEPs on each host create the overlay via a VXLAN tunnel.\nEach container then gets its own virtual Ethernet (veth) adapter that is also plumbed into the local Br0 virtual switch.\nLet\u0026#8217;s go over an example in the following diagram, where container C1 with an overlay IP needs to communicate to another container C2, with a different overlay IP, sitting on a different node (Docker host). Each node has its own underlay IP.\nIP communication details:\nC1 creates the IP datagram with destination IP (C2) and sends it over its veth interface, which is connected to the Br0 virtual switch on the host node. The virtual switch doesn\u0026#8217;t know where to send the datagram, as it doesn\u0026#8217;t have an entry in its ARP table that corresponds to the destination IP address. As a result, it floods the packet to all ports. The VTEP interface connected to Br0 knows how to forward the frame, so responds with its own MAC address. This is a proxy APR reply and results in the Br0 switch learning how to forward the packet. So it updates its ARP mapping the destination IP address to the MAC address of the local VTEP. The VTEP knows about C2 because all newly started containers have their network details propagated to the other nodes in the Swarm using the network\u0026#8217;s built-in gossip protocol. When the packet arrives at node2 The VTEP encapsulates the frame so it can be sent over the underlay transport infrastructure, by adding a VXLAN header to the Ethernet frame. The VXLAN header contains the VXLAN network ID (VNID) which is used to map frames from VLANs to VXLANs and vice versa. Each VLAN gets mapped to VNID, so that the packet can be de-encapsulated on the receiving end and forwarded to the correct VLAN. This is how network isolation is maintained. The encapsulation also wraps the frame in a UDP packet with the IP address of the remote VTEP on node2 in the destination IP field, and the UDP port 4789 socket information. The underlying network does not know that it is transporting data frames for the overlay network. When the packet arrives at node2, the kernel sees that it\u0026#8217;s addressed to UDP port 4789. The kernel also knows that it has a VTEP interface bound to this socket. As a result, it sends the packet to the VTEP, which reads the VNID, de-encapsulates the packet, and sends it on to its own local Br0 switch on the VLAN that corresponds the VNID. From there it is delivered to container C2 Docker also supports Layer 3 routing within the same overlay network. For example, you can create an overlay network with two subnets, and Docker will take care of routing between them. Two subnets will require two virtual switches, Br0 and Br1, being created inside the sandbox, and routing happens by default.\nPlugging into existing vLANs The built-in MACVLAN driver was created for onnect containerized apps to external physical network. A good example is partially containerized app, in which the containerized parts will need a way to communicate with the non-containerized parts still running on existing physical networks.\nTo connect the container interface through the host interface to an external network, the host NIC needs to be in promiscuous mode. For public cloud, this is most likely prohibited. For data centers, this depends on the network policy.\nDocker MACVLAN driver is built on top of Linux kernel driver with the same name. As such, it supports VLAN trunking. This means we can create multiple MACVLAN networks and connect containers on the same Docker host to them.\nFor connectivity issues between containers, it\u0026#8217;s worth checking both the daemon logs (on host) and container logs.\nService discovery allows all containers and Swarm services to locate each other by name, as long as they are on the same network. This leverages Docker\u0026#8217;s embedded DNS server as well as a DNS resolver in each container.\nEach Swarm Service and standalone container started with the \u0026#8211;name flag will register its name and IP address with the Docker DNS service.\nThis name resolution, however, only works within the same network.\nIt is also possible to configure Swarm services and standalone containers with customized DNS options in case embedded Docker DNS server cannot resolve a query (/etc/resolv.conf)\nIngress load balancing Services published via ingress mode (by default, as opposed to host mode) can be accessed from any node in the Swarm, even nodes not running a service replica. Ingress mode uses a layer 4 routing mesh called the Service Mesh or the Swarm Mode Service Mesh.\nUpdates:\nThe most common network modes that I use are host and bridge. With host network mode, container exposes ports on the interface of the host machine. Containers talk to each other via that interface. With bridge network, containers have their own namespace of networking separate from the one from the interface of the hosts, with a bridge getting the two networks connected.\nReference Docker Deep dive\nPrevious PostDataStax Python Driver Next PostEMC Isilon storage product ","date":"2020-07-01T20:19:00-04:00","permalink":"/2020/07/dockersnetwork/","title":"Docker network in different modes"},{"content":"For someone with relational database background, analyzing data in Cassandra isn\u0026#8217;t intuitive. There are two reasons. First, Cassandra data table is hardly updated or deleted in avoidance of tombstones. Insertion is the only action on the table resulting in multiple versions of each record all stored in the same table, thus a much longer table than its relational counterpart. Second, Cassandra schema is designed around how end-user will query the database, rather than a modelling of entity-relations. There are less fields, but some field may contain large data chunk, such as an entire XML document being stored in a column.\nData engineers with Cassandra may need to run full table scan, and extract values from wide columns of XML document by drilling down the XML tree structure, in order to produce a data frame (two-dimensional mutable, possibly heterogeneous tabular data structure with labeled rows and columns). I\u0026#8217;ve came across this task in the past and the duration of a full table scan on Cassandra table is in the order of hours, which is beyond what the built-in cqlsh tool can handle. I had to use Python to iterate through 200 million rows. Datastax Provides Cassandra client driver as a Python3 package, known as DataStax Python Driver. It allows us to build a simple Python3 script to complete a full table scan. The driver can be installed with pip3:\npip3 install cassandra-driver With the driver installed, we can start to pull data from Cassandra table into Python client class. here is a basic example of how to print the rows into a file:\n#! /usr/bin/python3 from cassandra.query import SimpleStatement from cassandra.cluster import Cluster from cassandra import ConsistencyLevel import datetime if __name__ == \u0026#34;__main__\u0026#34;: cluster = Cluster([\u0026#39;cass_host\u0026#39;],port=9042,protocol_version=3) try: print (datetime.datetime.now().strftime(\u0026#34;%Y-%m-%d %H:%M:%S\u0026#34;)+\u0026#34; start\u0026#34;) session = cluster.connect(\u0026#39;myownkeyspace\u0026#39;, wait_for_all_pools=True) query = \u0026#34;SELECT * FROM mytable\u0026#34; statement = SimpleStatement(query, fetch_size=50, consistency_level=ConsistencyLevel.ONE) csv_file = open(\u0026#39;result.csv\u0026#39;,\u0026#39;w\u0026#39;,8192) csv_file.write(\u0026#34;header\u0026#34;) for tbrow in session.execute(statement,timeout=2.0): csv_file.write(tbrow.user_id+\u0026#34;\\n\u0026#34;) except Exception as ex: print(ex) except KeyboardInterrupt: print(\u0026#34;Task Interrupted by SIGINT.\u0026#34;) finally: cluster.shutdown() csv_file.close() print (datetime.datetime.now().strftime(\u0026#34;%Y-%m-%d %H:%M:%S\u0026#34;)+\u0026#34; finish\u0026#34;) Note that the fetch_size can be set to larger number, but it may increase the chance of server read timeout (code=1200) in the middle of execution.\nThe processing logic can be implemented in the loop while each record in the table is being pulled out. The logic is repeated for every row so it will have a significant impact on the overall execution time.\nSome the data needs to be ported into pandas data frame for further engineering, instead of being printed out to file. The following snippet will do the trick:\n#! /usr/bin/python3 from cassandra.query import SimpleStatement from cassandra.cluster import Cluster from cassandra import ConsistencyLevel import datetime import pandas as pd def pandas_factory(colnames,rows): res = [] res.append(pd.DataFrame(rows, columns=colnames)) return res if __name__ == \u0026#34;__main__\u0026#34;: cluster = Cluster([\u0026#39;cass_host\u0026#39;],port=9042,protocol_version=3) try: print (datetime.datetime.now().strftime(\u0026#34;%Y-%m-%d %H:%M:%S\u0026#34;)+\u0026#34; start\u0026#34;) session = cluster.connect(\u0026#39;myownkeyspace\u0026#39;, wait_for_all_pools=True) session.row_factory = pandas_factory query = \u0026#34;SELECT * FROM mytable\u0026#34; statement = SimpleStatement(query, consistency_level=ConsistencyLevel.ONE,fetch_size=50) df=pd.DataFrame() for tbrow in session.execute(statement): df=df.append(tbrow.user_id,ignore_index=True) except Exception as ex: print(ex) except KeyboardInterrupt: print(\u0026#34;Task Interrupted by SIGINT.\u0026#34;) finally: cluster.shutdown() print (datetime.datetime.now().strftime(\u0026#34;%Y-%m-%d %H:%M:%S\u0026#34;)+\u0026#34; finish\u0026#34;) In my test environment with 180 million rows in the table, the execution of the first script takes 37 minutes (of course there\u0026#8217;s a lot of factors at play). I experimented several approaches to improve the speed, such as tuning the buffering options for file write. However, It turns out that the speed bottleneck of the script is not even file IO, but rather pulling data out of Cassandra.\nThe output can be stored as CSV file, which can be lately loaded to relational database for analysis. PostgreSQL would be a good open-source choice because it is both transactional and analytical.\nPrevious PostPerformance Analysis Next PostDocker network in different modes ","date":"2020-06-27T14:20:34-04:00","permalink":"/2020/06/iterate-through-cassandra-table-with-datastax-python-driver/","title":"DataStax Python Driver"},{"content":"Overview In 2015, Brendan Gregg posted two great articles on Netflix blog: Linux Performance Analysis in 60 seconds, and Linux Perfomrance Tools. They have great value when I was in a urgency to spot performance issues. The articles cover the essential tools for performance troubleshooting, including:\nCheck out load averages: w or uptime Print kernel ring buffer: dmesg -T Virtual memory status: vmstat 1 Multiple processor staticstics: mpstat -P ALL 1 Task status: pidstat 1 CPU and I/O status: iostat -xz 1 Free memory check: free -m Network Activity record: sar -n DEV 1 TCP activity record: sar -n TCP,ETCP 1 Display processes: top We will dive into each of them in the next section.\nBasic Troubleshooting The command w is equivalent of uptime (which shows uptime since boot) and who (which shows logged-in users). It also displays load average for the last 1 minute, 5 minutes and 15 minutes. The number of load average reflects the overall system load (CPU + disks), and it is further discussed in this post with a simple take away:\nIf the averages are 0.0, then your system is idle. If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing. If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing. If they are higher than your CPU count, then you might have a performance problem (it depends). When Linux load averages increase, you know you have higher demand for resources (CPUs, disks, and some locks), but you aren\u0026#8217;t sure which. You will need to switch to other metrics. Brendan recommend don\u0026#8217;t spend more than 5 seconds on these numbers.\n[dhunch@c7v-ghintapp01 ~]$ w 12:14:10 up 46 days, 16:41, 3 users, load average: 2.69, 2.44, 2.29 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT dhunch pts/0 w6v-ghas01 24Jun20 3days 0.36s 0.30s ssh c7v-bastion dhunch pts/1 202.95.88.111 12:02 2.00s 0.00s 0.00s w dhunch pts/4 w6v-ghas01 17Jun20 15days 0.15s 0.07s view readme.txt Before moving to more insightful metrics, it is also worth a quick look into the kernel ring buffer with dmesg command (dmesg -T | less +G). This will allow us to capture obvious issues such as oom-killer or TCP request dropping.\nThe vmstat tool reports the statistics of virtual memory. Servers have a fixed amount of physical memory, but they can run a set of applications that use a much larger amount of virtual memory. Application tend to reserve more memory than they need, and they usually operate on only a subset of their memory. In both cases, the operating system can keep the unused parts of memory on disk, and page it into physical memory only if it is needed. For the most part, this kind of memory management works well. But it doesn\u0026#8217;t always with Java applications due to Java heap. Once a system start swapping \u0026#8211; moving pages of data from main memory to disk, and vice versa, the performance tend to be bad. Systems must be configured so that swapping never occurs.\n[dhunch@c7v-ghintapp01 ~]$ vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 239360 385928 0 36734692 0 0 1 200 0 0 6 1 93 0 0 3 0 239360 387732 0 36734704 0 0 0 43 8017 8524 14 0 85 0 0 3 0 239360 387608 0 36734904 0 0 0 57 6768 7680 14 1 86 0 0 2 0 239360 389008 0 36734904 0 0 0 44 6366 7300 14 0 86 0 0 3 0 239360 421728 0 36700144 0 0 0 0 8141 7957 13 0 86 0 0 2 0 239360 421984 0 36702048 0 0 0 467 8994 8362 14 0 85 0 0 The tool prints key server statistics each line, with the first line showing the average since boot. Here lists the explanation of some columns:\nr: number of processes running on CPU and waiting for a turn. This provides a better signal than load averages for determining CPU saturation, as it does not include I/O. To interpret: an “r” value greater than the CPU count is saturation. swpd: the amount of virtual memory used. This number should align with the used column for Swap row from free command. buff, cache: buffer and cache. They should align with the buff/cache column form Mem row from free command. free: free memory in kilobytes. This number should align with the free column for Mem row from free command. si, so: swap-ins and swap-outs. As mentioned, if these are non-zero, you\u0026#8217;re out of memory. bi, bo: blocks received from and sent to a blcok device (# of block per second) in, cs: number of interrupt, and context switches per second. us, sy, id, wa, st: user, system, idle, wait I/O and stolen times. These are breakdowns of CPU time, on average across all CPUs. They should add up to 100% (or close). stolen time is amount of CPU time needed by a guest virtual machine that is not provided by the host. IO wait time is the CPU time waiting for I/O activity. Idle time could be several things: the process may be waiting for something (e.g. a response from database); the process may be blocked by a thread lock; or the process simply has nothing to do. user and system times are CPU times spent on user tasks and kernel tasks, respectively. Out of these columns, watch for columns r, free, buff, cache, us, sy, id and wa at minimum. The combination of us and sy confirms if CPUs are busy. A constant degree of wa points to a disk bottleneck with too much time spent on pending disk I/O. The sy (kernel time) is necessary for I/O processing but sy stays high (e.g. constantly over 20%), it becomes interesting. Perhaps the kernel is processing I/O inefficiently.\nFor further per-CPU stats, use mpstat command (-P ALL), to prind CPU time breakdowns per CPU and check for imbalance. A single host CPU can be evidence of a single-threaded application. Here is an example output from a system of 16 CPU cores.\n[dhunch@c7v-ghintapp01 ~]$ mpstat -P ALL 1 Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU) 13:47:23 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 13:47:24 all 13.77 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 86.05 13:47:24 0 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98 13:47:24 1 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:47:24 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:47:24 3 98.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 13:47:24 4 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98 13:47:24 5 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98 13:47:24 6 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:47:24 7 2.94 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 96.08 13:47:24 8 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:47:24 9 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:47:24 10 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 13:47:24 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 98.02 13:47:24 12 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98 13:47:24 13 1.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.02 13:47:24 14 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:47:24 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 For a per process summary of CPU consumption, use pidstat command. It can be thought of a periodical snapshot of top command, allowing you to watch for patterns. The %CPU column is the total across all CPUs so 5 CPUs have a maximum value of 500.\nIf vmstate indicates some I/O issue, iostat tool can help us understand block devices, on both the workload applied and the resulting performance. Key columns are:\nr/s, w/s, rkB/s, wkB/s: delivered reads, writes, read Kbytes, and write Kbytes per second to the device. Use these for workload characterization. A performance problem may simply be due to an excessive load applied. await: the average wait time for I/O in milliseconds. This is the time that the application suffers, as it includes both time queued and time being serviced. Larger than expected average times can be an indicator of device saturation, or malfunction. avgqu-sz: the average number of requests issued to device. Values greater than 1 can be evidence of saturation (although devices can typically operate on requests in parallel, especially virtual devices which front multiple back-end disks.) %util: device utilization. This is really a busy percent, showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance (which should be seen in await), although it depends on the device. Values close to 100% usually indicate saturation. I/O problem may either be inefficiencies in application that issues I/O request, or slowing disk unable to keep up with I/O requests. We review two examples here to illustrate each situation. The first output is as follows:\n% iostat -xm 5 avg-cpu: %user %nice %system %iowait %steal %idle 23.45 0.00 37.89 0.10 0.00 38.56 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 11.60 0.60 24.20 0.02 0.14 13.35 0.15 6.06 5.33 6.08 0.42 1.04 In the first example, the disk stat loosk up at first glance. The w_await (time to service I/O write) is fairly low at 6.08ms. However, the system is spending 37.89% of its time in the kernel. If all that system time is from the application, it suggest something inefficient is happening. The fact that the system is doing 24.2 writes per second is another clue: that is alot when writing only 0.14 MB per second (MBps). I/O has become a bottleneck, and the next step would be to look into how the application is performing its writes.\nThe second example output is as follows:\n% iostat -xm 5 avg-cpu: %user %nice %system %iowait %steal %idle 35.05 0.00 7.85 47.89 0.00 9.20 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.20 1.00 163.40 0.00 81.09 1010.19 142.74 866.47 97.60 871.17 6.08 100.00 In this example, it tells us that processes are spending 47.89% of their time in iowait, and the data to complete the I/O (w_await) is 871ms, the queue size is large, and the disk is writing at 81MB per second. This all points to disk I/O as a problem and that the amount of I/O in the application (or elsewhere in the system) must be reduced.\nBear in mind that poor performing disk I/O isn\u0026#8217;t necessarily an application issue. Many techniques are typically used to perform I/O asynchronously, so that the application doesn\u0026#8217;t block and suffer the latency directly (e.g. read-ahead for reads, and buffering for writes, also refer to \u0026#8220;tail latency\u0026#8220;). Note that the acceptable utilization metric depends on the configuration of block device. If the storage is a logical disk device fronting many back-end disks (e.g. RAID 0), then 100% utilization may just mean that some I/O is being processed 100% of the time, however, the back-end disks may be far from being saturated, and may even be able to handle more work.\nThe free command gives the breakdown of memory usage. The right two columns are:\nbuffers: for the buffer cache, used for block device I/O. cached: for the page cache, used by file systems. We just want to check that these aren\u0026#8217;t near-zero in size, which can lead to higher disk I/O (confirm using iostat), and worse performance. Linux uses free memory for the caches, but can reclaim it quickly if applications need it. So in a way the cached memory should be included in the free memory column. In this case, it\u0026#8217;s included in the available column. This website has further details.\nTo check interface stat, nicstat is a great tool but it isn\u0026#8217;t available by default in Linux. Instead we can run sar (-n DEV) to retrieve stats. [dhunch@c7v-ghintapp01 ~]$ sar -n DEV 1 Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU) 16:51:25 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 16:51:26 eth0 3089.00 934.00 3815.33 834.61 0.00 0.00 0.00 16:51:26 lo 464.00 464.00 2289.07 2289.07 0.00 0.00 0.00 16:51:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 16:51:27 eth0 956.00 586.00 826.66 211.34 0.00 0.00 0.00 16:51:27 lo 213.00 213.00 196.00 196.00 0.00 0.00 0.00 16:51:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 16:51:28 eth0 349.00 181.00 52.32 147.19 0.00 0.00 0.00 16:51:28 lo 244.00 244.00 81.13 81.13 0.00 0.00 0.00 Here rxkB/s/s and txkB/s represents receive and transmission throughput, as a measure of workload. If they reach the limit then the interface is the bottleneck.\nOn top of interface is the TCP layer, which can be monitored with sar again (-n ECP, ETCP). The key metrics are:\nactive/s: number of locally-initiated (e.g. via connect()) TCP connections per second passive/s: number of remotely-initiated (e.g. via accept()) TCP connections per second retrans/s: number of TCP retransmits per second The active and passive counts are often useful as a rough measure of server load. It might help to think of active as outbound, and passive as inbound, but this isn\u0026#8217;t strictly true (e.g. consider a localhost to localhost connection). Retransmits are a sign of network or server issue; it may be an unreliable network (e.g. public Internet), or it may be due to a server being overloaded and dropping packets.\n[dhunch@c7v-ghintapp01 ~]$ sar -n TCP,ETCP 1 Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU) 16:52:29 active/s passive/s iseg/s oseg/s 16:52:30 0.00 1.00 28.00 35.00 16:52:29 atmptf/s estres/s retrans/s isegerr/s orsts/s 16:52:30 0.00 0.00 0.00 0.00 0.00 16:52:30 active/s passive/s iseg/s oseg/s 16:52:31 8.00 8.00 200.00 317.00 16:52:30 atmptf/s estres/s retrans/s isegerr/s orsts/s 16:52:31 0.00 1.00 1.00 0.00 3.00 Last but not least is our favourite command top, which includes many of the metrics covered in previous tools. The downside to top is it is harder to see patterns over time, which may be more clear in tools like vmstat and pidstat, both of which produce rolling output.\nSeveral tools introduced here involves sar, which is a great monitoring tool on its own that we need to be familiar with.\nSystem Activity Report (SAR) Further to the basic tools, the sar command is very helpful as it stores historical stat every 10 minutes. Sar keeps 18 types of reports, all stored in /var/log/sa/. When viewing the report file, you may pipe the result to less command so it only prints header once. For example, if you would like to print CPU report for the 2nd of the month:\n# sar -u -f /var/log/sa/sar02 | less If you check NFS client statistics for the 31st\n# sar -n NFS -f /var/log/sa/sar31 If you need to check network server statistics for today\n# sar -n NFS -f /var/log/sa/sar31 Below are all types of reports:\n-u CPU utilization -w task creation and system switching activity -W swapping statistics -B report paging -b report I/O and transfer rate statistics -R report memory statistics -r memory utilization -S swap space utilization -H huge pages utilization statistics -v inode -q queue length and load average -y TTY device activity -d activity for each block device -n network statistics, DEV (per interface) -n network statistics, EDEV (error per interface) -n network statistics, NFS (NFS client) -n network statistics, NFSD (NFS server) -n network statistics, SOCK (socket usage) Berkeley Packet Filter (BPF) Compiler Collection (bcc) tools For advanced, low-level performance troubleshooting, the BCC tools provide a suite of tools. Here we only cover the installation of it.\nIn CentOS, install bcc-tools package with yum. When you try to run a command, such as cachestat, if you come across this error:\n-bash: cachestat: command not found Then you will need to add its path to default:\nexport PATH=$PATH:/usr/share/bcc/tools Now if you run into this error:\nchdir(/lib/modules/3.10.0-1062.12.1.el7.x86_64/build): No such file or directory Traceback (most recent call last): The file listed is a symbolic link, and if it is missing, you just need to install kernel-headers that matches the kernel version:\nyum install kernel-headers Then you may use tools in /usr/share/bcc/tools. For example, cachestat help you display page cache hit ratio; gethostlatency shows DNS resolution latency; tcpconnect prints out active tcp connections (made via connect system call):\n[root@dhunch ~]# /usr/share/bcc/tools/tcpconnect -t -P 8080 | gawk \u0026#39;{ print strftime(\u0026#34;%F %T \u0026#34;), $0 }\u0026#39; 2020-06-13 00:16:57 TIME(s) PID COMM IP SADDR DADDR DPORT 2020-06-13 02:16:57 0.000 15241 QNetworkAcce 4 10.100.22.21 10.101.84.10 8080 2020-06-13 02:16:57 0.064 15241 QNetworkAcce 4 10.100.22.21 10.101.84.10 8080 2020-06-13 02:16:57 0.438 15241 QNetworkAcce 4 10.100.22.21 10.101.84.10 8080 The command above outputs a\u0026nbsp;time and pid stamped log line every time\u0026nbsp;a TCP connection is made to port 8080; tcpaccept traces passive tcp connections (via accept system call). These tools are not as intrusive as tcpdump.\nIt is beyond the purpose of this article to get into details of each tool in the BFP suite. The tools are covered in detail in books \u0026#8220;BPF Performance Tools\u0026#8221; and \u0026#8220;Linux Observability with BPF\u0026#8221;.\nPrevious PostCapture filter and Display filter in Network Analyzer Next PostDataStax Python Driver ","date":"2020-06-19T16:47:01-04:00","permalink":"/2020/06/performance-analysis-tools/","title":"Performance Analysis"},{"content":"Capture filter is set before collecting packets. It is applied at the time of data acquisition and it impacts the size of the capture. It does not have as many variations as display filter and is usually not aware of protocols above TCP/UDP layer. A common form of capture filter is BPF (Berkerly Packet Filter) which is used in Linux Socket Filtering (e.g. tcpdump).\nBasic form is:\n[tcp|udp] [src|dst] host 192.168.1.2 port 1234\nFor example:\n\u0026#39;tcp dst port 8080 and src host 147.206.160.9\u0026#39; Display filter is set after packet collection. It is applied at the time of data manipulation. It does not impact the size of capture, but it controls how the data is presented (typically for analysis purpose). Display filter may support a variety of expressions that are interpreting data at TCP/UDP layer or above, for example HTTP. Here are some examples:\n(tcp.flags.syn == 1) || (tcp.flags.reset == 1) (tcp.flags.reset == 1) || (http.request.method==GET) || (tcp.flags.reset == 1)||(http.request.uri contains \u0026#34;/box/url/string\u0026#34;) ||(http.response.code == 200) Here are some further examples provided by Wireshark.\nFor more details about the usage of capture filter and display filter, here is a page with cheatsheet. Example for tcpdump on the left and wireshark in the middle and on the right.\nTo view http packet in shell terminal, there is also a helpful tool called httpry. You can applied BPF styled filter for capture, and organize display column. The drawback is there is no display filter so you\u0026#8217; would have to use grep. Here is an example:\nhttpry -i eth0 \u0026#39;tcp dst port 8080 and src host 147.206.160.9\u0026#39; -m GET -f Timestamp,x-correlation-id,x-userid,Request-URI | grep -v -P \u0026#39;\\t\\-\\t\u0026#39; Previous PostSetup WSL2 (and Docker) on Windows 10 Next PostPerformance Analysis ","date":"2020-06-10T21:21:18-04:00","permalink":"/2020/06/network-analyzer-capture-filter-and-display-filter/","title":"Capture filter and Display filter in Network Analyzer"},{"content":"This is not for Linux snobs, but rather for those who are stuck with a Windows work laptop, have to deal with Linux on a daily basis, and are not a fan of PuTTY. This posting provides the steps to setup Windows 10 so you get a work environment closer to a Linux one. The environment to begin with should be Windows 10 version 2004 and above in order to use WSL2. Here\u0026#8217;s the comparison between WSL and WSL2. This presentation is a great deep dive into how WSL2 works. The architecture diagram below is from that presentation. Note that WSL2 operates on a \u0026#8220;true\u0026#8221; Linux Kernel, therefore giving WSL2 the ability to run Docker.\nInstall WSL2 WSL2 was officially released in Windows 10 version 2004 (build 19041 or higher) and we will use Ubuntu 20.04 LTS. If you\u0026#8217;re upgraded from older version of Windows 10 you will need to upgrade. The steps are:\nInstall Ubuntu 20.04 LTS from Microsoft Store, which requires Windows components \u0026#8220;Windows Subsystem for Linux\u0026#8221; and \u0026#8220;Virtual Machine Platform\u0026#8221;; or, if you already have an older version of Ubuntu such as 18.04, you will need to run a distribution upgrade with \u0026#8220;sudo apt-get dist-upgrade\u0026#8221;; Upgrade Linux Virtual Machine from WSL to WSL2, following the official instruction here; Once completed, use this command to confirm the WSL version installed: wsl.exe \u0026#8211;list -v There\u0026#8217;s also plenty of Youtube videos with step-by-step instruction on upgrading to WSL2 and its advantage over WSL. Once you\u0026#8217;re there, you will realize one problem: the command terminal for Ubuntu is ugly:\nThe colour looks awful and it does not support multi-tabs. If you want better palette, there are a number of options. Check out ColorTool project and iTerms2 Colors project. The former gives you away to configure color scheme (from command prompt but takes effect in WSL as well) and the latter gives you rich choices of color schemes. My personal favourite is \u0026#8220;Banana Blueberry\u0026#8221;. The command to set color scheme is: C:\\ColorTool.exe -b \u0026#34;iTerm2-Color-Schemes-master\\schemes\\Banana Blueberry.itermcolors\u0026#34; Windows Terminal For a better terminal, consider Windows Terminal from Microsoft Store. Windows terminal is an open-source project and still a little glitchy as of June 2020. However it is heading to the right direction. Once installed, you may customize it by clicking settings:\nA json file will open and you may edit it for customization, for example, you can adjust each profile (e.g. PowerShell, Windows command or WSL); you can specify default profile and default open location. // This file was initially generated by Windows Terminal 1.0.1401.0 // It should still be usable in newer versions, but newer versions might have additional // settings, help text, or changes that you will not see unless you clear this file // and let us generate a new one for you. // To view the default settings, hold \u0026#34;alt\u0026#34; while clicking on the \u0026#34;Settings\u0026#34; button. // For documentation on these settings, see: https://aka.ms/terminal-documentation { \u0026#34;$schema\u0026#34;: \u0026#34;https://aka.ms/terminal-profiles-schema\u0026#34;, \u0026#34;defaultProfile\u0026#34;: \u0026#34;{2c4de342-38b7-51cf-b940-2309a097f518}\u0026#34;, // You can add more global application settings here. // To learn more about global settings, visit https://aka.ms/terminal-global-settings // If enabled, selections are automatically copied to your clipboard. \u0026#34;copyOnSelect\u0026#34;: false, // If enabled, formatted data is also copied to your clipboard \u0026#34;copyFormatting\u0026#34;: false, // A profile specifies a command to execute paired with information about how it should look and feel. // Each one of them will appear in the \u0026#39;New Tab\u0026#39; dropdown, // and can be invoked from the commandline with `wt.exe -p xxx` // To learn more about profiles, visit https://aka.ms/terminal-profile-settings \u0026#34;profiles\u0026#34;: { \u0026#34;defaults\u0026#34;: { // Put settings here that you want to apply to all profiles. }, \u0026#34;list\u0026#34;: [ { \u0026#34;guid\u0026#34;: \u0026#34;{2c4de342-38b7-51cf-b940-2309a097f518}\u0026#34;, \u0026#34;hidden\u0026#34;: false, \u0026#34;name\u0026#34;: \u0026#34;Ubuntu\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;Windows.Terminal.Wsl\u0026#34;, \u0026#34;startingDirectory\u0026#34;: \u0026#34;//wsl$/Ubuntu/home/myuser/\u0026#34; }, { // Make changes here to the powershell.exe profile. \u0026#34;guid\u0026#34;: \u0026#34;{61c54bbd-c2c6-5271-96e7-009a87ff44bf}\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Windows PowerShell\u0026#34;, \u0026#34;commandline\u0026#34;: \u0026#34;powershell.exe\u0026#34;, \u0026#34;hidden\u0026#34;: false }, { // Make changes here to the cmd.exe profile. \u0026#34;guid\u0026#34;: \u0026#34;{0caa0dad-35be-5f56-a8ff-afceeeaa6101}\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Command Prompt\u0026#34;, \u0026#34;commandline\u0026#34;: \u0026#34;cmd.exe\u0026#34;, \u0026#34;hidden\u0026#34;: true }, { \u0026#34;guid\u0026#34;: \u0026#34;{b453ae62-4e3d-5e58-b989-0a998ec441b8}\u0026#34;, \u0026#34;hidden\u0026#34;: true, \u0026#34;name\u0026#34;: \u0026#34;Azure Cloud Shell\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;Windows.Terminal.Azure\u0026#34; } ] }, // Add custom color schemes to this array. // To learn more about color schemes, visit https://aka.ms/terminal-color-schemes \u0026#34;schemes\u0026#34;: [], // Add custom keybindings to this array. // To unbind a key combination from your defaults.json, set the command to \u0026#34;unbound\u0026#34;. // To learn more about keybindings, visit https://aka.ms/terminal-keybindings \u0026#34;keybindings\u0026#34;: [ // Copy and paste are bound to Ctrl+Shift+C and Ctrl+Shift+V in your defaults.json. // These two lines additionally bind them to Ctrl+C and Ctrl+V. // To learn more about selection, visit https://aka.ms/terminal-selection { \u0026#34;command\u0026#34;: {\u0026#34;action\u0026#34;: \u0026#34;copy\u0026#34;, \u0026#34;singleLine\u0026#34;: false }, \u0026#34;keys\u0026#34;: \u0026#34;ctrl+c\u0026#34; }, { \u0026#34;command\u0026#34;: \u0026#34;paste\u0026#34;, \u0026#34;keys\u0026#34;: \u0026#34;ctrl+v\u0026#34; }, // Press Ctrl+Shift+F to open the search box { \u0026#34;command\u0026#34;: \u0026#34;find\u0026#34;, \u0026#34;keys\u0026#34;: \u0026#34;ctrl+shift+f\u0026#34; }, // Press Alt+Shift+D to open a new pane. // - \u0026#34;split\u0026#34;: \u0026#34;auto\u0026#34; makes this pane open in the direction that provides the most surface area. // - \u0026#34;splitMode\u0026#34;: \u0026#34;duplicate\u0026#34; makes the new pane use the focused pane\u0026#39;s profile. // To learn more about panes, visit https://aka.ms/terminal-panes { \u0026#34;command\u0026#34;: { \u0026#34;action\u0026#34;: \u0026#34;splitPane\u0026#34;, \u0026#34;split\u0026#34;: \u0026#34;auto\u0026#34;, \u0026#34;splitMode\u0026#34;: \u0026#34;duplicate\u0026#34; }, \u0026#34;keys\u0026#34;: \u0026#34;alt+shift+d\u0026#34; } ] } With these configuration, we have a quite comfortable Linux work environment on Windows.\nUpdate to Zsh and beautify it I\u0026#8217;m no so big a fan of zsh (for lack of default wildcard support) but I do like one of its themes. We can install zsh on WSL2 (assuming Ubuntu distribution). This is a good instruction. I do like the theme name agnoster. However, the theme displays username at the beginning of the prompt, which takes a lot of screen real estate if your username is long. To remove it, the trick is to add a line in ~/.zshrc, as answered in this thread. Docker runtime There is a lot of fun to have in WSL2 on Windows 10. For example, you can configure Docker runtime on Windows according to this post. However, you cannot route traffic to the container in the absence of docker0 bridge for Docker on WSL2, as indicated in the known limitations.\nUpdate: Docker also brings a Kubernetes cluster named docker-desktop. This allows you configure Kubernetes cluster, or install Helm and Rancher (instruction) for cluster management.\nDual-Boot? With WSL2 on top of a real Linux kernel, I do not find a need for a dual-boot system on my workstation. To be fair, only in the following scenarios should one consider installing a dual-boot system.\nYou need to work on a specify version of Linux Kernel. The kernel provided by Microsoft is called \u0026#8220;microsoft-standard-WSL2\u0026#8221;, and its pretty up to date (5.x) You need to run GUI applications on Linux. Update: Microsoft has started developing GUI application support on WSL2, even though it usually takes time to mature. You need to work on a distribution not available in Windows Store. WSL2 provides a kernel and it\u0026#8217;s up to developers to provide Linux distros in Windows store. Popular (and official) ones are: Ubuntu 20.04 LTS by Canonical Group, Debian by the Debian project, SUSE Linux Enterprise Server by SUSE, Kali Linux. Certificate Issuers I have to use WSL2 mostly because I\u0026#8217;m forced to use a Windows laptop on my contracts. That also means that the Windows laptop has been configured to corporate firewall\u0026#8217;s packet inspection. When that is the case, they corporate IT usually automatically imports a firewall certificate to the Windows trust store. However, the WSL is missing this certificate. The symptom would be that you cannot run certain curl command. For example, when I do the followings:\ncurl -LO \u0026#34;https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\u0026#34; curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. This error message basically means that the website certificate cannot be trusted. To view the certificate, use Open SSL command:\nopenssl s_client -connect dl.k8s.io:443 -servername dl.k8s.io | openssl x509 -text -noout | less In my case, I noticed that the certificate is a Zscaler issued certificate (firewall). While running the same command from my personal device returns a Goolge Trust issued certificate. This indicates that the certificate is changed by the proxy managed by the corporation. To fix this I need to export the certificate from Windows, and then import it in WSL. The answer in this post has good detailed steps.\nPrevious PostAnsible at scale 2 of 2 Next PostCapture filter and Display filter in Network Analyzer ","date":"2020-06-02T22:01:00-04:00","permalink":"/2020/06/wsl2-environment-on-windows-10/","title":"Setup WSL2 (and Docker) on Windows 10"},{"content":"Template (with Jinja2) and files In an Ansible role, we can use files or templates to achieve similar results for configuration files. If the configuration file is the same across all targets then we can place it in files directory to push out. If the content of configuration file varies depending on the cluster size, we use Jinja2 template. For example, when you configure zookeeper configuration, a first entry may require total number of nodes in the cluster, a second entry may require the hostname of the server itself; and a third entry may require a comma separated line with hostnames of all nodes in the cluster. This is a typical use case of Jinja template.\nWe need to make sure Jinjas version is above 2.11.2 (as of May 2020) because older version such as 2.7.2 has known issues with namespaces. To check version and then upgrade Jinja2, we need to use pip:\npip show Jinja2 pip install -U Jinja2 The Ansible template module takes Jinja2 file as input and delivers result file on target host. Note that if the template references host variables from Ansible playbook, then you need to gather facts about host. This means you will have to use a basic playbook like below instead of adhoc command.\nA basic playbook to test Jinja2 template is:\n- hosts: \u0026#39;{{ansible_limit}}\u0026#39; gather_facts: yes tasks: - template: src: cassandra_xml.j2 dest: /tmp/cassandra.xml Although Jinja2 offers a lot of flexibility with loop and if-else statement, it is just a templating language and not a programming language. It requires some tricks to achieve what you may otherwise easily do with programming language. One example is persisting a variable outside of a loop. As per the document, it is not possible to set variables inside a block and have them show up outside of it. This also applies to loops. The only exception to that rule are if statements which do not introduce a scope. To achieve that, you would have to use namespace, for each loop where you need to access the variable afterwards from outside of the loop.\n{% block db_cluster_config_nobackup %} {% set ns=namespace(nodeid=0) %} {% for host in groups[my_db_group]|sort %} \u0026lt;var name=\u0026#34;DBHost{{ns.nodeid+1}}\u0026#34; value=\u0026#34;{{hostvars[host].inventory_hostname}}\u0026#34; /\u0026gt; {% set ns.nodeid=ns.nodeid+1 %} {% endfor %} \u0026lt;var name=\u0026#34;DBClusterHosts\u0026#34; value=\u0026#34;{% for i in range(ns.nodeid) %} ${DBHost{{i+1}}}{% if not loop.last %},{% endif %} {% endfor %}\u0026#34; /\u0026gt; {% endblock cass_cluster_config_nobackup %} For the same reason, you might as well clearly define the start and end of each block in order to not run into trouble with scoping behaviours of variables. These limitations makes Jinja2 template not easy to read and may take several rounds of playbook runs to troubleshoot.\nHandler vs conditional task Sometimes you only want to run a task when its previous task results a change. There are two ways to achieve this: conditional task and handler.\nWith conditional task, we register the result of previous task to a variable, and execute the ensuing tasks conditionally based on assessment of the variable. We\u0026#8217;d have to specify the condition for each of the subsequent tasks that needs to execute conditionally. These tasks, if condition is met, can execute immediately after the first task that registers the variable.\nThe alternative is through an Ansible mechanism called handler. Handler implements a series of tasks in a separate yaml file in the handers directory under the role. In the triggering task we need to notify the handler. The tasks in the hander will fire if the triggering task returns \u0026#8220;changed\u0026#8221; in its result. Handler is a great way to shorten the length of task or Playbook. However, we need to understand several subtleties with regard to handlers: Although handler is notified during a task run, it is not fired until the end of each block of tasks in a play. They are not immediately fired after triggering task. A handler will only execute once at the end of play, even if it was notified multiple times by different tasks during the play run. Handler tasks are executed in the order of declaration, not in the order of notification. In summary, Ansible\u0026#8217;s notification handling mechanism is asynchronous, once-only, and out of sequence. The points above are illustrated in the following playbook:\n--- - hosts: ghdocker tasks: - name: CopyFile3 copy: src: ~/ansible/file3.txt dest: /tmp/file3.txt notify: - handler3 - handlergeneral - name: CopyFile2 copy: src: ~/ansible/file2.txt dest: /tmp/file2.txt notify: - handler2 - handlergeneral - name: CopyFile1 copy: src: ~/ansible/file1.txt dest: /tmp/file1.txt notify: - handler1 - handlergeneral - debug: msg=\u0026#34;end of play!\u0026#34; handlers: - name: handler1 debug: msg=\u0026#34;file1.txt has been copied.\u0026#34; - name: handler2 debug: msg=\u0026#34;file2.txt has been copied.\u0026#34; - name: handler3 debug: msg=\u0026#34;file3.txt has been copied.\u0026#34; - name: handlergeneral debug: msg=\u0026#34;A file has been copied\u0026#34; Here is the output of the playbook run: PLAY [ghdocker] ****************************************************************************** TASK [Gathering Facts] ****************************************************************************** ok: [ghdocker] TASK [CopyFile3] ****************************************************************************** changed: [ghdocker] TASK [CopyFile2] ****************************************************************************** changed: [ghdocker] TASK [CopyFile1] ****************************************************************************** changed: [ghdocker] TASK [debug] ****************************************************************************** ok: [ghdocker] =\u0026gt; { \u0026#34;msg\u0026#34;: \u0026#34;end of play!\u0026#34; } RUNNING HANDLER [handler1] ****************************************************************************** ok: [ghdocker] =\u0026gt; { \u0026#34;msg\u0026#34;: \u0026#34;file1.txt has been copied.\u0026#34; } RUNNING HANDLER [handler2] ****************************************************************************** ok: [ghdocker] =\u0026gt; { \u0026#34;msg\u0026#34;: \u0026#34;file2.txt has been copied.\u0026#34; } RUNNING HANDLER [handler3] ****************************************************************************** ok: [ghdocker] =\u0026gt; { \u0026#34;msg\u0026#34;: \u0026#34;file3.txt has been copied.\u0026#34; } RUNNING HANDLER [handlergeneral] ****************************************************************************** ok: [ghdocker] =\u0026gt; { \u0026#34;msg\u0026#34;: \u0026#34;A file has been copied\u0026#34; } PLAY RECAP ****************************************************************************** ghdocker : ok=9 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Handler is a good way to keep idempotency. For example, Ansible does not have a way to import a yum .repo file to create a repo. We have to take two steps: use get_url module to download the repo file (e.g. to /tmp), use shell module to call yum-config-manager. The problem is these two steps are not idempotent. If you repeat them, it will attempt to import the same repo file again. A little trick here is to use force=no option on get_url so it does not attempt to download if the file is already present in target directory. Then notify a handler to import repo file so the shell command is only called if there is a change.\nThe task looks like this:\n- name: download repo file get_url: url: https://download.docker.com/linux/centos/docker-ce.repo dest: /tmp/docker-ce.repo mode: \u0026#39;0755\u0026#39; force: no notify: - Add docker repository The handler looks like this:\n- name: Add docker repository shell: yum-config-manager --add-repo=/tmp/docker-ce.repo The handler is only fired when it is notified after get_url module returns changed in its result. Running the task again will not cause it to attempt to add the same repo again.\nNote that when you use command or shell module, Ansible typically reports changed status. If this is not desired (e.g. you don\u0026#8217;t want it to notify handler all the time), this behaviour can be overridden with changed_when parameter. You can specify conditions to meet in order to consider the shell/command module to have a changed result. Here is an example.\nAnsible commands In operation, our engineer needs to run a command on a group of servers. I encourage the use of Ansible adhoc command whenever possible. I recommend start with the following two commands:\nansible-inventory --graph ansible all -m ping The ping module triggers an \u0026#8220;Ansible ping\u0026#8221; to targets in the specified group. Over the years, Ansible community developed many helpful modules, such as yum, yum_repository, apt_rpm, uri, synchronize, fine, copy, etc and many can be used instead of bash command. However, sometimes, the expected Ansible module is either unavailable or missing function. For example, Ansible\u0026#8217;s uri module cannot replace curl command with the following switches:\ncurl -s -XGET http://{{inventory_hostname}}:8080/objects/{{object_id}}/binary/all -o /dev/null -w \u0026#39;%{response_code} %{size_download} %{time_total} %{speed_download}\\n\u0026#39; | awk \u0026#39;{if ($1==200) print \u0026#34;size=\u0026#34;$2/1048576\u0026#34;MB,time=\u0026#34;$3\u0026#34;s,speed=\u0026#34;$4/1048576\u0026#34;MB/s\u0026#34;; else if($1==404) print \u0026#34;Cannot find object {{object_id}}\u0026#34;; else print \u0026#34;Unknown error. Code \u0026#34;$1 \u0026#34; when retrieving object {{object_id}}\u0026#34;;}\u0026#39; To leverage all these curl options, we still need to use the shell module in Ansible to call the command in shell. Other helpful Ansible commands include ansible-pull for pulling playbooks from VCS repo, and ansible-console for interactive adhoc command execution.\nTags and extra variables Both tags(-t) and extra variables (-e) are great ways to achieve flow control in playbooks. You can specify to run tasks with certain tags or skip tasks with certain tags. Extra variables can override the default variables from the host or the group. Both are great tools to improve re-usability of a Playbook.\nSpeed up execution To speed up execution of Ansible tasks, there are several ways. For example, we can disable fact gathering by default so it only gathers fact if explicitly specified. This can be set in gathering=explicit under defaults section of ansible configuration file. If you have to gather facts, you may cache the facts using the following:\n[defaults] gathering = smart fact_caching_timeout = 86400 fact_caching = jsonfile fact_caching_connection = /tmp/ansible_fact_cache Other than caching, Ansible allows you to select from several execution strategies for playbook. The linear strategy introduces configurable parallelization per task. The free strategy introduces parallelization per play. linear (by default): Up to the fork limit of hosts will execute each task at the same time and then the next series of hosts until the batch is done, before going on to the next task. This mode ensures the progress is synchronized at each task. free: as specified above, this is preferred when there is no need to coordinate the progress between each host target. It is a \u0026#8220;free run\u0026#8221; for each host all the way till the end of the playbook. debug: essentially linear strategy except that the progress is controlled by an interactive debug session The fork limit, with a conservative default of 5, can be adjusted in Ansible configuration. The execution strategy can be either specified in Ansible configuration, or specified per play. For example, the following snippet sets the strategy to free for the current play:\n--- - hosts: all strategy: free tasks: ... Ansible documentation also mentions some play-level keywords to control execution. The serial keyword, is one of them. It can be set along with any strategy above, and it introduces the effect of hosts batching. The value can be a single number, a percentage, or even a list of numbers (if size for each batch is different). Note that the batch size should not exceed the fork limit. This is particularly useful in rolling upgrades. For example:\n--- - name: test play hosts: webservers serial: \u0026#34;30%\u0026#34; With the parallelization capacity outlined above, a potential concern is some heavy-lifting task may consume a lot of resources, if being executed for all hosts at the same time. Luckily, Ansible has a task/block level keyword throttle, which \u0026#8220;de-parallelize\u0026#8221; the multi-host progress at a particular task, or block. Here is an example provided by Ansible documentation:\ntasks: - command: /path/to/cpu_intensive_command throttle: 1 If there are long running tasks, we can specify async and poll values so Ansible leaves a task running and check back later. For example, the following task allows Ansible to move on and check back every 5 seconds, if the task takes longer than 45 seconds, it is considered failed:\n--- - hosts: all remote_user: root tasks: - name: simulate long running task for 15 sec, wait for up to 45 sec, poll every 5 sec command: /bin/sleep 15 async: 45 poll: 5 Python Version The recommendation is to use Python3 for any new development because there is no dependency. If there is no preference specified, Ansible tries to find out the appropriate interpreter and it can be seen in the response of ansible ping module. You can also force the interpreter by providing additional parameter ansible_python_interpreter. To change default interpreter, specify interpreter_python in ansible.cfg. For example:\n[defaults] inventory=~/ansible/inventories/site.yml library=~/ansible/library/ vault_password_file = ~/ansible/.vault_key host_key_checking = False display_skipped_hosts = False retry_files_enabled = False interpreter_python=/usr/bin/python3 [privilege_escalation] become_method=sudo [ssh_connection] ssh_args = -C -o ControlMaster=auto -o ControlPersist=1h pipelining = True My open issues I have some minor details that I have not been able to address, after a lot of time googling around. So I have to leave them for future reference.\nIf an Ansible playbook involves multiple plays (i.e. each with their own host), there is no way to persist a variable across different plays. A dumb alternative is to make all the variables to use available for every single host (under all directory).\nIn Jinja2 template, if I need to access the group of a target host (as defined in inventory), and the target belongs to multiple groups, I cannot filter to match the group I need.\nPrevious PostDocker Compose, Docker Stack and Docker Swarm Next PostSetup WSL2 (and Docker) on Windows 10 ","date":"2020-05-25T22:08:54-04:00","permalink":"/2020/05/ansible-directory-for-scalability-2-of-2/","title":"Ansible at scale 2 of 2"},{"content":"This posting covers some basic docker orchestration tools.\nDocker Compose Docker Compose\u0026#8217;s predecessor is a tool called Fig developed by Orchard, which was acquired by Docker in 2014, with Fig renamed to Docker Compose. Docker Compose is the official container management tool. It is essentially a python script that parses yaml file, to make Docker API calls to manage containers dynamically. It is installed along with Docker on MacOS and Windows. On Linux, you will have to download package with curl command and install manually. Docker Compose has three versions so far and we should create new template with v3. The Docker compose yaml template consists of three parts:\nservices: similar to docker run build: specify Dockerfile to build image cap_add, cap_drop: specify kernel capabilities (e.g. NET_ADMIN, SYS_ADMIN) command: override default startup command by container container_name depends_on devices: map host device to container dns dns_search: entryppoint: override entry point from image env_file: specify file that stores environment variable environment: specify environment variable image: specify the location of image pid: share the PID namespace with host ports: expose network ports. HOST:CONTAINER networks volumes: mount host volume to container networks: similar to docker network create volumes: similar to docker volume create Here is a typical structure of docker compose yaml template (wordpress):\nversion: \u0026#34;3.8\u0026#34; services: mysql: image:mysql:5.7 volumes: - mysql_data:/var/lib/mysql restart: always environment: MYSQL_ROOT_PASSWORD:root MYSQL_DATABASE:mywordpress MYSQL_USER:digihunch MYSQL_PASSWORD:hunchdigi wordpress: depends_on: - mysql image: wordpress:php7.4 ports: - \u0026#34;8080:80\u0026#34; restart:always environment: WORDPRESS_DB_HOST:mysql:3306 WORDPRESS_DB_USER:digihunch WORDPRESS_DB_PASSWORD: hunchdigi WORDPRESS_DB_NAME: digihunch networks: frontend: backend: volumes mysql-data: {} In summary, Docker Compose is an orchestration tool for single host, typically seen in development and test environment with dependencies between services.\nDocker Stack A stack is a set of related services and infrastructure that gets deployed and managed as a unit. A docker stack file has the same format as Docker Compose file, with the only requirement that the version: key specify a value of 3.0. The other difference between Docker Stacks and Docker Compose, is that stacks do not support builds. All images have to be built prior to deploying the stack.\nFrom the stack file, Docker first executes the network section and create networks that do not exist. Then it goes through other elements. A service is a JSON collection(dictionary) that contains a bunch of keys. The image key is the only mandatory key in the service objects, which will be pulled from Docker Hub by default. Ports key maps the port of Swarm to the port of each service replica. By default, all ports are mapped using ingress mode. This means they\u0026#8217;ll be mapped and accesible from every node in the Swarm -even nodes not running a replica. The alternative is host mode, where ports are only mapped on Swarm nodes running replicas for the service.\nThe environment key lets you inject environment variables into services replica.\nThe secrets key defines two secrets \u0026#8211; revprox_cert and revprox_key. These must be defined in the top-level secrets key, and must exist on the system. Secrets get mounted into service replicas as a regular file. The secrets defined in this service will be mounted in each service replica as /run/secrets/revprox_cert and /run/secrets/revprox_key, unless otherwise specified.\nThe volumes key is used to mount pre-created volumes and host directories into a service replica.\nThe networks key ensures that all replicas for the service will be attached to the front-tier network. The network specified here must be defined in the networks top-level key, and if it doesn’t already exist, Docker will create it as an overlay.\nThe service also defines a placement constraint under the deploy key. This ensures that replicas for this service will always run on Swarm worker nodes. Placement constraints are a form of topology-aware scheduling, and can be a great way of influencing scheduling decisions.\nWhen Docker stops a container, it issues a SIGTERM to the process with PID 1 inside the container. The container (its PID 1 process) then has a 10-second grace period to perform any clean-up operations. If it doesn’t handle the signal, it will be forcibly terminated after 10 seconds with a SIGKILL. The stop_grace_period property overrides this 10 second grace period.”\nAlthough you may scale a docker service as part of a stack with scale command, it is not recommended. Instead, stack file should be used as the ultimate source of truth (declarative method vs imperative method). All changes to the stack should be made to the stack file, and the updated stack file used to redeploy the app.\nDocker Swarm For multi-host cluster, Docker Swarm facilitates the deployment of micro-services. Docker Swarm is:\na cluster of Docker hosts: enterprise-grade, secure communication, PKI with automation, dynamic addition of nodes an orchestration engine, with deployment automation, deploying native swarm apps (using Docker API) and Kubernetes apps. A Docker nodes can be physical servers, VMs, cloud instances, etc. Nodes are configured as managers or workers. Managers look after the control plane of the cluster, and dispatches tasks to workers. Managers forms a distributed management cluster on its own, and they use Raft protocol to ensure consistency. Workers accept tasks from managers and execute them. Swarm mandatorily uses TLS to encrypt communications, authenticate nodes, and authorize roles, with Automatic key rotation.\nThe atomic unit of scheduling on a swarm is the service. When a container is wrapped in a service, we call it a task or a replica, and the service construct adding things like scaling, rolling updates and simple rollbacks.\nTo initialize a swarm, we need to have the following ports open. Then we can initialize the first manager node, join additional manager nodes, and then join workers.\n2377/tcp: for secure client-to-swarm communication 7946/tcp \u0026amp; udp: for control plane gossip 4789/udp: for VXLAN-based overlay networks A Docker node can exist either in single-engine mode as stand alone, or in swarm mode as part of a swarm. Service only exist in swarm mode. Running docker swarm init on a Docker host in single-engine mode will switch that node into swarm mode, create a new swarm, and make the node the first manager of the swarm. Then additional nodes can be joined as managers or workers.\nSwarm managers have native support for high availability, through an active-passive, multi-manager HA. Only one manager is considered active (the leader), which is the only one that will ever issue live commands against the swarm. If a passive manager receives commands for the swarm, it proxies them across to the leader.\nManagers are either leaders or followers. This is Raft terminalogy because swarm uses an impelementation of the Raft consensus althorithm to power manager HA. As to HA, the following two best practices apply:\ndeploy an odd number of managers don\u0026#8217;t deploy too many managers (3 or 5 recommended, never more than 7) Having an odd number of managers reduces the chances of split-brain conditions. Having less than 7 managers ensures that achieving consensus is quick.\nWith a service, we can specify name, port mappings, network to attach to, and images, as well as desired state for an application service. It is recommended in production environment to use docker-compose template to specify service. Services have replication mode, and the default is replicated. This will deploy a desired number of replicas and distribute them as evenly as possible across the cluster. The other mode is global, which runs a single replica on every node in the swarm.\nRunning \u0026#8220;docker service scale\u0026#8221; command can scale the number of service replicas from 5 to 10, which in the background updates the service\u0026#8217;s desired state to the newly specified number of replicas. Behind the scenes, Swarm also runs a scheduling algorithm that defaults to balancing replicas as evenly as possible across the node in the swarm. Docker makes it super easy to push updates to deployed applications. With rolling update, you may specify number of replicas to update at a time, and cool-off period per update.\nHere is a great article on the difference between Docker Swarm and Kubernetes.\nPrevious PostAnsible at scale 1 of 2 Next PostAnsible at scale 2 of 2 ","date":"2020-05-24T21:58:03-04:00","permalink":"/2020/05/docker-swarm-brief-notes/","title":"Docker Compose, Docker Stack and Docker Swarm"},{"content":"The Ansible In Depth white paper outlines Ansible\u0026#8217;s use cases in four categories:\nConfiguration managementApplication deploymentOrchestration: for coordinating a multi-machine process such as interacting with load balancer and rolling cluster upgradeAs-needed task execution: ad-hoc tasks on large number of hosts At work, my original automation scheme involves several Ansible Playbooks that started off simple but have been sprawling ever since. I have to spend some time to revamp the Ansible code base, following the best practices from the official documentation. The goal is to:\nReduce the number of Playbooks;Improve code re-usability (by Ansible roles);Increase portability across different customer environment;Improve security;Re-organize the directory so that more team members can contribute to different parts of it. The Ansible code base is used by our customer service engineers, some of whom are ingrained with the established way they have been using certain Playbooks in their daily tasks. This requires me, throughout the development initiative, to ensure a consistent interaction between them and their Playbook commands.\nInventory and variables We cannot guarantee that every custom environment are identical, but what we can do is make sure that for a new environment, the only change to make is inventory and variables. This is where we can strike a balance between portability and customization. No changes should be made to tasks, roles or Playbooks when the Ansible directory is deployed at a different customer environment.\nIf the total number of servers to manage are under 100, all can be listed in a single inventory file, and specify the only inventory file as default so the -i switch is not required for every Ansible command run. If there are more than 100 servers, it is advisable to separate them out into several inventory files in YAML, each less than 200 lines. You will have to specify inventory file with -i each time you run Ansible command.\nThe inventory may contain a hierarchy of groups, in order to facilitate command calls to specific groups of servers. For example:\nall: children: prod: children: prod_app: children: prod_app_dc1: hosts: apphost01: apphost03: zk_id: 1 apphost05: apphost07: zk_id: 2 apphost09: apphost11: zk_id: 3 apphost13: prod_app_dc2: hosts: apphost02: apphost04: zk_id: 1 apphost06: apphost08: zk_id: 2 apphost10: apphost12: zk_id: 3 apphost14: vars: has_app: yes has_nginx: yes has_db: no prod_db: children: prod_db_dc1: hosts: dbhost01: dbhost03: dbhost05: dbhost07: dbhost09: dbhost11: vars: clustered: yes bk_node: dbhost11 prod_db_dc2: hosts: dbhost02: dbhost04: dbhost06: dbhost08: dbhost10: dbhost11: vars: clustered: yes bk_node: dbhost12 vars: has_app: no has_nginx: no has_db: yes vars: ansible_become_pass: \u0026#39;{{site_prod_root_pw}}\u0026#39; test: children: test_dc1: hosts: tapphost01: test_dc2: hosts: tapphost02: vars: ansible_become_pass: \u0026#39;{{site_test_root_pw}}\u0026#39; has_app: yes has_nginx: yes has_db: yes vars: ansible_become: yes ansible_become_method su ansible_become_user: root User of Ansible Playbook can use -l to specify a pattern that matches a single or multiple groups, such as prod_db_dc*. In the above example, the password is not stored in clear text. They should reference a variable from a separate file encrypted by ansible-vault.\nFrom Playbooks to roles As a refresher from the white paper, a single \u0026#8220;task\u0026#8221; in Ansible is essentially a module call with parameters. A \u0026#8220;play\u0026#8221; consists of a series of tasks (defined under \u0026#8220;tasks\u0026#8221; section) all to execute on a specified host (defined under \u0026#8220;hosts\u0026#8221; section). A Playbook consist of one or several plays, as shown in this example. In reality though, a Playbook usually contains only one play. Even that one play can grow to an unmanageable length, as complexity increases over time. This is where we need to change our approach towards scalability and manageability.\nThe Ansible community advocates the use of roles in place of Playbooks. The concept of Ansible \u0026#8220;role\u0026#8221; seems fairly abstract and confusing at the beginning. The word \u0026#8220;role\u0026#8221; pictures a static server state, whereas our existing Playbooks are full of actions (think of shell scripts). How would one convert an action list into static states? After some thought, I came to the understanding that roles should be thought of as desired end state. Yes, the end state is static, but that\u0026#8217;s all we care about. This is essentially the whole idea of Ansible\u0026#8217;s desired state configuration: you start from the end state and leave it to modules to complete what needs to be done to reach that state. The concept of role perfectly reflects how Ansible wants you to think about solving an infrastructure problem \u0026#8211; stop thinking about what you need to do. Instead, think about what you ultimately want, start from the desired state and work backwards.\nIn our own setup, the best practice turned out to be: if the Playbook involves a single play with less than 5 tasks, just stick to Playbook. We don\u0026#8217;t get rid of Playbooks just for the sake of it. Otherwise, if a Playbook has grown to more than 5 tasks, we need to think about our desired state, and either implement a new role, or incorporate it into an existing role. This is the time we have to transition from the Playbook oriented thinking to the role oriented thinking. Each role directory can include a task sub-directory with main.yml that references the rest of the tasks. Each role can define its own role-related variables. If there\u0026#8217;s a lot in common between two roles, we can even have a common role with or without its main.yml.\nA simplified version of our Ansible directory structure looks like this:\n├── deploy-app.yml ├── deploy-db.yml ├── inventories │ ├── group_vars │ │ ├── all │ │ │ ├── all.yml │ │ │ └── vault_all.yml │ │ ├── prod_dc1_db.yml │ │ ├── prod_dc2_db.yml │ │ ├── test_dc1_db.yml │ │ └── test_dc2_db.yml │ ├── host_vars │ └── site_inventory.yml ├── roles │ ├── common │ │ ├── files │ │ └── tasks │ │ ├── log.yml │ │ ├── skip_self.yml │ │ └── validate_path.yml │ ├── db_conf │ │ ├── defaults │ │ │ └── main.yml │ │ ├── files │ │ ├── handlers │ │ │ └── main.yml │ │ ├── meta │ │ │ └── main.yml │ │ ├── README.md │ │ ├── tasks │ │ │ ├── main.yml │ │ │ ├── start_db.yml │ │ │ ├── stop_db.yml │ │ │ └── update_cluster_var.yml │ │ ├── templates │ │ │ ├── myid.j2 │ │ │ ├── db_properties.j2 │ │ │ └── zookeeper_properties.j2 │ │ └── vars │ │ └── main.yml │ └── app_conf │ ├── defaults │ │ └── main.yml │ ├── files │ ├── handlers │ │ └── main.yml │ ├── meta │ │ └── main.yml │ ├── README.md │ ├── tasks │ │ ├── bk_app_conf.yml │ │ ├── empty_app_conf.yml │ │ ├── main.yml │ │ ├── push_app_conf.yml │ │ ├── start_app.yml │ │ ├── stop_app.yml │ │ ├── tar_app_conf.yml │ │ ├── untar_app_conf.yml │ │ └── update_cluster_var.yml │ ├── templates │ │ └── dbref_xml.j2 │ └── vars │ └── main.yml ├── service-app.yml └── service-db.yml Variables specific to a group of hosts or individual hosts can be included in different yml files. When the entire directory is moved to a different customer environment, our engineers will need to update the inventory and variable files. The task, roles and Playbooks should build their logics using those variables. Vault Our previous implementation of Ansible Playbook stores sudo password base64 encoded and use no_log to avoid displaying values. Now we move those to encrypted variable yml file using ansible-vault. We reference the value to encrypt as regular variable:\nansible_become_pass: \u0026#39;{{passtoencrypt}}\u0026#39; ansible_become_method: sudo ansible_become: yes Then we run the following:\nansible-vault create vault_all.yml This prompt for a key, and once you type in the key it opens a text editor where we can store the real password. For example:\npasstoencrypt: MyP@ssw0rd4real! Use the text editor to save file. The file is now saved encrypted and must be open with correct key (aka vault password). If we call Ansible Playbook with \u0026#8211;ask-vault-pass switch then the Playbook will prompt for key input, or use include_vars to include variable from vault file (example). If we want to even skip this, we can store the key in a file and reference them from vault_password_file in ansible.cfg\nAnsible Vault has more commands to edit or view the encrypted variables in the documentation.\nOptimize connection OpenSSH 5.6 and later supports multiplexing where multiple SSH sessions share a TCP connection. This can be turned on so that the following SSH connections save the time of TCP handshake. This can be configured in ansible configuration file under ssh_connection. Below is an example of this option with ControlPersist=1h. So the TCP connection is torn down after 1 hour. [ssh_connection] ssh_args = -C -o ControlMaster=auto -o ControlPersist=1h The other option we can leverage is pipelining. Ansible takes three steps to execute a task:\nbuild a python script based on module usedcopy the python script to remote hostexecute the python script on the remote host If pipelining is turned on, the python script is passed in along with the SSH session, this would save a roundtrip and increase performance. Pipelining can be configured under ssh_connection in Ansible configuration file:\n[ssh_connection] pipelining = True In the example below we can see by pipelining we cut the number of connection in half:\n# with pipelining [ghunch@control-host ~]$ ansible remote-host -vvvv -m ping | grep EST \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch # without pipelining [ghunch@control-host ~]$ ansible remote-host -vvvv -m ping | grep EST \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch \u0026lt;remote-host\u0026gt; ESTABLISH SSH CONNECTION FOR USER: ghunch Note that if we use sudo command, then we need to disable requiretty in /etc/sudoers on the remote host.\nCustom Module It\u0026#8217;s fairly straightforward to build a custom module in Ansible. Just place the module file (modulename.py) in inventory directory and use it as you would with regular Ansible module. The module file needs to be completed in Python with certain return value. Before creating custom module, you should look for existing modules to avoid re-inventing the wheel. You may also need to determine whether you simply need to run a python script on target host (with Ansible\u0026#8217;s script module), or you really need an Ansible module. The former is procedural, and the latter focus on desired state. Custom module is more used in proprietary development. Previous PostBalloon steals memory from virtual machines Next PostDocker Compose, Docker Stack and Docker Swarm ","date":"2020-05-17T19:38:34-04:00","permalink":"/2020/05/revamp-ansible-directory-for-scalability-1-of-2/","title":"Ansible at scale 1 of 2"},{"content":"This article is my experience with memory balloon on virtual machine.\nI came across an ElasticSearch server (ESXi guest with 32GB physical memory) where the main process keeps dying of OOM. Even worse, after the OOM event, the free memory left is about 10G and Elastic Search cannot start because its JVM is set with -Xms16g in /etc/elasticsearch/jvm.options. So I need to address the OutOfMemory error.\nTo understand what triggered OOM, we can use dmesg or just check /var/log/message, where the memory snapshot by OOM killer is displayed during the kernel panic:\nApr 26 04:18:15 elastichost kernel: kworker/7:1 invoked oom-killer: gfp_mask=0x200d2, order=0, oom_score_adj=0 Apr 26 04:18:15 elastichost kernel: kworker/7:1 cpuset=/ mems_allowed=0 Apr 26 04:18:15 elastichost kernel: CPU: 7 PID: 13968 Comm: kworker/7:1 Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1 Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 Apr 26 04:18:15 elastichost kernel: Workqueue: events_freezable vmballoon_work [vmw_balloon] Apr 26 04:18:15 elastichost kernel: Call Trace: Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa2961e41\u0026gt;] dump_stack+0x19/0x1b Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa295c86a\u0026gt;] dump_header+0x90/0x229 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa2301052\u0026gt;] ? ktime_get_ts64+0x52/0xf0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23581ef\u0026gt;] ? delayacct_end+0x8f/0xb0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23ba4e4\u0026gt;] oom_kill_process+0x254/0x3d0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23b9f8d\u0026gt;] ? oom_unkillable_task+0xcd/0x120 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23ba036\u0026gt;] ? find_lock_task_mm+0x56/0xc0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23bad26\u0026gt;] out_of_memory+0x4b6/0x4f0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa295d36e\u0026gt;] __alloc_pages_slowpath+0x5d6/0x724 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23c1105\u0026gt;] __alloc_pages_nodemask+0x405/0x420 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa240df68\u0026gt;] alloc_pages_current+0x98/0x110 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffc05f1f84\u0026gt;] vmballoon_work+0x454/0x6ff [vmw_balloon] Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa22b9d4f\u0026gt;] process_one_work+0x17f/0x440 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa22bade6\u0026gt;] worker_thread+0x126/0x3c0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa22bacc0\u0026gt;] ? manage_workers.isra.25+0x2a0/0x2a0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa22c1c31\u0026gt;] kthread+0xd1/0xe0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa22c1b60\u0026gt;] ? insert_kthread_work+0x40/0x40 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa2974c37\u0026gt;] ret_from_fork_nospec_begin+0x21/0x21 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa22c1b60\u0026gt;] ? insert_kthread_work+0x40/0x40 Apr 26 04:18:15 elastichost kernel: Mem-Info: Apr 26 04:18:15 elastichost kernel: active_anon:3649243 inactive_anon:439380 isolated_anon:0#012 active_file:480 inactive_file:693 isolated_file:0#012 unevictable:0 dirty:2 writeback:0 unstable:0#012 slab_reclaimable:77845 slab_unreclaimable:10219#012 mapped:10086 shmem:9596 pagetables:16727 bounce:0#012 free:50116 free_pcp:238 free_cma:0 Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993 Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122768kB min:5972kB low:7464kB high:8956kB active_anon:471052kB inactive_anon:471040kB active_file:232kB inactive_file:460kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:584kB shmem:300kB slab_reclaimable:23908kB slab_unreclaimable:3872kB kernel_stack:656kB pagetables:6440kB unstable:0kB bounce:0kB free_pcp:288kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:214 all_unreclaimable? no Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163 Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61804kB min:61576kB low:76968kB high:92364kB active_anon:14125920kB inactive_anon:1286480kB active_file:1688kB inactive_file:2312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:8kB writeback:0kB mapped:39760kB shmem:38084kB slab_reclaimable:287472kB slab_unreclaimable:36988kB kernel_stack:5680kB pagetables:60468kB unstable:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:698 all_unreclaimable? no Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0 Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 407*4kB (UEM) 506*8kB (UEM) 431*16kB (UM) 364*32kB (M) 272*64kB (UM) 179*128kB (UM) 99*256kB (UM) 35*512kB (M) 11*1024kB (M) 2*2048kB (M) 0*4096kB = 123164kB Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15576*4kB (UM) 39*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62616kB Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Apr 26 04:18:15 elastichost kernel: 44652 total pagecache pages Apr 26 04:18:15 elastichost kernel: 34162 pages in swap cache Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884045, delete 10845872, find 3294122/3678936 Apr 26 04:18:15 elastichost kernel: Free swap = 0kB Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly Apr 26 04:18:15 elastichost kernel: 193436 pages reserved Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 12996 100 6314 0 systemd-journal Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 83 32 88 -900 dbus-daemon Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 50 15 41 0 irqbalance Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 47 18 41 0 systemd-logind Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 113 58 246 0 vmtoolsd Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 39 27 140 0 ntpd Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 132 62 970 0 snmpd Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 4770 100 482 0 rsyslogd Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6332 100 781 0 sh-metricbeat Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 30 18 129 0 crond Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3994129 15027 457539 0 java Apr 26 04:18:15 elastichost kernel: [16264] 495 16264 18032 0 31 162 0 controller Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16070 (java) score 512 or sacrifice child Apr 26 04:18:15 elastichost kernel: Killed process 16264 (controller) total-vm:72128kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Apr 26 04:18:15 elastichost kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Apr 26 04:18:15 elastichost kernel: java cpuset=/ mems_allowed=0 Apr 26 04:18:15 elastichost kernel: CPU: 2 PID: 16265 Comm: java Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1 Apr 26 04:18:15 elastichost kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 Apr 26 04:18:15 elastichost kernel: Call Trace: Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa2961e41\u0026gt;] dump_stack+0x19/0x1b Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa295c86a\u0026gt;] dump_header+0x90/0x229 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa2301052\u0026gt;] ? ktime_get_ts64+0x52/0xf0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23581ef\u0026gt;] ? delayacct_end+0x8f/0xb0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23ba4e4\u0026gt;] oom_kill_process+0x254/0x3d0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23b9f8d\u0026gt;] ? oom_unkillable_task+0xcd/0x120 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23ba036\u0026gt;] ? find_lock_task_mm+0x56/0xc0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23bad26\u0026gt;] out_of_memory+0x4b6/0x4f0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa295d36e\u0026gt;] __alloc_pages_slowpath+0x5d6/0x724 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23c1105\u0026gt;] __alloc_pages_nodemask+0x405/0x420 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa240df68\u0026gt;] alloc_pages_current+0x98/0x110 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23b6347\u0026gt;] __page_cache_alloc+0x97/0xb0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23b8fa8\u0026gt;] filemap_fault+0x298/0x490 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffc0484d0e\u0026gt;] __xfs_filemap_fault+0x7e/0x1d0 [xfs] Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa22c2dc0\u0026gt;] ? wake_bit_function+0x40/0x40 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffc0484f0c\u0026gt;] xfs_filemap_fault+0x2c/0x30 [xfs] Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23e444a\u0026gt;] __do_fault.isra.59+0x8a/0x100 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23e49fc\u0026gt;] do_read_fault.isra.61+0x4c/0x1b0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23e93a4\u0026gt;] handle_pte_fault+0x2f4/0xd10 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa23ebedd\u0026gt;] handle_mm_fault+0x39d/0x9b0 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa296f5e3\u0026gt;] __do_page_fault+0x203/0x500 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa296f915\u0026gt;] do_page_fault+0x35/0x90 Apr 26 04:18:15 elastichost kernel: [\u0026amp;lt;ffffffffa296b758\u0026gt;] page_fault+0x28/0x30 Apr 26 04:18:15 elastichost kernel: Mem-Info: Apr 26 04:18:15 elastichost kernel: active_anon:3607073 inactive_anon:480522 isolated_anon:0#012 active_file:8 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:0 writeback:1 unstable:0#012 slab_reclaimable:75170 slab_unreclaimable:10131#012 mapped:2070 shmem:9592 pagetables:16696 bounce:0#012 free:50006 free_pcp:72 free_cma:0 Apr 26 04:18:15 elastichost kernel: Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 2829 31993 31993 Apr 26 04:18:15 elastichost kernel: Node 0 DMA32 free:122560kB min:5972kB low:7464kB high:8956kB active_anon:471000kB inactive_anon:471076kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2897760kB mlocked:0kB dirty:0kB writeback:0kB mapped:292kB shmem:288kB slab_reclaimable:22832kB slab_unreclaimable:3752kB kernel_stack:672kB pagetables:6408kB unstable:0kB bounce:0kB free_pcp:272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:215 all_unreclaimable? yes Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 29163 29163 Apr 26 04:18:15 elastichost kernel: Node 0 Normal free:61572kB min:61576kB low:76968kB high:92364kB active_anon:13957292kB inactive_anon:1451012kB active_file:32kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29866500kB mlocked:0kB dirty:0kB writeback:4kB mapped:7988kB shmem:38080kB slab_reclaimable:277848kB slab_unreclaimable:36756kB kernel_stack:5664kB pagetables:60376kB unstable:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:500 all_unreclaimable? yes Apr 26 04:18:15 elastichost kernel: lowmem_reserve[]: 0 0 0 0 Apr 26 04:18:15 elastichost kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB Apr 26 04:18:15 elastichost kernel: Node 0 DMA32: 393*4kB (UEM) 490*8kB (UEM) 453*16kB (EM) 369*32kB (UEM) 269*64kB (M) 180*128kB (M) 98*256kB (M) 35*512kB (M) 12*1024kB (UM) 1*2048kB (M) 0*4096kB = 122148kB Apr 26 04:18:15 elastichost kernel: Node 0 Normal: 15440*4kB (UM) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61768kB Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Apr 26 04:18:15 elastichost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Apr 26 04:18:15 elastichost kernel: 42975 total pagecache pages Apr 26 04:18:15 elastichost kernel: 33308 pages in swap cache Apr 26 04:18:15 elastichost kernel: Swap cache stats: add 10884233, delete 10846914, find 3294127/3678944 Apr 26 04:18:15 elastichost kernel: Free swap = 0kB Apr 26 04:18:15 elastichost kernel: Total swap = 2097148kB Apr 26 04:18:15 elastichost kernel: 8388478 pages RAM Apr 26 04:18:15 elastichost kernel: 0 pages HighMem/MovableOnly Apr 26 04:18:15 elastichost kernel: 193436 pages reserved Apr 26 04:18:15 elastichost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Apr 26 04:18:15 elastichost kernel: [ 3164] 0 3164 47420 4016 100 6314 0 systemd-journal Apr 26 04:18:15 elastichost kernel: [ 3192] 0 3192 11158 2 24 174 -1000 systemd-udevd Apr 26 04:18:15 elastichost kernel: [ 3198] 0 3198 66023 0 31 117 0 lvmetad Apr 26 04:18:15 elastichost kernel: [ 6051] 0 6051 15511 20 29 138 -1000 auditd Apr 26 04:18:15 elastichost kernel: [ 6073] 81 6073 14557 78 32 88 -900 dbus-daemon Apr 26 04:18:15 elastichost kernel: [ 6075] 32 6075 18412 16 39 166 0 rpcbind Apr 26 04:18:15 elastichost kernel: [ 6076] 0 6076 50404 0 37 171 0 gssproxy Apr 26 04:18:15 elastichost kernel: [ 6077] 0 6077 5422 46 15 41 0 irqbalance Apr 26 04:18:15 elastichost kernel: [ 6078] 0 6078 118943 155 85 362 0 NetworkManager Apr 26 04:18:15 elastichost kernel: [ 6086] 0 6086 6594 42 18 41 0 systemd-logind Apr 26 04:18:15 elastichost kernel: [ 6087] 0 6087 24892 0 42 402 0 VGAuthService Apr 26 04:18:15 elastichost kernel: [ 6088] 0 6088 56746 91 58 246 0 vmtoolsd Apr 26 04:18:15 elastichost kernel: [ 6089] 999 6089 153086 130 61 1787 0 polkitd Apr 26 04:18:15 elastichost kernel: [ 6129] 38 6129 11817 36 27 140 0 ntpd Apr 26 04:18:15 elastichost kernel: [ 6464] 0 6464 56962 130 62 970 0 snmpd Apr 26 04:18:15 elastichost kernel: [ 6468] 0 6468 97733 1215 100 485 0 rsyslogd Apr 26 04:18:15 elastichost kernel: [ 6470] 0 6470 470477 6230 100 781 0 sh-metricbeat Apr 26 04:18:15 elastichost kernel: [ 6472] 0 6472 250084 2279 72 2367 0 sh-filebeat Apr 26 04:18:15 elastichost kernel: [ 6483] 0 6483 28189 26 57 231 -1000 sshd Apr 26 04:18:15 elastichost kernel: [ 6485] 0 6485 143455 107 97 2667 0 tuned Apr 26 04:18:15 elastichost kernel: [ 6569] 0 6569 76290 625 41 188 0 avagent.bin Apr 26 04:18:15 elastichost kernel: [ 6574] 29 6574 12239 1 27 254 0 rpc.statd Apr 26 04:18:15 elastichost kernel: [ 6601] 0 6601 31572 27 18 129 0 crond Apr 26 04:18:15 elastichost kernel: [ 6609] 0 6609 27523 1 10 32 0 agetty Apr 26 04:18:15 elastichost kernel: [ 7451] 494 7451 315970 36412 364 8550 0 node Apr 26 04:18:15 elastichost kernel: [11272] 0 11272 39154 0 80 336 0 sshd Apr 26 04:18:15 elastichost kernel: [11277] 0 11277 28885 2 12 112 0 bash Apr 26 04:18:15 elastichost kernel: [16070] 495 16070 47122631 3993992 15027 457699 0 java Apr 26 04:18:15 elastichost kernel: Out of memory: Kill process 16291 (java) score 512 or sacrifice child Apr 26 04:18:15 elastichost kernel: Killed process 16070 (java) total-vm:188490524kB, anon-rss:15975968kB, file-rss:0kB, shmem-rss:0kB Apr 26 04:18:16 elastichost systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL Apr 26 04:18:16 elastichost systemd: Unit elasticsearch.service entered failed state. Apr 26 04:18:16 elastichost systemd: elasticsearch.service failed. The line \u0026#8220;Free swap = 0kB\u0026#8221; suggest the trigger of OOM is out of swap. So does ElasticSearch contribute to the swap shortage?\nElasticSearch was the main process on the VM and I noticed that the OS did not disable swap, which does not align with the best practice from ElasticSearch community. In order to find out whether Elastic Search is pushed to use swap, we can get the process ID:\npidof java ElasticSearch happens to be the only Java based process and the PID is 2283, the following command shows the swap usage by this process:\ncat /proc/2283/status | grep VmSwap If it shows a non-zero value, then ElasticSearch is using swap and you should expect some performance issues. We definitely should disable swap on ElasticSearch but does that solve the problem? Probably not because we didn\u0026#8217;t address what caused the memory pressure in the first place. Disabling swap most likely makes ElastciSearch last longer before something else such as shortage of available memory triggers OOM. In search for the source of memory shortage, I checked the result of free command and the top command. The free command (-mh) shows the following:\ntotal used free shared buff/cache available Mem: 31G 20G 10G 117M 243M 10G Swap: 2.0G 249M 1.8G I tried to follow my example from a previous post to make sense of the memory reads. When I monitor process with top and watch for RSS column, I cannot identify a single process that even takes more than a few hundred megabytes. I\u0026#8217;ve downloaded a tool smem from epel repo, and the result of \u0026#8220;smem -kt\u0026#8221; suggest that the total RSS is about 372.2M:\nSo there is about 19.6GB of memory usage unaccounted for. There is something that takes this much memory in the VM and haunting round even after OOM killer! Inspired by this post, I was able to identify the culprit, that is the memory balloon. We don\u0026#8217;t have access to the host but from the Guest OS, we can tell by vmware-toolbox-cmd (need to install yum package open-vm-tools):\nvmware-toolbox-cmd stat balloon The result displays a whopping 20807 MB as memory balloon! This needs to be sent to PaaS vendor for investigation but it is likely a result of memory over-allocation/over-commitment at host level, as well as the setup where the memory of guest is not reserved. Looking at the original log snippet, the line with \u0026#8220;events_freezable vmballoon_work\u0026#8221; is also an indicator of balloon causing the OOM.\nHypervisor needs memory ballooning to reclaim memory from guest. Since the guest OS does not expect the amount of physical memory to change, hypervisor has to maintain the illusion that the guest has its fixed amount of physical memory. The hypervisor first computes the amount of memory that needs to reclaim, then it leverage some low-level mechanism such as a balloon driver (a pseudo-device driver) installed on guest. The driver communicates with hypervisor and is told to allocate or de-allocate memory. If the driver is told to allocate memory to host, it tells the guest OS to pin the allocated pages into physical memory so they are locked and the physical memory available to guest OS is decreased. All these low level mechanisms explains why it is hard to account for memory consumption based on process running in Guest OS.\nMemory balloon is a dynamic process and the hypervisor adjusts the size of balloon. However, the hypervisor, the guest OS and the application process (ElasticSearch in this case) may enter a nuance interaction:\nThe hypervisor reclaims memory from guest OS (ballooning); Guest OS panicked with OOM exception; OOM picks a process to kill, based on oom_score. Unfortunately, the true culprit (balloon) is not visible to guest OS, hence exempted from being considered to kill. By killing the application process (usually the main application for VM), a fair chunk of guest memory is freed up; the balloon then became more aggressive on the guest as seeing more memory becomes \u0026#8220;available\u0026#8221;; the application on guest then does not have minimum memory on guest OS to start; How to solve this problem? There are several things to consider. On the hypervisor, keep the entire or part of the guest memory reserved for start of application. Tune the OOM scoring so the non-critical application get killed. Implement application daemon, etc.\nPrevious PostCassandra data model (as opposed to relational model) Next PostAnsible at scale 1 of 2 ","date":"2020-05-07T19:49:00-04:00","permalink":"/2020/05/understanding-where-the-memory-goes-on-linux-vm/","title":"Balloon steals memory from virtual machines"},{"content":"Bad data model design with Cassandra causes chronic pains as application scales. I had to re-read about data model design in \u0026#8220;Cassandra \u0026#8211; the Definitive Guide\u0026#8221; and keep my notes and thoughts in this post.\nThe data modelling in the relational world is indoctrinated to every students out of university. It embraces several things:\nEntity-Relation: we typically start with tables that represents entities, and then tables that expresses relations; Query design after table design: we can join multiple tables, index certain fields for better query performance; Data normalization: several normal forms (NFs) are brought up to better organize data; de-normalization only occurs when 1) performance bottleneck reached; and 2) specific requirement on retaining snapshots of previous (un-updated) value in a field; referential integrity: we can specify foreign keys on a table to reference the primary key of a record in another table; we can configure cascading deletes, etc; Anybody with years of experience with relational database may have all these built in their instinct. Unfortunately, Cassandra does not follow any of these patterns. For someone with relational database background, the trip entering the Cassandra design is very counter-intuitive.\nNo joins In Cassandra you have very limited options to achieve what you can do with joins in relational realm. One option is to duplicate the data column on different tables (a pattern against the \u0026#8220;normalization\u0026#8221; best practice). The second table is denormalized and it represents the join results. The other option rarely applied is to do the work on the client side.\nQuery-driven data modelling In relational database, you start writing queries after tables are laid out to pull together disparate data, using the relationship defined by the keys. The queries is secondary concern. It is assumed that you can always get the data you want as long as you have your tables modelled properly, even if you have to use several complex subqueries or join statements.\nIn Cassandra, You do not start with tables to represent entity. Instead, you would start with queries, and then organize data around the queries. This means an upfront effort must be made to investigate what queries the client application may perform, and work backwards with tables that answers those queries in the most efficient manner. Table names in Cassandra often takes names such as: hotels_by_poi, avaialbe_rooms_by_hotel_date, reservations_by_guest, reservations_by_hotel_date. On the tables the selection of partition key and clustering keys should also consider best query performance and avoids wide partitions. It should also ensure with best effort that a query should not have to travel across multiple partitions in order to return results.\nDenormalization Due to the query-driven modelling approach, Cassandra usually need to be designed with denormalization. The entire concept of normalization applies only to relational world and in Cassandra it\u0026#8217;s perfectly normal to organize data that are against NFs.\nDesigning for optimal storage Cassandra tables are each stored in separate files on disk. Its best practice to keep related columns defined together in the same table. We need to minimize the number of partitions that must be searched in order to satisfy a given query. Because the partition is a unit of storage that does not get divided across nodes, a query that searches a single partition will typically yield the best performance.\nThe book \u0026#8220;Cassandra: the definitive guide\u0026#8221; contains a great example of modelling hotel reservation system. This article is also a good guideline.\nPrevious PostHow memory usage adds up in Linux Next PostBalloon steals memory from virtual machines ","date":"2020-04-29T18:45:00-04:00","permalink":"/2020/04/cassandra-data-model-as-opposed-to-relational-database/","title":"Cassandra data model (as opposed to relational model)"},{"content":"There are too many metrics that describes some aspects about memory in Linux. This posting will make sense of those common metrics in Linux, CentOS as an example.\nThe most fundamental command is free and my favourite switch is -h for human readable reads. You can use -m, -k, -b for different units. The result looks like this:\ntotal used free shared buff/cache available Mem: 32780168 16832160 3200408 101356 12747600 15399528 Swap: 2097148 2055148 42000 Swap is essentially disk space and many application such as Cassandra, ElasticSearch recommend disabling swap as best practice and they do not want disk speed to drag the performance of memory. Many suggest that swap is not needed in today\u0026#8217;s era at all given the amount of memory for cheap. This is debatable. With the row for Mem, the four columns should add up to the total, as suggested in the chart below. total = free + used + shared + buff/cache The four columns from free command output are supposed to always add up to the physical memory size. This command simplifies things quite a bit and each of these values are actually taken from certain lines in /proc/meminfo:\nMetric from free commandMetric in /proc/meminfototalMemTotalused??freeMemFreesharedShmembuff/cacheCached + SlabavailableMemAvailable The buffer and cache (and even swap) can be freed by command. The value of used doesn\u0026#8217;t seem to come from anwhere in /proc/meminfo, but it should be calculable from the memory used per process, which can be seen from top command.\nIn the result of top command, the column RSS (resident set size) is from the VmRSS value in /proc/\u0026lt;pid\u0026gt;/status, it is the actual physical memory consumed by the process. This value is originally from the second read in /proc/\u0026lt;pid\u0026gt;/statm, which represents the number of pages. For example:\n[ghunch@centos ~]$ cat /proc/6495/status | grep VmRSS ; cat /proc/6495/statm VmRSS:\t20852916 kB 49829626 5213229 1212275 1 0 5773980 0 [ghunch@centos ~]$ getconf PAGE_SIZE 4096 Linux default page size is 4096 or 4K, so in the result from above, 5213229 x 4kB = 20852916 kB, which is the size of memory taken by process ID 6459. Therefore if we go through all processes and add up the VmRSS, we should get (close to) the used memory?\nBut wait a second, we have not account for slab info (memory used by kernel) yet, which is displayed in /proc/slabinfo. To calculate the total size taken by slab, we use \u0026lt;num_objs\u0026gt; and \u0026lt;objsize\u0026gt; columns from /proc/slabinfo.\nApart from that there is page table, the table that stores the mapping between virtual address and physical address, is stored in the physical memory as well and the size is specified in the PageTables entry in /proc/meminfo. Now our equation becomes:\nUsed Memory = (RSS for all processes) + (all objects in slab) + (page table)\nWe can use the following script to calculate the used memory and compare it with free command output.\n#/bin/bash for PROC in `ls /proc/|grep \u0026#34;^[0-9]\u0026#34;` do if [ -f /proc/$PROC/statm ]; then TEP=`cat /proc/$PROC/statm | awk \u0026#39;{print ($2)}\u0026#39;` RSS=`expr $RSS + $TEP` fi done RSS=`expr $RSS \\* 4` PageTable=`grep PageTables /proc/meminfo | awk \u0026#39;{print $2}\u0026#39;` SlabInfo=`cat /proc/slabinfo |awk \u0026#39;BEGIN{sum=0;}{sum=sum+$3*$4;}END{print sum/1024/1024}\u0026#39;` echo $RSS\u0026#34;KB\u0026#34;, $PageTable\u0026#34;KB\u0026#34;, $SlabInfo\u0026#34;MB\u0026#34; printf \u0026#34;rss+pagetable+slabinfo=%sMB\\n\u0026#34; `echo $RSS/1024 + $PageTable/1024 + $SlabInfo|bc` free -m Running it require root access and the bc package installed. The result is most likely greater than the used memory value. Below is the result from my server:\n89925884KB, 201788KB, 3303.92MB rss+pagetable+slabinfo=91318.92MB total used free shared buff/cache available Mem: 128772 87032 726 498 41013 40346 Swap: 2047 3 2044 So the result is over by (91318 \u0026#8211; 87032) = 4286M. This is due to shared memory. The RSS value from above include memory from shared libraries as long as the pages from those libraries are in the memory. If multiple processes use the same library, the memory from shared library is counted multiple times. Check out the difference between RSS and PSS (proportional set size) [Disclaimer] The chart and script are stolen from this authors post.\nPrevious PostCommon local Git operations Next PostCassandra data model (as opposed to relational model) ","date":"2020-04-19T21:12:56-04:00","permalink":"/2020/04/how-memory-usage-adds-up-in-linux/","title":"How memory usage adds up in Linux"},{"content":"This is a summary of concepts in common Git operations. We will discuss brach, merge, rebase, cherrypick, stash and reset. Then we\u0026#8217;ll discuss pull, fetch, and push.\nCommit, Branch and HEAD When you run \u0026#8220;git commit\u0026#8221;, the following happens:\nGit checksums each subdirectory, and stores them as a tree object (file path and name) and blob object (file content) in Git repository;Git creates a commit object that has the metadata and a pointer to the root project tree; or if this is not the first commit, the pointer will point to the commit immediately before it The operations above should form a chain of commit. It can be a long chain and may diverge into branches. In Git semantics however, a branch is simply a lightweight, movable pointer to one of the commits. The default branch name in Git is master. A Git repository may contain multiple branches and the name master itself does not suggest any privilege. There is also a special pointer called HEAD, which indicates the branch you are currently working on. So branch is essentially a pointer to a commit; HEAD is essentially a pointer to a branch. \u0026#8220;git checkout\u0026#8221; can switch branch that HEAD points to. Basic Merge One type of basic merge simply moves branch pointer from one commit to another (along the same chain) without creating any commit. Here is a diagram before basic merge:\nBefore basic merge, Hotfix branch is based on\u0026nbsp;master The following command performs basic merge:\n$ git checkout master $ git merge hotfix Updating f42c576..3a0874c Fast-forward index.html | 2 ++ 1 file changed, 2 insertions(+) Then Git simply moves the pointer (named master) forward. There is no divergent work to move together, hence no chance of merge conflict. This type of basic merge is also called \u0026#8220;fast-forward\u0026#8221; merge.\nAfter basic merge, master\u0026nbsp;is fast-forwarded to\u0026nbsp;hotfix The other type of merge involves reconciling divergent work together, which may or may not involve conflict. Suppose this is the commit tree to start with:\nThree snapshots used in a typical merge The following commands perform the merge:\n$ git checkout master Switched to branch \u0026#39;master\u0026#39; $ git merge iss53 Merge made by the \u0026#39;recursive\u0026#39; strategy. index.html | 1 + 1 file changed, 1 insertion(+) Instead of just moving the branch pointer forward, Git creates a new snapshot that results from this three-way merge and automatically creates a new commit that points to it. This is referred to as a merge commit, and is special in that it has more than one parent.\nA merge commit Now that your work is merged in, you have no further need for the\u0026nbsp;iss53\u0026nbsp;branch. You can close the issue in your issue-tracking system, and delete the branch:\n$ git branch -d iss53 If commits from two respective branches changes the same file in different ways, then there is a merge conflict. In this case, Git cannot just create a merge commit. Instead it asks the user to resolve the conflict first. You have to choose either side of the change, or just merge the content yourself. At this point, if you introduce a change that does not appear in any parent, it is referred to as an evil merge. Beyond the basic merge, there are more sophisticated merge conflict resolution tools covered in advanced merging.\nMerge and Rebase There are two ways to integrate changes from one branch to another. Merge and rebase. Suppose your commit chain diverge into a master branch and a feature branch. Merging (from feature to master) takes the content of feature branch and integrate it with master branch. $ git checkout master $ git merge feature When you rebase a feature branch onto master, you move the base of the feature branch to master branch’s ending point.\n$ git checkout feature $ git rebase master After merge, you are still on the same branch. The commits from other branch are integrated into the branch that you are already on. There is no change in any existing commits (history). After rebase, your base will be moved to a different branch, along with the commits that you have made in the previous branch (since the diverge). In other words, by re-playing those commits on a different branch, it changed history.\nmerge vs rebase The chart above is stolen from this article, which does a better job explain in detail the difference, pros and cons of merge and rebase. Merge does create a \u0026#8220;merge commit\u0026#8221;, and a git history full of merges can be cluttered. Rebase does not create an extra commit but since it changes the history of a branch, it has impact to other collaborators. It can be done in an interactive way (with -i switch). The golden rules of rebasing is covered in this article. One of the principles is that never perform a rebase on a public branch.\ngit operations Cherrypick In a cherrypick operation, the current branch does not change. You simply pick interested commits from other branches to re-apply to your current branch. You may pick a single or a series of commits from other branch. These commits are not \u0026#8220;moved\u0026#8221; to your current branch. They remain intact. They are just re-played as new commit to current branch. Unlike rebase, there is no re-writing of history, hence not as dangerous.\nReset and Stash Suppose you are working on a part of a project and it starts getting messy. There has been an urgent bug that needs your immediate attention. It is time to save your changes and switch branches. If you are okay to give up your uncommitted work, you may perform a reset, in one of the three modes covered in a previous article.\nBut most likely, you don’t want to do a commit of half-done work. The solution is git stash. Stashing is handy if you need to quickly switch context and work on something else but you\u0026#8217;re mid-way through a code change and aren\u0026#8217;t quite ready to commit. In the most basic workflow, you need to run this command to save your uncommitted (but staged) work. As soon as you stash your change, the working directory is clean with all uncommitted local changes saved elsewhere. You can perform any other Git operations, such as change branch. When you\u0026#8217;re ready to resume, you may pop the stash. Here is an example:\n$ git add . $ git stash $ git checkout correctbranch $ git stash pop Instead of pop, you can also use apply to keep the changes in working directory. $ git stash apply More details are on this page from Bitbutket.\nFetch and Pull A git fetch simply downloads blob data from remote so the .git directory comes in sync with the server. It does not attempt to update the local working directory. If there is staged or uncommitted local changes, fetch will not impact them. A git pull is essentially git fetch followed by git merge. In addition to downloading blob data, it also updates local working directory. Therefore, there is a chance of merge conflict when the same file has been modified locally. Git will usually guide you through the merge conflict by flagging the conflict area in the file and let you decide the survival changes. For example:\n#! /usr/bin/env ruby def hello \u0026lt;\u0026lt;\u0026lt;\u0026lt;\u0026lt;\u0026lt;\u0026lt; HEAD puts \u0026#39;hola world\u0026#39; ======= puts \u0026#39;hello mundo\u0026#39; \u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt;\u0026gt; mundo end hello() You will be prompted in an editor session to reconcile the conflict. Once the file is saved, you will also need to do a \u0026#8220;merge commit\u0026#8221;, before you can pull again.\nPush Git push is the opposite of pull, where you merge local branch to the remote. (There is no opposite of fetch because there is no point to merge to remote without updating working directory, no collaborator works on the working directory on the server after all). If the local branch has fallen out of sync with the remote, there is a chance of merge error during git push. To minimize the chance of a merge during push, we can run a git pull before and reconcile any potential conflict locally. This is known as a pre-merge.\nVisualizer I came across a great visualizer of commit chain here. In the command panel type some git command and it will print the commit graph for you\nPrevious PostNTLM and Kerberos protocols Next PostHow memory usage adds up in Linux ","date":"2020-04-08T11:54:00-04:00","permalink":"/2020/04/common-local-git-operations/","title":"Common local Git operations"},{"content":"This article explains how two most common authentication mechanisms (NTLM and Kerberos) work. Both authentication protocols are based on symmetric key cryptography. The protocols themselves are platform independent. NTLM Authentication NTLM is the default authentication protocol prior to Windows 2000 and still prevalent today as backup to Kerberos. It is based on challenge/response mechanism.\nUserUserClientClientServerServerLog onLog onDomain ControllerDomain ControllerActive DirectoryActive Dire\u0026#8230;(2) Challenge(2) Challenge(3) Response(3) Response(1) Username(1) Username(4) Challenge from (2) and Response from (3)(4) Challenge from (2) and Response from (3)(5) Authentication Result(5) Authentication Result(6) Authentication result(6) Authentication resultNTLM AuthenticationNTLM AuthenticationViewer does not support full SVG 1.1\nAuthentication starts with user trying to log on from a client computer providing the username and password. The following steps will occur:\nThe client application hashes user\u0026#8217;s password (with HMAC-MD5) and then discard the password. Then the client only sends the username to the server;The server generates a challenge (16-byte random number) and sends it to the client;The client encrypts this challenge with the hash (of user\u0026#8217;s password from step 1), and send the result back to the server as response (encrypted challenge);The server sends the following three items to the domain controller:username (in plain text)the challenge it had sent to the clientthe response from the client (encrypted challenge)The domain controller has the hash of user\u0026#8217;s password stored, and retrieved (by username). It uses the hash to encrypt the challenge by itself and get its own version of encrypted challenge. It then compares it with the one passed on from the server. If identical, the user is authenticated and the domain controller notifies the user;The server sends the appropriate response back to the client. Essentially, NTLM mechanism is comparing the result encrypted by the client the result encrypted by the credential stored in itself. The results are expected to be identical because the encryption key \u0026#8211; hash of user\u0026#8217;s password \u0026#8211; should be identical. In this whole process, there is no password transmission on the wire. However, the disadvantage with this challenge-response based mechanism is that it does not let client authenticates the server, and is therefore considered less secure. This is what Kerberos is built to address.\nKerberos Authentication Kerberos is a ticket-based authentication mechansim. In Kerberos, a Key Distribution Centre (KDC) consists of AS (Authentication Service) and TGT (Ticket Granting Service). Authentication takes place in the following steps:\nKey Distribution Centre (KDC)Key Distribution Centre (KDC)ServerServerAuthentication ServiceAuthentication Ser\u0026#8230;Ticket-Granting ServiceTicket-Granting Se\u0026#8230;DatabaseDatabase1) KRB_AS_REQ1) KRB_AS_REQ2) KRB_AS_REP2) KRB_AS_REP3) KRB_TGS_REQ3) KRB_TGS_REQ4) KRB_TGS_REP4) KRB_TGS_REP5) KRB_AP_REQ5) KRB_AP_REQ6) KRB_AP_REP6) KRB_AP_REPClientClientUserUserLog onLog onKerberos AuthenticationKerberos AuthenticationViewer does not support full SVG 1.1\nThe authentication starts with user trying to log on from a client computer. Note that the \u0026#8220;client\u0026#8221; here refers to Kerberos client relative to Kerberos server (KDC). The client machine can serve as application server that runs kerberos library. The user provides username and password.\nThe client sends KRB_AS_REQ as plain text to AS including:usernametimestampAS verifies timestamp, and validates username exists. If timestamp is too far (i.e. over 5 min) from current time, or user is not a legal principal, KRB_AS_REQ will be rejected. Otherwise, AS generates a random TGS session key and uses it to build a TGT. In KRB_AS_REP AS sends two messages to the client:Message 1 is the TGT, which is encrypted with TGS secret key (so the client cannot decrypt TGT). It includes:usernameTGS nametimestampclient network addresslifetime of TGTTGS session keyMessage 2 is encrypted with client secret key (stored in AS)TGS nametimestamplifetimeTGS session key (same as message 1)The client receives both message 1 (TGT) and message 2. It decrypts message 2 with its own secret key and obtains TGS session key. For message 1 (TGT), the client cannot decrypt it. The client simply stores it in the credential cache. Then the client prepares two messages to send to the KDC:Message 3 contains:TGS namelifetimeTGT (message1, encrypted with TGS secret key)Message 4 (aka Authenticator) is encrypted with TGS session key and contains:usernametimestampKDC processes message 3, determines TGS name is valid and forward TGT to TGS. TGS decrypts TGT using its own secret key and obtains TGS session key, along with username and timestamp. Then TGS decrypts message 4 using the newly obtained TGS session key, in order to get username and timestamp from message 4 as well. At this point, TGS has two sources of username and timestamp: one from TGT; the other from message 4. The TGS make sure they are identical, check if TGT is expired, and confirm that authenticator is not in the cache (to prevent replay). If all checks pass, the TGS then generates a random service session key. It will send two messages back to the client:Message 5 (the service ticket) is encrypted with service secret key (stored in TGS) and contains:usernameservice nametimestampclient network addresslifetimeservice session keyMessage 6 is encrypted with the TGS session key containing:service nametimestamplifetimeservice session keyThe client receives message 5 and message 6 but it cannot decrypt message 5. The client cached TGS session key from previous step so it can decrypt message 6 and obtain service session key. Now the client contact the server by sending the following two messages:Message 7: a new authenticator message encrypted with service session key that contains:usernametimestampMessage 8: the same as message 5, encrypted with service secret keyThe server now receives message 7 and message 8. It decryptes message 8 to get service session key, along with username, service name, timestamp, etc. Using the service session key it descrypts message 7 to get a second source of username and timestamp. Similar to what happened in TGS, now the server compares username from the authenticator and from the ticket, checks if ticket is expired, and confirms that authenticator is not already in cache (to prevent replay attack). If all checks turn out okay. The service confirms its identity to the client with:Message 9: an authenticator message encrypted with service session key that contains:service nametimestampLastly, the client receives message 9 and decrypts it with the service session key in cache. The client then confirms the service name and timestamp are valid/expected. If they are good, the authentication is completed and the client starts to communicate with the server. The steps above borrowed some information from this page. The Keberos authentication process involves many steps and several keys:\nsession (shared) keys:TGS session keyService session keysecret keys:client secret keyTGS secret keyService secret key By using these keys, no password is ever transmitted across the wire in the clear. The client and the server authenticate each other (mutual authentication). With a trusted third party, Kerberos ensures that the service ticket is only used by the intended client, and that only the intended server can validate the requested service ticket. Although, this sounds similar in two way authentication in TLS handshake, Kerberos does not encrypt the application traffic, neither is it intended to.\nKerberos Implementations The most popular implementation is MIT Kerberos 5. The other well-known implementation is Heimdal. In addition, it is supported as API in GSS-API. In Windows applications, SSPI (Security Support Provider Interface) provides similar functionality to the GSS-API. SSPI can be viewed as Microsoft\u0026#8217;s implementation of GSS-API, which can be virtually regarded as Kerberos API.\nKerberos is supported by many application protocols through GSS-API. If you build a Linux application, the server where the application is hosted acts as Kerberos client (i.e. requiring krb-libs package and /etc/krb5.conf configured correctly) to interact with customer\u0026#8217;s Active Directory environment. If you need to configure Kerberos servers, you will need other packages such as krb5-admin-server, krb5-kdc, and krb5-user.\nThis page lists some pros and cons of Kerberos. Some important takeaways are:\nKerberos is legacy protocol, complex to set-up and maintain. It requires user accounts, user clients and the services on the server to all have a trusted relationship to the Kerberos token server. All must be in the same Kerberos realm or in domains that have a trust relationship between each other. Kerberos cannot be used in a scenario where users want to connect to services from unknown/untrusted clients as in a typical Internet or cloud computing environment, where authentication provider typically does not have knowledge about the users client system. This implies Kerberos does not work will with modern REST applications and Authentication Methods such as SAML, and OAuth 2.0 Kerberos was created to accomplish authorization back in the days when no-one used a secure network connections. Previous PostIntroduction to Authentication Frameworks (PAM and SSPI) Next PostCommon local Git operations ","date":"2020-03-25T20:33:00-04:00","permalink":"/2020/03/ntlm-and-kerberos/","title":"NTLM and Kerberos protocols"},{"content":"This article gives a very brief high-level introduction to PAM (Pluggable Authentication Module) and SSPI (Security Support Provider Interface) as authentication frameworks in Linux and Windows respectively. PAM The Pluggable Authentication Module (PAM) architecture provides a powerful abstraction for user IAM using pluggable authentication model Unix platforms. It defines a generic API for authentication and hides the underlying mechanisms. Thanks to PAM, administrators can plug different authentication modules and protocols into Linux. This makes different authentication methods and protocols available to applications running on Linux. Here is some of authentication methods and protocols that PAM supports:\nUnix file-based authentication (using /etc/passwd or /etc/shadow) LDAP-based authentication Kerbero-based authentication NTLM-based authentication PAM obviates the need for a separate authentication schemes. It exports methods of the various libraries under its auspices to calling applications. Here is a diagram for PAM on Redhat.\nPAM can also enbable single-sign-on(SSO) on the UNIX platform. If the password used for different services are identical, PAM can be used to share the password transparently between the application\u0026#8217;s possibly different authentication mechanisms. PAM is configured in /etc/pam.d/ directory.\nThe Windows equivalent of PAM is the Security Support Provider Interface (SSPI) and its Security Support Provider (SSP) Modules. For example /etc/pam./login manages login module, /tec/pam.d/imap manages imap module.\nSSPI Windows Authentication Architecture involves Local Security Authority (LSA, to authenticate users to local computer only and is managed in local security policy) and Security Support Provider Interface (SSPI).\nSSPI is the API that obtains integrated security service for authentication, message integrity, message privacy, and security quality-of-service for any distributed application protocol. SSPI is the implementation of the Generic Security Service API (GSS-API) in Windows operating system. Applications and infrastructure services authenticate users by using the SSPI to abstract calls for authentication. This way, developers do not need to understand the complexities of specific authentication protocols or build authentication protocols into their applications. Here is the architecture diagram:\nThe SSPI in Windows provides a mechansim that carries authentication token over the existing communication channel between the client computer and the server. When two computers or devices need to be authenticated so that they can communicate securely, the requests for authentication are routed to the SSPI, which completes the authentication process, regardless of the network protocol currently in use. Here is some exampls of SSPs that are supported by SSPI:\nKerberos SSP (default for Active Directory) NTLM SSP Digest SSP Negotiate SSP (based on SPNEGO, RFC4178) Credential SSP Negotiate Extension SSP Some SSPs such as Kerberos SSP and NTLM SSP use a single protocol. Some (e.g. Negotiate SSP and Credential SSP) combine several protocols to allow application to select what security mechanism they wish to use and negotiate with authentication services.\nPrevious PostSASL Authentication Mechanisms Next PostNTLM and Kerberos protocols ","date":"2020-03-24T20:19:00-04:00","permalink":"/2020/03/introduction-to-authentication-frameworks-pam-and-sspi/","title":"Introduction to Authentication Frameworks (PAM and SSPI)"},{"content":"Introduction Authentication is used in many protocols (such as LDAP binding) and it usually involves sending password. Given the nature of authentication protocol, its traffic encryption is usually mandatory. Simple Authentication and Security Layer (SASL) is introduced to ensure the security during authentication. It is not a single protocol, but rather a framework for authentication and data security involving many protocols. The intent is to decouple authentication mechanisms from application protocols, thus allowing any authentication mechanism (under SASL) to be used in any application protocol (that supports SASL). Application protocols that support SASL typically can also be built on Transport Layer Security (TLS), whose latest versions (1.2 and 1.3) are considered more secure.\nNone (ANONYMOUS) The server basically does not authenticate the client. The client connects to the server anonymously. Under SASL framework, this may also be referred to as ANONYMOUS mechanism.\nSimple (PLAIN) In simple authentication method the password is sent to server in the clear. This is subject to eavesdropping and is not secure. It is still surprisingly widespread in legacy configurations probably due to the simplicity of configuration. This option should not be available in cloud environment. Under SASL framework, this may also be referred to as PLAIN mechanism.\nCRAM-MD5 and DIGEST-MD5 CRAM-MD5: described in RFC 2195, using HMAC-MD5 algorithm. In this challenge-response scheme based mechanism, the client\u0026#8217;s password is protected during authentication, but the application session (e.g. LDAP) traffic is not encrypted. It includes random data from the server and is slightly better than Simple authentication. However, this authentication method is not recommended either.\nDIGEST-MD5: described in RFC 2831. This is very similar to CRAM-MD5 but is is somewhat stronger because it includes random data from both the client and server. In addition, it also provides a provision to ensure connection integrity and confidentiality (a data security layer).\nGSSAPI Generic Security Service Application Program Interface (GSS-API) is an API specification for programs to access security services. GSS-API by itself does not provide any security. Instead, security-service vendors provide GSSAPI implementations \u0026#8211; usually in the form of libraries installed with their security software. These libraries present a GSSAPI-compatible interface to application developers who can write their application to use only the vendor-independent GSSAPI. Under SASL, the dominant GSSAPI mechanism implementation in use is Kerberos version 5. GSSAPI allows Kerberos implementations to be API compatible. In many contexts, GSSAPI simply implies Kerberos.\nNTLM NT LAN Manager (NTLM) is a challenge-response based Microsoft security protocols. It is implemented in a Security Support Provider (SSP), which combines the older LAN Manager authentication protocol, NTLMv1, NTLMv2 and NTLM2 Session protocols in a single package. Group policy manages whether these protocols are used or can be used. NTLM passwords are considered weak because they can be brute-forced very easily with modern hardware. It might still be enabled in server configuration as a backup mechanism to Kerberos.\nTLS (EXTERNAL) CRAM-MD5, DIGEST-MD5, GSSAPI, and NTLM are more commonly referred to as SASL mechanisms (in loose terms). These mechanisms allow for a secure password exchange without requiring TLS by trying to address the authentication traffic encryption problem at application layer. Using TLS this can also be address at transport layer. TLS can be used in combination with any of the mechanisms above but usually TLS/Simple mechanism is sufficient. In many occasions the mechanisms under SASL can be replaced by simple authentication encrypted with TLS. Under the SASL framework, this may also be referred to as EXTERNAL mechanism so TLS (in strict terms) is also considered a SASL mechanism.\nSummary This article outlined several authentication protocols under SASL. Since SASL is the framework that intends to govern all authentication protocols, the use case of these protocol can be widespread. For example, in LDAP you can find all of them. Here is a comparison across them:\nPassword on wireSession SimpleClearNo Encryption SASL/CRAM-MD5EncryptedNo Encryption SASL/DIGEST-MD5EncryptedNo Encryption SASL/GSSAPIKerberosEncryption TLS:SIMPLEEncryptedEncrypted Common SASL implementation includes Cyrus SASL and GNU SASL. There are also some API implementations that supports some of SASL mechanisms, such as SSPI.\nPrevious PostOAuth 2.0 and OIDC 1 of 2 Next PostIntroduction to Authentication Frameworks (PAM and SSPI) ","date":"2020-03-19T22:53:00-04:00","permalink":"/2020/03/authentication-mechanisms-under-simple-authentication-and-security-layer-sasl/","title":"SASL Authentication Mechanisms"},{"content":"OAuth 2.0 and OpenID Connect (OIDC 1.0) are different but highly related protocols and they are often confused. When we talk about IAM (identity and access management), we should first distinguish between Authentication (AuthN) and Authorization (AuthZ):\nAutheNtication (AuthN, aka Identity Management) is about validating user\u0026#8217;s identity by verifying that the user trying to connect is actually who it claims itself to be; AuthoriZation (AuthZ, aka Access Management) refers to granting or denying access to specific resources based on the requesting user\u0026#8217;s identity. It is usually performed after a user is identified through authentication. The most common approach is Role-Based Access Control (RBAC). In a nutshell, OAuth 2.0 deals with authorization. OIDC is a layer later developed on top of OAuth 2.0, to deal with authentication. This post is greatly influenced by a presentation (1 hour) delivered by Nate Barbettini from Okta, with the slides available here. There is also an abridged and illustrated video (16 min) by Okta available here.\nOAuth 2.0 OAuth was originally developed by Twitter and Google in 2006 as an open standard for API authorization. OAuth 2.0 is published in 2012. It allows user to delegate authorization. The original scenario is a user signed up to a new application and allows it to automatically import her Gmail contact. The technical problem to solve is: how can a user (Resource Owner) let an app (Client) to access his contact list stored in Google server (Resource Server)? The proposal is that it redirects user to Google Account page (Authorization Server) for user to log in. Then Google Account issues token to the application (Client) with user\u0026#8217;s approval. Note that the user did NOT log in to the application itself with her Google account. From the application\u0026#8217;s standpoint, the user had been authenticated already, and was simply importing contact after logging in. The roles involved in OAuth 2.0 are:\nResource: the contact list of the user Resource owner: the user Client: the application Resource Server: contact.google.com Authorization Server: accounts.google.com The diagram below illustrates the interactions:\nOAuth 2.0 Authorization Code Flow This page has further details for each step. Note that at step 3 to 5 may seem unnecessary because Auth Server could have send Access Token Grant to Application via User-Agent at step 3, which could have eliminate the need for step 4 and 5. In fact this design is to avoid sending critical information (Access Token Grant) to User-Agent (browser) which is considered in secure. In other words, it avoids front channel (User-agent to auth server) and prefers back channel (Client to Auth server) for security. This is the difference between Authorization code flow and the implicit flow.\nWith OAuth 2.0, there are a number of flows:\nAuthorization code (front channel and back channel) Implicit (front channel only, token returned to user agent directly) Resource owner password credentials (back channel only) Client credentials (back channel only) In Authorization Code Flow, the Application (client) needs a one-time registration with the Auth Server and is given a client ID and client secret, which are sent to Auth Server at step 4 along with Access Token Request, to prove the identity of the client application.\nNote that OAuth 2.0 is an inherently insecure protocol since it does not support signature, encryption, channel binding or client verification. The protocol relies entirely on the underlying transport layer security (TLS) to provide confidentiality and integrity.\nAlso note that throughout the process (Authorization Code Flow as an example), the Client application eventually is granted access to user\u0026#8217;s data. However, it does not know anything about the user itself. Neither the authorization code grant, nor the access token grant is obligated to present information about the user itself. Therefore, OAuth 2.0 is designed strictly for permission purpose without the intent to address identity issue. In the flow, the auth server does the authentication (for the purpose of granting access to resources, but none of the authentication. A user logs in to client application as Bob, when he requests to imports contact, he is redirected to account.google.com and there he could put in the credential of Alice and therefore load Alice\u0026#8217;s Google contacts into Bob\u0026#8217;s App account!\nPseudo-authentication with OAuth 2.0 In many real life OAuth 2.0 implementations, at step 3, the Auth server chooses to include a field about the user\u0026#8217;s identity. This makes user\u0026#8217;s identity visible to the client, and the client is therefore able to confirm user\u0026#8217;s identity in its own code. This also allows client application to use OAuth 2.0 as an authentication method, which is referred to as pseudo-authentication. The access token acts as a kind of \u0026#8220;valet key\u0026#8221; that the application can include with its request to the auth server, as a proof that it has user\u0026#8217;s permission to access the resources (or APIs).\nBecause the identity provider (auth server) typically (but not always) authenticates the user as part of the process of granting an OAuth access token, it\u0026#8217;s tempting to view a successful OAuth access token request as an authentication method itself. However, because OAuth was not designed with this use case in mind, making this assumption can lead to major security flaws.\nNate\u0026#8217;s presentation outlined some scenarios where OAuth 2.0 is applied up to 2012, and which ones are misuses:\nSimple login \u0026#8211; pseudo authentication with OAuth 2.0 Single sign-on across sites \u0026#8211; pseudo authentication with OAuth 2.0 Mobile app login \u0026#8211; pseudo authentication with OAuth 2.0 Delegated authorization \u0026#8211; the only intended use case for OAuth 2.0 To address the authentication issue properly, and in a standard approach, we need OpenID Connect.\nOpenID Connect (OIDC) OpenID Connect is an open standard for authentication, promoted by the non-profit OpenID Foundation. It allows user to be authenticated using a third-party service called identity providers. User may choose to use their preferred OpenID Connect providers to log in to websites that accept the OpenID Connect authentication scheme. For example, a user uses her Facebook to login to an online application.\nOpenID Connect is an extension to OAuth 2.0 with a just few additions:\nIn addition to access token, an ID token is returned by the authorization server; Userinfo end point is provided in case Id token is not sufficient and more user information is needed; \u0026#8220;openid\u0026#8221; is passed as a parameter in the Scope during the initial call to the authorization server; Therefore OpenID Connect is considered an identity layer on top of OAuth 2.0. Many application supports OpenID Connect such as Apache Nifi. OIDC is comparable with SAML in the sense that both provide SSO feature (federated identity). Here is a comparison table:\nOpenID ConnectSAML Main PurposeSSO for consumer/mobile applicationsSSO for enterprise applications LoadRelatively light weightHeavy weight due to the size of XML messages Use caseSatisfies both authentication and authorization use cases, often combined with OAuth 2.0Generally not used for API security TransportHTTP GET and HTTP POSTHTTP Redirect (GET) binding, SAML SOAP binding, HTTP POST binding, et Here are more details about their differences. In general SAML is more common in the enterprise world for SSO and it has been around for a while. When developing new applications for enterprise it is advised to consider OIDC first.\nOIDC also has authorization code flow, with the additional fields on top of its counterpart in OAuth 2.0. The authorization server returns both access and ID tokens, wrapped in a data structure named JWT (JSON Web Token). The JWT includes a signature field, allowing the client application to verify it with authorization server\u0026#8217;s public key. Nate\u0026#8217;s presentation proposes the following flows for each application type:\nWeb application with server backend: authorization code flow Native mobile app: authorization code flow with PKCE Java Script app (SPA) with API backend: implicit flow Microservices and APIs: client credential flow In addition, this page from Okta developer has a good summary of how to select flow type (grant) based on each use case.\nSummary OpenID Connect is an authentication protocol for the purpose of validating user\u0026#8217;s identity. OAuth 2.0 is an authorization protocol. You should use OAuth 2.0 for granting access to your API, or access to user data in other systems. If you need to log user in, or make your accounts available in other systems, you need OIDC.\nThe next post about OAuth and OIDC was posted in 2023.\nPrevious PostSecurity Assertion Markup Language (SAML) Next PostSASL Authentication Mechanisms ","date":"2020-03-14T21:10:00-04:00","permalink":"/2020/03/oauth-and-openid-connect/","title":"OAuth 2.0 and OIDC 1 of 2"},{"content":"SAML is an XML-based standard for exchanging authentication and authorization data between IdP (identity provider) and service provider. We can compare SAML with LDAP (as authentication protocol) as both are to provide single-sign-on (SSO) feature.\nLDAP is considered traditional configuration in on-premise operation for organizations. The configuration can be complex and administrators needs to complete significant work upfront. AD is notoriously hard to integrate into the cloud. On the other hand, LDAP gives the organization greater level of control over authentication and authorization due to its tighter integration with domain controller. It is prevalent in on-premise enterprise infrastructure and integrate well with OpenVPN, Jenkins, Docker, Kubernetes, etc.\nOn the other hand, SAML was created in early 2000s with the exclusive purpose of federating identities to web applications. The protocol was introduced assuming an IdP already exists in an organization. The SAML protocol doesn\u0026#8217;t intend to replace the IdP, but rather use it to assert the validity of a user\u0026#8217;s identity. This timed assertion (declaration that user\u0026#8217;s identity is valid for a period of time) will be delivered to a service provider via secure XML exchange. The benefit of SAML is that an on-premise identity typically stored in Active Directory (AD), could be extended to authenticate its users against web applications. ISVs (independent software vendor) can build web applications that integrates with on-premise AD server to achieve SSO feature. The AD server in this case provides IDaaS (Identity as a service) using its FS (Federation Service) module. Examples of web applications that support SAML integration include Confluence, Zendesk, Slack, Bombgar, etc. In the configuration, you typically need to specify who is the IdP (e.g. Microsoft AD, Okta, etc) and it\u0026#8217;s SSO URL.\nAt a high level, an SSO process using SAML takes places in the following steps:\nUser tires to reach web application (service provider);Web application redirects user browser to SSO URL;User provide credential in the SSO URL;IdP authenticates the user;IdP produces SAML response to browser;Browser passes the SAML response to service provider\u0026#8217;s dedicated endpointService provider permits user access to web application Here is an example of SAML integration guide from an G Suite. The guide outlines how it works and the assertion requirements.\nThe Wikipedia page for SAML 2.0 provide an example of assertion message.\nPrevious PostLightweight Directory Access Protocol (LDAP) Next PostOAuth 2.0 and OIDC 1 of 2 ","date":"2020-03-08T22:23:00-04:00","permalink":"/2020/03/saml-security-assertion-markup-language/","title":"Security Assertion Markup Language (SAML)"},{"content":"Introduction Originally LDAP only refers to the connectivity protocol to the directory server. This term is being used loosely today and it also refers to the actual directory service that supports and complies with LDAP. LDAP v3 is the current version developed in RFC 2251.\nA directory is information about some set of entities such as people, organization, or stones. An example of directory would be /etc/passwd file in Linux. A directory server is simply an application with the main purpose of maintaining directories. Typically, the read traffic is high whereas write traffic is low. LDAP is a general-purpose directory server. It can store information about people, or cars, or rocks. You just need to define what a person\u0026#8217;s entry looks like as well as what a rock\u0026#8217;s entry looks like. The general architecture of LDAP provides the capability nedded for managing large amount of diverse directory entries.\nAn LDAP entry consists of DN (distinguished name) and attributes. An attribute may have one or more attribute names and they are defined in attribute definitions. Attribute names are not case-sensitive. An attribute may have one or more values if multiple values are allowed for that attribute. Attribute values may be case-sensitive depending on the definition.\nA special attributed named objectclass attribute provides information about what type of record it is, and what attributes canbe given to the record. For example, the organization name (o) is required for any entry with an organization object class. While a record may have multiple object classes, one of these object classes must be the structural object class for the record. A structural object class determines what type of object the record is.\nIn addition to regular attributes, the directory server may also attach special operational attributes to an entry. Operational attributes are used by the directory server itself to store information about entries. Such attributes are not designed for use by end users, and are usually not returned during LDAP searches.\nAn LDAP schema defines types of records in a directory and how those records might relate to each other. Information in an LDAP directory is organized into one or more hierarchies where, at the top of the hierarchy, there is a base entry, and other entries are organized in tree-like structures beneath the base entry. Each node on the hierarchy is an entry, with a DN and more than one attributes. This hierarchically organized collection of entries is called a directory information tree (DIT). In DIT, LDAP directories stores data in hierarchical relationships. The root entry sits at the top and subordinate entry is beneath that, which in turn may have its own subordinate entries. Each of these records has its own DN, and its own attributes. The DN of each entry is composed of two parts: the relative DN (RDN) and the full DN of the superior entry.\nLDAP is nothing other than a special sort of database that organizes data into tree structures, like a file system hierarchy. This view is more easily seen by comparing an LDAP directory to a relational database system (RDB), where SQL is the protocol and RDBMS is the service. LDAP refers to both the protocol and the service.\nOpenLDAP A common LDAP implementation is openldap. OpenLDAP suite can be broken up into four components:\nServers: slapd (stand-alone LDAP Daemon) provides LDAP services.Clients: ldapsearch is used to manipulate LDAP dataUtilities: support LDAP serversLibraries: provide programming interfaces to LDAP Installing OpenLDAP requires libldap-2.3-0, slapd, ldap-utiles packages. It is configured in /etc/ldap/. An HDB (hierarchical database) needs to be specified in the configuration.\nTo test as a client, the first thing that must happen is the client must authenticate to the server (via simple bind or SASL Bind). LDAP server verifies the identity, permission as well as password provided by the client.\nLDAPTLS_REQCERT=never ldapsearch -x -o ldif-wrap=256 -H ldaps://ldap.digihunch:636/ -b \u0026#34;OU=Admin,OU=Service Department,DC=digihunch,DC=com\u0026#34; -D \u0026#34;gh\\ldap-bind-user\u0026#34; -w \u0026#39;S@f35+P@55w0rd\u0026#39; \u0026#34;(objectclass=user)\u0026#34; -s sub -d 9 The command above first sets client environment variable LDAPTLS_REQCERT to never, in case the client is being asked to provide certificate. Then the ldapsearch command performs the bind.\nTo search the directory, the client needs to provide the followings:\nBase DN: where in the directory to start fromScope: how deep in the tree to lookAttributes: what information to be retrieved per resultFilter: what to look for Below is an example of ldapsearch (-b for Base DN, -s for Scope, -S for attributes, stdin for filter):\nldapsearch -x -o ldif-wrap=256 -H ldaps://ldap.digihunch:636/ -b \u0026#34;OU=Admin,OU=Service Department,DC=digihunch,DC=com\u0026#34; -D \u0026#34;gh\\ldap-bind-user\u0026#34; -w \u0026#39;S@f35+P@55w0rd\u0026#39; \u0026#34;(memberof=CN=Security-Admin,OU=Admin,OU=Service Department,DC=digihunch,DC=com)\u0026#34; -s sub -S name Users with appropriate permissions may also other directory operations using ldapadd, ldapmodify, ldapdelete, ldapcompare, ldapmodrdn, ldappasswd, ldapwhoami, etc\nApart from those in Openldap toolkits, there are many other tools such as Apache Directory Studio that allows you to perform similar functionality with a user interface.\nLDAP security Historically LDAP servers listens to port 389 through which traffic is sent in clear text. This is a bad security practice known as \u0026#8220;insecure bind\u0026#8221;. To secure LDAP traffic, two prevalent approaches are Secure LDAP and StartTLS.\nSecure LDAP was the original attempt to secure LDAP traffic as an addition to LDAP v2. It is also known as LDAPS, LDAP over TLS/SSL or LDAP channel binding (“channel binding” just refers to the establishment of encrypted channel following TLS handshake. It provides a facility to tie an authentication exchange to security services provided at a lower layer. Defined in RFC 5056). Secure LDAP operates on port 636 on the server side and TLS handshake must be established for traffic encryption. Client application usually need to import the certificate of LDAP server. As part of TLS 1.2 protocol, the server may also request client certificate during ServerHello message. The presence of CertificateRequest means the server either demands client certificate, or tries to get client certificate (i.e. TLSVerifyClient is set to demand or try, which is only visible on the server). If client cert is only attempted, the LDAP client may choose to ignore it. If client cert is demanded, then a two-way TLS authentication is required and thus the client must proof its identity to the server. This Secure LDAP configuration requires the server to listen to both 389 and 636 ports on the same server to support both secure and legacy applications, which is unnecessary. Secure LDAP therefore is not the preferred approach. The standardized way of implementing SSL/TLS in LDAP v.3 is to use the StartTLS method. This method should be implemented whenever possible. If an AD server supports StartTLS, the client can start with a STARTTLS command to the server so that the server begins the TLS encryption process. In the binding phase, TLS handshake follows a LDAP_START_TLS_OID command through port 389.\nHere’s the summary of the three LDAP configuration mode:\nLegacySecureLDAP (aka LDAPS, LDAP over TLS/SSL)StartTLS Listening port389636389 Traffic EncryptedNoYesYes StandardYes but this should always be avoided since it is insecureIntroduced in the time of LDAP v2, but the option is deprecated (although still supported) by RedHatIntroduced in LDAP v3. This may be left as the only valid option. Note that one of the recent changes that drives may customer away from the legacy mode is the requirement for LDAP channel binding on Windows servers, with a target date of March 2020. Our current strategy at CS is to direct customer towards Secure LDAP as we do not support StartTLS yet and we know we do support LDAPS. Although Secure LDAP itself is somewhat legacy this would not hold long. According to this Wikipedia page:\nThe use of LDAP over SSL was common in LDAP Version 2 (LDAPv2) but it was never standardized in any formal specification. This usage has been deprecated along with LDAPv2, which was officially retired in 2003. The trade off between StartTLS and TLS/SSL exists not only in LDAP protocol, but also in many other protocols such as SMTP (port 2525, 25, 587). StartTLS is also called Opportunistic TLS. The standard is in the relevant RFC documents.\nPrevious PostIntroduction to Active Directory (AD) Next PostSecurity Assertion Markup Language (SAML) ","date":"2020-03-02T21:11:00-04:00","permalink":"/2020/03/lightweight-directory-access-protocol-ldap/","title":"Lightweight Directory Access Protocol (LDAP)"},{"content":"Workgroup, homegroup and Windows Domain A workgroup is a group of computers on the same local network. A Windows computer not joined to a domain is part of a workgroup. In a workgroup, no computer has control over any other computer and it does not require a password. Any computer can join or leave a workgroup any time. Workgroup was previously for home file and printer sharing and Microsoft later introduced homegroup for more security. Compared to workgroup, all computers in a homegroup needs to be on the same home network (instead of local network). Homegroup is password protected. New computer needs to join homegroup by providing the password.\nWindows domains (or domains for short) provide network administrators with a way to manage a large number of PCs and control them from one place and remotely.\u0026nbsp;One or more servers — known as domain controllers — have control over the domain and the computers on it. Computers on a domain has to be on the same local network, either physically or over VPN. Centralized control is essential for corporate operation.\nAdministrators can join a Windows PC with professional or enterprise license to a domain. Once joined, the computer does not use its own local user accounts. When a user logs into a computer on that domain, the computer authenticates the user account name and password with the domain controller. Also, the computer cannot just leave the domain without administrator access. Network administrators can change group policy settings on the domain controller. Each computer on the domain will get these settings from the domain controller and they’ll override any local settings users specify on their PCs. All the settings are controlled from a single place. This prevents from users from changing many system settings on a computer joined to a domain. The domain controller is in charge of what a user can do. Apart from centralized administration, the benefit the users is that they can log in with the same username and password on any computer joined to the domain, if permission allows.\nDomain Controller In order to achieve centralized administration and log-in from any computer in the corporate world, a centralized service called domain controller is introduced. At a high level, a domain controller maintains a list for each of the followings:\nUsers and their passwordsComputers and their credentials This is because in a domain, not only the users, but also the computers (workstations or servers) need to be authenticated. For example, when a Windows server boots up, it needs to log on to the domain with its own credential. This way we can control whether the server is allowed to query the domain for information about users. If it is allowed to query the domain, then we can determine whether the user is allowed to log on that server, and eventually, authenticate the user. For a domain controller, it responsibility to credentials for users and computers, and respond to log in requests (authentication service) is a critical commitment in the enterprise environment. Domain controller is therefore commonly built with high availability and fault tolerance.\nAdministrators needs to add each new user to the user directory in domain controller. They also needs to register each new computer with the domain controller by joining them to the domain. Joining a Domain As explained earlier, joining a Window domain means register a computer in the domain so it has the permission to query the domain to validate users identity and permissions. Both Windows server and Linux server can join a domain. Windows servers usually provide a path through UI to join a domain and password is required. To join a Linux (e.g. Redhat) server to a domain, we can use a tool called adcli. Here is a good example of using this command to join a domain.\nAuthentication A Windows or Linux server in the domain needs to go to the domain controller to authenticate itself and the users. Authentication involves several protocols, including kerberos, NTLM, TLS/SSL and Digest, as part of an extensible architecture. In addition, some protocols are combined into authentication packages such as Negotiate and the Credential Security Support Provider.\nThe MIT Kerberos Documentation provides some tools (e.g. kinit, klist) to configure and troubleshoot Kerberos protocol.\nActive Directory Since Windows 2000, Active Directory is a complete redesign and re-branding of the entire Windows Domain system. The term Active Directory now refer to either the entire domain system, or the actual database that comprises the Windows Domain information or both.\nAll of the information that makes up an Active Directory is stored in an X.500 compatible database, typically replicated between domain controllers to ensure high availability and fault tolerance. X.500 is a set of network directory standards. A Windows Domain is a kind of network directory, hence the name Active Directory for its replacement. Active Directory introduced one important new type of object and concept, Forests. An Active Directory Forest is kind of a list of lists, meaning, it is a collection of Domains that are all related to each other for both security and management purposes.\nHere is more details about Active Directory.\nLightweight Directory Access Protocol As mentioned above, X.500 is a series of computer networking standards covering electronic directory services. ISO incorporated it into OSI suite of protocols. The protocols defined by X.500 include DAP (Directory Access Protocol), DSP (Directory System Protocol), DISP (Directory Information Shadowing Protocol) and DOP (Directory Operational Bindings Management Protocol). DAP is a heavyweight protocol that operates over a full OSI protocol stack and requires a significant amount of computing resources. LDAP (Lightweight Directory Access Protocol), as its alternative, is designed to operate over TCP/IP and provides most of the functionality of DAP at a much lower cost.\nTechnically speaking, LDAP is a directory access protocol to an X.500 directory service. In early days, the typical architecture involves a proxy. Client connects to the proxy in LDAP and the proxy connects to X.500 server in DAP. Nowadays, it is common that LDAP is directly implemented in X.500 servers. Because LDAP is based on a simpler subset of the standards contained within the X.500 standard, LDAP was sometimes called X.500-lite. While DAP and the other X.500 protocols can now use the TCP/IP networking stack, LDAP remains a popular directory access protocol.\nPrevious PostHigh Availability and Load Balancer Next PostLightweight Directory Access Protocol (LDAP) ","date":"2020-02-28T21:36:00-04:00","permalink":"/2020/02/everything-about-the-domain/","title":"Introduction to Active Directory (AD)"},{"content":"Overview Fault tolerance and high availability are two architectural characteristics that people often confuse with each other. High availability focuses on minimizing downtime. It guarantees uptime, but not performance in the event of component failures. Fault tolerance, on the other hand, focuses on stable capacity even in the event of component failures. Fault tolerance has higher bar, and therefore is more expensive. Suppose an application requires four servers to meet performance goal. Placing two servers in each of the two AZs will meet HA criteria but not FT requirement. In the event of an AZ failure, application can operate at degraded performance yet still be highly available. However, FT requires stable capacity and to meet FT requirement, we\u0026#8217;d have to place four servers in each AZ. High availability can be achieved either by clustering, or load balancing. A cluster involves several nodes, all able to perform the same function, but may take different roles at different times (e.g. primary, standby) in order for the cluster to perform its function as a single system. In Linux, clustering is implemented by pacemaker or corosync. With a high load system, it is common to set up load balancing system to achieve high availability (and fault tolerance).\nLoad balancing The idea of load balancing is simple: load goes high and we want to scale horizontally instead of simply upgrading server hardware. At a high level, there has been three approaches to load balancing:\nDNS rotating: (aka. DNS round robin) DNS record resolves to multiple IPs, very simple and cheap to implement. Since DNS is cached, the load distribution will come imbalanced and it\u0026#8217;s hard to re-balance, making this a very limited approach; Hardware Load Balancer: using dedicated hardware device to configure load balancing. This option is expensive and only enterprises can afford it (here\u0026#8216;s some pricing information). A classic load balancer operates at layer 3 and 4, which is also known as POLB (plain old load balancer). It is the core functionality of hardware load balancer. The hardware load balancer on the market today usually come with a variety of add-on features, such as advanced load balancing (L4, L7 path-based, script driven), compression, caching, SSL offloading, and even DDoS mitigation, etc. The whole suite of features makes it an Application Delivery Controller (ADC). Therefore many refer to hardware load balancer as hardware-based ADC to highlight the features in addition to POLB. Hardware-based ADCs ship with manufactures hardware, with specialized processors, advanced network hardware, and often ASIC (application specific integrated circuit). At a higher expense, they have better reliability and capacity. Some major market players are: F5 \u0026#8211; Big IP, F5 also has a good article about history of load balancer. Cisco \u0026#8211; Citrix Ahttps://www.citrix.com/products/citrix-adc/DC (formerly NetScaler ADC) A10 Networks \u0026#8211; Thunder (general) and Lightning (cloud) Software Load Balancer: using software to achieve load balancing. These solutions are affordable, and usually open-source. They can be loaded on commodity hardware (including NIC). Some (e.g. Nginx) refers to themselves as software-based ADC. Major players are: HA Proxy Nginx Linux Virtual Server (LVS, L4 only) The hardware ADCs are usually supported commercially and there are plenty of resources from their white papers. There is an ongoing debate about whether one is better than the other. However, there is no doubt that a software-based load balancer is more approachable as open-source tools. The line between software and hardware load balancers becomes blurred today as hardware vendors try to adapt their software appliance to commodity hardware. Check out this article. The rest of this post, will focus on software-based load balancer. Software-based load balancer We explained that ADC (application delivery controller) is an expanded set of features from load balancer, and will only cover the load balancer part of the feature set in this article.\nHAProxy supports both layer 4 and layer 7 load balancing. It supports load balancing based on cookie and session, as well as health check. Since it is layer 4 load balancing, it supports any TCP protocol such as read traffic for MySQL. \u0026nbsp;\nNginx is a high-performance, event-driven, cross-platform layer 7 load balancing application. It works as a reverse proxy where it receives request for the Internet and forwards it to (upstream) internal servers. It consumes less memory than many of its alternatives for layer 7 load balancing. There are many strategies for load balancing such as round robin, by weight, by hash of requesting IP, by upstream response time, or by URL hash. It supports 20-30 k concurrent connections, and support compression and health check. It is known to be very stable and common for small and medium volume. Nginx has a commercial counterpart Nginx Plus with advanced features.\nNginx and HA proxy are commonly used in front end load balancing. For backend traffic such as database (e.g. separating read write traffic), LVS can be used.\nLinux Virtual Server LVS (Linux Virtual Server) is part of standard Linux kernel. It performs layer 4 load balancing based on TCP or UDP and therefore consumes less memory and CPU. Compared to layer 7 load balancing, the performance is generally higher, and the configuration is less complex (with simpler routing rules). LVS is usually configured in a common cluster architecture involving these components:\nLoad balancer: the front-end machine of the whole cluster systems, and balances requests from clients among a set of servers, so that the clients consider that all the services is from a single IP address. Server cluster: set of servers running actual business workload Shared storage: a shared storage space for the servers, such as NFS Load balancer is the single entry-point of server cluster systems, it can run\u0026nbsp;IPVS\u0026nbsp;that implements IP load balancing techniques inside the Linux kernel, or\u0026nbsp;KTCPVS\u0026nbsp;that implements application-level load balancing inside the Linux kernel. When IPVS is used, all the servers are required to provide the same services and contents, the load balancer forward a new client request to a server according to the specified scheduling algorithms and the load of each server. No matter which server is selected, the client should get the same result. When KTCPVS is used, servers can have different contents, the load balancer can forward a request to a different server according to the content of request. Since KTCPVS is implemented inside the Linux kernel, the overhead of relaying data is minimal, so that it can still have high throughput.\nIPVS is also called layer-4 switching, it directs TCP/UDP requests to the real servers behind load balancer. It works in three modes:\nNetwork Address Translation (NAT) Direct Routing (DR) Tunnel mode (TUN) These are three packet-forwarding methods in IPVS. The IPVS is implemented as a module over the netfilter framework, similar to iptables, which is also built on top of netfilter, based on chain and rules.\nSummary We had an overview of high availability, and then expanded on load balancing, an important mechanism to implement high availability. We touched on both hardware-based and software-based load balancing technologies, and dived a little more into Linux Virtual Server. It is worth-noting that LVS is also the foundation of kube-proxy, the load balancing mechanism used in Kubernetes.\nPrevious PostNginx as a reverse proxy for Nifi web UI and Kibana Next PostIntroduction to Active Directory (AD) ","date":"2020-01-22T20:49:00-04:00","permalink":"/2020/01/several-ways-to-ensure-high-availability/","title":"High Availability and Load Balancer"},{"content":"Nginx can act as a application neutral proxy. One example is to front Nifi. The nifi default configuration provides an HTTP access point, specified in the following entries in nifi.properties:\nnifi.web.http.host=192.168.133.5 nifi.web.http.port=8080 Nifi can provide secure port by commenting out the lines above and provide the followings:\nnifi.web.https.host=192.168.133.5 nifi.web.https.port=8083 However, it does require configuring JKS keystore for Java, as well as authentication. Customers with existing AD servers are likely to require authentication via LDAP. While Nifi does support LDAP integration according to its administration guide. The configuration is quite involving. You need to configure the identity provider, as well as authorizes. I have personally spent a couple days on this without much progress. The information in the logging isn\u0026#8217;t to the point. Restarting nifi also is a long process, making it painful to troubleshoot. I then moved to Nginx (open-source) as an alternative and it is quite enlightening. I already knew that the SSL termination in nginx is super easy to configure. This time I learned that the opensource community even has a support for LDAP integration. Here is a diagram of how it works:\nClientClientActive DirectoryActive DirectoryContainer1Container1NifiNifiContainer2Container2NifiNifiContainerContainerNginx processNginx processhttp_auth_requesthttp_auth_requestldap-auth daemonldap-auth daemonhttphttpdhttpdhttphttphttpLDAPLDAPhttpshttps\nThis approach is outlined in a blog post on Nginx website. The ldap-auth daemon is implemented in Python can can be wrapped up as a systemd service. Once a client sends a request in https, security layer is terminated in nginx, and an authentication request in http is sent to the ldap-auth daemon, which proxies converts the request into LDAP searches and proxies it over to customer\u0026#8217;s Active Directory server, for authentication. Once authenticated, the http request can make to one of the backend container or server which hosts Nifi. Below is an example of how this can be configure on RedHat.\nInstall python3 and python-ldap RedHat may have both python2 and python3 pre-installed, python2 being the default. We do not want to change the default because other applications such as yum still depends on python2 as of early 2020.\nyum -y install python3 yum -y install gcc python3-devel openldap-devel pip3 install python-ldap Once python3 is installed, pip3 will be available and we use that to install python-ldap. This is a module in Python3 that will be used by the script that act as ldap daemon.\nConfigure ldap-auth daemon as systemd service In the github project for ldap-auth, download nginx-ldap-auth-daemon.py to local location such as /usr/bin, then we create nginx-ldap-auth.service in /etc/systemd/system/ with the following content.\n[Unit] Description=LDAP authentication helper for Nginx After=network.target network-online.target [Service] Type=simple User=root Group=root WorkingDirectory=/var/run ExecStart=/usr/bin/python3 /usr/bin/nginx-ldap-auth-daemon.py KillMode=process KillSignal=SIGINT Restart=on-failure [Install] WantedBy=multi-user.target Then, run the following command to load, start and check nginx-ldap-auth service.\nsystemctl reload-daemon systemctl start nginx-ldap-auth systemctl status nginx-ldap-auth This service will be up and listening to port 8888 for http traffic.\nConfigure Nginx Then configure nginx with the following entries in its default.conf file, typically located in /etc/nginx/conf.d.\u0026nbsp; upstream nifibackend { # default: round robin server container1.nifi.digihunch.com:8080; server container2.nifi.digihunch.com:8080; } proxy_cache_path cache/ keys_zone=auth_cache:10m; # nifi proxy server { listen 8083ssl; include /etc/nginx/ssl/default.conf; location / { auth_request /auth-proxy; proxy_pass http://nifibackend; proxy_set_header Host $host:$server_port; proxy_set_header X-ProxyScheme https; proxy_set_header X-ProxyHost $1; proxy_set_header X-ProxyPort 8083; proxy_set_header X-ProxyContextPath /; } location /auth-proxy { internal; proxy_pass http://127.0.0.1:8888; proxy_pass_request_body off; proxy_set_header Content-Length \u0026#34;\u0026#34;; proxy_cache auth_cache; proxy_cache_valid 200 10m; proxy_cache_key \u0026#34;$http_authorization$cookie_nginxauth\u0026#34;; proxy_set_header X-Ldap-URL \u0026#34;ldaps://ldap.digihunch.com:636\u0026#34;; proxy_set_header X-Ldap-BaseDN \u0026#34;OU=Corporate User Accounts,DC=digihunch,DC=org\u0026#34;; proxy_set_header X-Ldap-BindDN \u0026#34;CN=Digi Hunch Service Account,OU=Digi,OU=ServiceAccounts,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org\u0026#34;; proxy_set_header X-Ldap-BindPass \u0026#34;myownpasswordtricks\u0026#34;; proxy_set_header X-CookieName \u0026#34;nginxauth\u0026#34;; proxy_set_header Cookie nginxauth=$cookie_nginxauth; proxy_set_header X-Ldap_Starttls \u0026#34;true\u0026#34;; proxy_set_header X-Ldap-Template \u0026#34;(\u0026amp;(sAMAccountName=%(username)s)(objectClass=organizationalPerson)(memberOf=CN=GH_SYSADMIN,OU=GHCO,OU=Groups,OU=Digi,OU=Digi Applications,DC=digihunch,DC=org))\u0026#34;; } } We need the full distinguished name of bind user to get this to work. Once configured properly, and user attempts to connect through a browser, Nginx will pop up a prompt for username and password. The username will be plugged into the X-Ldap-Template for further queries. The same HTTP header also allows you to filter by membership that the user is associated with.\nPrevious PostNetworking Basics 3 of 3 – common network protocols and technologies Next PostHigh Availability and Load Balancer ","date":"2020-01-16T22:22:51-04:00","permalink":"/2020/01/nginx-as-a-reverse-proxy-for-nifi/","title":"Nginx as a reverse proxy for Nifi web UI and Kibana"},{"content":"The 5 layer TCP/IP model (or its more rigorously defined alternative OSI model) leads to a whole world of network protocols. Understanding these new protocols requires one to map it out agains the network layers (e.g. at Layer 4 whether it is TCP or UDP, etc) .\nVPN is a whole family of technologies with many flavours of implementation. The previous posting covered some basics of the idea, as well as the two common forms (remote access VPN and site-to-site VPN). The VPN implementation protocols vary a lot. PPTP(Point-to-Point Tunnelling Protocol) is outdated and less secure\u0026nbsp;IPSec (Internet Protocol Security)L2TP (Layer 2 Tunnelling Protocol) replacement of PPTP, more secure, more overhead and slightly slower.OpenVPN \u0026#8211; very secure, and reliable and supported by communities all over the world.TLS/SSL and SSH connections may be considered VPN as well. Phone service protocols:\nVoIP (voice over IP, operating at network layer) \u0026#8211; allows one to make and receive phone calls over the network. Communication on the IP network is perceived as less reliable in contrast to the circuit-switched public telephone network because it does not provide a network-based mechanism to ensure that data packets are not lost, and are delivered in sequential order. It is a best-effort network without fundamental Quality of Service (QoS) guarantees. Voice, and all other data, travels in packets over IP networks with fixed maximum capacity. This system may be more prone to data loss in the presence of congestion[a] than traditional circuit switched systems; a circuit switched system of insufficient capacity will refuse new connections while carrying the remainder without impairment, while the quality of real-time data such as telephone conversations on packet-switched networks degrades dramatically. Therefore, VoIP implementations may face problems with latency, packet loss, and jitter.SIP (session initiation protocol, operating at application layer) \u0026#8211; a VOIP signaling protocol responsible for the creation and tearing down of media connections. So it supports all types of media. ALG (application layer gateway, aka proxy server) \u0026#8211; a software component that manages specific application protocols such as SIP and FTP. An ALG acts as an intermediary between the Internet and an application server that can understand the application protocol. ALG proxies connection to destination on behalf of client. This adds Application Layer Gateway DDNS (dynamic domain name service) \u0026#8211; a router service that assigns your device a fixed domain name even though you are using dynamic IP.\nNAT \u0026#8211; another family of technologies, usually implemented in the following\nPort preservation: source port chosen by a client is the same port used by the router\u0026nbsp;Port forwarding: NAT application redirects a communication request from one address and port number combination to another while the packets are traversing a network gateway, such as a router or firewall. Port triggering: a dynamic form of the port forwarding model. Generally, port triggering is used when the user needs to use port forwarding to reach multiple local computers. Port are close when they aren\u0026#8217;t in use (more secure) protocol used is UPnP\u0026nbsp; DMZ \u0026#8211; a physical or logical subnet that contains external facing service to untrusted network (e.g. Internet). The purpose is to add an additional layer of security so an external network can assess what is exposed in DMZ while the rest of network remains firewalled.\u0026nbsp;\nVPN passthrough \u0026#8211; a feature that allows any device connected to the router to establish outbound VPN connections. Most modern router already have this built in.\u0026nbsp;\nWAN optimization \u0026#8211; a collection of techniques for increasing data transfer efficiencies across wide-area networks.\nDeduplicationCompressionLatency optimizationCaching/proxyProtocol spoofingTraffic shaping Network performance tuning\nPerformance tuning in network covers a variety of skills. It is important to understand in which layer the problem occurs. Anything above layer 4 is more likely to be an application issue. For ethernet performance tuning, I found this page and this page to be helpful in my practices.\nPrevious PostNetworking basics 2 of 3 – Layer 4 and common network configurations Next PostNginx as a reverse proxy for Nifi web UI and Kibana ","date":"2019-12-20T10:28:00-04:00","permalink":"/2019/12/networking-basics-3-of-3-common-network-technologies/","title":"Networking Basics 3 of 3 – common network protocols and technologies"},{"content":"Transport Layer Transport Layer handles multiplexing \u0026amp; de-multiplexing through ports. Port is more or less a virtual concept. Source port is usually ephemeral. Two dominant protocols are TCP and UDP.\nTCP relies on acknowledgement. TCP control flags are SYN, ACK, FIN, URG, PSH, RST, ECE, CWR. TCP connection is established by 3-way handshake and torn down by 4-way termination.\nTCP handshake and termination Socket \u0026#8211; the instantiation of an end-point in a potential TCP connection. A socket can be in one of the following states:\nLISTEN: a TCP socket is ready and listening for incoming connections; SYN_SENT: a SYNC request has been sent but connection hasn\u0026#8217;t been established yet; SYN_RECEIVED: a socket previously in a LISTEN state has received a SYNC request and sent a SYN/ACK back; ESTABLISHED: connection is up; FIN_WAIT: FIN sent, ACK hasn\u0026#8217;t been received yet; CLOSE_WAIT: connection has been closed at the TCP layer but the application that opened the socket hasn\u0026#8217;t release the hold on the socket yet; CLOSED: connection fully terminated; TCP packet format TCP is a connection-oriented protocol\nTransport layer is responsible for re-sending data if data is lost Sequence # is important because packet may arrive out of sync but receiver reassemble them in order There is a lot of overhead (acknowledgement, establish connection first, tear down connection afterwards) On the other hand, UDP is connectionless. A good example is video streaming, where it is okay to lose a few packet along the way, in exchange of bandwidth saving.\nFirewall may operate at different layers but it is most commonly used at transport layer, to block traffic based on port.\nApplication Layer There is no dominant protocol at this layer. IIS, Nginx and Apache are examples of applications operating at this layer.\nBasic network configurations Standard modern network configuration involves: IP address, subnet mask, gateway and DNS server. DNS \u0026#8211; global and highly distributed network service that resolves domain name into IP address. There are many steps in DNS resolution. DNS service listens on port 53. Two famous free public DNS servers are 8.8.8.8 and 8.4.4.4. DNS servers have five categories:\nCaching name servers: store known domain name lookups in cache. TTL today can be a few hours, much shorter than what it used to be in early days; Recursive name servers: perform full DNS resolution request; Root name servers; TLD name servers; Authoritative name servers; DNS resolution steps DNS uses UDP protocol and it can generate a lot of traffic (TCP is impractical. If implemented in TCP, it would have required 44 packet for a DNS query, which is very expensive considering DNS query is just a precursor of the real traffic)\nAnycast DNS \u0026#8211; any one of a number of DNS servers can respond to DNS queries, and typically the one that is geographically closest will provide the response. This reduces latency, improves uptime for the DNS resolving service and provides protection against DNS flood DDoS attacks.\nDNS record types:\nA record: domain name to IP address. DNS service round robin across multiple A records AAAA (quad A) record: domain name to IPv6 address CNAME: redirect traffic from one domain to another (e.g. test.com to www.test.com so you can minimize IP references) MX record SRV record TXT record: originally for human consumption, freeform text for configuration purpose. A FQDN (fully qualified domain name) can have up to 127 domains, but only three in most cases. (i.e. subdomain.domain.topleveldomain)\nDNS zones \u0026#8211; allow for easier control over multiple levels of a domain. DNS zones are configured in zone files. Domains vs zones \u0026#8211; Domains are broken into zones for which individual DNS servers are responsible. A domain represents the entire set of names/machines that are contained under an organizational domain name. For example, all domain names ending with \u0026#8220;.com\u0026#8221; are part of the \u0026#8220;com\u0026#8221; domain. A \u0026#8220;zone\u0026#8221; is a domain less any sub-domains delegated to other DNS servers. A DNS server could be responsible (authoritative) for all records under the \u0026#8220;xyz.com\u0026#8221; domain, but by defining NS-records for \u0026#8220;abc.xyz.com\u0026#8221;, this part of the domain is delegated to other DNS servers \u0026#8211; and possibly a different company/entity. A zone contains exactly one SOA-record describing the general properties of the zone, and any number of other DNS records. Entire zones can transferred from a primary DNS server to secondary DNS servers through Zone Transfers. A domain administrator would be responsible for creating zones, and delegating responsibility for these zones to an administrator and DNS server.\nReverse DNS lookup \u0026#8211; query for FQDN by IP. This is commonly used by email servers where anti-spam mechanism on the receiver needs to validate that sender\u0026#8217;s IP is associated with a domain as claimed. This is also used in logging application to convert IP into human-readable domains in the log data. Reverse DNS lookups query DNS server for a PTR (pointer reserve record). If the server does not have a PTR record, it cannot resolve a reverse lookup.\nDHCP\nDHCP operates at application layer and helps you to configure\u0026nbsp; IP automatically with a lease, through automatic allocation, or fixed allocation based on MAC, etc. DHCP process involves address allocation, renewal, and release. Address allocation takes four steps:\nClient sends a broadcast to discover DHCP server; DHCP server broadcast a DHCP offer; Client requests IP address from the DHCP server; Server acknowledged the DHCP request; Here is an illustration of DHCP address allocation.\nDHCP can also be used to set NTP address.\nNAT\nNetwork Address Translation (NAT), can be implemented in many different ways in different OS. Essentially, it is a technology that allows a gateway, usually a router or firewall, to rewrite the source IP of an outgoing IP datagram while retaining the original IP in order to rewrite it into the response.\u0026nbsp; Two categories of NAT are:\nBasic NAT: provides a one-to-one translation of IP addresses, aka one-to-one NAT. Basic NATs can be used to interconnect two IP networks that have incompatible addressing. One-to-many NAT: maps multiple private hosts to one publicly exposed IP address, aka IP masquerading. This can be a security measure so that no external host can establish to your computer without knowing your actual IP. Source port conflict can be managed in two ways: port preservation: When making outgoing connection, NAT preserves the ephemeral port number used by internal client that initiates the connection; if two clients happen to use the same ephemeral port, then NAT picks a random port to initiate outgoing TCP connection; port forwarding (port mapping): Forward traffic to certain destination based on the port of incoming request that NAT receives. Proxy\nProxy refers to a concept rather than a specific implementation. It exists on almost every layer in the network model, and act on behalf of a client in order to access other service. For example, Web proxy used to be used to cache web traffic data in slow Internet but it is not necessary any more because 1) there is not much speed benefit; 2) website today is much more dynamic. Reverse proxy is a popular architecture of web server, such as Nginx, to act as a front end of web servers, as well as point of decryption so web servers are free to just serve the content. VPN\nVPN \u0026#8211; allows for extension of a private or local network to host that might not be on that local private network by using encrypted tunnel. There are many flavours of implementation for many purposes. It is a general concept rather than a specific protocol (just like NAT). VPN client provisions the computer with a virtual interface with an IP that matches the address space of the private network, and establish a VPN tunnel to it. Most VPNs work by using the payload section of transport layer to carry an encrypted payload that actually contains an entire second set of packets: the network, the transport and the application layers of a packet intended to traverse a network. Basically, this payload is carried to the VPN\u0026#8217;s endpoint where all the other layers are stripped away and discarded. Then, the payload is unencrypted, leaving the VPN server with the top three layers of a new packet. This gets encapsulated with the proper datalink layer information and sent across the network. This process is completed in the inverse in the opposite direction. VPN usually requires strict authentication procedures and encryption. VPN can also be used to establish site-to-site connection (aka point-to-point VPN) where individual user doesn\u0026#8217;t have to establish connections on their own. Both sites needs specialized hardware to achieve this. Site-to-site VPN is a good alternative to WAN when two site don\u0026#8217;t need to transfer large amount of data for very fast speed.\nWAN\nWide Area Network \u0026#8211; act like a single network, but span across multiple physical locations, it requires that you contract the link across the internet with ISP. ISP handles data link from one site to another.\nWireless Network\nWireless protocol (802.11 family) defines how Wifi operates at physical and data link layer. Wifi networks operates on 2.4GHz and 5GHz frequency bands. Wireless frame is fairly different from Ethernet frame due to the nature of wireless transmission.\nWireless frame Wireless access point is a device that bridges the wireless and wired portions of a network. A single wired network might have many wireless access points to cover a large area.\nWireless network can be configured in a few main ways:\nAd-hoc network \u0026#8211; nodes all directly speak to each other. No supporting infrastructure is needed but not most common. It can be powerful tool during disasters. Wireless LAN (WLAN) \u0026#8211; one or more access points act as abridge between wireless and wired network. This is the most common type in business world where the wired LAN provides link to the Internet. Mesh networks \u0026#8211; a hybrid of the two above Wireless Security\nWireless transmission is across the air so encryption is more important. The number of bit in the encryption key corresponds to how secure the encryption is.\nWEP (encryption technology) provides low level of privacy (40-bit encryption) and it is not preferred today; WPA provides 128-bit key encryption; WPA2 provides 256-bit key encryption and is most common today. MAC filtering also help security in wireless Basic networking troubleshooting\nICMP ping to test general quality of connection; traceroute discovers the path between two nodes and give you the information along the way; netcat checks port and host address (telnet is retiring); nslookup: very powerful in interactive mode for resolution tools; IPv6 IPv4 address running out in November 2019 is a major crisis. IPv6 becomes more critical to implement for many organizations. IPv6 address is very long with numbers. Numbers represents a natural way that computer thinks and operates, but not for human. So IPv6 comes with two rules to shorten the address:\nYou may remove any leading 0; Any number of consecutive groups can be replaced with two colons; IPv6 header looks like below:\nIPv6 header Coexistence \u0026#8211; the optimal approach for existing networks is to focus not on transition but on coexistence. Coexistence may live a long period with these phases: 1) Turn on IPv6 routing in their existing IPv4 networks and start using it; 2) Contract IPv6 service with their upstream, peer, and downstream neighbours; 3) Use the IPv6 protocol in addition to IPv4 in their applications and services both on server equipment and on their clients; 4) turn off IPv4 at some point when it is no longer a business requirement.\nPrevious PostStorage Nitty-Gritty 5 of 5 – Replication Next PostNetworking Basics 3 of 3 – common network protocols and technologies ","date":"2019-12-07T23:19:00-04:00","permalink":"/2019/12/tcp-ip-basics-2-of-3-layer-4-and-common-technologies/","title":"Networking basics 2 of 3 – Layer 4 and common network configurations"},{"content":"Replication Terms PIT (point in time) replica \u0026#8211; snapshot of the source at some specific timestamp;\nContinuous Replica \u0026#8211; always in-sync with the production data;\nRecoverability \u0026#8211; enables restoration of data from the replica to the source if data loss or corruption occurs;\nRestartability \u0026#8211; enables restarting business operations using the replicas;\nLocal Replication Use Case:\nAlternative source for backupFast recoveryDecision-support activities such as data warehousingTesting platformData migration Consistency in file system replication File systems buffer the data in the host memory to improve the application response time. The buffered data is periodically written to the disk. In UNIX operating systems, sync daemon is the process that flushes the buffers to the disk at set intervals. In some cases, the replica is created between the set intervals, which might result in the creation of an inconsistent replica. Therefore, host memory buffers must be flushed to ensure data consistency on the replica, prior to its creation.\nFlushing the file system buffer In the illustration above, If the host memory buffers are not flushed, the data on the replica will not contain the information that was buffered in the host. If the file system is unmounted before creating the replica, the buffers will be automatically flushed and the data will be consistent on the replica.\nConsistency in database replication\nWhen a database is replicated while it is online, changes made to the database at this time must be applied to the replica to make it consistent. A consistent replica of an online database is created by using the dependent write I/O principle or by holding I/Os momentarily to the source before creating the replica.\nA dependent write I/O principle is inherent in many applications and database management systems (DBMS) to ensure consistency. According to this principle, a write I/O is not issued by an application until a prior related write I/O has completed. For example, a data write is dependent on the successful completion of the prior log write.\nFor a transaction to be deemed complete, databases require a series of writes to have occurred in a particular order. These writes will be recorded on the various devices or file systems.\nAnother way to ensure consistency is to make sure that the write I/O to all\nsource devices is held for the duration of creating the replica. This creates a\nconsistent image on the replica. However, databases and applications might time out if the I/O is held for too long.\nLocal Replication Technologies\nHost-based Local Replication\nLVM-based replication: logical volume manager (LVM) is responsible for creating and controlling the host-level logical volumes. Each logical block in a logical volume is mapped to two physical blocks on two different physical volumes. LVM-based replication is part of operating system and comes without additional license cost. However, every write generated by application translates into two writes on the disk, and thus, an additional burden is placed on the host CPU. This can degrade application performance. Presenting an LVM-based logical replica to another host is usually not possible because the replica will still be part of the volume group, which is accessed by one host at any given time. You can\u0026#8217;t track changes on LVMs either so it does not support incremental resynchronization.\nFile system snapshot: a pointer-based replica that requires a fraction of the space used by the production FS. This snapshot can be implemented by either FS or by LVM. It uses the Copy on First Write (CoFW) principle to create snapshot. When a snapshot is created, a bitmap and blockmap are created in the metadata of the Snap FS. The bitmap is used to keep track of blocks that are changed on the production FS after the snap creation. The blockmap is used to indicate the exact address from which the data is to be read when the data is accessed from the Snap FS. Immediately after the creation of the FS snapshot, all reads from the snapshot are actually served by reading the production FS. In a CoFW mechanism, if a write I/O is issued to the production FS for the fi rst time after the creation of a snapshot, the I/O is held and the original data of production FS corresponding to that location is moved to the Snap FS. Then, the write is allowed to the production FS. The bitmap and blockmap are updated accordingly. Subsequent writes to the same location do not initiate the CoFW activity. To read from the Snap FS, the bitmap is consulted. If the bit is 0, then the read is directed to the production FS. If the bit is 1, then the block address is obtained from the blockmap, and the data is read from that address on the Snap FS. Read requests from the production FS work as normal. File system snapshot Storage Array-based local replication\nthe array-operating environment performs the local replication process. The host resources, such as the CPU and memory, are not used in the replication process. Consequently, the host is not burdened by the replication operations. The replica can be accessed by an alternative host for other business operations.\nFull-Volume Mirroring \u0026#8211; the target is attached to the source and established as a mirror of the source. After all the data is copied and both the source and the target contain identical data, the target can be considered as a mirror of the source. After the synchronization is complete, the target can be detached from the source and made available for other business operations. The target becomes a point-in-time (PIT) copy of the source. After detachment, changes made to both the source and replica can be tracked at some predefined granularity. This enables incremental resynchronization (source to target) or incremental restore (target to source). The granularity of the data change can range from 512 byte blocks to 64 KB blocks or higher.\nFull volume mirroring Pointer-based, Full-Volume Replication \u0026#8211; the target is immediately accessible by the BC host after the replication session is activated. Therefore, data synchronization and detachment of the target is not required to access it. Pointer-based, Virtual Replication \u0026#8211; at the time of the replication session activation, the target contains pointers to the location of the data on the source. The target does not contain data at any time. Therefore, the target is known as a virtual replica. the target is immediately accessible after the replication session activation. A protection bitmap is created for all data blocks on the source device. Granularity of data blocks can range from 512 byte blocks to 64 KB blocks or greater. Network-based local replication: the replication occurs at the network layer between host and storage arrays. By offloading replication from servers and arrays, network-based replication can work across a large number of server platforms and storage arrays, making it ideal for highly heterogeneous environments.\nContinuous Data Protection: CDP provides the ability to restore data to any previous PIT. In CDP, data changes are continuously captured and stored in a separate location from the primary storage. With CDP, recovery from data corruption poses no problem because it allows going back to a PIT image prior to the data corruption incident. CDP uses a journal volume to store all data changes on the primary storage. The journal volume contains all the data that has changed from the time the replication session started. The amount of space that is configured for the journal determines how far back the recovery points can go. CDP appliance is an intelligent hardware platform that runs the CDP software and manages local and remote data replications. Write splitters intercept writes to the production volume from the host and split each write into two copies. Write splitting can be performed at the host, fabric, or storage array. CDP Local Replication Operation: before the start of replication, the replica is synchronized with the source and then the replication process starts. After the replication starts, all the writes to the source are split into two copies. One of the copies is sent to the CDP appliance and the other to the production volume. When the CDP appliance receives a copy of a write, it is written to the journal volume along with its timestamp. As a next step, data from the journal volume is sent to the replica at predefi ned intervals.\nContinuous Data Protection Tracking Changes to Source and Replica\nChanges can occur on the replica device if it is used for other business operations. To enable incremental resynchronization or restore operations, changes to both the source and replica devices after the PIT should be tracked.\nThis is typically done using bitmaps, where each bit represents a block of data. For example, if the block size is 32 KB, then a 1-GB device would require 32,768 bits (1 GB divided by 32 KB). The size of the bitmap would be 4 KB. If the data in any 32 KB block is changed, the corresponding bit in the bitmap is flagged. If the block size is reduced for tracking purposes, then the bitmap size increases correspondingly.\nThe bits in the source and target bitmaps are all set to 0 (zero) when the replica is created. Any changes to the source or replica are then fl agged by setting the appropriate bits to 1 in the bitmap. When resynchronization or restore is required, a logical OR operation between the source bitmap and the target bitmap is performed. The bitmap resulting from this operation references all blocks that have been modifi ed in either the source or replica.\nThis enables an optimized resynchronization or a restore operation because it eliminates the need to copy all the blocks between the source and the replica. The direction of data movement depends on whether a resynchronization or a restore operation is performed.\nIf resynchronization is required, changes to the replica are overwritten with the corresponding blocks from the source. If a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.\nIf a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.\nTracking Changes Comparison of local replication technologies Local Replication in a Virtualized Environment\nTypically, local replication of VMs is performed by the hypervisor at the compute level. However, it can also be performed at the storage level using array-based local replication, similar to the physical environment. In the array-based method, the LUN on which the VMs reside is replicated to another LUN in the same array. VM Snapshot captures the state and data of a running virtual machine at a specifi c point in time. The VM state includes VM files, such as BIOS, network confi guration, and its power state (powered-on, powered-off, or suspended). The VM data includes all the files that make up the VM, including virtual disks and memory. A VM Snapshot uses a separate delta file to record all the changes to the virtual disk since the snapshot session is activated. Snapshots are useful when a VM needs to be reverted to the previous state in the event of logical corruptions. Reverting a VM to a previous state causes all settings confi gured in the guest OS to be reverted to that PIT when that snapshot was created. There are some challenges associated with the VM Snapshot technology. It does not support data replication if a virtual machine accesses the data by using raw disks. Also, using the hypervisor to perform snapshots increases the load on the compute and impacts the compute performance.\nRemote Replication Synchronous remote replication \u0026#8211; writes must be committed to the source and remote replica (or target), prior to acknowledging \u0026#8220;write complete\u0026#8221; to the host. Additional writes on the source cannot occur until each preceding write has been completed and acknowledged. This ensures that data is identical on the source and replica at all times. Further, writes are transmitted to the remote site exactly in the order in which they are received at the source. Therefore, write ordering is maintained. If a source-site failure occurs, synchronous remote replication provides zero or near-zero RPO. However, application response time is increased with synchronous remote replication because writes must be committed on both the source and target before sending the “write complete” acknowledgment to the host. The degree of impact on response time depends primarily on the distance between sites, bandwidth, and quality of service (QOS) of the network connectivity infrastructure.\nSynchronous replication In asynchronous remote replication, a write is committed to the source and immediately acknowledged to the host. In this mode, data is buffered at the source and transmitted to the remote site later. Asynchronous replication eliminates the impact to the application’s response time because the writes are acknowledged immediately to the source host. This enables deployment of asynchronous replication over distances ranging from several hundred to several thousand kilometers between the primary and remote sites.\nAsynchronous replication Below are the bandwith requirement for both:\nBandwidth requirement for synchronous replication Bandwidth requirement for asynchonous replication Asynchronous replication implementation can also take advantage of locality of reference (repeated writes to the same location). If the same location is written multiple times in the buffer prior to transmission to the remote site, only the final version of the data is transmitted. This feature conserves link bandwidth.\nRemote Replication Technologies\nHost-Based Remote Replication\nLVM-based remote replication: performed and managed at the volume group level. Writes to the source volumes are transmitted to the remote host by the LVM. The LVM on the remote host receives the writes and commits them to the remote volume group.\nLVM-based remote replication supports both synchronous and asynchronous modes of replication. LVM-based remote replication is independent of the storage arrays and therefore supports replication between heterogeneous storage arrays.\nThe replication process adds overhead on the host CPUs. CPU resources on the source host are shared between replication tasks and applications. Because the remote host is also involved in the replication process, it must be continuously up and available.\nLVM based remote replication Host-Based Log Shipping\nDatabase replication via log shipping is a host-based replication technology supported by most databases. Transactions to the source database are captured in logs, which are periodically transmitted by the source host to the remote host. The remote host receives the logs and applies them to the remote database.\nRPO at the remote site is fi nite and depends on the size of the log and the frequency of log switching. Available network bandwidth, latency, rate of updates to the source database, and the frequency of log switching should be considered when determining the optimal size of the log file. Host-based log shipping requires low network bandwidth because it transmits only the log fi les at regular intervals.\nHost based log shipping Storage Array-Based Remote Replication\nSynchronous replication mode\nTo optimize the replication process and to minimize the impact on application response time, the write is placed on cache of the two arrays. The intelligent storage arrays destage these writes to the appropriate disks later.\nIf the network links fail, replication is suspended; however, production work can continue uninterrupted on the source storage array. The array operating environment keeps track of the writes that are not transmitted to the remote storage array. When the network links are restored, the accumulated data is transmitted to the remote storage array. During the time of network link outage, if there is a failure at the source site, some data will be lost, and the RPO at the target will not be zero.\nArray-based remote synchronous replication Asynchronous replication mode Fig 12-8\nData is buffered at the source and transmitted to the remote site later. The source and the target devices do not contain identical data at all times. The data on the target device is behind that of the source, so the RPO in this case is not zero. Asynchronous replication writes are placed in cache on the two arrays and are later destaged to the appropriate disks. Some implementations of asynchronous remote replication maintain write ordering. A timestamp and sequence number are attached to each write when it is received by the source. Writes are then transmitted to the remote array, where they are committed to the remote replica in the exact order in which they were buffered at the source. This implicitly guarantees consistency of data on the remote replicas.\nArray-based asynchronous replication Disk-buffered replication mode: a combination of local and remote technologies. A consistent PIT local replica of the source device is fi rst created. This is then replicated to a remote replica on the target array.\nAt the beginning of the cycle, the network links between the two arrays are suspended, and there is no transmission of data. While production application runs on the source device, a consistent PIT local replica of the source device is created. The network links are enabled, and data on the local replica in the source array transmits to its remote replica in the target array.\nDisk buffered remote replication Network-based Remote Replication\nCDP remote replication\nFig 12-10\nThree site replication\nCascade/Multihop: data fl ows from the source to the intermediate storage array, known as a bunker, in the fi rst hop, and then from a bunker to a storage array at a remote site in the second hop. Replication between the source and the remote sites can be performed in two ways: synchronous + asynchronous or synchronous + disk buffered. Replication between the source and bunker occurs synchronously, but replication between the bunker and the remote site can be achieved either as disk-buffered mode or asynchronous mode.\nThree-site remote replication cascade/multihop Triangle/Multitarget: data at the source storage array is concurrently replicated to two different arrays at two different sites. The source-to-bunker site (target 1) replication is synchronous with a near-zero RPO. The source-to-remote site (target 2) replication is asynchronous with an RPO in the order of minutes. The distance between the source and the remote sites could be thousands of miles. The key benefit of three-site triangle/multitarget replication is the ability to failover to either of the two remote sites in the case of source-site failure, with disaster recovery (asynchronous) protection between the bunker and remote sites. Three-site replication triangle/multitarget Data migration solutions\nData mobility refers to moving data between heterogeneous storage arrays for cost, performance, or any other reason. It helps implement a tiered storage strategy. Data migration refers to moving data from one storage array to other heterogeneous storage arrays for technology refresh, consolidation, or any other reason. The array performing the replication operations is called the control array.\nData migration solutions perform push and pull operations for data movement.\nThese terms are defined from the perspective of the control array. In the push operation, data is moved from the control array to the remote array.\nThe control device, therefore, acts like the source, while the remote device is the target.\nIn the pull operation, data is moved from the remote array to the control array.\nThe remote device is the source, and the control device is the target.\nThe push and pull operations can be either hot or cold. These terms apply to the control devices only. In a cold operation the control device is inaccessible to the host during replication. Cold operations guarantee data consistency because both the control and the remote devices are offl ine. In a hot operation the control device is online for host operations. During hot push and pull operations, changes can be made to the control device because the control array can keep track of all changes and thus ensure data integrity.\nRemote replication and migration in a virtualized environment\nIn hypervisor-to-hypervisor VM migration, the entire active state of a VM is moved from one hypervisor to another. This method involves copying the contents of virtual machine memory from the source hypervisor to the target and then transferring the control of the VM’s disk fi les to the target hypervisor. Because the virtual disks of the VMs are not migrated, this technique requires both source and target hypervisor access to the same storage.\nHypervisor-to-hypervisor VM migration In array-to-array VM migration, virtual disks are moved from the source\narray to the remote array. This approach enables the administrator to move VMs across dissimilar storage arrays. Array-to-array migration starts by copying the metadata about the VM from the source array to the target. The metadata essentially consists of configuration, swap, and log files. After the metadata is copied, the VM disk file is replicated to the new location.\nArray-to-array VM migration Related Postings Disk and RAIDSANNAS and Object StorageBackup and Archive Solution Previous PostNetworking Basics 1 of 3 – Layer 1 through Layer 3 Next PostNetworking basics 2 of 3 – Layer 4 and common network configurations ","date":"2019-11-19T00:10:23-04:00","permalink":"/2019/11/storage-nitty-gritty-5-of-5-replication/","title":"Storage Nitty-Gritty 5 of 5 – Replication"},{"content":"What layer model works the best? Back in university my textbook was based on OSI 7-layer model. It is rigorously defined and often used in academics. When it comes to day-to-day operation, the 5-layer TCP/IP model is more useful. It combines Application, Presentation and Session layers in OSI model into a single Application layer.\nLayerNameProtocolProtocol Data UnitAddressingDeviceDescription 5ApplicationHTTP, FTP, etcMessageN/AN/A 4TransportTCP and UDPSegmentPort NumberGatewaySort out which application on the same host receives incoming data 3NetworkIPDatagramIP addressRouter and (layer-3) switchAllows devices across different networks to talk to each other 2Data LinkEthernet, WiFiFrameMAC addressBridge and (layer-2) SwitchDefines common way of interpreting signals so devices in the network can communicate 1Physical10baseT, 802.11bitN/AHubHardware: cables, signal connector Physical Layer Crosstalk \u0026#8211; electrical pulse on one wire is accidentally detected on another wire. This was a common challenge when the industry started. The most common cable is UTP (Unshielded Twisted Pair) cable such as Cat 5, Cat 5e cables.\nHub \u0026#8211; a physical layer device that allow for connectivity from many computers at once. It is up to each device to determine if incoming data is for them, or to ignore it. Because this slows down transmission, hubs are hardly used any more. Collision domain \u0026#8211; A network segment where only one device can communicate at a time. The device sending signal is occupying the entire media, within its time-sharing window. All devices connected to a hub are in the same collision domain. Ethernet nodes use CSMA/CD to detect collisions and re-transmit when the wire becomes available again. Wireless Channels \u0026#8211; individual, smaller sections of the overall frequency band used by a wireless network. Collision is very common in wireless communication. So the channel selection should minimize collision. Data Link Layer In a LAN, NICs talk to each other via MAC address (first 3 octets is organization unique identifier; the last 3 octets are assigned by vendor). Two NICs communicate through twisted pair cable in one of the following modes:\nSimplex: data is sent in one directly only; Half duplex: one line and transmission in each direction takes turns; Full duplex: two lines, one for each direction, simultaneous; Network Switch \u0026#8211; connects to many devices as well but it determines which device the data is intended for and only send that data to that device. Switch is a layer-2 device. (However today as a network device, many switches have layer-3 capability so it is important to be specific when talking about switch)\nEthernet Frame Format\nEthernet Frame Ethernet address types:\nUnicast address: points to one receiving end; it contains a unique MAC address the frame is intended for; Multicast address: multicast frame is identified by FF as the first 8-bit, followed by a 4-bit flag field, a 4-bit scope field, and a 112-bit group ID. Broadcast address: for special destination such as ARP; it contains all Fs in the destination address. Ethernet frame types include, but not limited to:\nEthernet II frame (most common type in use today used directly by the Internet Protocol) Novell raw IEEE 802.3 non-standard variation frame IEEE 802.2 Logical Link Control (LLC) frame IEEE 802.2 Subnetwork Access Protocol (SNAP) frame Virtual LAN \u0026#8211; any broadcast domain that is partitioned and isolated in a computer network at the data link layer. It is a technique that allows you to have multiple logical LANs operating on the same physical equipment, to segregate traffic.\nNetwork Layer Router connects between LANs. A router needs at least two NICs. The steps to route are:\nReceive data packet Examines destination IP Look up IP destination network in routing table Forward traffic to destination; Routing can be complex but it is mostly handled by ISPs now. A routing table may have millions of rows (use route command to check). Here is an example of routing tables. Example of routing table Autonomous system \u0026#8211; a collection of networks that fall under the control of a single network operator (i.e. large corporation)\nRouting protocol \u0026#8211; specifies how routers communicate with each other, distributing information that enables them to select routes between any two nodes on a computer network. Interior Gateway Protocols are used by routers to share routing information within a single autonomous system. Exterior Gateway protocols are used across autonomous system. Interior Gateway Protocol (link-state routing): OSPF, IS-IS Interior Gateway Protocol (distance-vector): RIP, RIPv2, IGRP Exterior Gateway Protocol: BGP (Border Gateway Protocol) \u0026#8211; allows routers (e.g. Internet) to learn from each other about the most optimal paths to forward traffic. IP datagram\nIP datagram (payload at the bottom) An IP datagram contains a lot more compared to Ethernet frames. One place called type of service field (8-bits) specifies priority. QoS technologies are mostly built on this field, to allow routers to determine which datagram is more important.\nIP Fragmentation: an Internet Protocol (IP) process that breaks packets into smaller pieces (fragments), so that the resulting pieces can pass through a link with a smaller maximum transmission unit (MTU) than the original packet size. The fragments are reassembled by the receiving host. If a receiving host receives a fragmented IP packet, it has to reassemble the packet and pass it to the higher protocol layer. Reassembly is intended to happen in the receiving host but in practice it may be done by an intermediate router, for example, network address translation (NAT) may need to reassemble fragments in order to translate data streams.\nIP address class\nNon-routable IPv4 address spaces belong to no one. Any one can use them in their private network:\n192.168.0.0/16 172.16.0.0/12 10.0.0.0/8 Subnetting \u0026#8211; splitting large network into smaller ones. Incorrect subnetting setups are a common problem you might run into as an IT support. Each subnet has their ingress routers, subnet ID and subnet mask. CIDR is a better way to describe subnet because router only need one entry in their routing table to know where to deliver the traffic.\nNAT allows communicate between non-routable addresses.\nARP (address resolution protocol) table \u0026#8211; maps IP address to MAC address. It is kept on each device (run arp -a to check) and expires after short period of time.\nPrevious PostClean up Git repository Next PostStorage Nitty-Gritty 5 of 5 – Replication ","date":"2019-11-10T20:18:00-04:00","permalink":"/2019/11/networking-basics-layer-1-and-layer-2/","title":"Networking Basics 1 of 3 – Layer 1 through Layer 3"},{"content":"A BitBucket repo has a hard limit of 2GB in size, and soft limit of 1GB. This is\u0026nbsp;not expandable\u0026nbsp;as per\u0026nbsp;Bitbucket and contributors will start receiving warnings once soft limit is reached. We can tell the usage of a repo from the landing page of the repo in BitBucket.\nGit is a distributed version control system for source code management, which implies the followings:\nIt is intended for source code, or configuration code; but not for storing build artifacts, or installers; Git remembers every single commit, including the ones associated with large files; Even a contributor deletes a large file (\u0026#8220;git rm filename\u0026#8221;) after commit, the large file is only removed from the HEAD. The historical commit still stores the file. After all, the whole point of version control is to survive crazy deletion. distributed means that those large files will be pulled down to contributors laptop (waste everybody\u0026#8217;s space although up to 2G:); With all these implications, shrinking the size of a repo isn\u0026#8217;t as straightforward as just removing large files from current commit. We\u0026#8217;d have to\u0026nbsp;rewrite the commit history. Here are the steps we should take once repo size grows over the soft limit.\nClean up remote orphaned branches Removing these branches (remotes/origin/branchname) per se does not free up space. It simplifies the branch structure, leaving /remote/origin/HEAD the only branch left to cleanse for the rest of the steps.\n# git push origin --delete branchname Remove useless files in current commit (HEAD) In this step we remove useless files in current commit. Again we should not expect much space freed because all file committed previously, even deleted, are still stored. They are just now showing up in the working directory. For this step, we can create a separate local dir on Mac:\nmkdir -p /Users/digihunch/repo-cleanup cd /Users/digihunch/repo-cleanup Now within the new directory, we create a bare repo and then the full repo:\ngit clone --mirror https://gh@bitbucket.org/digihunch/source.git git clone https://gh@bitbucket.org/digihunch/source.git Then we dive into the full repo and identify the large files:\nfind . -type f -size +1000k -exec ls -lh {} \\; |awk \u0026#39;{print $9\u0026#34;:\u0026#34; $5}\u0026#39; We can run \u0026#8220;git rm \u0026#8221; against the files identified as too large or deletable. Then commit and push to remote repo. This removes large files from current commit.\nRemove large file and the relevant commits in the history As previously mentioned, we have to re-write the history so history forget about the large files. After this step, the historical commits that large files are associated with will all be deleted. Compare the two charts below to understand what the effect is:\nWe can use the bare repo created in the last step, with \u0026#8220;git filter-branch\u0026#8221; tool to cleanse the branch tree. Some advocate as a faster third party tool\u0026nbsp;BFG Repo-Cleaner\u0026nbsp;as a faster, third-party alternative but I usually lean towards native tool.\u0026nbsp;This\u0026nbsp;article\u0026nbsp;explains the command switches.\ngit filter-branch -f --tree-filter \u0026#34;rm -rf \\large_file.zip\u0026#34; --prune-empty -- --all After this steps the repo should be cleansed. According to this\u0026nbsp;guide\u0026nbsp;from BitBucket, we still need to contact their support to run a garbage collection for us in order to see the size change. It even takes time for the size to be reflected after garbage collection. This\u0026nbsp;reference\u0026nbsp;also does great job explaining what we need to do.\nOther contributors re-sync history It is important to understand that the step above modifies history. Although the commit hash did not change, they are assigned with different commit-ids and you can tell from the commit history where it displays former commit id.\nThis activity only affects remote repository. Each contributor\u0026#8217;s local repository still stores the old commits and should be sync\u0026#8217;ed with the remote origin by deleting the entire repo and run \u0026#8220;git clone\u0026#8221; again. Although not welcomed by every individual contributors, but it is a necessary evil and better approached with explicit instruction.\nBecause this activity takes higher risks, changes each commit, involves vendor support and requires activities by each contributor, the support team should focus on preventing this from happening instead of fixing it.\nConfigure pre-commit hook as a preventive measure As we have more Ansible tasks related, working directory becomes complicated and sometimes contributors accidentally committed large unwanted files (and pushed into the remote repo).\u0026nbsp;Down the road, the best practice is to prevent contributors from committing junks.\u0026nbsp;\nThe best spot to detect this should be a pre-receive hook on the server side, which is only available with self-hosted Bitbucket Server. Unfortunately, this is not a viable option for\u0026nbsp;Bitbucket cloud. Our best bet is client-side pre-commit hook, in which a script\u0026nbsp;performs size check when contributors run \u0026#8220;git commit\u0026#8221;. The purpose is to fail the commit if total file size is over the limit (20M), and the hook itself should be version controlled as well. Compared to (server side) pre-receive hook, the drawback of (client side) pre-commit hook is it requires initial client configuration. The upside is it captures large files before commit.\u0026nbsp;\nThis hook can be a shell script as simple as this:\ncommitsizelimit=20 stagedfilelist=$(git diff --name-only --cached) stagedfilecnt=`echo \u0026#34;$stagedfilelist\u0026#34;| sed \u0026#39;/^\\s*$/d\u0026#39; |wc -l` if [[ $stagedfilecnt -gt 0 ]]; then totalcommitsize=$(du -cm $stagedfilelist | tail -1 | cut -f 1) # Redirect output to stderr. exec 1\u0026gt;\u0026amp;2 if [[ $totalcommitsize \u0026gt; $commitsizelimit ]]; then echo \u0026#34;Warning: Total size of all files in staging area is \u0026#34;$totalcommitsize\u0026#34;MB, exceeding the limit of \u0026#34;$commitsizelimit\u0026#34;MB.\u0026#34; echo \u0026#34; To list files by size, run \u0026#39;du -ch \\$(git diff --name-only --cached)\u0026#39;\u0026#34; echo \u0026#34; To drop large ones from staging area with \u0026#39;git rm -f filename\u0026#39;\u0026#34; echo \u0026#34; To bypass this limit, use \u0026#39;git commit --no-verify\u0026#39;\u0026#34; exit 1 fi fi In the repo we will have a .githook directory to store hooks (e.g. ~/source/.githooks/pre-commit) and point to the hooks directory using the following command:\ngit config core.hooksPath .githooks Previous PostStorage Nitty-Gritty 4 of 5 – Backup and Archive Solutions Next PostNetworking Basics 1 of 3 – Layer 1 through Layer 3 ","date":"2019-10-26T20:33:00-04:00","permalink":"/2019/10/clean-up-your-git-repository/","title":"Clean up Git repository"},{"content":"Business Continuity Information Availability IA = MTBF/(MTBF+MTTR), where\n* MTBF (Mean Time Between Failure) \u0026#8211; average time available for a system or component to perform its normal operations between failures.\n* MTTR (Mean Time to Repair) \u0026#8211; the average time required to repair a failed component.\nDisaster Recovery \u0026#8211; the coordinated process of restoring systems, data, and the infrastructure required to support ongoing business operations after a disaster occurs. It is the process of restoring a previous copy of the data and applying logs or other necessary processes to that copy to bring it to a known point of consistency.\nRecovery-Point Objective (RPO) \u0026#8211; the point in time to which systems\nand data must be recovered after an outage. It defi nes the amount\nof data loss that a business can endure. A large RPO signifi es high tolerance\nto information loss in a business.\nRecovery-Time Ojbective (RTO) \u0026#8211; The time within which systems and applications must be recovered after an outage. It defi nes the amount of downtime that a business can endure and survive. Businesses can optimize disaster recovery plans after defi ning the RTO for a given system.\nStrategies to meet RTO and RPO Data Vault: a repository at a remote site where data can be periodically or continuously copied so a copy is always available in that site.\nHot site: A backup site running all the time.\nCold site: A backup site with minimum infrastructure, to be activated for operation in the event of disaster.\nServer Clustering: a group of servers and relevant resources coupcled to operate as a single syste. Clusters can ensure high availability and load balancing.\nSingle Point of Failure \u0026#8211; failure of a component that can terminate the availability of the entire system or IT service. To mitigate single point of failure, systems are designed with redundancy. This includes:\nredundant HBA on server NIC teamingredundant switchmultiple storage array portsRAID and hot spare configurationRedundant storage arrayserver clustering (e.g. clustered servers exchange heartbeat to inform each other about their health. If one of the servers fails, other server can take up the workload.VM Fault ToleranceMultipathing software: If one path fails, I/O does not reroute unless the system recognizes that it has an alternative path. Multipathing software provides the functionality to recognize and utilize alternative I/O paths to data. Multipathing software also managees the load balancing by distributing I/Os to all available, active paths. Backup Backup is an additional copy of production data created and retained for the sole purpose of recovering lost or corrupted data. Backup are typically performed for the following purposes:\nDisaster recovery. e.g. the backup copies are used for restoring data at an alternate site, when the primary site is incapacitated due to disaster.Operational recovery. e.g. accidental deletion, file corruptionArchival. e.g. data is not changed or accessed any more. Common considerations for backup includes: time interval between two backups (to meet RPO), retention period, media type (to meet RTO), granularity, compression and deduplication\nBackup Granularity\nFull backup: backup of the complete data on the production volumes.Incremental backup: copies the data that has changed since the last full or incremental backup, whichever occurred more recently.Cumulative backup: copies the data that has changed since the last full backup. Backup Methods\nhot backup/online backup: backup is completed while application is up and running;cold backup/offline backup: backup is completed while the application is shutdown for the backup window. The hot backup of online production data is challenging because data is actively used and changed. If a file is open, it is normally not backed up during the backup process. In such situations, an open file agent is required to back up the open file. These agents interact directly with the operating system or application and enable the creation of consistent copies of open files. In database environments, To ensure a consistent database backup, all files need to be backed up in the same state. That does not necessarily mean that all files need to be backed up at the same time, but they all must be synchronized so that the database can be restored with consistency. The disadvantage associated with a hot backup is that the agents usually affect the overall application performance. If this is not acceptable, PIT (point-in-time) copy method can be utilized to create a PIT copy from the production volume and use it as the source for the backup. PIT copy method can reduce impact on production volume.\nTypical Backup Architecture Typical Backup Architecture Typical Backup steps Typical Restore steps Backup topologies\nDirect-attached backup: the storage node is configured on a backup client, and the backup device is attached directly to the client;\nLAN-based backup: the clients, backup server, storage node, and backup device are connected to the LAN;\nSAN-based backup (LAN-free): The SAN-based backup topology is the most appropriate solution when a backup device needs to be shared among clients;\nMixed topology: mix of LAN-based and SAN-based topologies;\nNDMP protocol is for backup in NAS environment\nBackup media\nTape: for long-term offsite storage due to low cost. data access is sequential which implies slowness for both backup and restore. Tapes are susceptible to wear and tear.Disk: fast backup and retrieve to improve RPT and RTO. No offsite capability.Virtual Tape: virtual taps are disk drives emulated and presented as tapes to the backup software. VTL (virtual tape library) has the same components as that of a physical tape library.\nFig 10-18 Data deduplication \u0026#8211; identify and eliminate redundant data to reduce backup window and size. Common data deduplication methods:\nfile-level deduplication (aka. single-instance storage) detects and removes redundant copies of identical files. It enables storing only one copy of the file; the subsequent copies are replaced with a pointer that points to the original file.subfile deduplication breaks file into smaller chunks and then uses a specialized althorithm to detect redundant data within and across the file. This eliminates duplicate data across files. This has two forms:\n* fixed-length block deduplication \u0026#8211; divides the files into fi xed length blocks and uses a hash algorithm to fi nd the duplicate data. * variable-length segment deduplication \u0026#8211; if there is a change in the segment, the boundary for only that segment is adjusted, leaving the remaining segments unchanged. Data deduplication implementation\nsource-based data deduplication \u0026#8211; eliminates redundant data at the source before it\ntransmits to the backup device. This requires less bandwidth and shortens backup window. It increases the overhead on the backup client and could impact the performance of the backup and application running on the client.target-based data deduplication \u0026#8211; deduplication occurs at the backup device, which offloads the backup client from the deduplication process. This takes two forms:inline deduplication \u0026#8211; performs deduplication on the backup data before it is stored on the backup device. this reduces storage need, but introduces time overhead to identify and remove duplication. best for large backup windowpost-process deduplication \u0026#8211; enables backup data to be stored on backup device first, and then deduplicate later. This is suitable for tighter backup windows, but requires more storage. In virtualized environments, backup agent can be installed on the hypervisor, where the VMs appear as a set of files to the agent. VM files can be backed up by performing a file system backup from a hypervisor. For example, Image-based backup operates at hypervisor level and essentially takes a snapshot of the VM. It creates a copy of the guest OS and all the data associated with it (snapshot of VM disk files), including the VM state and application configurations. The backup is saved as a single file (an image) and mounted on a separate server as proxy, which acts as backup client. Image Based Backup Data archive Archive \u0026#8211; a repository where fixed content is stored. Fixed content can be data that were changed but will not be changed anymore.\nOnline archive: A storage device directly connected to a host that makes\nthe data immediately accessible.\nNearline archive: A storage device connected to a host, but the device where the data is stored must be mounted or loaded to access the data.\nOffline archive: A storage device not ready to use. Manual intervention is required to connect, mount or load the storage device before data can be accessed.\nAn archiving agent is software installed on application server. The agent is responsible for identify data that can be archvied based on policy. After the data is identified for archiving, the agent sends the data to the archiving server. Then the original data on the application server is replaced with a stub file, which contains the address of the archived data. Archiving Solution Architecture An archiving server is software installed on a host that enables administrators to configure the policies for archiving data. An archiving storage device stores fixed content.\nRelated Postings Disk and RAIDSANNAS and Object StorageReplication Previous PostPersonal Vim cheatsheet Next PostClean up Git repository ","date":"2019-10-14T19:42:00-04:00","permalink":"/2019/10/storage-nitty-gritty-4-of-5-backup-and-archive-solutions/","title":"Storage Nitty-Gritty 4 of 5 – Backup and Archive Solutions"},{"content":"This is my personal cheatsheet as intermediate Vim user so I skipped the ones that I consider basic. All the commands listed are used in command mode for fast editing.\nCommand execution In command mode, use colon to start ex command. Here are some examples of ex commands:\ne: edit fileg: global commandq: quitw: writes: substitute The commands (g)lobal and (s)ubstitute are heavily used in string manipulation. The rest are summarized here:\n:e!reload current file discarding all unsaved changes:e newfile.txtopen file newfile.txt for editing :e .load current directory:w!force write (if permission allows)shift + zzequivalent to :wq!shift + zqequivalent to :q! Operator .repeat last operationfxfind next character x on the same line Text Editing y for yank(copy), i for inside, a for around, d for delete, w for word, p for paragraph or paste. Examples:\nyypcopy current line and insert afterdiwdelete the entire word where the cursor sits in (dw deletes from cursor to end of word; db deletes from cursor to beginning of word)shift + Vselect entire line in visual mode (v selects character in visual mode) 2\u0026gt;visual mode: indent twice on all selected lines\nedit mode: indent once for 2 lines3\u0026lt;visual mode: outdent three times on all selected lines\nedit mode: outdent once for 3 linesdi\u0026#8221;delete everything between the double quotes surrounding the cursor (exclusive); use c instead of d to finish the same effect with insert modedi\u0026gt;delete everything between \u0026lt; and \u0026gt; surrounding the cursor (exclusive); use (c)hange instead of (d)elete to finish the same effect with insert modeditdelete everything between tags. e.g. \u0026lt;xml\u0026gt;contenttodelete\u0026lt;/xml\u0026gt;dipdelete the entire paragraphda\u0026#8217;delete everything between the single quote surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert modeda}delete everything between { and } surrounding the cursor (inclusive); use c instead of d to finish the same effect with insert modedt.delete all characters until the next .iinsert at cursor locationshift + Imove cursor to first non-blank character of line and start in insert modeainsert at the location next to cursorshift + Amove cursor to last non-blank character of line and start in insert mode0move to beginning of line. ^ moves to first non-blank character in the line. $ moves to the end of line Note: wherever d is used in this table, c can be used instead for the same effect but switch to editing mode at the end.\nString Manipulation The general patterns are:\n[range]g/pattern/cmd[range]s/match/replacement/option If range is not specified, it applies to current line only! To specify the whole file, use % range. You may also specify line range such as \u0026#8220;10,20\u0026#8221;. Here are some examples: :%s/bacon/lettuceFor every line of the file, replace the first occurrence of bacon in each line to lettuce:%s/bacon/lettuce/gFor every line of the file, replace all occurrences of bacon to lettuce:s/bacon/lettuceFor current line, replace the first occurrence of bacon to lettuce:s/bacon/lettuce/giFor current line, replace all occurrences of bacon to lettuce, case insensitive:g/bacon/ddelete all lines that contain pattern \u0026#8216;bacon\u0026#8217;:g!/lettuce/ddelete all lines that do not contain pattern \u0026#8216;lettice\u0026#8217;; or use :v/lettuce/d instead:g/^\\s*$/ddelete all blank lines. \\s* represents zero or more white spaces Bookmarking mamark cursor line as bookmark a`ajump to cursor position at line a\u0026#8216;ajump to beginning of line a`.jump to last line where change occurred\u0026#8220;jump back Insert Mode Ctrl + NAuto complete Edit and run While tmux and screen can help split screen in Bash, we sometimes need to split a bash screen to run a quick command when we\u0026#8217;re already in vim. This can be done with some simple commands.\n:termOpen up a terminal above vim. You can also spell :ter or :terminalCtrl+W; Ctrl+WPress Ctrl+W twice can help you toggle between the terminal and vim bufferCtrl+DClose the terminal This can be very helpful when you are debugging code and need it run repeatedly. You don\u0026#8217;t need to exit vim just to run a command and come back. Note that while you\u0026#8217;re in terminal, you can\u0026#8217;t use Ctrl+W as a shortcut key to backspace a word. Use Alt + Delete instead.\nPrevious PostCryptography basics 2 of 2 Next PostStorage Nitty-Gritty 4 of 5 – Backup and Archive Solutions ","date":"2019-10-07T17:52:00-04:00","permalink":"/2019/10/personal-vim-cheatsheet/","title":"Personal Vim cheatsheet"},{"content":"My previous post outlines several core concepts around cryptography, such as asymmetric key encryption, digital certificate, the encoding formats and relevant file extensions. In this article, we continue to explore cryptography use cases, where these concepts are connected and put into application.\nThe most important use case is TLS handshake. I cannot stress enough how paramount this scenario is. This use case and its variation can be found in almost every situation where connection needs to be secured.\nTLS handshake: the process in which client and server establish secure connection. During the handshake, two parties agree on TLS version, decide on cipher suite, authenticate the identity of each other (although client identity authentication is less common), and generate session key for symmetric encryption after the handshake. Details steps are very important. Here is a fairly thorough reference, and here is a great diagram:\nTLS handshake Secure browser connection: The most widespread use of TLS handshake is to secure browser connection with HTTPS:\nBrowser initiates connection to the server; Server sends browser its certificate (public key + digital signature signed by CA); Browser has preloaded public key of CA and uses it to decrypt digital signature and get the digest of public key;\u0026nbsp; Browser calculate digest of received public key and compares it against the digest from the previous step; If the result is the same, public key is trusted, a green lock is displayed; otherwise, a warning is displayed; proceed to the rest of the steps in TLS handshake. Secure connection in Java: Java applications manages keys and certificates through two classes: java.security.KeyStore and java.security.TrustStore. Suppose a Java application client initiates TLS connection to server. The server application will present its certificates from server\u0026#8217;s Key Store. The client will use certificates stored in client\u0026#8217;s Trust Store to verify the identity of the server. Once validated, the client then presents certificates stored in client\u0026#8217;s Key Store back to the server for validation.\nA KeyStore keeps keys and certificates for your own application. Typically, you store a KeyPair in a KeyStore file. A TrustStore keeps the certificates of external systems that your application trusts.\u0026nbsp; Below is a great diagram:\nSecure Java application in TLS JKS file is Java\u0026#8217;s version of PKCS#12 (private key + certificate, password protected). Entries in a JKS file must have an \u0026#8220;alias\u0026#8221; that is unique. The JKS file type can be used for both Key Store and Trust Store. When it\u0026#8217;s used in Key Store, it contains a certificate and private key for the Java application. When it\u0026#8217;s used in Trust Store, it only contains certificate from external trusted applications. Note that JKS is the default keystore format until Java 8. Since Java 9 the default keystore format is PKCS12.\u0026nbsp;\nFinally, we can introduce some tools:\nCommand line Tools\nopenssl is a versatile tool for cryptography and keytool is a similar tool for Java applications.\nkeytool is for store keys/certificates in Java Key Store or Trust Store.\nConclusion\nI do not include any openssl or keytool command in this article in order to remain theoretical. However, once through the conceptual hurdles, one should become fairly comfortable picking up the tools and understand why each command is needed to achieve its purpose.\nPrevious PostAWS Certified DevOps Engineer Exam Tips Next PostPersonal Vim cheatsheet ","date":"2019-09-08T21:21:41-04:00","permalink":"/2019/09/cryptographic-concepts-for-busy-it-professionals-2-of-2/","title":"Cryptography basics 2 of 2"},{"content":"The last 30 days have been exhausting for me. I studied hard on the new AWS Certified DevOps Engineer exam and thank goodness I passed (750 out of 1000 is required). This was the hardest professional certification I ever worked on. The exam was re-launched recently in March 2019 so there is still a shortage of information around the community. I was hoping to share my experience to help demystify this new exam. I had a somewhat solid background to begin with, having taken the AWS Certified Solution Architect Professional exam (before the 2019 update), and worked quite a bit on CloudFormation, automation and Git. However, I still did not anticipate the exam to be this difficult until I was halfway through and already had the exam and materials paid for.\nWithout clear guideline on study material other than the white papers, I first skimmed through the ACloudGuru course, which helped me form a high level sense of exam coverage. Nonetheless it does not cover any topic in-depth and therefore by no means makes an essential part of my study. I checked out LinuxAcademy course and they are much more in-depth for the major topics indeed. I like the course material in Lucid chart. However, the LinuxAcademy course along does not cover everything you need to know.\nWhat I found extremely helpful is the free training videos from the official training website. I strongly recommend the 7-hour course Exam Readiness: AWS Certified DevOps Engineer – Professional. The instructor did a great job outlining the services and knowledge areas in the assessment . The course also comes with quality sample questions with answers and explanations on what the thinkings are behind the correct answers or why some choices are obviously wrong. I went through these questions twice and feel much better at not only understanding the question, but also understanding the intent of the question.\nApart from the Exam Readiness course, other free introductory courses from the official training website are helpful as well especially for those services that you only need to know the basics. Most of those courses are 5 ~ 10 minutes long, with brief but sufficient introduction and a demo session. The other extremely helpful resource is the official practice questions. The practice exam is harder than the actual exam but they closely resemble the actual question style in the exam. Unfortunately, no answer is provided but they made me spend time finding answers across the documentations and blogs. It is worth-noting that the AWS blogs provides plenty of use cases that are covered in the questions.\nWhen it comes to the real exam, it covers a lot more topics than its predecessor. Many questions are long and confusing. And I wish I could run a diff command to highlight the differences between choices. During exam preparation you really need to train yourself on reading efficiently. I found myself sometimes eyeball through all four choices at the same time, which get my mind scattered.\nDuring my study, I divide all services into three categories based on the level of familiarity, and here is my list:\nCategory 1. Know these services very well, in and out:\nAWS ElasticBeanstalk, OpsWorks Stacks, OpsWorks Chef Automate, CloudFormation, CloudWatch, CodeBuild, CodeCommit, CodeDeploy, CodePipeline, CodeStar, Lambda, API Gateway, Config, Trusted Advisor, CloudTrail, Systems Manager, Autoscaling Group in EC2, DynamoDB\nCategory 2. Know these services well, but not necessarily down to every single detail:\nAmazon Kinesis Firehose, Kinesis Analytics, Kinesis Streams, Step Functions, Elastic Load Balancer, Secrets Manager, Serverless Application Model (SAM), Route53, RDS, Certificate Manager, ElasticSearch, ECS, ECR\nCategory 3. Know about these services at a high level, but do not skip any:\nAWS Organization, X-Ray, GuardDuty, Macie, Inspector, Service Catalog, KMS, Batch, Athena, Single-Sign-On, Data LifeCycle Manager, CloudSearch, Health Dashboard, Glue, QuickSight, LightSail\nAlthough I did not mention much about the white papers, I want to highlight their importance again. I would not attempt the exam without reading and understanding the required white papers. If you aspire to take the AWS certified DevOps Engineer exam I hope this helps you a little bit. Good luck.\nPrevious PostStorage Nitty-Gritty 3 of 5 – NAS and Object Storage Next PostCryptography basics 2 of 2 ","date":"2019-08-17T23:07:59-04:00","permalink":"/2019/08/aws-certified-devops-engineer-exam-tips/","title":"AWS Certified DevOps Engineer Exam Tips"},{"content":"NAS (network attached storage) NAS server is dedicated to file-serving. NAS device runs its own specialized operating system that is optimized for file I/O, integrated hardware and software component that meets specific file-service needs, and performs file I/O better than a general-purpose server. NAS device can serve more clients than general-purpose servers and provide the benefit of server consolidation (centralized storage).\nNAS uses network and file-sharing protocols to provide access to the file data. These protocols include TCP/IP for data transfer, and Common Internet File System (CIFS) and Network File System (NFS) for network file service.\nNetwork File Sharing \u0026#8211; user who creates a file determines the type of access to be given to other user. When multiple users try to access a shared file at the same time, a locking scheme is required to maintain data integrity and, at the same time, make this sharing possible. Examples of file sharing method (FTP, DFS, NFS, CIFS, P2P)\nComponents of NAS \u0026#8211; NAS head (CPU, memory, NIC, optimized OS, ports, applications that supports CIFS/NFS) and Storage Array\nTypical NAS components NAS I/O operation:\nClient packages an I/O request into TCP/IP and forwards it through network stack. NAS head receives this request from network; NAS head converts the I/O request into an appropriate physical storage request, which is a block-level I/O, and then performs the operation on the physical storage; When NAS head receives data from the storage array, it processes and repackages the data into an appropriate NFS/CIFS response; NAS head packages this response into TCP/IP again and forwards it to the client through the network NAS I/O operation NAS implementation\nUnified NAS \u0026#8211;\u0026nbsp; consolidate NAS-based and SAN-based data access within a unified storage platform and provides a unified management interface for managing both the environments. Unified NAS connectivity Gateway implementation \u0026#8211; similar to unified NAS, the storage is shared with other applications that use block-level I/O. The gateway NAS is more scalable compared to unified NAS because NAS heads and storage arrays can be independently scaled up when required. For example, NAS heads can be added to scale up the NAS device performance.\nWhen the storage limit is reached, it can scale up, adding capacity on the SAN, independent of NAS heads. Similar to a unified NAS, a gateway NAS also enables high utilization of storage capacity by sharing it with the SAN environment.\nGateway NAS connectivity Scale-out NAS \u0026#8211; enables grouping multiple nodes together to construct a clustered NAS system. A scaled-out NAS provides the capability to scale its resources by simply adding nodes to a clustered NAS architecture. The cluster works as a single NAS device and is managed centrally. Scaled-out NAS creates a single file system that runs on all nodes in the cluster. All information is shared among nodes, so the entire file system is accessible by clients connecting to any node in the cluster. Scale-out NAS stripes data across all nodes in a cluster along with mirror or parity protection. As data is sent from clients to the cluster, the data is divided and allocated to different nodes in parallel. When a client sends a request to read a file, the scale-out NAS retrieves the appropriate blocks from multiple nodes, recombines the blocks into a file, and presents the file to the client. As nodes are added, the file system grows dynamically and data is evenly distributed to every node. Each node added to the cluster increases the aggregate storage, memory, CPU, and network capacity. Hence, cluster performance also increases.\nScale-out NAS use separate internal and external networks for back-end and front-end connectivity, respectively. The internal network offers high throughput and low-latency and uses high-speed networking technology, such as InfiniBand or Gigabit Ethernet.\nScale-out NAS with dual internal and single external networks NFS protocol \u0026#8211; originally based on UDP, uses RPC as a method of inter-process communication between two computers. NFS provides a set of RPCS to access remote file system for the following operations:\nSearching files and directories Opening, reading, writing to and closing a file Changing file attributes Modifying file links and directories NFSv3 and earlier is stateless protocol. Each call provides a full set of arguments to access files on the server. NFSv3 is most commonly used version, based on UDP or TCP.\nNFSv4 uses TCP and is based on stateful protocol design.\nCIFS \u0026#8211; a public, or open variation of SMB protocol. Filenames in CIFS are encoded using unicode characters. It is stateful protocol because the server maintain connection information regarding every connected client. If a network failure or CIFS server failure occurs, the client receives a disconnection notification. If application has embedded intelligence to restore the connection, then the storage solution is fault tolerant. If the embedded intelligence is missing, the user must take steps to reestablish the CIFS connection.\nNAS Performance \u0026#8211; network congestion is one of the most significant sources of latency in NAS environment. Other factors\nnumber of hops authentication with AD Retransmission \u0026#8211; speed and duplex settings on the network devices and NAS heads must match Over-utilized routers and switches File system lookup and metadata request \u0026#8211; deep directory structure could cause delay. Over-utilized NAS devices \u0026#8211; client accessing multiple files can cause high utilization levels on a NAS device Over-utilized clients \u0026#8211; if a client is busy itself, it requires a longer time to process the request and responses. NAS latency NFS server manages privilege and does not require username and password from the client at the time of mounting. CIFS share does require username and password.\nCommon network optimization practices for network contestion:\nA VLAN is a logical segment of a switched network or logical grouping of end devices connected to different physical networks. The segmentation or grouping can be done based on business functions, project teams, or applications. VLAN is a Layer 2 (data link layer) construct and works similar to a physical LAN. A network switch can be logically divided among multiple VLANs, enabling better utilization of the switch and reducing overall cost of deploying a network infrastructure.\u0026nbsp;\nThe broadcast traffic on one VLAN is not transmitted outside that VLAN, which substantially reduces the broadcast overhead, makes bandwidth available for applications, and reduces the network\u0026#8217;s vulnerability to broadcast storms.\nMTU setting determines the size of the largest packet that can be transmitted without data fragmentation. Path maximum transmission unit discovery is the process of discovering the maximum size of a packet that can be sent across a network without fragmentation. The default MTU setting for an Ethernet interface card is 1,500 bytes. A feature called jumbo frames sends, receives or transports Ethernet frames with an MTU of more than 1,500 bytes. The most common deployments of jumbo frames have an MTU of 9,000 bytes. However, not all vendors use the same MTU size for jumbo frames. Servers send and receive larger frames more efficiently than smaller ones in heavy network traffic conditions. Jumbo frames ensure increased efficiency because it takes fewer, larger frames to transfer the same amount of data. Larger packets also reduce the amount of raw network bandwidth being consumed for the same amount of payload. Larger frames also help to smooth sudden I/O burst.\nThe TCP window size is the maximum amount of data that can be sent at any time for a connection. For example, if a pair of hosts is talking over a TCP connection that has a TCP windows size of 64KB, the sender can send only 64KB of data and must then wait for an acknowledgement from the receiver. If the receiver acknowledges that all the data has been received, then the sender is free to send another 64 KB of data. If the sender receives an acknowledgment from the receiver that only the first 32 KB of data has been received, which can happen only if another 32 KB of data is in transit or was lost, the sender can send only another 32 KB of data because the transmission cannot have more than 64 KB of unacknowledged data outstanding.\nIn theory, the TCP window size should be set to the product of the available bandwidth of the network and the round-trip time of data sent over the network. For example, if a network has a bandwidth of 100 Mbps and the round-trip time is 5 milliseconds, the TCP window should be as follows:\n100 Mb/s x .005 seconds = 524,288 bits or 65,536 bytes\nThe size of the TCP window fi eld that controls the fl ow of data is between 2 bytes and 65,535 bytes\nLink aggregation is the process of combining two or more network interfaces into a logical network interface, enabling higher throughput, load sharing or load balancing, transparent path failover, and scalability. Due to link aggregation, multiple active Ethernet connections to the same switch appear as one link. If a connection or a port in the aggregation is lost, then all the network traffic on that link is redistributed across the remaining active connections.\nFile-level virtualization\nFile-level virtualization eliminates the dependencies between the data accessed at the file level and the location where the files are physically stored. Implementation of file-level virtualization is common in NAS or file-server environments. It provides non-disruptive file mobility to optimize storage utilization.\nIt provides user or application independence from the location where the files are stored. File-level virtualization creates a logical pool of storage, enabling users to use a logical path, rather than a physical path, to access files. While the files are being moved, clients can access their files non-disruptively. Clients can also read their files from the old location and write them back to the new location without realizing that the physical location has changed. A global namespace is used to map the logical path of a file to the physical path names.\nFile-serving environment before and after file-level virtualization Object-based storage In NAS, metadata are stored as part of the file distributed throughout the environment, which adds to the complexity and latency in searching and retrieving files. Object-based storage, on the other hand, stores file data in the form of objects based on its content and other attributes, rather than the name and location.\nHierarchical File System and Flat Address Space OSD \u0026#8211; object-based storage devices, stores data in the form of objects using flat address space. There is no hierarchy of directories and file. Object is identified by objectID, which is usually generated using hash function.\nIn block storage, when file system receives the IO from an application, the file system maps the incoming I/O to the disk blocks. The block interface is used for sending the I/O over the channel or network to the storage device. The I/O is then written to the block allocated on the disk drive. When an application accesses data stored in OSD, the request is sent to the file system user component. The file system user component communicates to the OSD interface, which in turn sends the request to the storage device. The storage device has the OSD storage component responsible for managing the access to the object on a storage device.\nBenefit of object storage\nsecurity and reliability: OSD can use special algorithm for strong encryption capacity. Request authentication is performed at the storage device rather than with an external authentication mechanism platform independence: standard web access via REST or SOAP scalability: Both storage and OSD nodes can be scaled independently in terms of performance and capacity Block-level access vs object-level access OSD components:\nnodes: a server with OSD operating environment to provide services to store, retrieve and manage data. Two key services are metadata service (generating objectID and maintaining the mapping between objectID and file) and storage service (manage a set of disks where data are stored). private network: provides node-to-node connectivity and node-to-storage connectivity. storage device OSD system components Storage mechanism\nThe application server presents the file to be stored to the OSD node. The OSD node divides the file into two parts: user data and metadata. The OSD node generates the object ID using a specialized algorithm. The algorithm is executed against the contents of the user data to derive an ID unique to this data. For future access, the OSD node stores the metadata and object ID using the metadata service. The OSD node stores the user data (objects) in the storage device using the storage service. An acknowledgment is sent to the application server stating that the object is stored. OSD: object storage Retrieval mechanism\nThe application server sends a read request to the OSD system. The metadata service retrieves the object ID for the requested file. The metadata service sends the object ID to the application server. The application server sends the object ID to the OSD storage service for object retrieval. The OSD storage service retrieves the object from the storage device. The OSD storage service sends the file to the application server. OSD object retrieval OSD usage: data archival, especially long-term; and cloud storage, storage as service\nCAS \u0026#8211; content addressed storage, a special type of OSD designed for secure online storage and retrieval of fixed content. Data access in CAS differs from other OSD devices. In CAS, the application server access the CAS device only via the CAS API running on the application server. However, the way CAS stores data is similar to the other OSD systems.\nCAS Use case Healthcare: storing patient studies \u0026#8211; size of radiology study ranges from 15MB to more than 1GB. Newly acquired studies are retained for 60 days and moved to long term storage. Finance: storing financial records \u0026#8211; bank stores images of cheques (~25KB each) for about 90 millions a month. Images are processed in transaction system for 5 days. For the next 60 days images are requested for verifications. After 60 days access requirements drop drastically. Retention policy manages life-cycle of the images. Unified storage\nComponents\nstorage controller: The storage controller provides block-level access to application servers through iSCSI, FC, or FCoE protocols. NAS head: a dedicated file server that provides file access to NAS clients OSD node: accesses the storage through the storage controller using a FC or FCoE connection. Storage Unified storage platform Related Postings Disk and RAID SAN Backup and Archive Solutions Replication Previous PostCryptography Basics 1 of 2 Next PostAWS Certified DevOps Engineer Exam Tips ","date":"2019-07-13T23:31:00-04:00","permalink":"/2019/07/storage-nitty-gritty-3-of-5-nas-and-object-storage/","title":"Storage Nitty-Gritty 3 of 5 – NAS and Object Storage"},{"content":"I have been dabbling with OpenSSL commands to achieve what I needed during IT implementation, but I decided to spent some time to overcome the conceptual hurdles around cryptography. In this domain, following other people\u0026#8217;s instructions through the project does not produce much learning value when too many concepts cloud around. Let\u0026#8217;s take the bull by the horn.\nThis article is purely conceptual. There are already lots of step-by-step guideline about acquiring a website certificate. The intention is to elucidate the core concepts on IT cryptography, and then connect the dots to form the big picture in cryptography.\nFirst, let\u0026#8217;s distinguish three basic concepts: encoding, hashing and encryption. They are in essence all mathematical functions, but one does not need to understand the underlying algorithm in order to understand what they are.\nEncoding \u0026#8211; Transform data into a format so it is readable by external system. Encoding is about interoperability. It is not about security whatsoever. Example: ASCII, BASE64, UNICODE.\nHashing \u0026#8211; Mathematic algorithms to generate digest of content. A digest is usually fixed-size, non-reversible and deterministic. You cannot restore content from its digest (non-reversible, one-way calculation). Two different contents results guarantees different digests (deterministic). Digest as a result of hashing is mostly about data integrity. For example, in file download you can calculate MD5 hash and compare the result against the digest given by the source. Another common use, is to hash all password in database. In that sense, hashing has to do with security. Popular algorithms are MD5 and SHA-256.\nEncryption \u0026#8211; Mathematic algorithms that only succeeds if correct parameter (key) is provided. The function is deterministic and reversible (two-way). The parameter (key) used for calculation (encryption) and reverse calculation (decryption) can either be the same or different. Cryptography involves all three concepts, but it mainly addresses issues around encryption. We break down encryption into two categories, symmetric key encryption (aka private key encryption) and asymmetric key encryption (aka public key encryption).\nSymmetric Key Encryption: using the same key (shared secret) for encryption and decryption. Both parties need to keep it secret. Popular algorithm is AES.\nAsymmetric Key Encryption: involves a pair of public key and private key. Public key is given out to external systems. Private key is kept secret. Popular algorithm is RSA. You can run some experiments on this page, and I\u0026#8217;ve made some additional notes (all based on RSA algorithm):\nPublic Key and Private Key are NOT interchangeable. The size of an RSA private key is usually much larger than its public key;You can encrypt with either key, and decrypt with the other, so long as you specify the key type at the time of encryption or decryption;You can generate public key from private key (ssh-keygen -y); but not the other way round;Using a wrong key to encrypt or decrypt leads to failure, instead of wrong result. Obviously symmetric key encryption is the original form of encryption and requires both parties to keep the key secret, permanently. This is not realistic in real life between organizations. This challenge leads to the adoption of asymmetric key where the public key can be published, for external party to encrypt outgoing messages, whereas the private key is kept secret within the owner, only to decrypt incoming messages. Knowing this distinction, we can introduce two concepts that are built on top of asymmetric key encryption.\nDigital Signature: if an entity signs a document digitally. The digital signature is the digest of the document (hash of the content) encrypted with the signer\u0026#8217;s private key.\nDigital Certificate: contains owner\u0026#8217;s public key and digital signature of issuer (digest of owner\u0026#8217;s public key encrypted by issuer\u0026#8217;s private key). Client (e.g. browser with CA\u0026#8217;s public key preloaded) should not trust the owner\u0026#8217;s public key until it compares its digest against decryption result of digital signature (using CA\u0026#8217;s public key).\u0026nbsp;\nThe key difference here is that a digital certificate involves third party. Digital signature by itself cannot address impersonation, which is addressed by digital certificate issued by third party (certified authority). This requires digital certificate must follow some standard, and the most prevalent one is:\nX.509: a standard format for digital certificates. It contains:\na public keyan identity (a hostname, or\u0026nbsp; an organization, or an individual)a signature (either signed by CA or self-signed) Since the advent of certificate, there are tons of global organizations that need to manage keys and certificates. When we create a secure connection now, we only need to get a certificate from an intermediate CA, thanks to existing certificate chain. It involves lots of work for large organizations to maintain keys and certificates, which requires:\nPublic Key Infrastructure (PKI): the IT infrastructure to create, manage, distribute, use, store and revoke digital certificates and public keys.\u0026nbsp;\nOne of the important technique to manage keys and certificates is the encoding. Keys and certificates are usually wrapped with different encoding formats based on what need to be done on them. Here is a summary of encoding formats\nFormatEncodingwhat is storedPossible SuffixPEMDER file (binary content) encoded in Base64. Certificate files typically include clear text statement \u0026#8220;BEGIN CERTIFICATE\u0026#8221; and \u0026#8220;END CERTIFICATE\u0026#8221;single certificate, certificate chains or private keys.pem, .crt, .cer, .keyDERBinary format in early days; it\u0026#8217;s complex and don\u0026#8217;t use unless with a specific purpose.single certificate, certificate chains, or private keys.der, .cerPKCS#7Base64 encoded ASCII file, typically include clear text statement \u0026#8220;BEGIN PKCS7\u0026#8221; and \u0026#8220;END PKCS7\u0026#8221;only certificates or certificate chains; no private keys.p7b .p7sPKCS#8Similar to PEM as base64 encoded format but for storing private key only, can be password protectedPrivate key.keyPKCS#12a binary format, heavily used by Microsoft productscertificate, certificate chains or private keys; public private key pair.pfx .p12OpenSSHused by OpenSSH to store public keys (as specified in RFC4253).pub Note: above are just encoding formats. Just by file extension you cannot tell whether the file is a key or a X.509 certificate. When you are configuring certificates, you may come across the following file extensions as well:\n.key this extension can indicate any kind of key, but usually it is a private key (used along with .crt file) .csr certificate signing request, including a public key and an identity required by CA. CA needs this file to issue a certificate. CSR could be encoded in Base-64 or DER\n.cer or .crt a certificate, usually in X.509 v3 (public key + identity + signature), the encoding could be PEM or DER. .jks -\u0026gt; java key store file type. It can be either a key store (private key along with certificate) or trust store (certificate) for Java application. Refer to the section for Java applications.\nIn the next article, we will examine some use case involving the concepts introduced in this post.\nPrevious PostGit Explained 2 of 2 Next PostStorage Nitty-Gritty 3 of 5 – NAS and Object Storage ","date":"2019-07-10T20:07:18-04:00","permalink":"/2019/07/practical-cryptography-for-it-professional/","title":"Cryptography Basics 1 of 2"},{"content":"This is a continuation from Git Explained 1 of 2 where the fundamental concepts are covered. In this article we introduce some tools for customization and maintenance.\nAs for Git configuration, there are two files to dictate your Git configuration. ~/.gitconfig and .git/config in project directory. Running `git config \u0026#8211;list \u0026#8211;show-origin` shows all config entries and where they are from. For example you can custom your Git hooks location. Neither of the two files are being version controlled, so the configuration is only effective in the client environment,\nServer side Git hooks Git implementation supports server side hooks (pre-receive, update, post-receive). They are bash scripts placed in .git/hooks with specific names, fired upon event occurrence. Exit code of 1 from the scripts fails the event. Since server side Git hook consumes server resources, many repository hosting vendors (e.g. BitBucket Cloud) do not support it. You will need to enable it in self-hosted servers (e.g. BitBucketServer).\nClient side Git hooks Since server side hooks are not widely supported in every vendor, client-side Git hooks is good alternative places to implement functions such as code style check, commit size check, etc\nThe default directory for hooks is .git/hooks/ under the project directory is not version controlled and not easy to share with the team. If the hook needs to be shared among project contributors, we can place hooks files in .githooks/ under project directory. This will make the hook files version controlled. In addition, we need to point the hooks to this directory in configuration, by running `git config core.hooksPath .githooks` from project directory.\nWeb hooks Web hooks can be thought of as an event notification mechanism. It is a common feature provided by VCS repository hosting providers. If a certain type of event occurs to the repo, web hook will fire an RESTful API call. The HTTP Endpoint, authentication secret and event payload are pre-configured in the repo settings. Web hooks are commonly supported by Git-based repo implementation, such as BitBucket cloud, GitHub, GitLab or AWS CodeCommit. It can also be enabled in self hosted Git repo. Web hook is a powerful tool to drive downstream event, such as Jenkins to start building the code. The major difference between server side hook and web hook is web hook is RESTful API driven, whereas server side hook is executing a script.\nPipelines Some repo hosting vendor also provide a feature named pipeline. BitBucket has Pipeline as a CI/CD tool, AWS has AWS Code Pipeline, and GitLab offers CI/CD pipeline as well. These pipelines are usually in the form of a YAML file in the repo with a special name. The YAML spell out the steps to perform along the pipeline.\nSquash Commits For small projects I was in the habit of committing to main branch. I often need to squash a number of commits into one to \u0026#8220;clean up\u0026#8221;. Usually a Pull Request (e.g. in GitHub) or Merge Request (e.g. in GitLab) have such option during approval. We can squash a few commits with git command as well (suppose we want to squash the most recent 16 commits):\ngit rebase -i HEAD~16 git push origin +main The command will open text editor to allow you to mark what to do with each commit. You can mark all except one commit as squash. For the commit to keep, mark it as pick. Then save the text editor. Git rebase will perform the squash for you. However, since this is a rebase, do not do this if there are other collaborators working on the same branch.\nCleanse a repository Take BitBucket cloud for example, the size of a remote repo has a non-expandable hard limit of 2GB, and a soft limit of 1GB. Once the soft limit is reached, a warning will be displayed on Bitbucket\u0026#8217;s landing page as well as when contributors pushes changes. Once the hard limit is reached, the entire repo will turn read-only mode.\nSpace consumption can be caused by accidental committing of large file. As covered in the previous post, files are stored as blob objects in .git directory. If a file was deleted by `git rm` command, it simply means it is de-referenced from the next commit and on. After all, Git as a distributed version control system, has the ability to magically restore the deleted file when we want. The cost of that magic, is that deleted file is permanently stored in the repo, in the form of blob object, although not present in the working directory. It consumes space not only in remote repo, but also in the local repo of each contributor.\nThe purpose of Git repository is to store source code which are fairly small texts. However if a contributor pushed in large files, it can be tricky to cleanse a Git repo on the remote side. Here is some guidelines:\nBefore cleansing, identify large unwanted files in current working directory, delete them with `git rm` and then commit this change in master branch. Consolidate branches (e.g. delete useless remote branches with `git push origin \u0026#8211;delete branch_name`). This step itself does not free up space in remote repo but it simplifies the branches; Remove large blob objects and commit objects that reference them. This step essentially is re-writing the commit history of repo. Given the risk, it is recommended to perform this step from a separate local project directory with bare repo only without working directory. This article presents some good advices and illustrations. The command recommended is `git filter-branch -f \u0026#8211;tree-filter \u0026#8220;rm -rf \\large_file.zip\u0026#8221; \u0026#8211;prune-empty \u0026#8212; \u0026#8211;all` ; and if that is too slow here is an alternative named BFG Repo-Cleaner. After this step, each commit will have a new hash id. For the repo size to reduce, either wait until the next garbage collection on the server side, or request cloud vendor to run a manual garbage collection. Previous PostGit Explained 1 of 2 Next PostCryptography Basics 1 of 2 ","date":"2019-06-18T17:06:00-04:00","permalink":"/2019/06/git-explained-2-of-2/","title":"Git Explained 2 of 2"},{"content":"In a nutshell, Git is a distributed version control system, commonly used as source control management. It places files in one of three logical areas (working dir, staging, repo) below depending where it is in its lifecycle. There are many cheetsheets out there but this article will just sort through some concepts unique to Git. To understand how Git works it is crucial to think in terms of Git data model.\nWorking directory a single checkout of one version of the project. These files are pulled out of the object database in the Git directory (upon checkout) and placed in the project directory on disk, for you to use or modify;\u0026nbsp;\nIndex a file contained in your Git directory (stored as binary data in file .git/index) that keeps information about what will go into your next commit.\u0026nbsp;To display what\u0026#8217;s in the index, run `git ls-files \u0026#8211;stage`. Read this post for further details\nRepository: where Git stores the metadata and object database for your project.\u0026nbsp;The local repository is in .git/ under the project directory.\nAdd \u0026#8211; register one or more modified files to staging area. You may edit several files with only a few needed registered for future commit. Add activity ensures the file edited are recorded in the index (as a preview of next commit). You technically need to run add against each file. But the command syntax with * or . allows you to capture all edits in the same directory or under.\nCommit \u0026#8211; persist the staged file edits to the repository (so they are stored in Git object database). A commit represents all the file edits that were staged by add command in previous steps.\nBranch \u0026#8211; a branch is simply a movable pointer to a commit. Default branch name created by git init is called \u0026#8220;master\u0026#8221;. Other than the name, there is nothing special about master branch. Everytime you commit, the master branch pointer moves forward automatically. Branch pointers are kept in .git/refs directory. Read this post for further details.\nHEAD \u0026#8211; the pointer to the current branch reference, which is in turn a pointer to the last commit made on that branch. Git use HEAD pointer to know what branch you\u0026#8217;re currently on. HEAD will be the parent of the upcoming commit.\nTag \u0026#8211; an annotated tag contains the SHA of the commit being tagged. Alias of a commit.\nMerge \u0026#8211; choose current commit of other branch and apply it onto your branch.\nRebase \u0026#8211; copy all commits from other branch to your branch. Compared to merge, rebasing forms a cleaner commit history.\nCherrypick \u0026#8211; choose a previous commit from other branch and apply it onto your branch.\nStash \u0026#8211; \u0026nbsp;temporarily stashes changes you\u0026#8217;ve made to working tree so you can work on something else, and then come back and re-apply them later on.\nReset \u0026#8211; at a high level, reset is to revert some operations. After pulling code, developer usually follow three steps: editing-\u0026gt;add-\u0026gt;commit. reset is to reverse these steps, based on different modes. The Pro Git reference has further details on the three different modes:\nsoft mode (reverse operation of commit): based on what branch HEAD points to, move where that branch points to (e.g. from latest commit, to a different commit several steps up the link); mixed mode (default; reverse operation of commit and add) \u0026#8211; in addition to soft mode, also update index; hard mode (reverse operation of commit, add and file editing)- in addition to mixed mode, also update working directory. Edit on files are discarded. Git object model \u0026#8211; In Git database, files, commits and directories are stored as objects, In Git object model, there are three types (to tell object type, run `git cat-file -t`). Read this post for further details:\nblob object \u0026#8211; stores file data with metadata; use `git show` to examine blob object; tree object \u0026#8211; represents a directory. It references other tree objects (sub-directories) or blob objects (files under the directory, of a certain version); use `git ls-tree` to examine tree object; commit object \u0026#8211; represents a commit. It references its parent commit, as well as a tree object that represents the entire project directory. use `git cat-file -p` to inspect commit object; This diagram from from Git Pro outlines the interactions amongst these types of objects.\nFor more details, the\u0026nbsp;official documentation\u0026nbsp;is actually the most helpful reference with illustrations. In addition, I find on Hakcermoon three excellent articles with thorough explanation on\u0026nbsp;data model,\u0026nbsp;branching\u0026nbsp;and\u0026nbsp;index.\nPrevious PostSet up automation with Ansible Next PostGit Explained 2 of 2 ","date":"2019-06-08T22:39:00-04:00","permalink":"/2019/06/git-explained-1-of-2/","title":"Git Explained 1 of 2"},{"content":"Ansible is very flexible automation tools with many benefits. The free version is command-line based and here is an example to set it up.\nEnvironment setup\nAnsible 2.8 is required or some command may not work.Ansible files (including playbooks, tasks and inventory files) are all located in /home/glowing/ansibleDefault inventory file needs to be referenced in Ansible configuration /etc/ansible/ansible.cfg. This ensures ansible or ansible-playbook command can pick up hosts or host patterns without requiring inventory file through -i on every execution. Here is what the inventory config looks like in ansible.cfg: [defaults] # some basic default values... inventory = /etc/ansible/hosts,/home/glowing/ansible/inventories/glowing_inventory.yml host_key_checking = False As best practice, servers involved should be able to ssh to each other on RSA key authentication. This can be achieved by adding a separate authorized keys file and reference it from /etc/ssh/sshd_config, at the line starting with AuthorizedKeysFile, and separated with the file name of existing authorized keys with a space. In this way, you may keep public keys of human user in one authorized key file and the public keys of machines in the other.Build inventory file in ~/ansible/inventories/glowing_inventory.yml. The inventory file can declare some variables to use across hosts. If password is involved (e.g. synchronize module still requires password), it can be stored here with base64 encoded. Here is an example of glowing_inventory.yml --- all: vars: ansible_user: glowing gh_sudo_pass: qGS0bWVuu3Jr gh_dir: /opt/glowing/etc tmp_dir: /tmp children: dc1_front_end: hosts: e9a-ghfe01: e9a-ghfe03: e9a-ghfe05: e9a-ghfe07: e9a-ghfe09: dc2_front_end: hosts: e9a-ghfe02: e9a-ghfe04: e9a-ghfe06: e9a-ghfe08: e9a-ghfe10: dc1_back_end: hosts: e9a-ghbe01: e9a-ghbe03: e9a-ghbe05: dc2_back_end: hosts: e9a-ghbe02: e9a-ghbe04: e9a-ghbe06: dc1_database: hosts: e9a-ghdb01: e9a-ghdb03: e9a-ghdb05: dc2_database: hosts: e9a-ghdb02: e9a-ghdb04: e9a-ghdb06: That is the basic steps to set up Ansible. Now we can run adhoc commands. The command below allows me to copy a file from executing server to all destination servers that match a pattern:\n# ansible \u0026#39;dc*_back_end:!\u0026#39;`hostname -s` -e \u0026#34;file_name={{gh_dir}}/test.zip\u0026#34; -m copy -a \u0026#34;src={{file_name}} dest={{file_name}}\u0026#34; In this command:\nvariable {{gh_dir}} is declared in the inventory file. It must be referenced by placing variable name between double curly bracket;An extra variable {{file_name}} is declared at run time because this is the dynamic part of the command;This adhoc ansible command uses copy module. With copy module, the src anddest files are in the same absolute path here so we use this variable to save some typing;reference to the host support wildcard such as dc*_back_end;hostname -s returns the host name of the server where the adhoc command is run:! excluds the running host from being matched as destination server. this is in case that running machine is already in the dc*_back_end group, where copy source and destination are identical Here is another example for deleting a file from destination servers:\n# ansible dc2_database -e \u0026#34;fn=/tmp/file_to_delete\u0026#34; -m file -a \u0026#34;path={{fn}} state=absent\u0026#34; If we can run adhoc command, then we can start writing some playbook and roles. Below is a simple playbook test-conn.yml to ping each server:\n--- - name: measure mint retrieval time from {{ ansible_limit }} hosts: \u0026#39;{{ ansible_limit }}\u0026#39; serial: 1 order: sorted gather_facts: no ignore_errors: yes tasks: - name: measure time command: curl -s -XGET http://{{ inventory_hostname }}:8080/index.html -o /dev/null delegate_to: localhost register: curlout no_log: true - name: display result debug: msg: \u0026#34;Time to load webpage from {{ inventory_hostname }} is {{ curlout.delta }}.\u0026#34; Then you can run that playbook with the following command:\n# ansible-playbook -l dc2_front_end test-conn.yml Ansible automation is essentially editing yaml files for playbooks. Writing playbook involves a lot of module interaction and one needs to follow best practices. Understanding Ansible roles can help reuse some code.\nIn addition here are some common playbook keywords you should be familiar with:\nserialordergather_factsignore_errorswhenrun_oncelocal_actionregister Here are common modules to know\nset_factstatetouchfailsynchoronizelineinfile Previous PostStorage Nitty-Gritty 2 of 5 – SAN Next PostGit Explained 1 of 2 ","date":"2019-05-22T22:11:00-04:00","permalink":"/2019/05/automation-with-ansible-a-primer/","title":"Set up automation with Ansible"},{"content":"In direct attached storage (DAS), storage is server centric and the host owns the storage. The storage is fully dedicated to the server that owns it.\nWith DAS, storage is server-centric Storage has evolved into information centric model. In this model, when a new server is deployed in the environment, storage is assigned from the same shared storage pool to the new server.\nNetwork based centralized storage solution A network-based storage solution is the centralized storage pool. No single host owns the entire storage pool. The storage solution consists of two categories based on the interface with the host: SAN (storage area network) and NAS (network attached storage). To a client OS on the host, SAN typically appears as a local disk, allowing block-level access from the client OS, and therefore is more suited for structured workload such as database storage. It operates on its own storage network independent of the host network. NAS on the other hand, typically appears as a file share to the client OS, identified by an IP address and path. This is because NAS operates on the same TCP/IP network where the hosts are operated on. The client has file level access to storage, therefore NAS is better for unstructured data such as video and medical images. It is very important to understand the difference between SAN and NAS.\nSAN deployment consists of two categories based on the connection technology. FC SAN is based on Fibre Channel network; and IP SAN is based on Internet protocol (iSCSI, FCIP, FCoE).\nFibre Channel SAN Cable types: MMF (multimode fibre, usually for short distance within data centre because of signal attenuation due to modal dispersion) and SMF (single mode fibre, carries a single ray of light, used for long-distance cable runs; Connector: standard connector (SC), lucent connector (LC) and straight tip connector (ST); Interconnect device FC hub \u0026#8211; for FC-AL implementation, but no longer in use FC switch \u0026#8211; directly route data from one physical port to another (more intelligent than hub) Director \u0026#8211; high end switches with redundant components to provide high availability FC connectivity point-to-point: two devices connected directly to each other; arbitrated loop (FC-AL): devices are attached to a shared loop; FC- AL (rarely used today) switched fabric (FC-SW): uses switches that can switch data traffic between nodes directly through switch ports. Frames are routed between source and destination by the fabric Fibre Channel Switched Fabric Protocol: Fibre Channel Protocol (FCP): defines protocol stack (five layers, FC-0 through FC-4), addressing, identification (world wide name), frame, data structure, flow control, fabric services Fibre Channel Protocol stack FC frame Topology: mesh topology and core-edge fabric topology Block-level virtualization: aggregates block storage devices (LUNs) and enables provisioning of virtual storage volumes, independent of underlying physical storage. The virtualization layer maps the virtual volumes to the LUNs on the individual arrays. Block-level storage virtualization not only enables extending the storage volumes online; it consolidates heterogeneous storage arrays and enables transparent volume access. It also provides the advantage of non-disruptive data migration, where the virtualization layer handles the back-end migration of data, which enables the LUNs to remain online during migration. Block-level virtualization (classic) Federation of block storage across data centers (new generation) Virtual SAN (VSAN, aka virtual fabric) \u0026#8211; a logical fabric on an FC SAN, which enables communication among a group of nodes regardless of physical location in the fabric. IP SAN iSCI (one of the IP SAN protocols) \u0026#8211; an IP based protocol that establishes and manages connections between host and storage over IP. iSCSI encapsulates SCSI commands and data into an IP packet and transport them using TCP/IP. It is relatively inexpensive and easy to implement so widespread in environments without FC SAN.\nTopology Native connectivity (without FC components) Bridged connectivity (including FC components in the configuration) Combined connectivity (most common because a storage array usually comes with both FC and iSCSI ports) iSCSI topologies Protocol stack: SCSI is the command protocol that works at the application layer of OSI model; iSCSI is session-layer protocol that initiates a reliable session between devices that recognize SCSI commands and TCP/IP. The iSCSI session-layer interface is responsible for handling login, authentication, target discovery, and session management. iSCSI protocol stack iSCSI session and PDU encapsulation PDU encapsulation iSCSI discovery \u0026#8211; an initiator must discover the location of its targets on the network and the names of the targets available to it before session establishment. Two types are SendTargets discovery and internet Storage Name Service iSCSI names: IQN, iSCSI Qualified Name such as iqn.2008-02.com.example:optional_string;\u0026nbsp; EUI, extended unique identifier such as eui.0300732A32598D26 iSCSI command sequencing \u0026#8211; A command sequence may generate multiple PDUs. A command sequence number (CmdSN) within an iSCSI session is used for numbering all initiator-to-target command PDUs belonging to the session. This number ensures that every command is delivered in the same order in which it is transmitted, regardless of the TCP connection that carries the command in the session. FCIP (one of the IP SAN protocols) \u0026#8211; transporting FC block data over the IP infrastructure.\nprotocol stack and packet encapsulation FCIP protocol stack FCIP encapsulation Topology (FCIP gateway involved): FCIP topology FCoE (one of the IP SAN protocols) \u0026#8211; consolidation of LAN and SAN traffic over a single physical interface infrastructure. FCoE helps organizations address the challenges of having multiple discrete network infrastructures.\nCNA (converged network adapters) replaces both HBAs and NICs in the server and consolidates both the IP and FC traffic Special requirement on cables and switches protocol stack and encapsulation FCoE field mapping CEE (converged enhanced Ethernet, or lossless Ethernet) provides new specification to existing Ethernet standard that eliminates the lossy nature of Ethernet. This makes 10Gb Ethernet a viable storage networking option, similar to FC. It features the following functionalities as part of IEEE 802.1: PFC (priority-based flow control) ETS (enhanced transmission selection) CN (congestion notification) Related Postings Disk and RAID NAS and Object Storage Backup and Archive Solution Replication Previous PostApplication I/O Characteristics Next PostSet up automation with Ansible ","date":"2019-05-06T22:06:02-04:00","permalink":"/2019/05/storage-nitty-gritty-2-5/","title":"Storage Nitty-Gritty 2 of 5 – SAN"},{"content":"There are many unknown factors and randomness when a solution architect determines storage requirement for an application. However, this process should still be as scientific as it can be and here are some of the important considerations regards application I/O characteristics.\nRandom and Sequential Random I/OSequential I/ODescriptionSuccessive read/write operations from noncontiguous addresses \u0026#8211; accesses that are spread across the addressable capacity of the LUN.Successive read/write operations from contiguous addresses: one logical block address after another. In sequential I/O access, disk seek time is reduced because the read/write head moves little to access the next block.ExampleMessaging\nOLTP applicationData Backup Reads and Writes Another aspect of the I/O workload is the ratio of read I/Os to write I/Os generated by application. The sum of the read and write rate is the I/O rate (number of I/O operations per second). The application\u0026#8217;s I/O rate is one of the important factors that determine the minimum number of disks required for application. In storage systems, cache plays an important role to improve the system performance. The table below summarizes how read I/O and write I/O interact with cache.\nI/O TypeREADWRITERandomHard to effectively cache because of difficulty in predicting prefetch;\nRequires multiple fast disks for good performanceCaching is effective, resulting in a response time better than disk response time.SequentialCaching is extremely effective due to predictability of prefetch;\nReads are done at cache speeds;Caching is effective; cache is flushed quickly because entire disk stripe can be written. Here are some typical read versus write ratio for common business applications:\nOLTP: 67% reads and 33% writesDSS (decision support, aka data warehouse or business intelligence): 80%~90% reads to data tables including frequent table scans (sequential reads)Backup: As long as the file system is not fragmented, file-based backups are sequential I/O Request Size The size of I/O generated by an application may vary depending upon the type of the application. Some of the overhead to execute an I/O is fi xed. If data exists in large chunks, it is more effi cient to transmit larger blocks because a host can move data faster by using larger I/Os than smaller I/Os. The response time of each large transaction is longer than the response time for a single small transaction, but the combined service time of many smaller transactions is greater than a single transaction that contains the same amount of data.\nApplicationSeek TypeI/O Request SizeProportion of I/O as writesMicrosoft ExchangeRandom32KBModerate to highSAP/Oracle ApplicationsRandom~8KBDepends on applicationRDBMS: Data entry/OLTPRandomDatabase or file system page size Moderate to high RDBMS: Online transaction logsSequential512 byte+High, except for archiving processRDBMS: Temp SpaceRandomDatabase or file system page sizeVery highWeb file server75% random, 25% sequential4KB, 8KB, 64KB95% read, 5% writeWeb server log100% sequential8KB100% writeMedia Streaming100% sequential64KB98% write; 2% readOS paging100% sequential64KB98% write; 2% read Previous PostLightsail – create a WordPress site in one hour Next PostStorage Nitty-Gritty 2 of 5 – SAN ","date":"2019-04-20T22:09:00-04:00","permalink":"/2019/04/application-i-o-characteristics/","title":"Application I/O Characteristics"},{"content":"My challenge with my just revived wordpress.com site is the plug-ins. Even paid premium users cannot install plug-ins for diagram, etc. I don\u0026#8217;t want to upgrade to business plan so I decided to build my own.\n14 years ago in university, I prototyped an intranet site using the Windows, Apache, MySQL, and PHP. It took a few weeks. Nowadays, this is referred to as WAMP stack with WordPress. AWS has a post on the best practices for hosting WordPress. However, I just need a single all-in-one server and there are some instruction on that. With Amazon Lightsail it was made a lot easier. The annual cost to host a WordPress site would be $55 in US, given that virtually nobody visits it except myself. The cost consists of:\nDomain registration for $13 a year Lightsail blueprint for $3.5 a month There are several benefit of using this Lightsail blueprint, such as:\nYou can actually SSH into the EC2 instance. This is important to myself in the habit of probing into how things work; Resources are all packaged into a fixed price plan. Remember the pesky accidental AWS charges? You have the whole LAMP stack pre-installed, with the bitnami image for WordPress; Start small but with scalability. To start with Lightsail is extremely intuitive. Just click and launch:\nThere is a good instruction here on YouTube. Once launched successfully, there are some post configurations:\nRequest a static IP and assign it to the EC2 instance; Register a domain (within Lightsail, with Route53 or any other domain registrar) and create an A record referencing the IP address; Create a certificate and set up anto-renewal. Here is an instruction but the steps differ if DNS is managed outside of Lightsail; Redirect http to https. Refer to the instruction from bitnami; Export from wordpress.com and import the xml into this little site. Notice that all services relevant to hosting are packaged into a single service called bitnami. In order to restart service, instead of restarting php, httpd and mysql individually, you can run the following:\n# service bitnami stop\n# service bitnami start\nIn fact, in /etc/rc5.d or /etc/init.d, I do not find the individual services for httpd or mysql. There is actually an instance of MySQL database in the Instance:\n# mysql -u root -p\nmysql \u0026gt; show databases;\nThere you go! Now the site is live. This little instruction will come in handy for rebuilding when this site is blown away 🙂\nMigrate to new LightSail instance I have been using the same LightSail instance for 3.5 years and it\u0026#8217;s been stable. By Dec 2022, the version of PHP (7.2.17) has been outdated and I decided to launch a new LightSail instance based on new version of image, which comes with newer version of PHP (8.1.13). Here is a note of my steps:\nCreate a new LightSail instance with the same SSH key, assign a new static IP to it. As soon as the new instance is created, we can access it by IP on HTTP, using the initial credential provided. We need the latest version of All-in-One WP Migration plugin to perform the migration. Ensure its latest version is installed on both old and new instance. Export site to file using All-in-One WP Migration plugin. It should produce a single file with wpress extension. For this site, the site is 450MB by Dec 2022 For the new site, try to upload the file and notice that the max size allowed is 80MB. I made the following changes: Change PHP attributes. Edit /opt/bitnami/php/etc/php.ini at the following attributes: post_max_size = 512M upload_max_filesize = 512M Change WordPress attributes. Create /opt/bitnami/wordpress/.htaccess with the following attributes: php_value upload_max_filesize 512M php_value post_max_size 512M php_value memory_limit 512M php_value max_execution_time 0 php_value max_input_time 300 Restart services related to wordpress Now upload the .wpress file and it should take less than 5 minutes to upgrade. After the process, the new site is imported. Delete useless plugins. Port the certificate files, including keys, certificate and certificate chains. In my case, I had to edit the following files: /opt/bitnami/apache/conf/vhosts/wordpress-https-vhost.conf /opt/bitnami/apache/conf/vhosts/wordpress-vhost.conf /opt/bitnami/apache/conf/bitnami/bitnami.conf /opt/bitnami/apache/conf/bitnami/bitnami-ssl.conf Configure HTTP-\u0026gt;HTTPS redirect on the new server. Revisit the files above. Change DNS record to point to the new server. Browse the site, check page and posts and pay attention to: images, tables, code blocks. One common issue is images do not display. Check the image URL. The URL might have been replaced by bad values during import. Use a plugin (e.g. Go Live Update Urls) to correct the URLs in tables. Use a plugin to find out broken links such as image, etc and fix the broken links. This should be an ongoing maintenance activity but is particularly worth doing on newly migrated site. Other little things There are many little things to fix. For example, I need to give transparent images white background. This is because when the site\u0026#8217;s background is black and the transparent area in the diagram will be black too, making diagrams (authored in white background) illegible. To do this, I have to add the following section to the additional CSS setting of the active theme:\nWordPress is dynamic site and if you don\u0026#8217;t edit the site often, and don\u0026#8217;t want to start a static site from scratch (e.g. using Hugo or Jekyll frameworks), you can use WP2Static plugin. Make sure to configure S3 bucket and CloudFront accordingly. Now, enjoy blogging.\nPrevious PostStorage Nitty-Gritty 1 of 5 – Disk and RAID Next PostApplication I/O Characteristics ","date":"2019-04-10T01:30:44-04:00","permalink":"/2019/04/build-a-wordpress-site-in-one-hour-with-lightsail/","title":"Lightsail – create a WordPress site in one hour"},{"content":"On my trip I have been through several pre-sales discussions on storage. Therefore I\u0026#8217;m taking this opportunities to write up a series of postings deep diving into storage technologies. In this first section, we lay out the foundation of storage technology, from physical device to RAID, focusing on the concepts. Some contents are excerpts from Information Storage and Management.\nVolume Manager \u0026#8211; In early days, a file system occupies the entire disk drive, and presents continuous disk blocks directly to operating systems. Logical Volume Manager (LVM) was then introduced to bring a layer of abstraction (logical volume) on top of disks. The layers are shown as follows:\nPhysical Hard DrivePhysical Hard DrivePhysical Hard DrivePhysical Hard DrivePhysical Hard DrivePhysical Hard DrivePartitionPartitionPartitionPartitionPhysical VolumePhysical VolumePhysical VolumePhysical VolumePhysical VolumePhysical VolumePhysical VolumePhysical VolumeVolume GroupVolume GroupVolume GroupVolume GroupLogical VolumeLogical VolumeLogical VolumeLogical VolumeLogical VolumeLogical VolumeFile SystemFile SystemFile SystemFile SystemFile SystemFile Systempvcreatepvcreatevgcreatevgcreatelvcreatelvcreatemkfsmkfsfdiskfdisk\nWith all these layers, a byte in user file maps to disk sectors through several layers:\nMapping from user file to physical disk File System \u0026#8211; a hierarchical structure of files. It organizes data in a structural hierarchical manner. It includes files, directories as well as metadata. metadata must be consistent for the file system to be considered healthy. In Linux, metadata consists of:\nSuperblock: important information about file system, e.g. type, creation and modification dates, size, mount status flag Inodes: a data structure that contains information associated with every file or directory list of data blocks free and in use Host connects to storage through various Interface Protocols. Common interface protocols include:\nIDE/ATA and Serial ATA SCSI (Small Computer System Interface) FC (Fibre Channel) IP (Internet Protocol per se is a network protocol traditionally used for host-to-host traffic in the early days. In the virtualization era, it has become a viable option for host-to-storage communication. Examples are iSCSI and FCIP) The most prevalent disk drive types are SSD (solid state drive) and HDD (hard disk driveA). SSD (solid state drive) is newer, flash-based technology. Without seek and rotational latencies they deliver a high number of IOPS with low response times. They are especially suited for applications with small block size and random-read workloads requiring constant latency \u0026lt;1ms.\nHDD is traditional and more cost effective. Its physical components are illustrated in the following two graphs:\nHard disk component For HDD, disk service time (time taken by a disk to complete an I/O request) is determined by the following factors:\nSeek time (aka access time) describes the time taken to position the R/W heads across the platter with a radial movement (moving along the radius of the platter). In other words, it is the time taken to position and settle the arm and the head over the correct track. Rotation latency is the time taken by the platter to rotate and position the data under the R/W head. It depends on the rotation speed of the spindle and is measured in milliseconds. (Data) transfer rate is the average amount of data per unit time that the drive can deliver from disk controller to the HBA (on the host). Zone Bit Recording \u0026#8211; a mechanism to use disk efficiently by grouping tracks into zones based on their distance from the disk.\nLogical Block Addressing (LBA) \u0026#8211; disk controller translates LBA to a physical address (CHS, cylinder, head and sector). The host only needs to know the size of disk drive in terms of number of blocks. The logical blocks are mapped to physical sectors on a 1:1 basis\nIO request processing \u0026#8211; I/O controller is introduce to improve response time for I/O request, in this model, The I/O requests arrive at the controller at the rate generated by the application. This rate is also called the arrival rate. These requests are held in the I/O queue, and the I/O controller processes them one by one, as shown here:\nThe relationship between controller utilization and average response time is: Average response time = Service time / (1 – Utilization) where service time is the time taken by disk controller to service the I/O request. This results in a classic relation between response time and utilization, as plotted below:\nThe graph indicates that the response time changes are nonlinear as the utilization increases. When the average queue sizes are low, the response time remains low. The response time increases slowly with added load on the queue and increases exponentially when the utilization exceeds 70 percent. Therefore, for performance-sensitive applications, it is common to utilize disks below their 70 percent of I/O serving capability. Command queuing is a technique implemented on modern disk drives that determines the execution order of received I/Os and reduces unnecessary drive-head movements to improve disk performance. When an I/O is received for execution at the disk controller, the command queuing algorithms assign a tag that defines a sequence in which the commands should be executed. With command queuing, commands are executed based on the organization of data on the disk, regardless of the order in which the commands are received. Below is an example:\nDisk Command Queuing RAID is a technology that leverages multiple drives as part of a set that provides data protection against drive failures. It may also improve performance by serving I/Os from multiple disks simultaneously. It is primarily used in HDD but SSD may still benefit from it. RAID may be implemented by software but hardware RAID with a controller is widespread. RAID is built on three basic techniques:\nStriping: spread data across multiple drives (more than one) to use the drives in parallel. Mirroring: same data is stored on two different disk drives, yielding two copies of the data. Parity: a method to protect striped data from disk drive failure without the cost of mirroring. An additional disk drive is added to hold parity, a mathematical construct that allows re-creation of the missing data. Basic RAID levels are summarized here:\nRAID summary RAID 0, 1, 5 and 6 are pretty common in data centre operations. In addition to these levels above. If you hear RAID 1+0, 5+0 and RAID 6+0, they are called nested RAID. They are simply a RAID 0 on top of RAID1, RAID 5 and RAID 6, respectively.\nRAID 0 \u0026#8211; data striping technique utilizes full capacity of drives. Although it is a good option for applications that need high I/O throughput. It lacks data protection so it cannot drive application requiring high availability.\nRAID 0 RAID 1 \u0026#8211; mirroring technique ensures data duplication. In the event of disk failure, it introduces minimal impact to the disk array. It is suitable for applications that require high availability and cost is no constraint.\nRAID 1 RAID 1+0 or RAID 10 \u0026#8211; performs well for workloads with small, random, write-intensive I/Os. Some applications that benefit from RAID 1+0 include the following:\nHigh transaction rate Online Transaction Processing (OLTP) Large messaging installations Database applications with write intensive random access workloads RAID 1+0 RAID 3 \u0026#8211; RAID 3 stripes data for performance and uses parity for fault tolerance. the total disk space required is 1.25 times the size of the data disks. RAID 3 always reads and writes complete stripes of data across all disks because the drives operate in parallel. RAID 3 provides good performance for applications that involve large sequential data access, such as data backup or video streaming.\nRAID 5 \u0026#8211; RAID5 is similar to RAID 4 because it uses striping. The drives (strips) are also independently accessible. The difference between RAID 4 and RAID 5 is the parity location. In RAID 4, parity is written to a dedicated drive, creating a write bottleneck for the parity disk. In RAID 5, parity is distributed across all disks to overcome the write bottleneck of a dedicated parity disk.\nRAID 5 RAID 5 is good for random, read-intensive I/O applications and preferred for messaging, data mining, medium-performance media serving, and relational database management system (RDBMS) implementations, in which database administrators (DBAs) optimize data access.\nRAID 6 \u0026#8211; works the same way as RAID 5, except that RAID 6 includes a second parity element to enable survival if two disk failures occur in a RAID set. The write penalty in RAID 6 is more than that in RAID 5; therefore, RAID 5 writes perform better than RAID 6. The rebuild operation in RAID 6 may take longer than that in RAID 5 due to the presence of two parity sets.\nRAID 6 Intelligent Storage System involves cache as the core component. An intelligent storage system involves front end, cache, back end and physical disks, as shown here:\nIntelligent Storage System components A variety of physical disk types and speed (e.g. mix of FC, SATA, SAS and flash) can be supported in a modern intelligent storage system.\u0026nbsp;The front end provides the interface between the storage system and the host. It consists of ports and controllers, with redundancy. The back end provides an interface between cache and the physical disks. It consists of ports and controllers. For high data protection and high availability, storage systems are configured with dual controllers with multiple ports.\nCache improves storage system performance by isolating hosts from mechanical delays associated with hard disks. In intelligent storage system, read and write are first attempted on cache.\nPage is the basic unit of cache, and the size of page is configured based on application I/O size. Cache consists of:\ndata store: holds the actual data temporarily tag RAM: mainly serves three purposes: it tracks locations of data in cache and on disk; it maintains dirty bit flag to indicate whether data in cache has been committed to disk; it keeps time based information such as last access time, for cache management Read Operation with Cache: When host issues a read request, the storage controller reads the tag RAM first to determine whether required data is available in cache:\nRead cache hit: data is sent to host without any disk operation; Read cache miss: back end access the disk to retrieve the requested data. Data is then placed in cache and sent to host through front end. Cache miss increases I/O response time, to increase read hit ratio, read-ahead algorithm can be used when read requests are sequential. In a sequential read request, a contiguous set of associated blocks is retrieved. Several other blocks that have not yet been requested by the host can be read from the disk and placed into cache in advance. When the host subsequently requests these blocks, the read operations will be read hits. Fixed prefetch \u0026#8211; the intelligent storage system prefetches a fixed amount of data. It is most suitable when host I/O sizes are uniform. Variable prefetch, the storage system prefetches an amount of data in multiples of the size of the host request. Maximum prefetch limits the number of data blocks that can be prefetched to prevent the disks from being rendered busy with prefetch at the expense of other I/Os. Write Operation with Cache: When an I/O is written to cache and acknowledged, it is completed in far less time (from the host’s perspective) than it would take to write directly to disk. Sequential writes also offer opportunities for optimization because many smaller writes can be coalesced for larger transfers to disk drives with the use of cache. Write operation with cache can be implemented in two ways:\nWrite-back cache: Data is placed in cache and an acknowledgment is sent to the host immediately. Later, data from several writes are committed (de-staged) to the disk. Write response times are much faster because the write operations are isolated from the mechanical delays of the disk. However, uncommitted data is at risk of loss if cache failures occur. Write-through cache: Data is placed in the cache and immediately written to the disk, and an acknowledgment is sent to the host. Because data is committed to disk as it arrives, the risks of data loss are low, but the write-response time is longer because of the disk operations. If the size of an I/O request exceeds the write aside size, writes are sent to the disk directly to reduce the impact of large writes consuming a large cache space. This is helpful where cache resources are constrained and cache is required for small random I/Os. Cache space can be assigned in two ways:\ndedicated cache: separate sets of locations are reserved for read and write; global cache: user may specify percentage of cache for read and write based on application workload pattern; or the system set is dynamically. Cache Management algorithm is used to determine when, and what pages of the cache need to be free up during maintenance. Most commonly used algorithms are:\nLRU (least recently used): assuming data not accessed for a while will not be requested by host any more; MRU (most recently used): assuming data recently accessed will not be requested by host again As cache fills, the storage system must take action to flush dirty pages by committing data from cache to disk. There are several triggers for cache management action:\nIdle flushing \u0026#8211; occurs continuously at modest rate when cache utilization level is between high and low watermark; High watermark flushing \u0026#8211; activated when utilization hits high watermark; and stops at low watermark; this has impact to I/O processing; Forced flushing \u0026#8211; occurs in the event of large I/O burst when cache reaches 100% capacity; this significantly impacts I/O response time Types of Flushing Cache data protection is the mechanism to prevent losing uncommitted data held in cache. Common mechanisms are:\nCache mirroring \u0026#8211; Each write to cache is held in two different memory locations on two independent memory cards. If a cache failure occurs, the write data will still be safe in the mirrored location and can be committed to the disk. The array operating environment needs to maintain cache coherency between the redundant memory locations. Read cache does not need mirroring.\nCache vaulting \u0026#8211; In the event of server power failure, use battery power to write the cache content to the disk (vault drive). When power is restored, data from these disks is written back to write cache and then written to the intended disks.\nRelated Postings SAN NAS and Object Storage Backup and Archive Solution Replication Previous PostPackage Repository Management for Linux Next PostLightsail – create a WordPress site in one hour ","date":"2019-03-09T22:25:52-05:00","permalink":"/2019/03/storage-nitty-gritty-1-5/","title":"Storage Nitty-Gritty 1 of 5 – Disk and RAID"},{"content":"RPM is the package manager tool in Linux. YUM is a repository management tool to fetch appropriate package for the particular version of Linux. YUM performs automatic dependency resolution when updating, installing or removing packages, and thus is able to automatically determine, fetch and install all available dependent packages. This posting is about common commands used for package management. Then we walk through the step to set up a local yum repository. Lastly, we summarize the best practice for patch management.\nTools In addition to the basic yum install, update and remove commands, here are some more tools for repository management:\n## check for installed packages with updates available yum check-update ## search for packages yum search package.name ## check for dependency yum deplist package.name ## list installed or available packages: yum list installed package.name yum list available package.name ## list all available packages of a repo: yum --disablerepo=* --enablerepo=repo.name list available ## display package info yum info package.name ## list all repositories yum repolist yum repoinfo ## list all transactions yum history list all ## display history of a package or transaction id yum history package-list package.name yum history info package.name yum history info transaction.id ## undo or redo a transaction yum history undo transaction.id yum history redo transaction.id RHEL provides a more complete cheatsheet for yum commands.\nIn addition to single package, YUM also manages package groups, and works with plugins. For example, you may enable security related packages only by using the yum-plugin-security. For security, YUM repositories can enable GPG check (based on public key cryptography).\nLocal Repo configuration Now we configure a YUM repository server and reference that from a client to update packages. To begin with, we collect all RPM packages in /home/dhunch/upgrade/downloads/, then we install createrepo package and use it to create metadata:\nyum install createrepo yum-utils createrepo /home/dhunch/upgrade/downloads/ Examine the directory and you will find a new repodata directory created. Now we can configure nginx with the following configuration:\nserver { listen 8088; server_name rpmsource.digihunch.com; root /home/dhunch/upgrade/downloads/; location / { autoindex on; #enable listing of directory index } } Restart nginx and browse to server name at port 8088, you should see the directory in html. If you\u0026#8217;re getting 403 error, most likely nginx has issues accessing the directory. Nginx process should be able to traverse each level of directory to serve the files. The parent directories should have executable permission for others. For example:\nchmod o+x /home/dhunch/ Here is a reference to nginx permission requirement. Now we can continue to configure the client.\nOn the client, we need to add a file in /etc/repos.d/yum\n[digirepo] name=Local YUM Repository baseurl=http://rpmsource.digihunch.com:8088/ enabled=1 gpgcheck=0 Last we can check the available packages from the client:\nyum --disablerepo=* --enablerepo=digirepo list available Note that in order to keep the local repo up-to-date, there is additional maintenance work on the server. You may need to sync from official source, such as:\nreposync -g -l -d -m --repoid=base-source --newest-only --download-metadata --download_path=/home/dhunch/upgrade/downloads/ You may also need to set up a daily job with yum-cron service so this is automated.\nSometimes the server is locked down and you need HTTP proxy to allow YUM to access repo. You may configure proxy in /etc/yum.conf in the proxy, proxy_username, and proxy_password entries. Also, you may export environment variable http_proxy. Here are more information.\nSometimes yum cache may introduce issues. To clean cache before installing, run:\nyum clean all \u0026amp;amp;\u0026amp;amp; yum -y install python3 Python packages Python3 is not installed by default on CentOS 7. So it needs to be installed with YUM. Do not replace the existing python2 with python3 by changing where symbolic link /bin/python points to, because YUM is dependent on python2.\nFor python3 use pip3 as package management. If the server does not have a public route, then you need to install pip3 package offline. For example, on a server with Internet, run\npip3 download -d ~/Downloads javaobj-py3 Then SCP the file to the offline server (e.g. to /home/dhunch/javaobj/), from which you can run:\npip3 install --no-index --find-links=/home/dhunch/javaobj/ javaobj-py3 --user This will install the pip3 package offline. Note it is recommended to not run pip3 installer as root user. The switch \u0026#8211;user allows you to run as non-root user.\nBest Practices To configure a patch management environment, we assume that the servers do not have access to the Internet, not even through proxy. This should be part of the security guideline anyways. This requires a local repository server to be setup. Administrators may use createrepo tool as outline above to create such repo, use reposync to synchronize local source from official source, or yum-cron to automate this for daily task.\nOn the clients, administrators should manage the repo file in /etc/yum.repo.d, to ensure they are pointing to the correct source, using gpg check, etc. A local script to enable and disable different repos are also helpful. Otherwise, the enabelrepo and disablerepo switch can be used per command.\nIt is important to catalogue the state of operating system, before and after each patching activity, for verification and auditing purposes. This usually requires scripting work using yum history or package-cleanup tools.\nThe actual patch work can be done either manually or automatically (e.g. with Ansible\u0026#8217;s yum module). RPM is also a good tool for troubleshooting. If the scope of patching is only for security, use the yum-plugin-security to limit the packages to only security related ones.\nRoll-back should be prepared in case of inadvertent outcome. Roll-back scripting relies heavily on yum history commands. If the OS needs to boot with a previous kernel, use the grubby tool.\nHappy Patching!\nPrevious PostInteresting terms and principles Next PostStorage Nitty-Gritty 1 of 5 – Disk and RAID ","date":"2019-02-09T22:14:56-04:00","permalink":"/2019/02/package-repository-management-in-linux/","title":"Package Repository Management for Linux"},{"content":"It is enlightening to find out that several things that I struggled at different times in professional life are actually experienced by many predecessors. Some smart people actually coined terms for these phenomena. Although they are somewhat negative, they won\u0026#8217;t go away simply because not being mentioned. Both technical staff and project management professions should be wary of them. Some came out of software project management and some grew out of behavioural economics context. Software Delivery Here we go the terms, mostly excerpts from Wikipedia:\nDeath march project \u0026#8211; a project that the participants feel is destined to fail, or that requires a stretch of unsustainable overwork. The general feel of the project reflects that of an actual death march because project members are forced by their superiors to continue the project against the member\u0026#8217;s better judgement. Death marches of the destined-to-fail type usually are the result of unrealistic or overly optimistic expectations in scheduling, feature scope, or both, and often include lack of appropriate documentation or relevant training and outside expertise that would be needed to accomplish the task successfully. Often the death march will involve desperate attempts to right the course of the project by asking team members to work especially grueling hours, or by attempting to \u0026#8220;throw (enough) bodies at the problem\u0026#8221;, often causing burnout.\nSoftware Peter Principle \u0026#8211; in software engineering, software Peter principle describes a dying project which has become too complex to be understood even by its own developers. It is well known in the industry as a silent killer of projects, but by the time the symptoms arise it is often too late to do anything about it. Good managers can avoid this disaster by establishing clear coding practices where unnecessarily complicated code and design is avoided. This term is derived from the Peter Principle \u0026#8211; a theory about incompetence in hierarchical organizations. There are mainly three causes of this:\nLoss of conceptual integrity: The conceptual integrity of software is a measure of how well it conforms to a single, simple set of design principles. When done properly, it provides the most functionality using the simplest idioms. It makes software easier to use by making it simple to create and learn. Conceptual integrity is achieved when the software design proceeds from a small number of agreeing individuals. For software to maintain conceptual integrity, the design must be controlled by a single, small group of people who understand the code in depth. In projects without a strong architecture team, the task of design is often combined with the task of implementation and is implicitly delegated among the individual software developers. Under these circumstances, developers are less likely to sacrifice personal interest in favour of the interests of the product. The complexity of the product grows as a result of developers adding new designs and altering earlier ones to reflect changes in fashion and individual taste. Programmer incompetence: the best developer should understand what details need to be communicated with people. Programmer inexperience: programmers sometimes make implementation choices that work but have unintended negative consequences. Over time, many such implementation choices degrade the software\u0026#8217;s design, making it increasingly difficult to understand. Brooks\u0026#8217; law \u0026#8211; an observation about software project management according to which \u0026#8220;adding human resources to a late software project makes it later\u0026#8221;. It was coined by Fred Brooks, according to who, there is an incremental person who, when added to a project, makes it take more, not less time. This is similar to the general law of diminishing returns in economics. Brooks admit the law is an outrageous oversimplification, but it captures the general rule. Brooks points the the main factors:\nIt takes some time for the people added to a project to become productive. Brooks calls this the \u0026#8220;ramp up\u0026#8221; time. Software projects are complex engineering endeavours, and new workers on the project must first become educated about the work that has preceded them; this education requires diverting resources already working on the project, temporarily diminishing their productivity while the new workers are not yet contributing meaningfully. New worker may even make negative contributions, for example, if they introduce bugs that move the project further from completion. Communication overhead increases as the number of people increase. Due to combinatorial explosion, the number of different communication channels increases rapidly with the number of people. Everyone working on the same task needs to keep in sync, so as more people are added they spend more time trying to find out what everyone else is doing. Adding more people to a highly divisible task, such as cleaning rooms in a hotel, decreases the overall task duration (up to the point where additional workers get in each other\u0026#8217;s way). However, other tasks including many specialties in software projects are less divisible; Brooks points out this limited divisibility; Brooks points out this limited divisibility with another example: while it takes one woman nine months to make one baby, nine women can\u0026#8217;t make a baby in one month. Escalation of commitment \u0026#8211; a human behaviour pattern in which an individual or group facing increasingly negative outcomes from a decision, action, or investment, nevertheless continues the behaviour instead of altering course. The actor maintains behaviours that are irrational, but align with previous decisions and actions. Cowboy coding \u0026#8211; the development process where autonomous developers in control of schedule, algorithms, frameworks and coding style work with minimal process or discipline. Usually it occurs when there is little participation by business users or fanned by management that controls only non-development aspects of the project, such as the broad targets, timelines, scope and visuals (the \u0026#8220;what\u0026#8221;, but not the \u0026#8220;how\u0026#8221;)\nConway\u0026#8217;s law \u0026#8211; organizations design systems which mirror their own communication structure. The law is based on the reasoning that in order for a software module to function, multiple authors must communicate frequently with each other. Therefore, the software interface structure of a system will reflect the social boundaries of the organization(s) that produced it, across which communication is more difficult. Goodhart\u0026#8217;s law \u0026#8211; When a measure becomes a target, it ceases to be a good measure.\nMurphy\u0026#8217;s law \u0026#8211; Anything that can go wrong will go wrong.\nKISS principle \u0026#8211; \u0026#8220;Keep it simple, stupid\u0026#8221;, as a design principle, it states that most systems work best if they are kept simple rather than made complicated; therefore, simplicity should be a key goal in design, and unnecessary complicity should be avoided.\nBoondoggle \u0026#8211; a project that is considered a waste of both time and money, yet is often continued due to extraneous policy or political motivations.\nOptimism bias \u0026#8211; a cognitive bias that causes someone to believe that they themselves are less likely to experience a negative event. Planning fallacy \u0026#8211; a phenomenon in which predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the time needed. This phenomenon sometimes occurs regardless of the individual\u0026#8217;s knowledge that past tasks of a similar nature have taken longer to complete than generally planned. The bias only affects predictions about one\u0026#8217;s own tasks. When outside observers predict task completion times, they show a pessimistic bias, overestimating the time needed. Separation of concerns \u0026#8211; a design principle for separating a computer program into distinct sections such that each section addresses a separate concern. A concern is a set of information that affects the code of a computer program. A concern can be as general as \u0026#8220;the details of the hardware for an application\u0026#8221;, or as specific as \u0026#8220;the name of which class to instantiate\u0026#8221;. A program that embodies SoC well is called a modular program. Modularity, and hence separation of concerns, is achieved by encapsulating information inside a section of code that has a well-defined interface. Encapsulation is a means of information hiding. Layered designs in information systems are another embodiment of separation of concerns (e.g., presentation layer, business logic layer, data access layer, persistence layer).\nDRY principle \u0026#8211; Don\u0026#8217;t Repeat Yourself. Every piece of knowledge must have a single, unambiguous, authoritative representation within a system. The principle has been formulated by Andy Hunt and Dave Thomas in their book The Pragmatic Programmer. They apply it quite broadly to include \u0026#8220;database schemas, test plans, the build system, even documentation\u0026#8221;.[3] When the DRY principle is applied successfully, a modification of any single element of a system does not require a change in other logically unrelated elements. Additionally, elements that are logically related all change predictably and uniformly, and are thus kept in sync. This principle is aimed at reducing repetition of software patterns, replacing it with abstractions or using data normalization to avoid redundancy.\nYAGNI principle \u0026#8211; You Aren\u0026#8217;t Gonna Need It. Always implement things when you actually need them, never when you just foresee that you need them.\nGolden Hammer \u0026#8211; aka Law of Instrument, a cognitive bias that involves an over-reliance on a familiar tool. It is a form of narrow-minded instrumentalism.\nShturmovshchina \u0026#8211; last minute rush, a common Soviet work practice of frantic and overtime work at the end of a planning period in order to fulfill the planned production target. The practice usually give rise too poor quality.\nPrinciples more relevant software development Least Knowledge (aka Law of demeter): this principle states that an object should never know the internal details of other objects. The dependencies between packages should be in the direction of the stability of the packages. A package should only depend upon packages that are more stable than it is.\nStable Dependencies: The dependencies between packages should be in the direction of the stability of the packages. A package should only depend upon packages that are more stable than it is. In Stable Dependencies principle, \u0026#8220;Stable\u0026#8221; roughly means \u0026#8220;hard to change\u0026#8221;, whereas \u0026#8220;instable\u0026#8221; means \u0026#8220;easy to change\u0026#8221;.\nSOLID: SOLID is an acronym for five principles for OOP. The five principles are: Single-responsibility, Open-closed, Liskov Substitution, Interface Segregation, Dependency Inversion. Inversion of Control (IoC): Inversion of control (IoC) is a design pattern in which custom-written portions of a computer program receive the flow of control from a generic framework.\nBoy Scout Rule: Always leave the code better than you found it.\nPersistence Ignorance (PI): The principle of Persistence Ignorance (PI) holds that classes modeling the business domain in a software application should not be impacted by how they might be persisted. For example, application code should not be affected by the chosen technology for underlying database or persistence storage. Business Logic should be independent of underlying technology for persistent storage.\nBounded Context: Bounded Context is a concept in domain driven design that is often used in microservice design. It provides a way of tackling complexity in large applications or organizations by breaking it up into separate conceptual modules. Each conceptual module then represents a context that is separated from other contexts (hence, bounded), and can evolve independently. Each bounded context should ideally be free to choose its own names for concepts within it, and should have exclusive access to its own persistence store.\nCohesion: cohesion refers to the degree to which the elements inside a module belong together.\nThat\u0026#8217;s it for now. There are some more software development philosophies on Wikipedia.\nPrevious PostJava version confusions Next PostPackage Repository Management for Linux ","date":"2019-01-26T18:10:05-04:00","permalink":"/2019/01/interesting-terms-about-unsuccessful-software-project-management/","title":"Interesting terms and principles"},{"content":"Anyone working with deploying Java applications inevitably came across one of these confusions with the terms. Let\u0026#8217;s clarify them. This clarification is not for Java developer and does not go deep with underlying technologies. This is for installation/DevOps engineers to understand Java environment.\nJava SE, EE and ME Java Platform, Standard Edition (Java SE) is a computing platform for development and deployment of portable code for desktop and server environments. Java SE was formerly known as Java 2 Platform, Standard Edition (J2SE). Java SE defines a range of general-purpose APIs, and also includes the Java Language Specification and the Java Virtual Machine Specification. Java Enterprise Edition (Java EE), formerly Java 2 Platform, Enterprise Edition, currently rebranded as Jakarta EE but the new brand is still being adopted. It is an extension to Java SE with specifications for enterprise features such as distributed computing, web services, XML processing, JMS (messaging). It is more widespread in enterprise contexts such as e-commerce, accounting, banking information systems. The specification defines APIs and their interactions for providers to meet in order to declare compliance with Java EE. For example Apache Tomcat is an implementation of a subset of Java EE. Java Platform, Micro Edition (Java ME, formerly knowned as Java 2 Platform, Micro Edition or J2ME) is a subset of Java SE for embedded and mobile devices. The advent of Android significantly de-popularized Java ME.\nJRE and JDK Java SE is the foundation for developing in Java language. The aforementioned platforms (Java SE, EE and ME) are just specifications, not implementations. Java Software Development Toolkit (SDK) is called JDK (Java Development Kit) for short. Strictly speaking, the JDK can be an implementation of any one of the platforms above. In every day language, people loosely refers to the implementation of Java SE as JDK, whereas Oracle\u0026#8217;s implementation of Java EE is referred to as Java EE SDK.\nJDK vs JRE JDK consists of Java Runtime Environment (JRE) along with tools to compile and debug Java code for developing Java applications. JRE consists of libraries, Java Virtual Machine (JVM), Java Pluging and Java Web Start to run Java applications. JRE alone does not contain compilers and debugging tools. The two most widespread JDKs are:\nOracle JDK: Oracle\u0026#8217;s official implementation of Java SE.OpenJDK: a free and open-source implementation of Java SE. They are both created and maintained by Oracle. Almost everything in Oracle JDK is from OpenJDK. The slight difference between them is an entirely separate topic itself but the idea is their binaries will be converged:\nOracle JDK was licensed under Oracle Binary Code License Agreement, whereas OpenJDK has the GNU General Public License (GNU GPL) version 2 with a linking exception. It is worth-noting that Oracle has announced that the Oracle JDK 8 builds released after Jan 2019 cease to be free for commercial use. This drives may application vendor to migrate from Oracle JDK to OpenJDK in their platforms.\nVersion History If what you have read so far is not confusing enough, here\u0026#8217;s some more muds. The version scheme for Java has changed in it\u0026#8217;s 20 years history. Here is a list of main versions.\nPlatform VersionInternal VersionRelease DateNotesJDK 1.01.0Jan 1996JDK 1.11.1Feb 1997J2SE 1.21.2Dec 1998In 1998 JDK splits into J2SE and J2EE. Code name for J2SE 1.2 is PlaygroundJ2SE 1.31.3May 2000Code name is KestrelJ2SE 1.41.4Feb 2002Code name is MerlinJ2SE 5.01.5Sep 2004In 2004, Sun introduced internal version and external version. Code name for this version is Tiger.Java SE 61.6Dec 2006Code name MustangJava SE 71.7Jul 2011Code name DolphinJava SE 81.8Mar 20145 year from previous version, LTSJava SE 91.9Sep 20173.5 year from previous version. Going forward new version will be released every six monthJava SE 1010Mar 2018It was proposed that versions should simply increase incrementallyJava SE 1111Sep 2018LTSJava SE 1212Mar 2019\u0026#8230;Java SE 1717Sep 2021LTS Since 2018, new version will be release every six month and the there is no longer a distinction between internal and external versions.\nMulti-version management We only cover Linux here to manage multiple versions of JDK. We use a tool named alternatives to maintain symbolic links determining default commands. How this works with Java is:\nMake /usr/bin/java a symbolic link pointing to /etc/alternatives/javaMake /etc/alternatives/java also a symbolic link pointing to the desired version of java To start configuration, run:\nalternatives --config java Then you will be given a list of Java versions to choose from. If the list does not have your desired version, and you confirm that the version has been installed. You will need to add this version by doing something like:\nalternatives --install /usr/bin/java java /usr/java/jdk1.6.0_25/bin/java 1000 The command executes and takes effect by modifying files under /var/lib/alternatives directory.\nAlso, one can overwrite environment variable $JAVA_HOME to force an application to use a different version of Java. This is because many application picks up JDK location from that variable.\nPrevious PostRedhat Firewall configuration: from iptables to firewalld Next PostInteresting terms and principles ","date":"2018-11-05T17:32:53-04:00","permalink":"/2018/11/the-java-confusions/","title":"Java version confusions"},{"content":"Tools to manage firewall Packet filter rules in Linux Kernel is managed by an user-space application named iptables in CentOS and RedHat. Since CentOS 7, firewalld is introduced as an alternative to iptables. Firewalld can be installed and executed as a systemd service, and it is supposed to replace iptables. This article describes how to configure both. There are several advantages in firewalld. One is is the support of zones. Here are some useful information. Also, iptables involves three different services for IPv4(iptables), IPv6(ip6tables), and software bridging (ebtables), whereas firewalld only involves a single service to manage all three. Firewalld allows user to add or remove rules/ports from running firewall, without restarting firewall. Unless you have specific reason to use iptables, always use firewalld service to manage firewall. Here is an instruction to firewalld service. In this posting however, we will be focusing on iptables to understand firewall managment. We also go through an example of opening a TCP port. How does iptables work When working with iptables, it is important to understand that its related concepts (tables-\u0026gt;chains-\u0026gt;rules-\u0026gt;criteria and targets) and how the order of rules plays a factor. There are five independent tables, each contains a number of chains, either built-in or user-defined. Administrators mostly deals with built-in chains in filter and nat tables. The five tables are:\nfilter: If -t isn\u0026#8217;t specified, this is the default table. It contains built-in chains:INPUT: for packet destined to local socketsFORWARD: for packets being routed through the boxOUTPUT: for locally-generated packetsnat: this table is consulted when a packet that creates a new connection is encountered. It has three built-in chains:PREROUTING: for altering packets as soon as they come inOUTPUT: for altering locally generated packets before routingPOSTROUTING: for altering packets as they are about to go outmangle: this table is used for specialized packet alternation, with five built-in chains (since kernel 2.4.18): PREROUTING and OUTPUT, INPUT, FORWARD, and POSTROUTINGraw: this table is mainly for configuring exceptions from connection tracking with two built-in chains: PREROUTING and OUTPUTsecurity: for Mandatory Access Control (MAC) networking rules, with three built-in chains: INPUT, OUTPUT, and FORWARD. Under the table (e.g. filter, nat), each chain (e.g. INPUT, OUTPUT, etc) consists of list of firewall rules. Each rule is made up of two parts defined for the packets:\nCriteria: if the packet does not match the criteria, the next rule in the chain is examined; if it does match, then the next rule is specified by the value of the target.Target: what to do if criteria is met. The target can be:user-defined chain, one of the target described in iptables-extensions, or in most cases, one of the special values ACCEPT, DROP or RETURNACCEPT \u0026#8211; let the packet throughDROP \u0026#8211; drop the packet on the floorRETURN \u0026#8211; stop traversing this chain, and resume at next rule in the previous (calling) chain The rules, defined in each chain under their tables, can be found in file /etc/sysconfig/iptables. You can find tables (prefix with asterisk *), chains (prefix with colon :), rules under their chains and a statement COMMIT after each table. The iptables process flow illustrates how a packet interact with all these rules under different chains and tables defined in this file: iptables Process Flow Although this big picture looks formidable, an administrator commonly only deals with the green and purple blocks (filter and nat), with the big picture in mind. Here is an example of /etc/sysconfig/iptables file from a newly installed system:\n# Generated by iptables-save v1.4.21 on Fri Sep 11 23:15:32 2017 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [132:17200] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Fri Sep 11 23:15:32 2017 The rule simply allows SSH traffic. This file will be loaded up on every reboot (specifically, restart of iptables service). So if you have made some changes to rules and you want the change picked up on reboot. The rules should be saved to this file:\n$ sudo iptables-save \u0026gt; /etc/sysconfig/iptables Other than saving rule for reboot, if you simply want to edit the rules (e.g. order of rules is incorrect), you can save the rules to file, modify the file and restore the rule from file:\n$ sudo iptables-save \u0026gt; ~/iptables.txt $ sudo vi ~/iptables.txt $ sudo iptables-restore \u0026amp;lt; ~/iptables.txt Here is some further reading about iptables architecture.\nAnatomy of a rule The man page for iptables species the following synopsis:\niptables [-t table] {-A|-C|-D} chain rule-specification rule-specification = [matches...] [target] match = -m matchname [per-match-options] target = -j targetname [per-target-options] So when you append (-A), delete (-D), insert (-I) or replace (-R) a rule, you need to specify rule specification. The man page further explains that the following parameters make up a rule specification:\nprotocol (-p): the protocol of the rule of the packet to check. value can be tcp, udp, icmp, all or any name defined in /etc/protocols.match (-m): specifies the name of a match to use and is followed by match options. The match refers to an extension module that tests for a specific property. Those extension modules are documented in the man page of iptables-extensions. You may specify -m multiple times for different match names, which together make up the condition under which a target is invoked. Matches are evaluated first to last as specified. We often use extensions tcp and state. According to iptables-extensions man page, we can specify \u0026#8211;dport followed by port number for the tcp extension, and \u0026#8211;state followed by value such as NEW or ESTABLISHED for the state extension.jump (-j): specifies the target of the rule, such as ACCEPT, REJECT or DROP.source and destination (-s and -d): source and destination IP address or masks. Hostname will work but not recommended since resolution is needed.inbound and outbound interface (-i and -o): name of interface via which the packet was received and is going to be sent.goto (-g): processing should continue in a user specified chainOther parameters: -4/\u0026#8211;ipv4, -6/\u0026#8211;ipv6, -c/\u0026#8211;set-counters, -f/\u0026#8211;fragment When we run iptables command to view rules, we need to specify the table (e.g. filter, nat, etc) followed by -S or \u0026#8211;list-rules:\n$ iptables -t nat -S If you do not specify -t switch, the default (-t filter) is applied. Be aware that in this case, you\u0026#8217;re only seeing rules under filter table, and not all rules under tall tables!\nIn the result, for example one line from command \u0026#8220;iptables -S\u0026#8221; may say:\n-A INPUT -p tcp -m state --state NEW -m tcp --dport 9200 -j ACCEPT The interpretation: appending a rule to INPUT chain of filter table (implicitly specified). The protocol is tcp. The first match extension is state, and the state value shall be NEW. The second match extension is tcp, and the dport value shall be 9200. If the packet is a match, then the target (action) is ACCEPT.\nManaging rules As mentioned earlier, rules can be dumped to any file or /etc/sysconfig/iptables, in which the rules are assessed in order. Below is a real life iptables file with a nat table as well. # Generated by iptables-save v1.4.21 on Wed Jan 15 13:58:39 2017 *filter :INPUT DROP [0:0] :FORWARD DROP [0:0] :OUTPUT ACCEPT [4:208] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7000:7001 -j ACCEPT -A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7199 -j ACCEPT -A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9042 -j ACCEPT -A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9160 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT -A INPUT -p udp -m state --state NEW -m udp --dport 161 -j ACCEPT -A INPUT -p udp -m state --state NEW -m udp --dport 162 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT *nat :PREROUTING ACCEPT [1:328] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] -A PREROUTING -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398 -A PREROUTING -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398 -A OUTPUT -o lo -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398 -A OUTPUT -o lo -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398 COMMIT # Completed on Wed Jan 15 13:58:39 2017 In this example, the nat table defines traffic forwarding: traffic arriving at TCP port 2392 and 2393 are forwarded to port 2398; outgoing traffic to port 2392 and 2393 are also redirected to port 2398. These rules do not overlap each other so the rules probably don\u0026#8217;t matter.\nOn the other hand, the tcp filter table lists the rules to open certain TCP and UDP ports. Its block starts with a couple accepting rules and ends with a couple reject rules (regardless of protocols or ports). This is a good way to close a chain of rules with security. However, if you need to add additional rules to open more TCP ports, the new rule should not be appended after the reject rules at the bottom since the order matter here!\nCorrect way to open a TCP port It\u0026#8217;s a common task for developers to open a TCP port simply for the purpose of bring up a web service and make it accessible to client. If we simply add a new rule to existing list, for example:\n# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9870 -j ACCEPT # iptables -S -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT -A FORWARD -j REJECT --reject-with icmp-host-prohibited # systemctl reload iptables You will notice that the rule is appended to the end of INPUT block, below the INPUT REJECT rule, which will never take effect.\nTo address this, you can use iptables-save and iptables-restore to export, edit to correct order and reload the rule, as illustrated above, instead of using iptables command to modify the rule directly. # iptables-save \u0026gt; /tmp/rule.list # vi /tmp/rule.list # iptables-restore \u0026lt; /tmp/rule.list # iptables -S -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited Alternatively, you could use some advanced iptables command switches to add the new rule to certain line number with \u0026#8211;line-number switch. Here is more information.\nPrevious PostLog shipping through ELK Next PostJava version confusions ","date":"2018-10-12T17:56:00-04:00","permalink":"/2018/10/redhat-firewall-configuration-firewalld-vs-iptables/","title":"Redhat Firewall configuration: from iptables to firewalld"},{"content":"A common devops task is build logging pipeline with ELK stack (Elasticsearch, Logstash, Kibana). Suppose the application is written in Java and currently use log4j\u0026#8217;s RollingFileAppender to generate log files locally. We can use log4j\u0026#8217;s socket appender to write to Logstash, which further pushes the log stream to Elasticsearch cluster. In this model, failure to push a log line results in the following in log4j output:\n2018-09-06 11:05:39,778,ERROR,stderr - [AsyncAppender-Dispatcher-Thread-672] log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed) The logstash log displays the socket exception as well:\n[2018-09-06T10:12:21,935][DEBUG][logstash.inputs.log4j ] Accepted connection {:client=\u0026gt;\u0026#34;192.168.111.56:58118\u0026#34;, :server=\u0026gt;\u0026#34;0.0.0.0:4560\u0026#34;} [2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] filter received {\u0026#34;event\u0026#34;=\u0026gt;{\u0026#34;method\u0026#34;=\u0026gt;\u0026#34;?\u0026#34;, \u0026#34;thread\u0026#34;=\u0026gt;\u0026#34;676774870@qtp-1804103302-6\u0026#34;, \u0026#34;message\u0026#34;=\u0026gt;\u0026#34;Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.\u0026#34;, \u0026#34;priority\u0026#34;=\u0026gt;\u0026#34;DEBUG\u0026#34;, \u0026#34;type\u0026#34;=\u0026gt;\u0026#34;log4j\u0026#34;, \u0026#34;path\u0026#34;=\u0026gt;\u0026#34;org.apache.shiro.session.mgt.DefaultSessionManager\u0026#34;, \u0026#34;@timestamp\u0026#34;=\u0026gt;2018-09-06T15:12:21.950Z, \u0026#34;file\u0026#34;=\u0026gt;\u0026#34;?:?\u0026#34;, \u0026#34;@version\u0026#34;=\u0026gt;\u0026#34;1\u0026#34;, \u0026#34;host\u0026#34;=\u0026gt;\u0026#34;192.168.111.56:58118\u0026#34;, \u0026#34;logger_name\u0026#34;=\u0026gt;\u0026#34;org.apache.shiro.session.mgt.DefaultSessionManager\u0026#34;, \u0026#34;class\u0026#34;=\u0026gt;\u0026#34;?\u0026#34;, \u0026#34;timestamp\u0026#34;=\u0026gt;1536246741950}} [2018-09-06T10:12:21,963][DEBUG][logstash.pipeline ] output received {\u0026#34;event\u0026#34;=\u0026gt;{\u0026#34;method\u0026#34;=\u0026gt;\u0026#34;?\u0026#34;, \u0026#34;thread\u0026#34;=\u0026gt;\u0026#34;676774870@qtp-1804103302-6\u0026#34;, \u0026#34;message\u0026#34;=\u0026gt;\u0026#34;Unable to resolve session ID from SessionKey [org.apache.shiro.web.session.mgt.WebSessionKey@71da9a4]. Returning null to indicate a session could not be found.\u0026#34;, \u0026#34;priority\u0026#34;=\u0026gt;\u0026#34;DEBUG\u0026#34;, \u0026#34;type\u0026#34;=\u0026gt;\u0026#34;log4j\u0026#34;, \u0026#34;path\u0026#34;=\u0026gt;\u0026#34;org.apache.shiro.session.mgt.DefaultSessionManager\u0026#34;, \u0026#34;@timestamp\u0026#34;=\u0026gt;2018-09-06T15:12:21.950Z, \u0026#34;file\u0026#34;=\u0026gt;\u0026#34;?:?\u0026#34;, \u0026#34;@version\u0026#34;=\u0026gt;\u0026#34;1\u0026#34;, \u0026#34;host\u0026#34;=\u0026gt;\u0026#34;192.168.111.56:58118\u0026#34;, \u0026#34;logger_name\u0026#34;=\u0026gt;\u0026#34;org.apache.shiro.session.mgt.DefaultSessionManager\u0026#34;, \u0026#34;class\u0026#34;=\u0026gt;\u0026#34;?\u0026#34;, \u0026#34;timestamp\u0026#34;=\u0026gt;1536246741950}} [2018-09-06T10:12:22,041][DEBUG][logstash.inputs.log4j ] Closing connection {:client=\u0026gt;\u0026#34;192.168.111.56:58118\u0026#34;, :exception=\u0026gt;java.io.InvalidObjectException: Object type java.util.Hashtable is not allowed.} The troubleshooting isn\u0026#8217;t very straightforward. So its alternative is preferred. The alternative is to keep the existing RollingFileAppender as well as the local log files, but use a filebeat agent for each application node. Here is the diagram:\nELK Log Shipping Pipeline Filebeat is a very light agent to be installed with the application on the same server or container. Streams of text information congregate to logstash nodes. These log stash service pushes the converged stream to ElasticSearch clusterElasticSearch cluster ingest the stream for indexingKibana is responsible for viewing. This is an example architecture with many potential variations. For example, log stash service may be deployed on the same hosts with ElasticSearch cluster. If there aren\u0026#8217;t many application nodes, filebeat may directly push its output to ElasticSearch. This architecture provides a lot of flexibility and scalability.\nWith log4j format, two challenges to address are:\nidentifying multi-line entry in the logmapping sections of a log entry into fields in Elastisearch For example, the following log entry reflects both challenges above:\n2018-09-06 11:52:18,022,DEBUG,service.dataportal.web.retrieve - [687285925@qtp-564051174-3350] creating streaming output request 9868594 and requid=fb9a2cad-e24b-4eb8-ad62-b027184e7b9b 2018-09-06 11:52:18,054,ERROR,org.glassfish.jersey.server.ServerRuntime$Responder - [687285925@qtp-564051174-3350] An I/O error has occurred while writing a re sponse message entity to the container output stream. org.glassfish.jersey.server.internal.process.MappableException: org.mortbay.jetty.EofException at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:789) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:568) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1010) at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:330) at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:287) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.flush(WriterInterceptorExecutor.java:305) at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) at org.glassfish.jersey.spi.ContentEncoder.aroundWriteTo(ContentEncoder.java:137) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) ... 40 more Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:122) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:723) ... 58 more To address multi-line entry we need to tell filebeat how to identify the start of a line through its multiline.pattern configuration option:\n- type: log enabled: true paths: - /var/log/dhunch/app.log multiline.pattern: \u0026#39;^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\u0026#39; multiline.negate: true multiline.match: after fields: product: name: dhunchapp log: type: app_log content: diagnostic When filebeat pushes log entries to logstash, there will be an additional \u0026#8220;time\u0026#8221; column added, which reflects the time the log is ingested. The entire log4j message line is all put in a single field called \u0026#8220;message\u0026#8221;, this message field contains the timestamp from application, the thread, class, logging level and the actual diagnostic message. These are not mapped to separate sections making it difficult to search in Elasticsearch.\nTo address section mapping, the configuration is made on the logstash side. We will take advantage of a plugin called grok filter. The function of grok filter here, is to parse the original \u0026#8220;message\u0026#8221; field, map different sections into separate columns, respectively called logtime, loglevel, logclass, logthread and logmsg. The grok filter expression is as below:\nfilter { grok { match =\u0026gt; { \u0026#34;message\u0026#34; =\u0026gt; \u0026#34;%{TIMESTAMP_ISO8601:logtime},%{LOGLEVEL:loglevel},%{NOTSPACE:logclass} - \\[%{DATA:logthread}\\] %{GREEDYDATA:logmsg}\u0026#34; } } date { match =\u0026gt; [ \u0026#34;logtime\u0026#34; , \u0026#34;yyyy-MM-dd HH:mm:ss,SSS\u0026#34; ] timezone =\u0026gt; \u0026#34;America/Chicago\u0026#34; target =\u0026gt; \u0026#34;@timestamp\u0026#34; } mutate { replace =\u0026gt; [ \u0026#34;message\u0026#34; , \u0026#34;%{logmsg}\u0026#34; ] } mutate { remove_field =\u0026gt; [ \u0026#34;logmsg\u0026#34; ] } } The date plugin indicates to Elasticsearch to treat this field as datetime with specified time zone. The mutate plugin below essentially renames the \u0026#8220;logmsg\u0026#8221; column to \u0026#8220;message\u0026#8221;, which allows Elasticsearch to understand this field as log message.\nPrevious PostA review of vSphere virtualization technologies Next PostRedhat Firewall configuration: from iptables to firewalld ","date":"2018-09-19T17:26:51-04:00","permalink":"/2018/09/log-shipping-through-elk/","title":"Log shipping through ELK"},{"content":"This post is a summary of VMware\u0026#8217;s white paper Introduction to VMware vSphere.\nESXi is the hypervisor (virtualization layer) on bare metal servers that abstracts processor, memory, storage and networking resources into multiple virtual machines. It was previously known as ESX and VMware discontinued ESX at version 4.1 so only ESXi is provided at and above version 5.0. vSphere is the platform to view, configure and manage the key aspects of virtualization, including:\ncomputing and memory resources (hosts, clusters and resource pools)storage resources (data stores)networking resources (networks)virtual machines Under vSphere product family, vCenter Server is the central point for configuring, provisioning and managing the virtual environment. vShphere client is a client application to connect remotely to vCenter Server, or ESXi from any Windows PC. There is also vSphere Web Access for users from non-Windows environment.\nFor each aspects of virtualization, there is some vSphere features.\nComputing A host is a virtual representation of of the computing and memory resources of a physical machine running ESXi. When two or more physical machines are grouped to work and be managed as a whole, the aggregate computing and memory resources form a cluster. Physical machines can be dynamically added to or removed from a cluster. A cluster acts and can be managed as a single entity. It represents the aggregate computing and memory resources of a group of physical x86 servers sharing the same network and storage arrays. Computing and memory resources from hosts and clusters can be finely partitioned into a hierarchy of resource pools. You can dynamically change resource allocation policies without shutting down the associated VMs. When reserved resources are not being used by a resource pool or a VM, the resources can be shared. This helps to maximize resource use while also ensuring that reservations are met and resource policies enforced.\nESXi provides a memory compression cache to improve VM performance when you use memory overcommitment. Memory compression is enabled by default. When a hosts memory becomes overcommitted, ESXi compresses virtual pages and stores them in memory. This is because accessing compressed memory is faster than accessing memory that has been swapped out to disk. Memory compression in ESXi allows you to overcommit memory without hindering performance. When a virtual page needs to be swapped, ESXi first attempts to compress the page. Pages that can be compressed to 2KB or smaller are stored in the VM\u0026#8217;s compression cache, increasing the capacity of the host.\nHigh Availability VMware vMotion enables the migration of running VMs from one physical server to another without service interruption. The effect is a more efficient assignment of resources across physical servers. Storage vMotion enables the migration of VMs from one datastore to another datastore without service interruption. This allows administrators to off-load VMs from one storage array to another. VMware DRS (distributed resource scheduler) helps you manage a cluster of physical hosts as a single compute resource. You can configure DRS to execute VM placement, VM migration, and host power actions. When you create a VM on a cluster, DRS places the VM in such a way as to ensure that load across the cluster is balanced, and cluster-wide resource allocation policies (e.g. reservations, priorities, and limits) are enforced. When you add a new physical server to a cluster, DRS enables VMs to immediately take advantage of the new resources. When a VM is powered on, DRS performs an initial placement of the VM on a host. As cluster conditions (e.g. load and available resources) change over time, DRS migrates (using vMotion) VMs to other hosts as necessary.\nVMware DRS When DPM (distributed power management) is enabled, the system compares cluster-level and host-level capacity to the demands of VMs running in the cluster. If the resource demands of the running VMs can be met by a subset of hosts in the cluster, DPM migrates the VMs to this subset and powers down the hosts that are not needed. When resource demands increase, DPM powers these hosts back on and migrates the VMs to them. This dynamic cluster right-sizing that DPM performs reduces the power consumption of the cluster, without sacrificing VM performance or availability.\nStorage I/O control congestion management allows cluster-wide storage I/O prioritization and enables administrator to set congestion thresholds for I/O shares.\nVMware HA enables quick automated restart of virtual machines on a different physical server within a cluster if a host fails. HA monitors all physical hosts in a cluster and detects host failures. An agent placed on each physical host maintains a heartbeat with the other hosts in the resource pool. Loss of a heartbeat initiates the process of restarting all affected VMs on that host. HA also provides a VM monitoring feature that monitors the status of VM in an HA cluster. If a VM does not generate heartbeats within a specified time, VM monitoring identifies it as having failed and restarts it. HA is configured centrally through vCenter Server and once configured, it operates continuously and in a distributed manner on every ESXi host without needing vCenter Server to stay up.\nVMware HA VMware vLockstep technology and VMware Fault Tolerance provides continuous availability by protecting a VM with a shadow copy that runs in virtual lockstep on a separate host. Inputs and events performed on the primary VM are recorded and replayed on the secondary VM to ensure identical state. The secondary VM in virtual lockstep can take over execution at any point without interruption or loss of data.\nStorage Datastores are virtual representations of combinations of underlying physical storage resources in the data center. These physical storage resources include:\nLocal SCSI, SAS, or SATA disks attached to the physical machinesFibre Channel or iSCSI SAN disk arraysNetwork Attached Storage (NAS) arrays Storage subsystem appears as a virtual SCSI controller connected to one or more virtual SCSI disks. These virtual controllers (BusLogic Parallel, LSI Logic Parallel, LSI Logic SAS and VMware Paravirtual) are the only types of SCSI controllers that a VM can see and access. The virtual SCSI disks are provisioned from datastore. This datastore abstraction is a model that assigns storage space to VMs while insulating the guest from the complexity of the underlying physical storage technology. The guest VM however, is not exposed to Fibre Channel SAN, iSCSI SAN, direct attached storage or NAS.\nStorage Architecture Each datastore is a VMFS volume on a storage device. Datastore can span multiple physical storage subsystems. A single VMFS volume can contain one or more LUNs from a local SCSI disk array on a physical host, a Fibre Channel disk farm, or iSCSI SAN disk farm. New LUNs added to any of the physical storage subsystems are detected and made available to all existing new datastores. Storage capacity on a previously created datastore can be extended without powering down physical hosts or storage subsystems. If any of the LUNs within a VMFS volume fails, only VMs that use that LUN are affected.\nEach VM is stored as a set of files in a directory in the datastore. The disk storage associated with each VM is a set of files within the guest\u0026#8217;s directory. You can operate on the guest disk storage as an ordinary file, which can be copied, moved, or backed up. New virtual disks can be added to a virtual machine without powering it down. In that case, a virtual disk file (.vmdk) is created in VMFS to provide new storage for the added virtual disk\nVMFS is a clustered file system that leverages shared storage to allow multiple physical hosts to read and write the same storage simultaneously. VMFS provides on-disk locking to ensure that the same virtual machine is not powered on by multiple servers at the same time. If a physical host fails, the on-disk lock for each VM is released so that VMs can be restarted on other physical hosts. VMFS also features failure consistency and recovery mechanisms, such as distributed journaling, a failure-consisten VM I/O path, and VM state snapshots. These mechanisms can aid quick identification of the cause and recovery from VM, physical host and storage subsystem failures. VMFS also supports raw device mapping (RDM), which is a mechanism for a VM to have direct access to a LUN on the physical storage subsystem (Fibre Channel or iSCSI only). An RDM is a symbolic link from a VMFS volume to a raw LUN. The mapping makes LUNs appear as files in a VMFS volume. The mapping file, not the raw LUN, is referenced in the VM configuration. When a LUN is opened for access, the mapping file is read to obtain the reference to the raw LUN. Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.\nRaw Device Mapping Networking Each VM has one or more vNICs (virtual network interface cards). The guest OS and application program communicate with a vNIC through either a commonly available device driver or a VMware device driver optimized for the virtual environment. In either case, communication in the guest OS occurs just as it would with a physical device. On the network, the vNIC responds to standard Ethernet protocol as would a physical NIC. An outside agent does not detect that it is communicating with a virtual machine.\nA virtual switch (vSwitch) works like a layer 2 physical switch. Each server has its own virtual switches. One one side of the virtual switch are port groups that connect to virtual machines. On the other side are uplink connections to physical Ethernet adapters on the physical server where the virtual switch resides. VMs connect to the outside world through the physical Ethernet adapters that are connected to the virtual switch uplinks. A virtual switch can connect its uplinks to more than one physical Ethernet adapter to enable NIC teaming.\nPort group is a unique concept in the virtual environment. A port group is a mechanism for setting policies that govern the network connected to it. A vSwitch can have multiple port groups. A VM connects its vNIC to a port group instead of to a particular port on the vSwitch, for better network segmentation.\nA vNetwork Distributed Swtich (vDs) function as a single virtual switch across all associated hosts. This functionality allows VMs to maintain consistent network configuration as they migrate across multiple hosts. Like vSwitch, each VDS is a network hub that VMs can use and it can route traffic internally between VMs or link to an external network by connecting to physical Ethernet adapters. Each vDS can also hae one or more dvPort groups assigned to it. dvPort groups aggregate multiple ports under a common configuration and provide a stable anchor point for VMs connecting to labeled networks.\nWhen network resource management is enabled, vDS traffic is divided into six network resource pools: FT traffic, iSCSI traffic, vMotion traffic, management traffic, NFS traffic, and VM traffic. You can control the priority of each of these network resource pools.\nvCenter vCenter Server provides centralized managed for data centers. It communicates with the ESXi host agent through the VMware vSphere API. When you first add a host to vCenter Server sends a vCenter Server agent to run on the host. The vCenter Server agent acts as a small vCenter Server to perform many fundamental management functions.\nPrevious PostDICOM data encoding Next PostLog shipping through ELK ","date":"2018-07-21T15:54:00-04:00","permalink":"/2018/07/overview-of-vsphere/","title":"A review of vSphere virtualization technologies"},{"content":"DICOM is a standard for medical imaging exchanges, originally in radiology, but later expanded into other departments where mass imaging data are acquired, such as cardiology. One part of the DICOM standard defines how to lay out the data without providing any official code implementation. It is up to each vendor to implement their application and declare what parts of DICOM standard they are compliant to in their individual conformance statement. Therefore it is common that different vendors have different perspectives of whether each other\u0026#8217;s implementation is compliant.\nMany arguments of this topic revolves around the encoding of DICOM data (or in strict term, Information Object Definition, IOD). In this article I try to clarify DICOM IOD encoding, best practices from vendor support. A more comprehensive coverage of this topic is chapter 5 of Oleg Pianykh\u0026#8217;s book, which discussed the basics such as implicit vs explicit VR, big vs little endian.\nThe very purpose of standard is to determine a common protocol in which two application communicate with each other. A strong standard leaves no room for ambiguity in implementation and unfortunately, many healthcare IT standards are weak standards. When two devices from two vendors fail to communicate properly, the healthcare provider (device buyer) should take the lead of moderation, because they suffer the most pain from proprietary implementation and they benefit the most from good interoperability. In reality however, healthcare organizations with insufficient technical competency in their information technology team, usually leave it in Vendors\u0026#8217; hands to configure integration, with minimum supervision on standard conformance. This allows vendor to put in technologies that are just made to work, but not fully up to standard. This is not optimal. Remember: Proprietary technology = Vendor Lock-In\nFrom vendor\u0026#8217;s perspective, the implementation should just comply with DICOM standard. They should not accommodate to third party application that are incorrectly implemented. Vendor\u0026#8217;s responsibility with customer is simply to proof that the data encoding is compliant with DICOM; or if otherwise is discovered, escalate to engineering with low level technical detail. It is a courtesy in the discretion of vendor\u0026#8217;s support operation, to investigate and advise on the integrity of externally sourced DICOM data. In reality however, vendors are pressured to just make it work.\nDICOM Objects In DICOM encoding, an IOD, either in a C-Store or a part 10 file consist of hundreds of data elements. A data element (uniquely identified by a tag) can be either:\nA single item;A sequence (SQ) of multiple items; Transfer Syntax Below is the table that summarizes the metadata and pixel data encoding under different transfer syntax UID specified in (0020,0010). It is not meant to be a completed list.\ntransfer syntax UIDtransfer syntax nameMetadata encodingPixel data encoding1.2.840.10008.1.2Implicit VR Endian: Default Transfer Syntax for DICOMImplicit VR Little EndianImplicit VR Little Endian1.2.840.10008.1.2.1Explicit VR Little EndianExplicit VR Little EndianExplicit VR Little Endian1.2.840.10008.1.2.4.70JPEG Lossless, Nonhierarchical, First- Order Prediction\n(Processes 14 [Selection Value 1]):\nDefault Transfer Syntax for Lossless JPEG Image CompressionExplicit VR Little EndianJPEG Lossless Compression1.2.840.10008.1.2.4.80JPEG-LS Lossless Image CompressionExplicit VR Little EndianJPEG-LS Lossless Compression Practically, Big Endian encoding is rarely used in DICOM. So is .99 so they are not covered. Little Endian simply refers to the reverse ordering of each pair of bytes. The rest of this article only discusses metadata encoding.\nEncoding of Data Element of single item Most DICOM parsers out in the market don\u0026#8217;t have a problem with data elements of single items. However it is important to understand the encoding of single item before trying to understand sequence. Regardless of implicit or explicit VR, big or little endian, a\u0026nbsp;single item is always encoded in the following sequence:\nLengthData formatExampleTagGroup number2-byteunsigned integerElement number2-byteunsigned integer0010,0010VR\n(present only for explicit VR)2-byte2 ASCII charactersPNLength of Value2-bytean even integer0x000AValuedetermined by length of valuedetermined by VRSmith^Joe\u0026nbsp; Note that in the example, length of value is 10 in decimal, and the value \u0026#8220;Smith^Joe \u0026#8221; contains a trailing space to make up for 10 character length. It is required by DICOM that the length be even number of characters, which sometimes omitted and tolerated by different implementations. The corresponding DICOM\u0026#8217;s guideline is\u0026nbsp;here. On this page please understand Figure 7.1-1, Table 7.1-1 and 7.1-2 before reading on.\nEncoding of data element with SQ type When it comes to SQ (sequence), there’s much confusion about what are valid options for sequence encoding. There is also a good chance that a third party DICOM interpreter is incompletely implemented, and mistakenly complains correctly-encoded sequence as bad data. Symptoms include, but not limited to, A-ABORT an association, silence a TCP connection, complaining in their logs that the data is “corrupted”.\nHere\u0026nbsp;is the reference to DICOM standard as to the valid options for sequence encoding. The language is fairly abstract and I’m making some addition to elucidate it:\nWhen determining the sequence encoding, DICOM needs to address two problems:Define how to start and end a data item;Define how to start and end the entire sequence;You can explicitly specify the length of a data item, or leave it undefined; Similarly for the entire sequence, you can explicitly specify the length upfront, or leave it undefined. This leads to four possible combinations but one of them is invalid. The following table points to an example of each based on the tables in DICOM document: Sequence Length is explicitSequence Length is undefinedData item length is explicitValid Format A\nExemplified in Table 7.5-1, when sequence length is explicit, length of each individual data item must be explicit as well.\n\u0026#8211; Sequence length is 0F00H, 3840 bytes\n\u0026#8211; Data Item length is 04F8H, 1272 bytes\n\u0026#8211; No delimiter (FFFE,E0DD or FFFE, E00D) is needed for sequence of data element\n\u0026#8211; Sum of unit length equals total length: (1272 + 4 + 4 ) x 3 = 3840\nEven though this particular example is implicit VR, the parser should know this is a sequence by the length calculation\na. the length of data element (sequence) is 0F00H Valid Format B\nExemplified in Table 7.5-2, as well as the first Item in Table 7.5-3\nSequence length is undefined, marked by (FFFF,FFFF) as the length value of data element\nData Item length is explicit defined, as follows:\n\u0026#8211; 98A5 and B321 for the two items in Table 7.5-2\n\u0026#8211; 17B6 for the first item in Table 7.5-3\n\u0026#8211; FFFE,E000 marks the start of a data itemData Item length is undefined This is NOT a valid encoding option.\nIt would be error prone, if the total length is explicitly defined but the unit length is not.\u0026nbsp; Valid Format C\nExemplified in the second Item in Table 7.5-3\n\u0026#8211; Sequence length is undefined, marked by (FFFF,FFFF) as the length value of data element\n\u0026#8211; Data ltem length is also undefined\n\u0026#8211; FFFE,E00D followed by 00000000H marks the end of the data item\n\u0026#8211; FFFE,E0DD followed by 00000000H marks the end of the sequence As shown above, there are multiple valid options (A, B and C) to encode data item in a sequence and the entire sequence. If sequence length is undefined, explicit and undefined data item length can even co-exist within the same sequence. (Table 7.5-3)if the length is left undefined at the beginning, you must clearly mark the end of the data item, or sequence using one of the special data elements. Special Data Element used in SQ encoding:\nFFFE,E000\u0026nbsp;(Data Item) – marks the start of each data item inside of SQ element; it shall be followed by a 4-byte field to indicate the length of the data item (either an explicit value or FFFFFFFFH to indicate undefined length)\nFFFE,E00D\u0026nbsp;(Item Delimitation) – marks the end of each data item\u0026nbsp;only if\u0026nbsp;the length of that data item is undefined; it shall follow the data item immediately and the length of itself shall be set to 00000000H\nFFFE,E0DD\u0026nbsp;(Sequence Delimitation) – marks the end of an entire sequence\u0026nbsp;only if\u0026nbsp;the length of that sequence is undefined; it shall follow the last item of the SQ element and the length of itself shall be set to 00000000H\nPrevious PostLinux Admin Basics 3 of 3 – text processing, regex, sed \u0026amp; awk Next PostA review of vSphere virtualization technologies ","date":"2018-06-30T20:42:36-04:00","permalink":"/2018/06/dicom-data-encoding/","title":"DICOM data encoding"},{"content":"Most of the text processing can be processed by awk and sed. Sed is non-interactive stream editor that allows you to specify all editing instructions in one place and execute them on a single pass through the file. Awk is a pattern-matching programming language.\nUsing sed and awk requires some understanding of regular expressions. Here\u0026#8217;s the basics of regular expression signs:\nsignoperation^matches beginning of line$matches end of line.matches any single character (wildcard)*repeat previous token zero, one or more times.+repeat previous token one or more times?repeat previous token zero or one time[\u0026#8230;]matches any one of the class of characters enclosed between the classes.\n^ as first character reverses the match\n\u0026#8211; is used to ndicate a range of characters()groups regex |either preceding or following regex can be matched{n,m}matches a range of occurrences of the single character that immediately precedes it. {n} will match exactly n occurrences\n{n,} will match at least n occurrences\n{n,m} will match any number of occurrences between n and m Common expressions\nexpinterpretation[^0-9]excluding number[15]00*matches \u0026#8220;10\u0026#8221;, \u0026#8220;50\u0026#8221;, \u0026#8220;100\u0026#8221;, \u0026#8220;500\u0026#8221;, \u0026#8220;1000\u0026#8221;, \u0026#8220;5000\u0026#8221;. Here the first 0 is literal, the second is modified by *, see the table above.*any number (including 0) of any character\u0026lt;.*\u0026gt;any html tags book matches book with preceding and following spaces books* matches books, or book, but not \u0026#8220;book.\u0026#8221; \u0026#8220;book?\u0026#8221; etc book.* matches book, followed by any number of characters, or none followed by a space Note that regular expression comes in several different flavours, which can be confusing and frustrating. This is a good summary. There are DFA (Deterministic Finite Automata) based engines and NFA (Non-Deterministic Finite Automata) based engines:\nNFA based engines can \u0026#8220;go back\u0026#8221; in the regex, used in Perl, Python, vim, sed and GNU grep.DFA based engines cannot \u0026#8220;go back\u0026#8221; in the regex, used in awk and BSD grep. StandardIEEE POSIX\nBREIEEE POSIX\nEREPCREDetailBasic Regular ExpressionsExtended Regular Expressions that add repetition, alternation on top of BREPerl Compatible regular expression.EngineDFADFANFAGNU grepgrep by default, or grep -Gegrep\ngrep -Egrep -PBSD grepgrepegrepGNU sedsedsed -rNA. Just use perlBSD sedsedsed -ENA. Just use perlawkawk The best way to check isn on BSD manual and GNU.\nHere are several examples of sed and awk I came across at work:\nFind and remove duplicate lines:\nawk \u0026#39;!x[$0]++\u0026#39; input_file.txt \u0026gt; output_file.txt Remove white spaces at the beginning and end of each line:\nawk \u0026#39;{$1=$1}1\u0026#39; input_file.txt \u0026gt; output_file.txt Print with multiple dilimiters (;, , , and |)\nawk -F \u0026#39;[;,|]\u0026#39; \u0026#39;{print $1, $3, $5}\u0026#39; Print with calculation between columns\nawk \u0026#39;{res=$1-$2;print res,$0}\u0026#39; Print rows conditionally\nawk \u0026#39;$1\u0026gt;20{print;}\u0026#39; Prefix each line of a file\nawk \u0026#39;$0=\u0026#34;PREFIX|\u0026#34;$0\u0026#39; input.txt \u0026gt; prefix.input.txt Replace string original to new in file\nsed -i \u0026#39;s/original/new/g\u0026#39; file.txt Remove multiple patterns\nsed \u0026#39;s/pattern1\\|pattern2\\|pattern3//g\u0026#39; Delete the first matching pattern only\nsed \u0026#39;s/pattern//\u0026#39; Remove blank lines:\nsed -i \u0026#39;/^$/d\u0026#39; input_file.txt \u0026gt; output_file.txt Copy from line 100 to line 500 of input file to output file\nsed -n 100,500p input.log\u0026gt;output.log Merge every three lines:\nsed \u0026#39;N;N;N; s/\\n/ /g\u0026#39; Previous Postcron and anacron in RedHat Linux (How logrotate works) Next PostDICOM data encoding ","date":"2018-06-06T19:56:00-04:00","permalink":"/2018/06/text-processing-with-linux-bash/","title":"Linux Admin Basics 3 of 3 – text processing, regex, sed \u0026 awk"},{"content":"Cron and anacron We all know cron is a job scheduler. Many admin uses crontab to manage scheduled task. It is also important to know that crontab works at different levels as well, as well as the distinction between cron and anacron. They are similar, but different, managed by different sets of files. Below is a brief description of how cron works from this article.\nAfter Cron starts, it searches its spool area to find and load crontab files into the memory. It additionally checks the /etc/crontab and or /etc/cron.d directories for system crontabs.\nAfter loading the crontabs into memory, Cron checks the loaded crontabs on a minute-by-minute basis, running the events which are due.\nIn addition to this, Cron regularly (every minute) checks if the spool directory’s modtime (modification time) has changed. If so, it checks the modetime of all the loaded crontabs and reloads those which have changed. That’s why we don’t have to restart the daemon when installing a new cron job.\nBasically, in cron, you specify a particular time at which a job will run. These jobs are managed by files in /var/spool/cron/ directory. In this directory, each file is named by the username that owns the crontab file. These files shall not be edited directly by respective users. Instead, they are edited by crontab by each user. To understand the syntax, one can refer to RedHat document for automating system tasks. Note that you can specify periodical jobs here with special syntax. For example, */5 at the minute slot indicates every five minutes.\nRunning cron jobs can be allowed or disallowed for different users. For this purpose, use the /etc/cron.allow and /etc/cron.deny files. If the cron.allow file exists, a user must be listed in it to be allowed to use cron If the cron.allow file does not exist but the cron.deny file does exist, then a user must not be listed in the cron.deny file in order to use cron. If neither of these files exists, only the super user is allowed to use cron.\nIn addition, there is a system-wide crontab file in /etc/crontab, in which you need to not only specify tasks, but also the user to run those tasks. By default, the schedule in this file is empty.\nThe limitation of cron is it assumes the servers is up all the time, if a script misses the schedule while the server is down, it will not be executed when the server comes back up. This is where anacron comes in handy. Although anacron can only be used by superuser, it doesn\u0026#8217;t expect system to be running 24\u0026#215;7. If a job is scheduled at a time system is down, it starts the job when system comes back up. Both cron and anacron are run by systemd service named crond.service. Although they require different packages installed (cronie vs cronie-anacron). They are also managed by different sets of files as explained below:\npackagefile or directorypurposeexample cronie/var/spool/cron/this directory accommodates files that represents cron jobs for each individual users.if a file named digihunch contains a valid line, it means that Linux user digihunch has a scheduled task for the time specified. /etc/crontabThis file keeps system-wide cronjob entries. Each line needs to sepcify users.if a line specifies schedule, user and command, it means that at the scheduled time, that user will execute the command. cronie-anacron/var/spool/anacron/This directory accommodates files such as cron.daily, in which a timestamp is kept to indicate last execution time.if cron.daily in this directory reads 20180418, it indicates last daily execution time stamp is 20180418 /etc/anacrontabThis file tells anacron where in the file system to go for directories for periodical jobs.Example:\n#period in days delay in minutes job-identifier command\n1 5 cron.daily nice run-parts /etc/cron.daily\n7 25 cron.weekly nice run-parts /etc/cron.weekly\n@monthly 45 cron.monthly nice run-parts /etc/cron.monthly\nThe file usually also indicates RANDOM_DELAY and START_HOURS_RANGE /etc/cron.hourly/\n/etc/cron.daily/\n/etc/cron.weekly/\n/etc/cron.monthly/These directories stores script files that anacron needs to execute at different intervals. this is configured in /etc/anacrontabif script logrotate is present in /etc/cron.daily/, it means the script is to be executed daily An anacron example: logrotate Rotating logs is a common task in Linux that can be done by logrotate. To understand how this works, first, make sure cronie-anacron package is installed and crond.service is up. Then examine the /etc/anacrontab file:\n# /etc/anacrontab: configuration file for anacron # See anacron(8) and anacrontab(5) for details. SHELL=/bin/sh PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # the maximal random delay added to the base delay of the jobs RANDOM_DELAY=45 # the jobs will be started during the following hours only START_HOURS_RANGE=3-22 #period in days delay in minutes job-identifier command 1 5 cron.daily nice run-parts /etc/cron.daily 7 25 cron.weekly nice run-parts /etc/cron.weekly @monthly 45 cron.monthly nice run-parts /etc/cron.monthly This indicates that daily, weekly and monthly jobs are active. Go into /etc/cron.daily/, and examine script logrotate:\n#!/bin/sh /usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate \u0026#34;ALERT exited abnormally with [$EXITVALUE]\u0026#34; fi exit 0 This indicates that logrotate loads configuration from /etc/logrotate.conf, the man page of logrotate explains how this configuration works, along with an example. If you have any custom application where the log file needs rotated, it can be configured in this file.\nPrevious PostBasics Terms in Linux OS Next PostLinux Admin Basics 3 of 3 – text processing, regex, sed \u0026amp; awk ","date":"2018-05-15T14:11:00-04:00","permalink":"/2018/05/cron-and-logrotate-in-centos/","title":"cron and anacron in RedHat Linux (How logrotate works)"},{"content":"These are the things quite confusing or abstract while I was at school but now makes lots of sense after many years working with different flavours of OS.\nGPL and BSD as software license types The main difference is that BSD (Berkeley Software Distribution) is a permissive (non-protective) license, while GPL(GNU General Public License) is a copyleft (protective) license.\nPermissive licenses do not protect the code from being used in non-open source apps and apply no restrictions on the derivatives, while copyleft licenses force the creator of derivatives or re-distributor of the software to open the modified code. Under GPL you can\u0026#8217;t sub-license, meaning, you can’t change any of the original license terms or introduce any of your own. You’re also required to state all the changes you make to the original code. That is why components licensed under GPL and other copyleft licenses should be avoided in commercial products that would later be distributed under proprietary licenses.\nThe BSD license family (including the Modified BSD License), on the other hand, doesn’t compel you to do any of the above. They have fairly relaxed redistribution terms.\nGNU/Linux and BSD as operating systems Unix \u0026#8211; the name of the original system designed at AT\u0026amp;T in the 1970s. At the time, it featured a great deal of novelties such as multi-tasking, multi-user support, time sharing, etc. It was made portable by using C language.\nLinux Kernel \u0026#8211; a free, open-source, monolithic, Unix-like OS kernel. It was conceived and created in 1991 by Linus Torvalds for his personal computer. The word Linux, technically, is just the kernel. By itself, Linux (Kernel) has no place for user to land (e.g. no apps, no commands)\nGNU/Linux \u0026#8211; GNU project has developed a comprehensive set of free software tools for use with Unix and Linux. GNU/Linux involves the software tools along with the Linux Kernel.\nGNU/Linux distribution \u0026#8211; ready-to-use full OS, including Linux Kernel, GNU library and tools, whose developers have made a commitment to follow GNU GFSD, X Window and desktop environment (e.g. KDE, GNOME) and includes many pieces of software. GNU/Linux distro is what many people refer to as \u0026#8220;Linux\u0026#8221;, and it includes Debian, Ubuntu, RedHat and CentOS. BSD (Berkeley Software Distribution) \u0026#8211; an OS based on Research Unix, originally developed at Bell Labs, eventually grown into a complete operating system. Today, \u0026#8220;BSD\u0026#8221; often refers to its decendants, such as FreeBSD, OpenBSD, NetBSD, or DragonFly BSD. Each of these are both a kernel and an operating system. Another famous BSD descendant is Darwin, which is what Mac OS X based on.\nComparing \u0026#8220;BSD\u0026#8221; with \u0026#8220;Linux\u0026#8221; \u0026#8211; Linux is more popular and tends to support new hardware sooner. Typical users usually don\u0026#8217;t feel the difference between them. FreeBSD as desktop OS uses the same GNOME, KDE, or Xfce desktop environments that many flavours of Linux use as well. Although you need to install the desktop environment yourself. Another important difference is the licensing model as mentioned above between GPL and BSD. This article is a good reference about all the differences bewteen \u0026#8220;BSD\u0026#8221; and \u0026#8220;Linux\u0026#8221;\nhttps://www.educba.com/linux-vs-bsd/\nSwap, Cache and Buffer Swap \u0026#8211; swap file or swap partition. The primary function is to substitute disk space for RAM memory when real RAM fills up and more space is needed. The kernel uses a memory management program that detects blocks, aka pages, of memory in which the contents have not been used recently. The memory management program swaps enough of these relatively infrequently used pages of memory out to a special partition on the hard drive specifically designated for “paging”, or swapping. This frees up RAM and makes room for more data to be entered into your spreadsheet. Those pages of memory swapped out to the hard drive are tracked by the kernel’s memory management code and can be paged back into RAM if they are needed.\nSwapping moves entire process between main memory and secondary storage; this is the original Unix method and can cause severe performance loss;Paging moves small unites of memory (i.e. pages with 4Kbytes). It is more efficient and was added to BSD In both cases, least recently used memory is moved to secondary storage and back to main memory only when needed again. In Linux, the term swapping is used to refer to paging. Older Unix-style swapping of entire thread and process is no longer supported.\nCPU Cache \u0026#8211; a hardware cache used by CPU to reduce the average time to access data from the main memory. A cache is a smaller, faster memory, located closer to a processor core, which stores copies of the data from frequently used main memory locations. Most CPUs have different independent caches, inclusing insructions and data caches, where the data cache is usually organized as a hierarchy of more cache levels (L1, L2, L3, L4, etc)\nPage Cache (or disk cache) \u0026#8211; kept by the OS in computer\u0026#8217;s main memory and controlled by the computer. The OS keeps a page cached in otherwise unused portions of the main memory, resulting in quicker access to the contents of cached pages and overall performance.\nDisk Buffer \u0026#8211; (or ambiguously called disk cache or cache buffer) the embedded memory in a hard disk drive acting as a buffer between the rest of the computer and the physical hard disk platter that is used for storage. Modern hard disk drives come with 8 to 256 MiB of such memory. Disk buffer is physically distinct from and is used differently from page cache. It is controlled by the microcontroller in the hard disk drive.\nGRUB (GNU GRand Unified Bootloader) \u0026#8211; a boot loader package from GNU project. It is predominantly used for Unix-like systems. Current version is GRUB2\nSystem call \u0026#8211; In computing, a system call is the programmatic way in which a computer program requests a service from the kernel of the operating system it is executed on. This may include hardware-related services (for example, accessing a hard disk drive), creation and execution of new processes, and communication with integral kernel services such as process scheduling. System calls provide an essential interface between a process and the operating system. Below is a list of key system calls:\nSystem CallDescriptionread()read byteswrite()write bytesopen()open a fileclose()close a filefork()create a new processexec()execute a new programconnect()connect to a network hostaccept()accept a network connectionstat()fetch file statisticsioctl()set I/O properties, or other miscellaneous functionsmmap()map a file to the memory address spacebrk()extend the heap pointer strace is the tool to trace system calls and signals in Linux. sysVinit, runit and systemd init \u0026#8211; in Unix-based OS, init is the first process started during booting of OS. Init is a daemon process that continues running until the system is shutdown. It is the direct or indirect ancestor of all other processes and automatically adopts all orphaned processes. Init is started by the kernel during the booting process; a kernel panic will occur if the kernel is unable to start it. Init is typically assigned process identifier 1 and its job is to start other programs that are essential to the operation of your system. All other processes are descended from init.\ninit systems\nLinux has several options as init systems. For example: sysvinit, runit, systemd and upstart. Here is a comparison of commands involved in managing each. http://unix.cafe/wp/en/2017/07/howto-manage-a-service-in-systemd-sysvinit-upstart-runit-and-openrc/\nSysV init \u0026#8211; System initialization process is handled by the init daemon. One of the original daemon is SysVinit, which is a collection of System V-style init programs. init process starts serially. It is a run-once process during the start of the OS. One task starts only after the last task startup was successful and it was loaded in the memory. This often resulted in delayed and long booting time. Runit \u0026#8211; an init scheme for Unix-like operating systems that initializes, supervises, and ends processes throughout the operating system. It is a replacement of sysvinit and features brevity and simplicity.\nSystemd \u0026#8211; A init replacement daemon designed to start process in parallel, implemented in a number of standard distribution – Fedora, OpenSuSE, Arch, RHEL, CentOS, etc. Its flexibility comes with more complexity. It is an event driven init system, that not only starts stuff at boot (hence managing dependencies), but also after that. It also keeps track of many things after boot, such as mounts, availability of services, integration with resource management, etc. Because of that, systemd is also good at logging and monitoring. Systemd allows services to start when:\nthe system bootsa hardware components attaches to the systemother service starteda timer fires To determine which system initialization method your current Linux distribution is using (SysVinit or systemd), simply check process id 1:\n$ ps -p 1\nSoft link and hard link inode \u0026#8211; stores the attributes and disk block locations of a file or directory\nSoft/Symbolic link \u0026#8211; essentially a shortcut to another file. The link itself is a separate file, pointing to the destination file or directory. The inode of the file is different from that of the symbolic link. Deleting the destination file will leave the symbolic link file a \u0026#8220;dangling link\u0026#8221;. Symbolic file may also have different permissions from the destination file or directory.\nHard link \u0026#8211; essentially an alias of a file. The link itself is not a separate file, and the destination can only be a file (no directory). The inode of the link the the same as the file itself. So there is actually no distinction between destination file and link. Both files are equal. If you delete the file, the link continue to work until the number of hard links to the file becomes zero.\nSoft link points to a file by name whereas hard link points by inode number.\nPrevious PostCentOS – Remove Swap safely Next Postcron and anacron in RedHat Linux (How logrotate works) ","date":"2018-04-24T19:16:00-04:00","permalink":"/2018/04/basics-of-linux-os/","title":"Basics Terms in Linux OS"},{"content":"If the default installation has swap on, you will see it from block device list:\n[root@server /]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 500G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 499G 0 part ├─centos-root 253:0 0 50G 0 lvm / ├─centos-swap 253:1 0 7.9G 0 lvm [SWAP] └─centos-home 253:2 0 441.1G 0 lvm /home sr0 11:0 1 1024M 0 rom [root@server /]# free -h total used free shared buff/cache available Mem: 7.6G 217M 7.2G 11M 206M 7.2G Swap: 7.9G 0B 7.9G Many installations require swap to be off for performance reasons (although some advocate turning off swappiness of the application, instead of removing swap partition from operating system, which is a separate topic). This can be turned off by a simple command:\nswapoff -a Then if you run free command after rebooting, you will see the Swap is set to 0G. Great. But lsblk and df commands still shows the space being used by swap. This disk space is not being used at all.\n[root@c7v-vitvcast02 ~]# free -m total used free shared buff/cache available Mem: 7802 237 7380 11 185 7333 Swap: 0 0 0 [root@c7v-vitvcast02 ~]# lvdisplay --- Logical volume --- LV Path /dev/centos/swap LV Name swap VG Name centos LV UUID 7zHtvT-zGYn-sNBc-JRlT-nTfU-O9hN-r07y3J LV Write Access read/write LV Creation host, time localhost, 2018-04-07 10:03:21 -0500 LV Status available # open 0 LV Size \u0026lt;7.88 GiB Current LE 2016 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:1 --- Logical volume --- LV Path /dev/centos/home LV Name home VG Name centos LV UUID 4XMTa7-uR44-qH0u-Oag4-PDyK-Z1BC-3oUSiz LV Write Access read/write LV Creation host, time localhost, 2018-04-07 10:03:22 -0500 LV Status available # open 0 LV Size \u0026lt;441.12 GiB Current LE 112926 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:2 --- Logical volume --- LV Path /dev/centos/root LV Name root VG Name centos LV UUID tkMCM4-cWxV-FaAC-hgJx-r1vS-DiY5-fyZvjt LV Write Access read/write LV Creation host, time localhost, 2018-04-07 10:03:22 -0500 LV Status available # open 1 LV Size 50.00 GiB Current LE 12800 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0 We can then remove the line for swap from /etc/fstab, this will keep the OS from mounting it upon reboot. However, the logical volume for swap is there, as a partition that\u0026#8217;s not exposed.\nIf you need to reclaim space from swap partition, delete the logical volume:\nlvremove /dev/centos/swap Do you really want to remove active logical volume centos/swap? [y/n]: y Logical volume \u0026#34;swap\u0026#34; successfully removed Now, using command lvdisplay or lvs, the swap space is not displayed anymore. So I rebooted the server in the hopes that the swap space is not being presented and everything is hunky-dory. But no\u0026#8230; the server doesn\u0026#8217;t boot up at all. It took a long time at the splash screen only to enter dracut, where /boot isn\u0026#8217;t present.\nOn further reading, this is because the grub file is still referencing /dev/centos/swap somewhere. I need to somehow get to the grub file and fix that before it can reboot again. Specifically, I pressed \u0026#8220;e\u0026#8221; at the menu, which gave me an opportunity to edit grub file. From there I removed section for swap. Then I saved it with \u0026#8220;Ctrl + X\u0026#8221; (as suggested on the screen) so the OS booted into normal mode. Alternatively, I entered single user mode in order to get to the point I can edit the grub file.\nOnce booted, I have to make change to the actual grub file at /boot/grub2/grub.cfg so the change persists. In the file, locate the lines starting with linux16 and remove the section \u0026#8220;rd.lvm.lv=centos/swap\u0026#8221;. There are two appearances of this section on my machine. Then the server will boot just normally.\nIn summary, the clean way to turn off swap on CentOS involves the following steps:\nturn off swapRemove swap mount from /etc/fstab (and umount)remove the logical volume for swapremove reference to swap from grub Out of those steps, 3 and 4 are needed to reclaim swap spaces. They must be done together. If you rebooted the server before step 4. The server will not be able to boot complaining about missing swap.\nPrevious PostCassandra Architecture Next PostBasics Terms in Linux OS ","date":"2018-04-14T16:15:43-04:00","permalink":"/2018/04/centos-remove-swap-safely/","title":"CentOS – Remove Swap safely"},{"content":"Excerpts from Cassandra The Definitive Guide\nGossip and Failure Detection Cassandra uses a gossip protocol that allows each node to keep track of state information about the other nodes in the cluster. The gossiper runs every second on a timer.\nGossip protocols assumes a faulty network, are commonly commonly employed in very large, decentralized network systems, and are often used as an automatic mechanism for replication in distributed databases. When a server node is started, it registers itself with the gossiper to receive endpoint state information. Because Cassandra gossip is used for failure detection, the Gossiper class maintains a list of nodes that are alive and dead.\nOnce per second, the gossiper will choose a random node in the cluster and initialize a gossip session with it. Each round of gossip requires three messages. The gossip initiator sends its chosen friend a GossipDigestSynMessage. When the friend receives this message, it returns a GossipDigestAckMessage. When the initiator receives the ack message from the friend, it sends the friend a GossipDigestAck2Message to complete the round of gossip. When the gossiper determines that another endpoint is dead, it “convicts” that endpoint by marking it as dead in its local list and logging that fact.\u0026nbsp;\nCassandra has robust support for failure detection, as specified by a popular algorithm for distributed computing called Phi Accrual Failure Detection. The traditional failure detection (based on whether heartbeat is received or not) is deemed naive.\u0026nbsp; Accrual failure detection determines suspicion level. Suspicion offers a more fluid and proactive indication of the weaker or stronger possibility of failure based on interpretation (sampling of heartbeats), as opposed to a simple binary assessment.\nAccrual Failure Detectors output a value associated with each process (or node). This value is called Phi. The value is output in a manner that is designed from the ground up to be adaptive in the face of volatile network conditions, so it’s not a binary condition that simply checks whether a server is up or down.\nThe Phi convict threshold in the configuration adjusts the sensitivity of the failure detector. Lower values increase the sensitivity and higher values decrease it, but not in a linear fashion.\nThe Phi value refers to a level of suspicion that a server might be down. Applications such as Cassandra that employ an AFD can specify variable conditions for the Phi value they emit. Cassandra can generally detect a failed node in about 10 seconds using this mechanism.\nSnitches A snitch determines relative host proximity for each node in a cluster, which is used to determine which nodes to read and write from. Snitches gather information about your network topology so that Cassandra can efficiently route requests. The snitch will figure out where nodes are in relation to other nodes. Snitch property can be adjusted (endpoint_snitch in cassandra.yaml)\nRings and Token A cassandra cluster presents itself as a ring. Each node in the ring is assigned one or more ranges of data described by a token, which determines its position in the ring. A token is a 64-bit integer ID used to identify each partition.\nA node claims ownership of the range of values less than or equal to each token and greater than the token of previous node. The node with lowest token owns the range less than or equal to its token and the range greater than the highest token, which is also known as the \u0026#8220;wrapping range\u0026#8221; In this way the token specifies a complete ring.\nData is assigned to nodes by using a hash function to calculate a token for the partition key. This partition key token is compared to the token values for the various nodes to identify the range, and therefore the node that owns the data.\nVirtual Nodes Instead of assigning a single token to a cassandra node, the token range is broken up into multiple smaller ranges, each represented by a vNode. By default a cassandra node will be assigned 256 vnodes (small range of tokens). Vnodes make it easier to maintain a cluster containing heterogeneous machines. Nodes in a cluster with more computing resources available can manage an increased number of vnode (num_tokens property in cassandra.yaml)\nReplication Strategies A node serves as a replica for different ranges of data. If one node goes down, other replicas can respond to queries for that range of data. Cassandra replicates data across nodes in a manner transparent to the user, and the replication factor is the number of nodes in your cluster that will receive copies (replicas) of the same data.\nThe first replica will always be the node that claims the range in which the token falls, but the remainder of the replicas are placed according to the replication strategy (sometimes also referred to as the replica placement strategy). Out of the box, Cassandra provides two primary implementations of this interface (extensions of the abstract class): SimpleStrategy and NetworkTopologyStrategy. They are specified at the time of keyspace creation.\nThe SimpleStrategy places replicas at consecutive nodes around the ring, starting with the node indicated by the partitioner. The NetworkTopologyStrategy allows you to specify a different replication factor for each data center. Within a data center, it allocates replicas to different racks in order to maximize availability.\nConsistency Levels Cassandra provides tuneable consistency levels that allow you to make trade-offs with CAP at a fine-grained level. You specify a consistency level on each read or write query that indicates how much consistency you require.\nFor read queries, the consistency level specifies how many replica nodes must respond to a read request before returning the data. For write operations, the consistency level specifies how many replica nodes must respond for the write to be reported as successful to the client. Because Cassandra is eventually consistent, updates to other replica nodes may continue in the background.\nConsistency levels include:\nresponse from an absolute number of nodes: ONE, TWO or THREE response from the majority of the replica nodes (e.g. replication factor/2+1): QUORUM response from all nodes: ALL ALL and QUORUM are considered strong consistency level. But in general we can consider a cluster of strong consistency if it meets this condition:\u0026nbsp;\nR + W \u0026gt; N\nwhere\nR is read consistency level W is write consistency level N is replication factor Queries and Coordinator Nodes A client may connect to any node in the cluster to initiate a read or write query. This node is known as the coordinator node. The coordinator identifies which nodes are replicas for the data that is being written or read and forwards the queries to them.\nFor a write, the coordinator node contacts all replicas, as determined by the consistency level and replication factor, and considers the write successful when a number of replicas commensurate with the consistency level acknowledge the write.\nFor a read, the coordinator contacts enough replicas to ensure the required consistency level is met, and returns the data to the client.\nMemtables, SSTables and Commit Logs When you perform a write operation, it’s immediately written to a commit log so the write operation is considered successful. If you shut down the database or it crashes unexpectedly, the commit log can ensure that data is not lost. That’s because the next time you start the node, the commit log gets replayed. In fact, that’s the only time the commit log is read; clients never read from it.\nAfter it’s written to the commit log, the value is written to a memory-resident data structure called the memtable. Each memtable contains data for a specific table. In early implementations of Cassandra, memtables were stored on the JVM heap, but\nimprovements starting with the 2.1 release have moved the majority of memtable data to native memory. (check out the memtable_allocation_type property: heap_buffers/offheap_buffers/offheap_objects). This makes Cassandra less susceptible to fluctuations in performance due to Java garbage collection.\nWhen the number of objects stored in the memtable reaches a threshold, the contents of the memtable are flushed to disk in a file called an SSTable. A new memtable is then created. This flushing is a non-blocking operation; multiple memtables may exist for a single table, one current and the rest waiting to be flushed. They typically should not have to wait very long, as the node should flush them very quickly unless it is overloaded.\nEach commit log maintains an internal bit flag to indicate whether it needs flushing. When a write operation is first received, it is written to the commit log and its bit flag is set to 1. There is only one bit flag per table, because only one commit log is ever being written to across the entire server. All writes to all tables will go into the same commit log, so the bit flag indicates whether a particular commit log contains anything that hasn’t been flushed for a particular table. Once the memtable has been properly flushed to disk, the corresponding commit log’s bit flag is set to 0, indicating that the commit log no longer has to maintain that data for durability purposes. Like regular logfiles, commit logs have a configurable rollover threshold, and once this file size threshold is reached, the log will roll over, carrying with it any extant dirty bit flags.\nThe SSTable is a concept borrowed from Google’s Bigtable. Once a memtable is flushed to disk as an SSTable, it is immutable and cannot be changed by the application. Despite the fact that SSTables are compacted, this compaction changes only their on-disk representation; it essentially performs the “merge” step of a mergesort into new files and removes the old files on success.\nCassandra supports the compression of SSTables in order to maximize use of the available storage. This compression is configurable per table. Each SSTable also has an associated Bloom filter, which is used as an additional performance enhancer.\nAll writes are sequential, which is the primary reason that writes perform so well in Cassandra. No reads or seeks of any kind are required for writing a value to Cassandra because all writes are append operations. This makes one key limitation on performance\nthe speed of your disk. Compaction is intended to amortize the reorganization of data, but it uses sequential I/O to do so. So the performance benefit is gained by splitting; the write operation is just an immediate append, and then compaction helps to organize for better future read performance. If Cassandra naively inserted values where they ultimately belonged, writing clients would pay for seeks up front.\nOn reads, Cassandra will read both SSTables and memtables to find data values, as the memtable may contain values that have not yet been flushed to disk.\nCaching Cassandra provides three forms of caching:\nKey cache: stores a map of partition keys to row index entries, facilicating faster read access into SSTables stored on disk. The key cache is stored on the JVM heap, configurable through key_cache_size_in_mb and key_cache_save_period in cassandra.yaml; Row cache: caches entire rows and can greatly speed up read access for frequently accessed rows, at the cost of more memory usage. The row cache is stored in off-heap memory, configurable through row_cache_size_in_mb and row_cache_save_period in cassandra.yaml; counter cache: improve counter performance by reducing lock contention for the most frequently accessed counters. By default, key and counter caching are enabled, while row caching is disabled, as it requires more memory. Cassandra saves its caches to disk periodically in order to warm them up more quickly on a node restart.\nHinted Handoff Hinted handoff mechanism is introduced to cope with the situation where a write request is sent to Cassandra but the replica node where the write belongs is not available. In this situation, the coordinator will create a hint to hang onto this write. Once the coordinator detects via gossip that the intended node is back online, the coordinator node will \u0026#8220;hand off\u0026#8221; to the intended node the \u0026#8220;hint\u0026#8221; regarding the write. Cassandra holds a separate hint for each partition that is to be written.\nThis allows Cassandra to be always available for writes, and generally enables a cluster to sustain the same write load even when some of the nodes are down. It also reduces the time that a failed node will be inconsistent after it does come back online.\nHints do not count as writes for the purposes of consistency level, except for consistency level ANY. Hinted handoff can be configured through properties hinted_handoff_enabled, max_hint_window_in_ms and hinted_handoff_throttle_in_kb, max_hints_delivery_threads and batchlog_replay_throttle_in_kb in cassandra.yaml.\nThere is a practical problem with hinted handoffs (and guaranteed delivery approaches, for that matter): if a node is offline for some time, the hints can build up considerably on other nodes. Then, when the other nodes notice that the failed node has come back online, they tend to flood that node with requests, just at the moment it is most vulnerable (when it is struggling to come back into play after a failure). To address this problem, Cassandra limits the storage of hints to a configurable time window. It is also possible to disable hinted handoff entirely.\nAlthough hinted handoff helps increase Cassandra’s availability, it does not fully replace the need for manual repair to ensure consistency.\nLightweight Transactions\u0026nbsp;\nIf a client is going to read (check existence) and then write a record (only if not existed already). We\u0026#8217;d like to guarantee linearizable consistency. In other words, no other client can come in between our read and write queries with their own modification. Lightweight transaction is a mechanism to support linearizable consistency based on Paxos. Paxos is a consensus algorithm that allows distributed peer nodes to agree on a proposal, without requiring a master to coordinate a transaction. It emerged as alternative to traditional two-phase commit.\nCassandra\u0026#8217;s lightweight transaction are limited to a single partition.\nTombstones When you execute a delete operation, the data is not immediately deleted. Instead, it\u0026#8217;s treated as an update operation that places a tombstone on the record. A tombstone is a deletion marker that is required to suppress older data in SSTables until compaction can run. The per-table setting gc_grace_period is the amount of time that the server will wait to garbage-collect tombstones. Once a tombstones ages over the grace period, they will be garbage-collected.\nBloom Filters Introduced to boost the performance of reads, Bloom filters are very fast, non-deterministic algorithms for testing whether an element is a member of a set. Being deterministic means false-positive is possible but not false-negative. In other words, if the filter indicates the given element exists in the set, cassandra needs to make sure by checking the set (disk); if the filter indicates the given element does not exist in the set, it certainly doesn\u0026#8217;t. Bloom filter is a special kind of cache, stored in memory to improve performance by reducing the need for disk access on key lookups. The accuracy can be increased (to reduce the chance of false positives) by increasing the filter size, at the cost of more memory. This is tunable per table using bloom_filter_fp_chance. Bloom filters are used in other distributed database and caching technologies as well such as Hadoop.\nCompaction A compaction operation in Cassandra is performed in order to merge SSTables. During compaction, the data in SSTables is merged: the keys are merged, columns are combined, tombstones are discarded, and a new index is created. Compaction is the process of freeing up space by merging large accumulated data files.\nThis is roughly analogous to rebuilding a table in the relational world. But the primary difference in Cassandra is that it is intended as a transparent operation that is amortized across the life of the server.\nAnother important function of compaction is to improve performance by reducing the number of required seeks. There are a bounded number of SSTables to inspect to find the column data for a given key. If a key is frequently mutated, it’s very likely that the mutations will all end up in flushed SSTables. Compacting them prevents the database from having to perform a seek to pull the data from each SSTable in order to locate the current value of each column requested in a read request.\nWhen compaction is performed, there is a temporary spike in disk I/O and the size of data on disk while old SSTables are read and new SSTables are being written. Cassandra supports multiple algorithms for compaction via the strategy pattern. The compaction strategy is an option that is set for each table. Strategies include:\nSizeTieredCompactionStrategy (STCS) is the default compaction strategy and is recommended for write-intensive tables; LeveledCompactionStrategy (LCS) is recommended for read-intensive tables; DateTieredCompactionStrategy (DTCS), which is intended for time series or otherwise date-based data. When compaction is performed, there is a temporary spike in disk I/O and the size of data on disk while old SSTables are read and new SSTables are being written.\nRepairs Replica synchronization is supported via two different modes known as read repair and antri-entropy repair.\nRead repair: the synchronization of replicas as data is read. Cassandra reads data from multiple replicas in order to achieve the requested consistency leve, and detects if any replicas have out of date values. If an insufficient number of nodes have the latest value, a read repair is performed to update the out of date replicas, either immediately or in the background. Anti-entropy repair (aka manual repair) is manually initiated operation performed on nodes as part of a regular maintenance process. This is initiated with nodetool repair command, which executes a major compaction. During a major compaction, the server initiates a TreeRequest/TreeResponse conversation to exchange Merkle trees with neighbouring nodes. The Merkel tree is a hash representing the data in that table. If the trees from different nodes don\u0026#8217;t match, they have to be reconciled (repaired) to determine the latest data values they should all be set to. DynamoDB also use Merkle tress for anti-entropy, with a slightly different implementation. Reference:\nPrevious PostLinux Admin Basics 2 of 3 – shell scripting Next PostCentOS – Remove Swap safely ","date":"2018-03-20T18:18:00-04:00","permalink":"/2018/03/cassandra-architecture-summary/","title":"Cassandra Architecture"},{"content":"Bash Options You can set bash option in two ways, with shopt or with set command. For example:\nshopt -s extglob set -o nounset Here is a list of options. I often use \u0026#8220;set -e\u0026#8221; right after shebang to tell the script to exit upon failed command, because the rest of the script will be error-prone after the failed command. I can use \u0026#8220;set +e\u0026#8221; to negate the flag.\nAlso, we can use the\u0026nbsp;unset\u0026nbsp;command to delete the variables during program execution, or the export command to export a variable or function to the environment of all the child processes running in the current shell.\nUser menu and argument processing It is a common task in bash scripting to display user menu or process argument. Both use case structure. Here are the examples:\nUser menu:\n#! /bin/bash select car in BMW TOYOTA TESLA do case $car in BMW) echo \u0026#34;X3\u0026#34;;; TOYOTA) echo \u0026#34;Camry\u0026#34;;; TESLA) echo \u0026#34;Some\u0026#34;;; *) echo \u0026#34;Unknown\u0026#34;;; esac done Argument processiong example:\n#! /usr/bin/bash while getopts \u0026#34;:ht\u0026#34; opt; do case ${opt} in h ) echo \u0026#34;option h\u0026#34; ;; t ) echo \u0026#34;option t\u0026#34; ;; \\? ) echo \u0026#34;Usage: cmd [-h] [-t]\u0026#34; ;; esac done Try to run the script with the following options:\n$ test.sh -t $ test.sh -h $ test.sh -ht $ test.sh -th $ test.sh -a Build-in commands declare, local, let, eval The declare command allow you to assign an attribute to a variable. For example:\n-a indexed array -A associative array -i integer -r read only The local descriptor declare a variable that is only effective in the code block. For example, within the function.\nWith command let, each arg is taken as arithmetic expression:\n$ A=1 $ B=2 $ let sum=$A+$B $ echo \u0026#34;A=$A, B=$B, C=$C\u0026#34; $ A=1, B=2, C=3 With command eval, each arg is taken as a string:\n$ A=1 $ B=2 $ eval sum=$A+$B $ echo \u0026#34;A=$A, B=$B, C=$C\u0026#34; A=1, B=2, C=1+2 Commands trap and inotifywait The trap command that you can use to catch signals from script and execute code. For example:\ntrap \u0026#34;info \u0026#39;caught interrupt, will stop\u0026#39;; exit 2\u0026#34; INT Inotify is a file system monitoring mechansim. The inotifywait can be used in many complex scenarios. It requires inotify-tools package. As an example, the following snippet use inotifywait to montor a directory, and trigger rsync upon changes\n#!/bin/bash DESTHOST=172.17.23.132 DESTHOSTDIR=/www/htdocs/ SRCDIR=/www/htdocs/ inotifywait -mr --timefmt \u0026#39;%d/%m/%y %H:%M\u0026#39; --format \u0026#39;%T %w %f\u0026#39; \\ -e create,delete,modify,attrib $SRCDIR | while read DATE TIME DIR FILE; do $FILECHANGE=${DIR}${FILE} rsync -avze \u0026#39;ssh\u0026#39; $SRCDIR root@${DESTHOST}:${DESTHOSTDIR} \u0026amp;amp;\u0026gt;/dev/null \u0026amp;amp;\u0026amp;amp; \\ echo \u0026#34;At ${TIME} on ${DATE}, file $FILECHANGE was backed up via rsync\u0026#34; \u0026gt;\u0026gt; /var/log/filesync.log done The example above shows how to copy a file to a remote server upon change. There is a simpler alternative to it, a tool called lsyncd, which combines rsync and inotify. lsyncd is a linux package (can be installed with yum) and remains an open source project. The configuration is done in LUA language. Here is a simple example:\nsettings { logfile = \u0026#34;/var/log/lsyncd/lsyncd.log\u0026#34;, statusFile = \u0026#34;/var/log/lsyncd/lsyncd.stat\u0026#34;, statusInterval = 1 } sync{ default.rsyncssh, source=\u0026#34;/etc/dhunch/\u0026#34;, host=\u0026#34;remote-host\u0026#34;, targetdir=\u0026#34;/etc/dhunch/\u0026#34;, delay = 10, exclude={\u0026#39;*.bak\u0026#39;, \u0026#39;*.tmp\u0026#39;, \u0026#39;deploy.log\u0026#39;, \u0026#39;themes/\u0026#39;}, rsync={ checksum=true, times=true, chown=\u0026#34;hunch:hunch\u0026#34;, chmod=\u0026#34;755\u0026#34; } } In this example, we tell lsyncd to use rsyncssh mechanism and it can use hosts defined in openssh configuration (e.g. ~/.ssh/config). For more than one hosts, we can declare the sync section more than once. Full documentation is here.\nPrevious PostTCPdump and Wireshark configuration Next PostCassandra Architecture ","date":"2018-03-18T15:18:00-04:00","permalink":"/2018/03/bash-tricks-continued/","title":"Linux Admin Basics 2 of 3 – shell scripting"},{"content":"This article explains how to troubleshoot TCP packet from Linux (CentOS) and Windows with TCP dump and wireshark. Both are important tools for troubleshooting. If you are troubleshooting a Windows server and have access to it to install Wireshark then there is nothing to worry about. Even if the server to troubleshoot is a Linux one with proper desktop (KDE/GNOME), you may still install the Wireshark UI on it and work from the server. If the server is Linux without any UI, this is where this article is trying to help because you need to run tcpdump on the server and somehow download the capture to your local computer for analysis.\nIf you work off of a MacOS, and need to capture in real time from a Linux server without a desktop (KDE/GNOME), then the best bet is to run tcpdump remotely from the server and pipe the result into Wireshark. This would require root access to the server. Tcpdump will require libpcap and tcpdump packages. Then from MacBook you can run:\n# ssh root@remote-server \u0026#34;tcpdump -w - -s0 -pi eth0 dst port 443 or src port 443\u0026#34;|wireshark -k -i - This will pipe the tcpdump result into Wireshark session in Mac in real time with a delay.\nIf you work off a Windows computer where plink.exe is available, you can run the following command if you know the root password:\nC:\\tools\\plink.exe -l root -pw rootpassword 192.168.117.12 -P 22 \u0026#34;tcpdump -w - -s0 -pi eth0 dst port 9042\u0026#34; |\u0026#34;C:\\Program Files\\Wireshark\\Wireshark.exe\u0026#34; -k -i - Both tricks above assumes that you have direct root log-in to the server, by RSA key or password. It is because running tcpdump requires root access on the server. It is not a good security practice to run tcpdump with a non-root user because it needs to scan the interface.\n-s: snap length in bytes. Setting to 0 is making it use default 65535-i: specify the interface to listen on. e.g. eth0 or ens192-p: no-promiscuous mode. this option asks tcpdump to not put interface in promiscuous mode-w: write the raw packets to file rather than parsing and printing them out. a hyphen indicates standard output here.-Z: drops the privileges of root and changes ownership to the specified user If you do not have direct root login access, but you can log in as a different user and su to root, you may run this once you are on root user:\necho \u0026#34;###Capture Begin: $(date \u0026#39;+%Y %b %d %H:%M:%S\u0026#39;)\u0026#34; \u0026amp;\u0026amp; tcpdump dst port 1524 or src port 1524 -s 0 -i eth0 -w \u0026#34;/tmp/cap.$(date +%Y%m%d_%H%M%S).cap\u0026#34; -Z linuser \u0026amp;\u0026amp; echo \u0026#34;###Capture End: $(date \u0026#39;+%Y %b %d %H:%M:%S\u0026#39;)\u0026#34; \u0026amp;\u0026amp; ls -ltr /tmp/cap*.cap To stop capture, you can use Ctrl-C but make sure that is passed to the server terminal or you will leave a zombie tcpdump process\nPrevious PostLinux Admin Basics 1 of 3 – Bash Next PostLinux Admin Basics 2 of 3 – shell scripting ","date":"2018-02-28T18:30:06-04:00","permalink":"/2018/02/tcpdump-and-wireshark/","title":"TCPdump and Wireshark configuration"},{"content":"This article summarized my time-saving Linux tips, mostly with CentOS environment.\nBash Shortcuts Bash shortcut illustration These shortcuts save a lot of arrow keystrokes. Some of them may require some tweaking to work on MacOS. ctrl \u0026#8211; wdelete wordctrl \u0026#8211; acursor to end of linectrl \u0026#8211; ecursor to beginning of linectrl \u0026#8211; kdelete to end of linectrl \u0026#8211; udelete to beginning of linectrl \u0026#8211; lclear screencd \u0026#8211; change to previous directoryalt \u0026#8211; bmove cursor back by a wordalt \u0026#8211; fmove cursor forward by a wordalt \u0026#8211; .type last parameter of previous command Job Control We need to first understand the following job states:\nActively runningDisplayed in Active Sessionforeground jobYesYessuspended jobNoNobackground jobYesNo These states can be managed by the following shortcut keys:\nctrl \u0026#8211; zsend active job to suspendedctrl \u0026#8211; csend SIGINT to active job to kill itjobslist jobs with idbgbring a job to background. the current shell still \u0026#8220;owns\u0026#8221; the jobfgbring a job to foreground (by id)disownremove job from current shell\u0026#8217;s job table. the job is still running and can be found by ps commandkillkill a job by id. the job is no longer running and won\u0026#8217;t be found by ps command Note 1: running a command with ampersand(\u0026amp;) at the end starts the process and pushes it to the background, so you can continue typing; Note 2: an example is to start vim with a file, ctrl-z to push to background, jobs to view, fg + job id to bring it back Share Screen Different persons may share screen on Linux shell and interact with each other. To do so, everyone need to log on the same server as the same linux user. Then the first person runs:\nscreen -S screen_name Then the second (and third, etc) person runs:\nscreen -x screen_name Now everyone can collaborate by seeing what each other is doing and type at the same time. I/O redirection file descriptorinput or output expressionstdin0to read file: 0\u0026lt; or \u0026lt; for short\nto read a single line: \u0026lt;\u0026lt;\u0026lt;\nto read multiple lines: \u0026lt;\u0026lt;stdout1to overwrite: 1\u0026gt; or \u0026gt; for short\nto append: 1\u0026gt;\u0026gt; or \u0026gt;\u0026gt; for shortstderr2to overwrite: 2\u0026gt;\nto append: 2\u0026gt;\u0026gt; For example, here is how you display all lines between \u0026#8220;ANYWORD\u0026#8221; (aka \u0026#8220;here document\u0026#8221;):\ncat \u0026lt;\u0026lt;ANYWORD paragraph after paragraph ANDWORD In addition, to redirect between commands, pipe (|) is used. You may also redirect stdout into stderr or the other way round. Below are several examples:\ncommand1 | command2redirect stdout of command1 to stdin of command2command3 \u0026gt; /dev/nullget rid of stdout from command3command4 2\u0026gt; file1redirect stderr of command4 to file1command5 | tee file2redirect stdout of command5 to stdin for tee command, which display stdout and write to file2 at the same timecommand6 2\u0026gt;\u0026amp;1redirect command6\u0026#8217;s stderr into stdout. command7 1\u0026gt;\u0026amp;2redirect command7\u0026#8217;s stdout into stderr Note: in the last two examples, the ampersand just indicates the following number is file descriptor instead of file name. For example, in the first command below, stderr is redirected to stdout, and stdout goes to file. The second and third commands are just two forms of shortcut for the first:\ncommand \u0026gt; file 2\u0026gt;\u0026amp;1 command \u0026gt;\u0026amp; file command \u0026amp;\u0026gt; file Here is an advanced example to compare two files from two different servers:\ndiff \u0026lt;(ssh user@host1 \u0026#39;cat /tmp/file1.txt\u0026#39;) \u0026lt;(ssh user@host2 \u0026#39;cat /tmp/file2.txt\u0026#39;) With the knowledge of I/O redirection, let\u0026#8217;s compare the efficiency of the following two commands in dealing with a huge file:\ncat huge.file | mycmd mycmd \u0026lt; huge.file In the first command, cat first causes an I/O read of huge.file, then stdout gets written into pipe buffer. Lastly, the mycmd reads it from its stdin. This is an example of inefficiency. In the second command, huge.file is provided as stdin to my cmd. Only one read is involved as compared with two read and one write in the previous command. The second command is expected to be three times as fast as the first.\nHere is a diagram to illustrate command and I/O redirection:\nCheck out the page for more scenarios. Operators First, let\u0026#8217;s introduce some special shell variable and operators:\nIFSA special variable indicating internal field separator;The ; token just separates commands to run. Use it when you\u0026#8217;d like to combine multiple lines of commands into a single line\u0026amp;\u0026amp;Logical AND operator. When you run command1 \u0026amp;\u0026amp; command2, command2 ONLY runs if command1 returns true (success, or exit status = 0).||Logical OR operator. When you run command1 \u0026amp;\u0026amp; command2, command2 ONLY runs if command1 does not return false (fail, or exit status != 0){}command combination operator. e.g. {command1;command2}()precedence operator Note the use of \u0026amp;\u0026amp; and || are fairly common. \u0026amp;\u0026amp; is used when you want to use the first command to provide a good status for the second. || is used when you want to report error about the first command. For example:\ntest -d \u0026#34;/tmp/newdir\u0026#34; || mkdir -p \u0026#34;/tmp/newdir\u0026#34; test -f \u0026#34;/var/run/app/app.pid\u0026#34; || echo \u0026gt;\u0026amp;2 \u0026#34;ERROR: cannot detect pid\u0026#34; Attributes I\u0026#8217;ve been using chown and chmod to manipulate owners and file permissions against owners, groups and other users. I recently realized another command chattr (change attribute) which controls file attributes, regardless of users or groups. In other words, if you change the attribute of a file to immutable, then even the owner isn\u0026#8217;t able to change it. Its manual has full list of attributes, but common ones are:\na -\u0026gt; Append only i -\u0026gt; Immutable c -\u0026gt; File automatically compressed in kernel. For example, the append only attribute is commonly used for log file to prevent any other users from modifying the file. Sometimes we want to keep system files from being changed as well:\nchattr -i /etc/resolv.conf echo \u0026#34;nameserver 8.8.8.8\u0026#34; \u0026gt; /etc/resolv.conf chattr +i /etc/resolv.conf Whatever attribute that we set with chattr, we can use lsattr to list them. These two commands are in Linux not BSD.\nLoop If you can tell how many iterations, then use for loop. For example, you want to execute a command for each file in a directory:\nfor file in /tmp/*.hl7 do echo \u0026#34;Picking File $file\u0026#34; \u0026gt;\u0026gt; output.txt hl7snd -f \u0026#34;$file\u0026#34; -d destionatiohost:2398 \u0026gt;\u0026gt; output.txt sleep 2 done or you may specify range and steps for a for loop: for i in {1..11..2} {13..23..2} do echo \u0026#34;hostmachine\u0026#34;$i scp local.file \u0026#34;hostmachine\u0026#34;$i:/tmp/ done Note that the way we use step here is only available on Bash 4.x (check version through variable $BASH_VERSION, reference)\nIf it\u0026#8217;s hard to tell how many iterations, then use while loop. Here are some examples\nExample 1. You want to execute the same command multiple times, each with a line in a file as parameter:\nwhile IFS= read -r var do echo $(date +%Y%m%d:%H%M%S) displaying line \u0026#34;$var\u0026#34; \u0026gt;\u0026gt; resend.log sleep 2 done \u0026lt; list.txt Example 2. One step further, if the input file is a CSV format like this:\nacc1,pid1 acc2,pid2 acc3,pid3 And you want to repeat a command multiple times, each execution with a column 1 as parameter 1, column 2 as parameter 2, etc. Then you can run something like this:\nIFS=,$(echo -en \u0026#34;\\r\\n\\b\u0026#34;); while read -r f1 f2; do echo processing $f1, $f2 curl -i -X PUT -H \u0026#34;Content-Type:application/json\u0026#34; -d \u0026#39;{\u0026#34;userId\u0026#34;:\u0026#34;dhunch\u0026#34;}\u0026#39; --url http://localhost:9876/monitoring/document/$f1/$f2/publish; done \u0026lt; input.txt \u0026gt;\u0026gt;out.txt In this example IFS assignment is done before while loop. So it will take effect outside of while loop. This is why \\r, \\n and \\b are specified in addition to comma. If this is not something you like, you may put IFS assignment after while and only specify comma as field separator.\nExample 3. If you just want to repeat the same command every 5 seconds, apart from watch command, you can leverage while loop doing something like this:\nwhile true; do df -Ph | grep \u0026#34;sda1\u0026#34;; sleep 10; done In fact in all these examples while loop can be expressed in a single line, with semicolon to indicate where you would have typed enter.\nfind, with xargs and -exec {} The find command offers flexibility searching files with certain conditions.\nExample 1. Find files from within last 5 * 24 hours in current directory:\nfind -maxdepth 1 -type f -mtime -5 Example 2. Find tar files older than 6 * 24 hours:\nfind . -type f -name \u0026#34;*.tar\u0026#34; -mtime +6 Notes:\nthe switch -iname is similiar to -name but case insensitive the switch -mtime goes by modified datetime; -atime goes by access time the switch -mmin measures in minutes the swtich -daystart makes it measure time from the beginning of current day (instead of 24 hours from current time of current day) Here are more information about find command, and more examples.\nTo execute command per result from find, we have the options of xargs and -exec {}. Both build command based on parameter input, instead of I/O redirection. xargs is considered more efficient and it also works with commands other than find. -exec {} only works with find command. Here\u0026#8217;s a basic example of -exec {}\nfind . -iname \u0026#39;*.dcm\u0026#39; -exec dcm2txt {} \\; On the other hand, xargs works with any command followed by a pipe. It takes input from previous command, split it by space or carriage return into a list, then build command with each item on the list. To help understand parameter passing with xargs, we examine two examples. The first example is a directory with three files in it: x.a, y.a, z.a\n$ ls x.a x.b x.c If we want to prefix each file with pre, we can do the following:\nls | xargs -I aa echo \u0026#34;mv aa prefix_aa\u0026#34; In the second example, suppose find command produces the following result:\nfirstdir seconddir Compare the following two commands:\nfind . -type d | xargs ls -l find . -type d | xargs -n 1 ls -l\tIn the first command, xargs invokes \u0026#8220;ls -l firstdir seconddir\u0026#8221; whereas in the second command, xargs invokes \u0026#8220;ls -l firstdir\u0026#8221; and then \u0026#8220;ls -l seconddir\u0026#8221;. The first command requires that the utility takes multiple parameters (in this case, ls does. Other commands such as wc, grep also do). The second command is particular helpful when the utility only takes one parameter. This is because the switch -n sets the maximum number of arguments taken from standard input for each invocation of utility.\nLast but not least, operators are quite useful in Bash scripting. Here is a good reference.\nNext PostTCPdump and Wireshark configuration\n","date":"2018-02-02T21:35:00-04:00","permalink":"/2018/02/linux-tips-and-tricks-in-shell/","title":"Linux Admin Basics 1 of 3 – Bash"},{"content":" On the Journey of an IT Architect, we share the expertise by:\nBlogs on this website and medium; Open source projects on GitHub; Professional IT consulting service History of this website:\nMay 2019: Moved to self-hosted WordPress May 2020: Moved to current domain name Dec 2022: Migration for PHP 8 upgrade Jul 2023: Security Enhancements Apr 2025: Performance Enhancement Privacy Thank you for choosing Hunch Digital Services (Digi Hunch). At Hunch Digital Services, we value your privacy and are committed to protecting your personal information. We collect the following information:\nPersonal Information: We may collect personal information such as your name, email address, phone number, and other contact details when you create an account or contact us. Log Data: When you use our services, we may collect information that your browser or device sends to us, including your IP address, browser type, and the pages you visit. Cookies: We may use cookies and similar tracking technologies to enhance your experience on our website. You can manage cookie preferences through your browser settings. We take reasonable measures to protect your information from unauthorized access, disclosure, alteration, or destruction. However, no method of data transmission over the internet or electronic storage is completely secure, and we cannot guarantee its absolute security.\nYou have the right to access, correct, or delete your personal information, subject to legal requirements. You may also unsubscribe from our promotional communications at any time. To exercise these rights or for any other privacy-related inquiries, please contact us at privacy@digihunch.com\n","date":"2010-11-13T03:29:50Z","permalink":"/about-digi-hunch/","title":"About"}]