iscsi on Digi Hunch

Kubernetes Storage Explained – from in-tree plugin to CSI

Sat, 12 Jun 2021 21:55:46 -0400

To support a variety of storage backend, Kubernetes abstract storage issues with several objects (volume, persistent volume, persistent volume claim, storage class) and adopts container storage interface. Unfortunately, the documents are not very well organized to deliver the idea of these concepts, most likely because features are introduced at very different times. Hence this article. At the bottom of this article, I also go through five examples of using volumes in different ways, taking azure disk (SSD as an example).

The first to think about is whether we need just ephemeral storage or persistent storage. Generic volume with ephemeral storage lives and dies with the Pod and we don’t really care where it is from. With persistent storage, we need to consider where it is from and how to create (provision) the storage. The storage can be created statically or dynamically.

PersistentVolume (PV) and PersistentVolumeClaim (PVC)

Regardlessly of static or dynamic storage provision, we first need to understand two objects before getting to that: Persistent Volume (PV) and Persistent Volume Claim (PVC).

We use PV object to represent external storage volume. A single external storage volume can be represented by a single PV. So PV goes with external volumes in 1 to 1 relationship. A 100G volume cannot be represented by two PVs each with 50G, unless the storage administrator divides it into two separate volumes, each with 50G.
PVC goes with Pod in 1 to 1 relationship. The Pods needs a PVC in order to claim ownership of a PV. A valid PVC allows a Pod to mount a PV as its volume.
Here we call storage volume external in relative to the pods. If the storage volume is mapped to a directory on the host file system, it is still considered an external storage.
A single PV can link to multiple PVCs, so long as the total request in PVCs does not exceed PV’s capacity. So PV and PVC are in 1 to many relationship.
How PVC binds to PV is defined by Access Mode, with three options. Note that the options are effective for the entire PV. You cannot have different options for each PVC linked to a PV:
- RWO (ReadWriteOnce): allowing the PV to be bound to a single PVC (for read write). This mode is typically used in block storage;
- RWM (ReadWriteMany): allowing the PV to be bound to multiple PVCs (for read write). This mode is only supported by file (e.g. NFS) and object storage;
- ROM (ReadOnlyMany): allowing the PV bound to multiple PVCs for read only.
When a PVC is released, what to do with the PV is defined as persistentVolumeReclaimPolicy, and the two options (effective at PV level) are:
- Delete
- Retain

Static Provisioning and Dynamic Provisioning

With static provisioning, the external storage volume must be pre-created. In this context, a PV object represents a pre-created external storage volume. So PVs must be explicit declared. The K8s literature also refers to such PVs as pre-created PV.

With dynamic provisioning, the external storage volume is provisioned dynamically. Therefore, you do not need to explicitly create PVs. By the same token, access mode does not apply.

Instead of PV, now we need to explicitly declare storage class, which specifies how to dynamically provision PVs, with the following properties:

volumeBindingMode defines when the binding and provisioning of a PersistentVolume occurs, with two options:
- Immediate (default)
- WaitForFirstConsumer (recommended): delays until a Pod using the PVC is created
reclaimPolicy (the equivalent of persistentVolumeReclaimPolicy for pre-created PV) with two options:
- Delete (default)
- Retain
provisioners: determines what volume plugin is used for provisioning PVs. There are two categories:
- Internal provisioner (prefixed with kubernetes.io): common ones are listed here. Note that there isn’t an internal provisioner for NFS any more. External NFS provisioner is needed.
- External provisioner: third-party out-of-tree plugins compliant to CSI. For example: Dell XtremIO CSI plugin, Dell Isilon plugin, PureStorage CSI driver, Scality Artesca (launched in Apr 2021), and NetApp Trident CSI drivers, and NFS subdir provisioner in Kubernetes-sigs repo.
parameters: each provisioner has its own set of mandatory and optional parameters;
allowVolumeExpansion: can be set to true if the underlying storage class supports volume expansion;
mountOptions: specify only if the storage class supports it;

With the information above, we can simplify the rules as follows:

In static provisioning, PV needs to be declared explicitly and SC is not needed
In dynamic provisioning, SC is required so we can specify provisioner and the parameters needed by the provisioner. PV doesn’t need to be explicitly declared, even though it exists in the interaction.

In real life however, you might come across the following edge cases which seems to contradict with the two generic rules above:

Local volume, currently does not support dynamic provisioning. However a StorageClass should still be created to delay volume binding until Pod scheduling. The volume binding mode WaitForFirstConsumer should be specified.
In dynamic provisioning, if a PVC does not explicitly define PVC, the administrator should have specified a default StorageClass in place for the cluster. You might also come across PVC with empty string (“”) as storageClassName, which indicates that no storage class will be used (i.e. dynamic provisioning is disabled for the PVC). According to this post, in a PVC:
- If storageClassName=””, then it is static provisioning
- If storageClassName is not specified, then the default storage class will be used.
- If storageClassName is set to a specific value, then the matching storageClassName will be considered. If no corresponding storage class exists, the PVC will fail.

The confusing “Volumes”

We’ve discussed PersistentVolume, which is a K8s object that represents an external storage volume. When the word Volume stands by itself, it generally refers to the part of storage exposed to the Kubernetes cluster, no matter what type of storage it is or where it comes from. We can distinguish them in the following table:

	Generic Volumes	Persistent Volumes
Pod assignment	Bound to a single pod, declared as part of a Pod.	A standalone resource type decoupled from Pod and can be bound to single, or multiple Pods via PVC
Lifecycle	Volume is deleted as the owner Pod dies. Data on the volume may or may not persist.	Assuming PVC is gone with Pod, the PV persists. Data on PV may or may not persist depending on ReclaimPolicy.
Configuration	Pod creator (e.g. app developer) needs to know the details of storage resource in the cloud environment. (e.g volume ID)	Pod creator does not need the details of storage resource in the cloud environment. K8s Cluster administrator can provision PV, either statically or dynamically for Pod creator.

If you want to use PeristentVolume to back a Volume in Pod, you’d have to use PersistentVolumeClaim. This means, some types of volumes (including hostPath) can be both mounted as a persistent volume as well as a regular volume. To compare the two ways of mount volume (direct vs via PVC), we take a look at the Kubernetes configuration examples for Azure Disk. The examples are provided at the bottom of this post.

Note that, no matter which method of using the volumes, some types of volumes just work natively, and some requires plugin to operate. The table below summarizes the mechanism behind common volume types.

Volume Types	Mechanism	Mountable as non-persistent volume	mountable as persistent volume (through PVC or SC)
emptyDir	A native volume type, for temporary data only. Data is wiped along with volume. The storage media is determined by the medium of the filsystem holding the kubelet root dir (typically /var/lib/kubelet). You can even set emptyDir.medium to “Memory”	YES	NO. By definition, emptyDir is not persistent.
ConfigMap, Secret	Native volume type to store non-sensitive or sensitive configuration data. ConfigMap and Secrets are stored in etcd.	YES	NO. However, by nature, ConfigMap and Secret are stored persistently. There is no need to mount them as PV.
HostPath	A native volume type to mount a file or directory from the host node’s filesystem into the Pod. In addition to path property, you may optionally specify a type for a hostPath volume (e.g. DirectoryOrCreate, Directory, FileOrCreate, etc). Note that there is also a type named empty string (“”) which is the default value. It means means that no checks will be performed before mounting the hostPath volume. In addition to the caveat with using hostPath from the documentation, we also need to understand that: 1. HostPath gives Pod the ability to maliciously modify files on the host system, or simply fill up the host file system; 2. As the document suggests, you may end up with multiple Pods trying to write simultaneously to a host path.	YES. Read this.	YES. Check out PersistentVolumes typed hostPath
Local	It represents a mounted local storage device such as a disk, partition, or directory. Compared to hostPath volumes, local volumes are used in a durable and portable manner, without manually scheduling pods to nodes. The system is aware of the volume’s node constraints by looking at the node affinity on the PV. You must set nodeAffinity on the PV when using local volumes. This also means local volumes are subject to the availability of the underlying node. Refer to this post. This is also referred to as Local persistent Volume.	NO	YES. Static provisioning only.
CephFS, NFS, GlusterFS, Ginder, RBD, FC, iSCSI……	These volume types are backed by legacy in-tree plugins. They are used to connect to external storage in self-hosted clusters.	YES	YES
awsElasticBlockStore, AzureDisk, AzureFile, GCEPersistentDisk	These volume types are backed by legacy in-tree plugins. They are used to connect to external storage in public cloud	YES	YES

Note that the table above does not list PersistenVolumeClaim as a volume type, because it obviously only support being mounted as persistent volume.

From in-tree plugins to out-of-tree CSI plugins

In the table above, the bottom two rows involves in-tree plugins (aka built-in plugins). In-tree means the volume plugins are built in the Kubernetes code repository. They were built, linked, compiled, and shipped with the core Kubernetes binaries. There has been 20+ in-tree plugins. The problems of this plugin development model are:

These in-tree plugins introduces risk to the stability of Kubernetes itself;
The maintenance and upgrade of plugin is tightly coupled with Kubernetes release
The Kubernetes community carries the burden of maintaining plugins for all storage backends.
Plugin developers have to open-source all their volume plugin code.

The Kubernetes community seeks better alternatives, and has stopped accepting any more in-tree plugins since GA 1.8. The first alternative paradigm for shipping storage plugin, is flexVolume, which existed since version 1.2. However, flexVolume is still not good enough. For example, some packages like Ceph requires dependency package (ceph-common), and the deployment of plugin requires elevated access to the worker node. For that reason, the community later shifted to the Container Storage Interface (CSI) paradigm. A CSI-compliant plugin allows the storage resource to be surfaced as volumes (be it persistent or not) in Kubernetes cluster. More details in this post and here is a list of supported CSI-compliant drivers.

Back to our azure disk example, this page provides examples for both dynamic and static provisioning.

CSI-compliant plugin development is more complicate but it offloads it the driver developer. The community hopes users to shift to CSI so the 20+ grandfathered in-tree plugins can eventually be phased out. With that as the goal, there are several types of volumes with the name “CSI migration”, allowing users to migrate from in-tree volume plugins to CSI-based plugins.

All the CSI-based plugins are fairly recent. As of today, the document outlines three ways to use CSI volume in a Pod:

through a reference to a PersistentVolumeClaim (examples 4 and 5 below)
with a generic ephemeral volume (alpha feature)
with a CSI ephemeral volume if the driver supports that (beta feature)

Examples

We’ll go over five examples, as listed in the able below. Note that out of all the combinations, you cannot mount a csi-based plugin as a volume. No such volume type supported by CSI exist.

Plug-in mechanism	Mount method	Example
In-tree legacy volume plug-in	as volume	#1. using azureDisk property of Volume
	as PV (static)	#2. using azureDisk property of PersistentVolume
	as PV (dynamic)	#3. using kubernetes.io/azure-disk as provisioner for SC
Out-of-tree CSI volume plugin	as volume	This mode does not exist. Example is not available
	as PV (static)	#4. using disk.csi.azure.com as csi driver of PV
	as PV (dynamic)	#5 using disk.csi.azure.com as provisioner for SC

Now, let’s take a look at the example code snippet. Some examples are from Azure documentation. Some are from the azure-disk-csi-driver repository. I’ve made minor modifications for conciseness.

Example 1 uses legacy in-tree plugin, and directly mount the volume. The example code is in Kubernetes repo.

apiVersion: v1
kind: Pod
metadata:
 name: mypod
spec:
 containers:
 - image: kubernetes/pause
 name: mypod
 volumeMounts:
 - name: azure
 mountPath: /mnt/azure
 volumes:
 - name: azure
 azureDisk:
 kind: Managed
 diskName: myAKSDisk
 diskURI: /subscriptions/&lt;subscriptionID>/resourceGroups/MC_myAKSCluster_myAKSCluster_eastus/providers/Microsoft.Compute/disks/myAKSDisk

Example 2 uses legacy in-tree plugin, and mount the PV statically via PVC. No storage class is used (as indicated by empty string in storage class property)

apiVersion: v1
kind: PersistentVolume
metadata:
 name: azure-disk-pv
spec:
 capacity:
 storage: 2Gi
 storageClassName: ""
 volumeMode: Filesystem
 accessModes:
 - ReadWriteOnce
 azureDisk:
 kind: Managed
 diskName: &lt;enter-disk-name>
 diskURI: &lt;enter-disk-resource-id>
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: azure-disk-pvc
spec:
 storageClassName: ""
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 2Gi
---
apiVersion: apps/v1
kind: Pod
metadata:
 name: logz-deployment
spec:
 containers:
 - name: pause
 image: kubernetes/pause
 volumeMounts:
 - name: azure-disk-vol
 mountPath: /mnt/logs
 volumes:
 - name: azure-disk-vol
 persistentVolumeClaim:
 claimName: azure-disk-pvc

Example 3 uses legacy in-tree plugin, and mount the PV dynamically and implicitly via SC. Note that Azure AKS will create several SCs for you by default so use existing ones whenever available.

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: managed-premium
parameters:
 cachingmode: ReadOnly
 kind: Managed
 storageaccounttype: Premium_LRS
provisioner: kubernetes.io/azure-disk
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: azure-managed-disk
spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: managed-premium
 resources:
 requests:
 storage: 5Gi
---
kind: Pod
apiVersion: v1
metadata:
 name: mypod
spec:
 containers:
 - name: mypod
 image: kubernetes/pause
 volumeMounts:
 - mountPath: "/mnt/azure"
 name: volume
 volumes:
 - name: volume
 persistentVolumeClaim:
 claimName: azure-managed-disk

Example 4 uses CSI-based plugin, and mount the PV statically via PVC

---
apiVersion: v1
kind: PersistentVolume
metadata:
 name: pv-azuredisk
spec:
 capacity:
 storage: 10Gi
 accessModes:
 - ReadWriteOnce
 persistentVolumeReclaimPolicy: Retain
 csi:
 driver: disk.csi.azure.com
 readOnly: false
 volumeHandle: /subscriptions/{sub-id}/resourcegroups/{group-name}/providers/microsoft.compute/disks/{disk-id}
 volumeAttributes:
 fsType: ext4
 partition: "1" # optional, remove this if there is no partition
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: pvc-azuredisk
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 10Gi
 volumeName: pv-azuredisk
 storageClassName: ""
---
kind: Pod
apiVersion: v1
metadata:
 name: nginx-azuredisk
spec:
 nodeSelector:
 kubernetes.io/os: linux
 containers:
 - image: kubernetes/pause
 name: mypod
 volumeMounts:
 - name: azuredisk01
 mountPath: "/mnt/azuredisk"
 volumes:
 - name: azuredisk01
 persistentVolumeClaim:
 claimName: pvc-azuredisk

Example 5 uses CSI-based plugin, and mount the PV dynamically and implicitly via SC

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: azuredisk-csi-waitforfirstconsumer
provisioner: disk.csi.azure.com
parameters:
 skuname: StandardSSD_LRS 
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: pvc-azuredisk
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 10Gi
 storageClassName: managed-csi
---
kind: Pod
apiVersion: v1
metadata:
 name: nginx-azuredisk
spec:
 nodeSelector:
 kubernetes.io/os: linux
 containers:
 - image: kubernetes/pause
 name: mypod
 volumeMounts:
 - name: azuredisk01
 mountPath: "/mnt/azuredisk"
 volumes:
 - name: azuredisk01
 persistentVolumeClaim:
 claimName: pvc-azuredisk

Bottomline

As of June 2021, the CSI support is still new. Generally, if a CSI-based plugin is available and in GA, you should consider using it. If you have existing legacy volume types using in-tree plugin, you should consider migration, and create a migration plan. Also, try to avoid the use case of mounting as generic volume (without PVC) because it is rare and not supported with CSI drivers. Without PVC, it also cannot take advantage of the volumeClaimTemplates property in StatefulSet object.

Storage Nitty-Gritty 2 of 5 – SAN

Mon, 06 May 2019 22:06:02 -0400

In direct attached storage (DAS), storage is server centric and the host owns the storage. The storage is fully dedicated to the server that owns it.

With DAS, storage is server-centric

Storage has evolved into information centric model. In this model, when a new server is deployed in the environment, storage is assigned from the same shared storage pool to the new server.

Network based centralized storage solution

A network-based storage solution is the centralized storage pool. No single host owns the entire storage pool. The storage solution consists of two categories based on the interface with the host: SAN (storage area network) and NAS (network attached storage). To a client OS on the host, SAN typically appears as a local disk, allowing block-level access from the client OS, and therefore is more suited for structured workload such as database storage. It operates on its own storage network independent of the host network. NAS on the other hand, typically appears as a file share to the client OS, identified by an IP address and path. This is because NAS operates on the same TCP/IP network where the hosts are operated on. The client has file level access to storage, therefore NAS is better for unstructured data such as video and medical images. It is very important to understand the difference between SAN and NAS.

SAN deployment consists of two categories based on the connection technology. FC SAN is based on Fibre Channel network; and IP SAN is based on Internet protocol (iSCSI, FCIP, FCoE).

Fibre Channel SAN

Cable types: MMF (multimode fibre, usually for short distance within data centre because of signal attenuation due to modal dispersion) and SMF (single mode fibre, carries a single ray of light, used for long-distance cable runs;
Connector: standard connector (SC), lucent connector (LC) and straight tip connector (ST);
Interconnect device
- FC hub – for FC-AL implementation, but no longer in use
- FC switch – directly route data from one physical port to another (more intelligent than hub)
- Director – high end switches with redundant components to provide high availability
FC connectivity
- point-to-point: two devices connected directly to each other;
- arbitrated loop (FC-AL): devices are attached to a shared loop;

FC- AL (rarely used today)

switched fabric (FC-SW): uses switches that can switch data traffic between nodes directly through switch ports. Frames are routed between source and destination by the fabric

Fibre Channel Switched Fabric

Protocol: Fibre Channel Protocol (FCP): defines protocol stack (five layers, FC-0 through FC-4), addressing, identification (world wide name), frame, data structure, flow control, fabric services

Fibre Channel Protocol stack

FC frame

Topology: mesh topology and core-edge fabric topology
Block-level virtualization: aggregates block storage devices (LUNs) and enables provisioning of virtual storage volumes, independent of underlying physical storage. The virtualization layer maps the virtual volumes to the LUNs on the individual arrays. Block-level storage virtualization not only enables extending the storage volumes online; it consolidates heterogeneous storage arrays and enables transparent volume access. It also provides the advantage of non-disruptive data migration, where the virtualization layer handles the back-end migration of data, which enables the LUNs to remain online during migration.

Block-level virtualization (classic)

Federation of block storage across data centers (new generation)

Virtual SAN (VSAN, aka virtual fabric) – a logical fabric on an FC SAN, which enables communication among a group of nodes regardless of physical location in the fabric.

IP SAN

iSCI (one of the IP SAN protocols) – an IP based protocol that establishes and manages connections between host and storage over IP. iSCSI encapsulates SCSI commands and data into an IP packet and transport them using TCP/IP. It is relatively inexpensive and easy to implement so widespread in environments without FC SAN.

Topology
- Native connectivity (without FC components)
- Bridged connectivity (including FC components in the configuration)
- Combined connectivity (most common because a storage array usually comes with both FC and iSCSI ports)

iSCSI topologies

Protocol
- stack: SCSI is the command protocol that works at the application layer of OSI model; iSCSI is session-layer protocol that initiates a reliable session between devices that recognize SCSI commands and TCP/IP. The iSCSI session-layer interface is responsible for handling login, authentication, target discovery, and session management.

iSCSI protocol stack

iSCSI session and PDU encapsulation

PDU encapsulation

iSCSI discovery – an initiator must discover the location of its targets on the network and the names of the targets available to it before session establishment. Two types are SendTargets discovery and internet Storage Name Service
iSCSI names:
- IQN, iSCSI Qualified Name such as iqn.2008-02.com.example:optional_string;
- EUI, extended unique identifier such as eui.0300732A32598D26
iSCSI command sequencing – A command sequence may generate multiple PDUs. A command sequence number (CmdSN) within an iSCSI session is used for numbering all initiator-to-target command PDUs belonging to the session. This number ensures that every command is delivered in the same order in which it is transmitted, regardless of the TCP connection that carries the command in the session.

FCIP (one of the IP SAN protocols) – transporting FC block data over the IP infrastructure.

protocol stack and packet encapsulation

FCIP protocol stack

FCIP encapsulation

Topology (FCIP gateway involved):

FCIP topology

FCoE (one of the IP SAN protocols) – consolidation of LAN and SAN traffic over a single physical interface infrastructure. FCoE helps organizations address the challenges of having multiple discrete network infrastructures.

CNA (converged network adapters) replaces both HBAs and NICs in the server and consolidates both the IP and FC traffic
Special requirement on cables and switches
protocol stack and encapsulation

FCoE field mapping

CEE (converged enhanced Ethernet, or lossless Ethernet) provides new specification to existing Ethernet standard that eliminates the lossy nature of Ethernet. This makes 10Gb Ethernet a viable storage networking option, similar to FC. It features the following functionalities as part of IEEE 802.1:
- PFC (priority-based flow control)
- ETS (enhanced transmission selection)
- CN (congestion notification)

Storage Nitty-Gritty 1 of 5 – Disk and RAID

Sat, 09 Mar 2019 22:25:52 -0500

On my trip I have been through several pre-sales discussions on storage. Therefore I’m taking this opportunities to write up a series of postings deep diving into storage technologies. In this first section, we lay out the foundation of storage technology, from physical device to RAID, focusing on the concepts. Some contents are excerpts from Information Storage and Management.

Volume Manager – In early days, a file system occupies the entire disk drive, and presents continuous disk blocks directly to operating systems. Logical Volume Manager (LVM) was then introduced to bring a layer of abstraction (logical volume) on top of disks. The layers are shown as follows:

With all these layers, a byte in user file maps to disk sectors through several layers:

Mapping from user file to physical disk

File System – a hierarchical structure of files. It organizes data in a structural hierarchical manner. It includes files, directories as well as metadata. metadata must be consistent for the file system to be considered healthy. In Linux, metadata consists of:

Superblock: important information about file system, e.g. type, creation and modification dates, size, mount status flag
Inodes: a data structure that contains information associated with every file or directory
list of data blocks free and in use

Host connects to storage through various Interface Protocols. Common interface protocols include:

IDE/ATA and Serial ATA
SCSI (Small Computer System Interface)
FC (Fibre Channel)
IP (Internet Protocol per se is a network protocol traditionally used for host-to-host traffic in the early days. In the virtualization era, it has become a viable option for host-to-storage communication. Examples are iSCSI and FCIP)

The most prevalent disk drive types are SSD (solid state drive) and HDD (hard disk driveA). SSD (solid state drive) is newer, flash-based technology. Without seek and rotational latencies they deliver a high number of IOPS with low response times. They are especially suited for applications with small block size and random-read workloads requiring constant latency <1ms.

HDD is traditional and more cost effective. Its physical components are illustrated in the following two graphs:

Hard disk component

For HDD, disk service time (time taken by a disk to complete an I/O request) is determined by the following factors:

Seek time (aka access time) describes the time taken to position the R/W heads across the platter with a radial movement (moving along the radius of the platter). In other words, it is the time taken to position and settle the arm and the head over the correct track.
Rotation latency is the time taken by the platter to rotate and position the data under the R/W head. It depends on the rotation speed of the spindle and is measured in milliseconds.
(Data) transfer rate is the average amount of data per unit time that the drive can deliver from disk controller to the HBA (on the host).

Zone Bit Recording – a mechanism to use disk efficiently by grouping tracks into zones based on their distance from the disk.

Logical Block Addressing (LBA) – disk controller translates LBA to a physical address (CHS, cylinder, head and sector). The host only needs to know the size of disk drive in terms of number of blocks. The logical blocks are mapped to physical sectors on a 1:1 basis

IO request processing – I/O controller is introduce to improve response time for I/O request, in this model, The I/O requests arrive at the controller at the rate generated by the application. This rate is also called the arrival rate. These requests are held in the I/O queue, and the I/O controller processes them one by one, as shown here:

The relationship between controller utilization and average response time is:

Average response time = Service time / (1 – Utilization)

where service time is the time taken by disk controller to service the I/O request. This results in a classic relation between response time and utilization, as plotted below:

The graph indicates that the response time changes are nonlinear as the utilization increases. When the average queue sizes are low, the response time remains low. The response time increases slowly with added load on the queue and increases exponentially when the utilization exceeds 70 percent. Therefore, for performance-sensitive applications, it is common to utilize disks below their 70 percent of I/O serving capability.

Command queuing is a technique implemented on modern disk drives that determines the execution order of received I/Os and reduces unnecessary drive-head movements to improve disk performance. When an I/O is received for execution at the disk controller, the command queuing algorithms assign a tag that defines a sequence in which the commands should be executed. With command queuing, commands are executed based on the organization of data on the disk, regardless of the order in which the commands are received. Below is an example:

Disk Command Queuing

RAID is a technology that leverages multiple drives as part of a set that provides data protection against drive failures. It may also improve performance by serving I/Os from multiple disks simultaneously. It is primarily used in HDD but SSD may still benefit from it. RAID may be implemented by software but hardware RAID with a controller is widespread. RAID is built on three basic techniques:

Striping: spread data across multiple drives (more than one) to use the drives in parallel.
Mirroring: same data is stored on two different disk drives, yielding two copies of the data.
Parity: a method to protect striped data from disk drive failure without the cost of mirroring. An additional disk drive is added to hold parity, a mathematical construct that allows re-creation of the missing data.

Basic RAID levels are summarized here:

RAID summary

RAID 0, 1, 5 and 6 are pretty common in data centre operations. In addition to these levels above. If you hear RAID 1+0, 5+0 and RAID 6+0, they are called nested RAID. They are simply a RAID 0 on top of RAID1, RAID 5 and RAID 6, respectively.

RAID 0 – data striping technique utilizes full capacity of drives. Although it is a good option for applications that need high I/O throughput. It lacks data protection so it cannot drive application requiring high availability.

RAID 0

RAID 1 – mirroring technique ensures data duplication. In the event of disk failure, it introduces minimal impact to the disk array. It is suitable for applications that require high availability and cost is no constraint.

RAID 1

RAID 1+0 or RAID 10 – performs well for workloads with small, random, write-intensive I/Os. Some applications that benefit from RAID 1+0 include the following:

High transaction rate Online Transaction Processing (OLTP)
Large messaging installations
Database applications with write intensive random access workloads

RAID 1+0

RAID 3 – RAID 3 stripes data for performance and uses parity for fault tolerance. the total disk space required is 1.25 times the size of the data disks. RAID 3 always reads and writes complete stripes of data across all disks because the drives operate in parallel. RAID 3 provides good performance for applications that involve large sequential data access, such as data backup or video streaming.

RAID 5 – RAID5 is similar to RAID 4 because it uses striping. The drives (strips) are also independently accessible. The difference between RAID 4 and RAID 5 is the parity location. In RAID 4, parity is written to a dedicated drive, creating a write bottleneck for the parity disk. In RAID 5, parity is distributed across all disks to overcome the write bottleneck of a dedicated parity disk.

RAID 5

RAID 5 is good for random, read-intensive I/O applications and preferred for messaging, data mining, medium-performance media serving, and relational database management system (RDBMS) implementations, in which database administrators (DBAs) optimize data access.

RAID 6 – works the same way as RAID 5, except that RAID 6 includes a second parity element to enable survival if two disk failures occur in a RAID set. The write penalty in RAID 6 is more than that in RAID 5; therefore, RAID 5 writes perform better than RAID 6. The rebuild operation in RAID 6 may take longer than that in RAID 5 due to the presence of two parity sets.

RAID 6

Intelligent Storage System involves cache as the core component. An intelligent storage system involves front end, cache, back end and physical disks, as shown here:

Intelligent Storage System components

A variety of physical disk types and speed (e.g. mix of FC, SATA, SAS and flash) can be supported in a modern intelligent storage system. The front end provides the interface between the storage system and the host. It consists of ports and controllers, with redundancy. The back end provides an interface between cache and the physical disks. It consists of ports and controllers. For high data protection and high availability, storage systems are configured with dual controllers with multiple ports.

Cache improves storage system performance by isolating hosts from mechanical delays associated with hard disks. In intelligent storage system, read and write are first attempted on cache.

Page is the basic unit of cache, and the size of page is configured based on application I/O size. Cache consists of:

data store: holds the actual data temporarily
tag RAM: mainly serves three purposes:
- it tracks locations of data in cache and on disk;
- it maintains dirty bit flag to indicate whether data in cache has been committed to disk;
- it keeps time based information such as last access time, for cache management

Read Operation with Cache: When host issues a read request, the storage controller reads the tag RAM first to determine whether required data is available in cache:

Read cache hit: data is sent to host without any disk operation;
Read cache miss: back end access the disk to retrieve the requested data. Data is then placed in cache and sent to host through front end.

Cache miss increases I/O response time, to increase read hit ratio, read-ahead algorithm can be used when read requests are sequential. In a sequential read request, a contiguous set of associated blocks is retrieved. Several other blocks that have not yet been requested by the host can be read from the disk and placed into cache in advance. When the host subsequently requests these blocks, the read operations will be read hits.

Fixed prefetch – the intelligent storage system prefetches a fixed amount of data. It is most suitable when host I/O sizes are uniform.
Variable prefetch, the storage system prefetches an amount of data in multiples of the size of the host request.
Maximum prefetch limits the number of data blocks that can be prefetched to prevent the disks from being rendered busy with prefetch at the expense of other I/Os.

Write Operation with Cache: When an I/O is written to cache and acknowledged, it is completed in far less time (from the host’s perspective) than it would take to write directly to disk. Sequential writes also offer opportunities for optimization because many smaller writes can be coalesced for larger transfers to disk drives with the use of cache. Write operation with cache can be implemented in two ways:

Write-back cache: Data is placed in cache and an acknowledgment is sent to the host immediately. Later, data from several writes are committed (de-staged) to the disk. Write response times are much faster because the write operations are isolated from the mechanical delays of the disk. However, uncommitted data is at risk of loss if cache failures occur.
Write-through cache: Data is placed in the cache and immediately written to the disk, and an acknowledgment is sent to the host. Because data is committed to disk as it arrives, the risks of data loss are low, but the write-response time is longer because of the disk operations.

If the size of an I/O request exceeds the write aside size, writes are sent to the disk directly to reduce the impact of large writes consuming a large cache space. This is helpful where cache resources are constrained and cache is required for small random I/Os.

Cache space can be assigned in two ways:

dedicated cache: separate sets of locations are reserved for read and write;
global cache: user may specify percentage of cache for read and write based on application workload pattern; or the system set is dynamically.

Cache Management algorithm is used to determine when, and what pages of the cache need to be free up during maintenance. Most commonly used algorithms are:

LRU (least recently used): assuming data not accessed for a while will not be requested by host any more;
MRU (most recently used): assuming data recently accessed will not be requested by host again

As cache fills, the storage system must take action to flush dirty pages by committing data from cache to disk. There are several triggers for cache management action:

Idle flushing – occurs continuously at modest rate when cache utilization level is between high and low watermark;
High watermark flushing – activated when utilization hits high watermark; and stops at low watermark; this has impact to I/O processing;
Forced flushing – occurs in the event of large I/O burst when cache reaches 100% capacity; this significantly impacts I/O response time

Types of Flushing

Cache data protection is the mechanism to prevent losing uncommitted data held in cache. Common mechanisms are:

Cache mirroring – Each write to cache is held in two different memory locations on two independent memory cards. If a cache failure occurs, the write data will still be safe in the mirrored location and can be committed to the disk. The array operating environment needs to maintain cache coherency between the redundant memory locations. Read cache does not need mirroring.

Cache vaulting – In the event of server power failure, use battery power to write the cache content to the disk (vault drive). When power is restored, data from these disks is written back to write cache and then written to the intended disks.