persistent disk on Digi Hunch

Docker storage

Tue, 03 Nov 2020 20:22:00 -0400

Microservices are all about stateless and ephemeral workloads, and containers are great microservices. This may suggest that that Docker is all about ephemeral storage. In fact, Docker supports both non-persistent and persistent storage, such as database, kafka, etc.

Non-persistent storage is automatically created, alongside the container and is tied to the lifecycle of the container. On Linux system, it is /var/lib/docker/ as part of container. This is referred to as local storage.

Docker has a concept of volume, which is essentially a file or a directory. Volumes are for persistent data. they are de-coupled from containers and are not tied to the lifecycle of any container. Volume allows process in docker container to bypass the default uionFS, and stores file or directory on host machine. It also allows different containers to share data. You may mount a volume to a container. even if container is deleted, volume persists.

By default, Docker creates new volumes with the built-in local driver. Local volumes are only available to containers on the node they’re created on. There are also third-party drivers as plugins that provides advanced options to integrate external storage system with Docker. (NAS, SAN, etc)

There are more than 25 volume plugins that you can specify with -d switch, to cover all three categories of storage

Block storage tends to be high performance and good for small-block random access workloads.
File storage is high performance, shared amongs multiple containers with NFS or SMB protocols.
Object storage is good for long term storage of large data blobs that do not change frequently. It is often content addressable and relatively low performance.

Note that if you share volume with multiple containers, the application needs to worry about data collision.

You may use docker volume create command to create volume. Note that there is no quota management within docker so the partition needs to be managed at operating system level.

Implementation of Volume

Remember that Docker image is built on multi-layer file system. When we run a container, Docker places a read-write layer on top of the image, such that the active files in running container are all placed in this read-write layer. When container is deleted, so are the files. The file system in Docker is a pseudo file system implemented in unionFS. Volumes bypasses the uionFS and directly accesses the host file system.

When we create a Docker volume, Docker places the volume data to /var/lib/docker/volumes and under each directory named after volume, creates a directory _data, which is attached to the corresponding container.

You can even mount an NFS volume to container. Reference here.

We mentioned UnionFS a couple times so far. UnionFS is a light-weight, layered file system. It can mount the contents of multiple directories to the same directory, to form a single file system. User can use unionFS like a directory. It is the foundation of Docker image and container and enables saving of spaces.

There are three common types of union FS: AUFS, DeviceMapper, and OverlayFS.

AUFS file system

AUFS is the earliest driver that Docker uses for file system, most common in Ubuntu and Debian. To check if the system support AUFS, check out the documentation here.

AUFS is recommended in Ubuntu or Debian. For CentOS and Redhat, it needs to be installed and make sure the command above returns aufs. To configure AUFS, create file /etc/docker/daemon.json and add:

{
 "storage-driver":"aufs"
}

Then restart docker service. Run “docker info” and examine the Storage Driver section, as documented here.

AUFS layers multiple directories on a single Linux host and presents them as a single directory. These directories are called branches in AUFS terminology, and layers in Docker terminology. The unification process is referred to as a union mount.

Layers of a Ubuntu container

This section describes how the layers work and this section describes how it reads and writes files (Copy-on-Write (CoW) strategy to maximize storage efficiency and minimize overhead). CoW characterized AUFS.

AUFS has not been adopted in the Linux kernel mainline for lack of maintainability. So for CentOS, the recommended file system driver is devicemapper.

Devicemapper file system

Devicemapper is a technical framework to map physical block device to virtual block device, introduced since kernel 2.6.9. So it’s essentially different from AUFS. The Logical Volume Manager (LVM) in Linux is also implemented based on devicemapper.

The three critical components in devicemapper are:

mapped device: a virtual device that devicemapper provides to client
target device: the underlying physical device or a section of it.
map table: keeps track of the offset, range, etc between mapped and target devices.

Devicemapper uses target driver to block, filter, and forward I/O requests (e.g. Raid, encryption, think provisioning, etc). In thin provisioning, storage driver only assigns spaces that are needed. Docker uses snapshot technology in thin provisioning. This part of the documentation provides further details as to how device mapper works.

Ubuntu and busybox image layers

Devicemapper has to modes:

loop-lvm: in dev and test environment
direct-lvm: recommended in production

Here is the performance best practice. To configure devicemapper, create /etc/docker/daemon.json file and add:

{
 "storage-driver":"devicemapper"
 "storage-opts":[
 "dm.directlvm_device=/dev/xdf",
 "dm.thinp_percent=95",
 "dm.thinp_metapercent=1",
 "dm.thinp_autoextend_threshold=80",
 "dm.thinp_autoextend_percent=20",
 "dm.directlvm_device_force=false"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure direct-lvm mode is on.

Since devicemapper uses block device to store files, it is faster than directly operate on file system. It is adopted as default driver as unionFS for a long time, ensuring stable performance under Red Hat and CentOS.

OverlayFS file system

Earlier versions of OverlayFS (known as overlay driver) is not stable. Later version is known as overlay2, which is very stable and recommended in overlay2. It requires:

Docker version higher than 17.06.02;
Kernel version higher than 3.10.0-514 for CentOS and RHEL; or higher than 4.0 for other distributions of Linux;
Using with xfs file system with d_type turned on

In production environment, it is recommended to moutn /var/lib/docker to separate disk or partition, to prevent the directory getting full from impacting the host OS. The option pquota is recommended for mounting options in /etc/fstab.

To configure storage driver, create file /etc/docker/daemon.json, with the following content:

{
 "storage-driver":"overlay2",
 "storage-opts":[
 "overlay2.size=20G",
 "overlay2.override_kernel_check=true"
 ]
}

Then restart docker service. Run “docker info” and examine the Storage Driver section to ensure storage driver is overlay2 and d_type is true.

The way overlay2 works is similar to AUFS, involving union mount process, with lowerdir, upperdir and merged. More details are here, including how overlay2 works with file read and file write (e.g. CopyOnWrite).

Today, overlay2 driver is officially recommended by Docker for its stability and performance, it should be used if all the conditions are met.

Cloud storage overview

Wed, 12 Aug 2020 22:19:00 -0400

In a narrow sense, cloud storage refers to object storage. In a broader sense, it refers to any storage service (block, file or object level) provided by cloud vendors, in a cloud business model. The underlying technology of storage, is the same be it in the cloud or on-premise.

	Block storage	File storage	Object
Interaction with OS	OS has direct byte-level access to disk blocks.	OS manages storage by file, or byte range of file. Files are organized in POSIX hierarchy.	OS reads and writes the entire object, or a byte range, via rest API calls.
Metadata	N/A	Stored in file system, for directory or file	customizable metadata
Common protocol	N/A	NFS	S3
Implementation	SAN (bock device is typically dedicated to a single VM) or DAS	NAS, file storage is usually shared amongst multiple VMs. Locking mechanism is usually in place to keep access in order.	S3
Workload	database storage, scratch data, etc	persistent data, content management, etc	archive data, media streaming, data analytics, static asset serving, etc

Below is a list of common storage services provided by public cloud vendors to day.

	Block Storage	File Storage	Object Storage	Other managed storage service
AWS	Elastic Block Store (EBS)	Elastic File System (EFS) FSx for Windows FSx for Lustre	Simple Storage Service (S3)	Storage Gateway Snow Family DataSync
Azure	Azure Managed Disks	Azure Files	Azure Blobs	Azure Table Azure Queues
GCP	Persistent Disk local SSD	Filestore	Cloud Storage	Cloud Storage for Firebase Data Transfer
Digital Ocean	Volumes Block storage local SSD	N/A	Space object storage (S3 compatible)	Content Delivery Network

Storage Products from common public cloud vendor

Since AWS is the first vendor that provides a full suite of storage service, this post will focus on the storage product lines, as a refresher of AWS cloud storage options: Simple Storage Service, Elastic File Storage and Elastic Block Storage). There will be some overlap with the AWS storage service whitepaper.

Before getting further to details, here’s a reminder of two types of policies in AWS:

	IAM policy	Resource-based policy
Principal	Must be attached to individual user, group, or role to take effect	Needs to be explicitly specified, can be ARN under other AWS account
Element	Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition	Principal/NotPrincipal Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition
Example	Managed policy, custom policy	File system policy, S3 bucket policy, access point policy, etc

Two types of policies

Although the resource is usually assumed in a resource-based policy, the policy usually target a sub-section of a resource (e.g. object with certain prefix), so resource section is still required in resource-based policy. In storage services, we may use S3 bucket policy, access point policy, or file system policy for EFS.

Below we go over the three families of storage service in AWS.

EBS (Elastic Block Storage)

EBS is a distributed system. Each volume is a logical volume, made up of multiple physical devices. EBS data is persistent, and access is dedicated to a single EC2 instance at a time. If EC2 instance failed, the attached EBS volume can be detached, and then re-attached to other instance, in the same Availability Zone. There are two types of EBS:

EC2 Instance store: ephemeral, block-level storage for EC2 instance, no replication by default, no snapshot support. Used as buffers, caches, scratch data, temporary content.
EBS volume (persistent) : used for database, dev/test, enterprise application, etc. There are two sub-categories:
- SSD-backed volumes:
  - Optimized for transnational workloads that requires very low latency
  - Dominant performance attribute is IOPS
  - For frequent, read/write with small size and random I/O
  - Typical use case include relational database (PostgresQL, MySQL) and NoSQL (Cassandra, Mongo)
  - gp2 (general purpose) and io1 (provisioned IOPS)
- HDD-backed volumes:
  - Optimized for large streaming workloads demanding throughput
  - Dominant performance attribute is throughput
  - For workloads with lots of sequential I/O
  - Typical use case icnlude big data, analytics (Kafka, Splunk, Hadoop, data warehousing), file/media server
  - st1 (throughput optimized0 and sc1 (cold HDD)

The four types of EBS are compared here:

Note that the volume can be modified (change type, increase size) after creation. However, you cannot decrease size. If you increase the size, the file system must be extended after the increase.

Another way to deliver better performance is to use EBS-optimized instances. These instances have dedicated network bandwidth for its I/O traffic to and from EBS. Without EBS-optimized instance, the traffic between EBS volume and EC2 instance uses shared network link with EC2, which is subject to latency during heavy traffic. This distinction is similiar to the difference between iSCSI SAN and FC SAN. Also, you may increase read-ahead buffer in OS for better EBS performance.

On EBS, users can create snapshot, a point-in-time incremental backup. When snapshot is restored to a volume, data is loaded lazily in the background, so that volume is available immediately. This also means that initial read of data that is not yet loaded will be subject to latency, known as first read penalty. To achieve target performance, user may run an initialization on the volume, by reading all blocks with data upfront.

In a newly created snapshot, only the data blocks modified since the previous snapshot are stored as is. The rest are pointers to unchanged data blocks in the original snapshot. When a previous snapshot is deleted, AWS ensures changes are reconciled into the newer snapshot so there is no loss of data. Creation of snapshots on many volumes can be automated with Data Lifecycle Manager (DLM).

As far as encryption goes, the best practice is to create your own master key. KMS uses envelop encryption, where the data key encrypts the data, and the master key encrypts the data key. The encryption key is stored in EC2 instance memory only and never written to disk, for security and performance considerations.

EFS (Elastic File Storage)

EFS is a managed implementation of file storage that supports NFS 4.0 and 4.1, with strong data consistency and file locking. An EFS includes a single mount target in (one subnet of) each availability zone. EC2 instance, or on-premise client via Direct Connect, can mount EFS volumes using amazon-efs-utils yum package. EC2 instance can also be configured to automatic mount EFS volume in launch wizard. EFS also has a lifecycle management policy, and a storage class for infrequent access.

Performance wise, EFS has two performance modes and two throughput modes. The two performance modes are:

General Purpose: for latency-sensitive applications and general-purpose workloads. limit of 7k ops/sec, best choice for most workloads
Max I/O: for large-scale and data-heavy applications, with virtually unlimited ability to scale out throughput/IOPS, but with slightly higher latencies. consider this for large scale-out workloads

The two throughput modes are:

Bursting throughput: recommended for the majority of workload. Since file system workload is typically spiky, aws use credit system to determine when the file system throughput can burst. credit accumates idle time, and consumed in retrieval
Provisioned throughput: recommended for higher throughput to storage ratio workload, can increase the provisioned throughput afterwards. but it incurs separate throughput charge

Other ways to achieve higher performance, include parallelization of file operation (e.g. multiple threads, more instances); and increase I/O size for better throughput.

In terms of security, EFS encryption at rest must be selected at the time of file system creation. There is an TLS mount option to encrypt traffic in transit. EFS involves its own resource-based policy called file system policy to manage file-level POSIX permissions. IAM policy is used to manage NFS administration access and client access. EFS access points is also a means to enforce the use of a specific operating system user, and group to access EFS.

S3 (Simple Storage Service)

S3 is one of the earliest and maturest AWS services for object storage. It is very cheap and easy to use, and supports user-defined metadata on objects as well as many peripheral features. There is no limit to the number of objects in a bucket. As the object in bucket increases, S3 scales to request rate by automatically creating more partitions to meet the target number of request per partition. There used to be a performance trick, that requires client to make object key naming pattern distribute across multiple prefixes. It is not required any more as of July 2018.

Versioning can be enabled at bucket level, and suspended afterwards. New version of object is created on every upload, without performance penalty.

S3 integrate well with other event-driven AWS services, such as SNS, SQS, Lambda, etc. Event can fire on request such as PUT, POST, COPY.

Object tags (not to be confused with object metadata) can help categorize storage. It also facilitates access control (i.e. by being referenced in bucket policy or IAM policy), lifecycle policy, analysis and CloudWatch configurations.

S3 select is a way to retrieve only a subset of data from an object based on a SQL expression, to reduce amount of data and help with performance. The input can be json or CSV and output will be in CSV.

S3 Inventory is a tool to audit object replication status and encryption status. It generates CSV report with all objects in the given bucket name, including: key name, version id, islatest, size, last modified date, etag, storage class, multipart upload flag, delete marker, replication status, encryption status. For storage-class analysis, S3 inventory is much faster than list-object API call which parses through all objects.

S3 also has access point, similar to EFS, with unique hostnames that customers create to enforce distinct permissions and network controls for any request made through the access point.

S3 transfer acceleration take advantage of edge locations (at additional charge) to speed up transfer of large object over long distance, by providing a separate end point. It is also helpful for faster uploads over long distances. Apart from transfer acceleration, for faster uploads for large object, user may also consider multi-part upload API when the object reaches 100MB. Orphaned uploaded parts can be cleaned up in lifecycle configuration. For better download performance, take advantage of CloudFront and byte range request.