network file system on Digi Hunch

Cloud storage overview

Wed, 12 Aug 2020 22:19:00 -0400

In a narrow sense, cloud storage refers to object storage. In a broader sense, it refers to any storage service (block, file or object level) provided by cloud vendors, in a cloud business model. The underlying technology of storage, is the same be it in the cloud or on-premise.

	Block storage	File storage	Object
Interaction with OS	OS has direct byte-level access to disk blocks.	OS manages storage by file, or byte range of file. Files are organized in POSIX hierarchy.	OS reads and writes the entire object, or a byte range, via rest API calls.
Metadata	N/A	Stored in file system, for directory or file	customizable metadata
Common protocol	N/A	NFS	S3
Implementation	SAN (bock device is typically dedicated to a single VM) or DAS	NAS, file storage is usually shared amongst multiple VMs. Locking mechanism is usually in place to keep access in order.	S3
Workload	database storage, scratch data, etc	persistent data, content management, etc	archive data, media streaming, data analytics, static asset serving, etc

Below is a list of common storage services provided by public cloud vendors to day.

	Block Storage	File Storage	Object Storage	Other managed storage service
AWS	Elastic Block Store (EBS)	Elastic File System (EFS) FSx for Windows FSx for Lustre	Simple Storage Service (S3)	Storage Gateway Snow Family DataSync
Azure	Azure Managed Disks	Azure Files	Azure Blobs	Azure Table Azure Queues
GCP	Persistent Disk local SSD	Filestore	Cloud Storage	Cloud Storage for Firebase Data Transfer
Digital Ocean	Volumes Block storage local SSD	N/A	Space object storage (S3 compatible)	Content Delivery Network

Storage Products from common public cloud vendor

Since AWS is the first vendor that provides a full suite of storage service, this post will focus on the storage product lines, as a refresher of AWS cloud storage options: Simple Storage Service, Elastic File Storage and Elastic Block Storage). There will be some overlap with the AWS storage service whitepaper.

Before getting further to details, here’s a reminder of two types of policies in AWS:

	IAM policy	Resource-based policy
Principal	Must be attached to individual user, group, or role to take effect	Needs to be explicitly specified, can be ARN under other AWS account
Element	Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition	Principal/NotPrincipal Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition
Example	Managed policy, custom policy	File system policy, S3 bucket policy, access point policy, etc

Two types of policies

Although the resource is usually assumed in a resource-based policy, the policy usually target a sub-section of a resource (e.g. object with certain prefix), so resource section is still required in resource-based policy. In storage services, we may use S3 bucket policy, access point policy, or file system policy for EFS.

Below we go over the three families of storage service in AWS.

EBS (Elastic Block Storage)

EBS is a distributed system. Each volume is a logical volume, made up of multiple physical devices. EBS data is persistent, and access is dedicated to a single EC2 instance at a time. If EC2 instance failed, the attached EBS volume can be detached, and then re-attached to other instance, in the same Availability Zone. There are two types of EBS:

EC2 Instance store: ephemeral, block-level storage for EC2 instance, no replication by default, no snapshot support. Used as buffers, caches, scratch data, temporary content.
EBS volume (persistent) : used for database, dev/test, enterprise application, etc. There are two sub-categories:
- SSD-backed volumes:
  - Optimized for transnational workloads that requires very low latency
  - Dominant performance attribute is IOPS
  - For frequent, read/write with small size and random I/O
  - Typical use case include relational database (PostgresQL, MySQL) and NoSQL (Cassandra, Mongo)
  - gp2 (general purpose) and io1 (provisioned IOPS)
- HDD-backed volumes:
  - Optimized for large streaming workloads demanding throughput
  - Dominant performance attribute is throughput
  - For workloads with lots of sequential I/O
  - Typical use case icnlude big data, analytics (Kafka, Splunk, Hadoop, data warehousing), file/media server
  - st1 (throughput optimized0 and sc1 (cold HDD)

The four types of EBS are compared here:

Note that the volume can be modified (change type, increase size) after creation. However, you cannot decrease size. If you increase the size, the file system must be extended after the increase.

Another way to deliver better performance is to use EBS-optimized instances. These instances have dedicated network bandwidth for its I/O traffic to and from EBS. Without EBS-optimized instance, the traffic between EBS volume and EC2 instance uses shared network link with EC2, which is subject to latency during heavy traffic. This distinction is similiar to the difference between iSCSI SAN and FC SAN. Also, you may increase read-ahead buffer in OS for better EBS performance.

On EBS, users can create snapshot, a point-in-time incremental backup. When snapshot is restored to a volume, data is loaded lazily in the background, so that volume is available immediately. This also means that initial read of data that is not yet loaded will be subject to latency, known as first read penalty. To achieve target performance, user may run an initialization on the volume, by reading all blocks with data upfront.

In a newly created snapshot, only the data blocks modified since the previous snapshot are stored as is. The rest are pointers to unchanged data blocks in the original snapshot. When a previous snapshot is deleted, AWS ensures changes are reconciled into the newer snapshot so there is no loss of data. Creation of snapshots on many volumes can be automated with Data Lifecycle Manager (DLM).

As far as encryption goes, the best practice is to create your own master key. KMS uses envelop encryption, where the data key encrypts the data, and the master key encrypts the data key. The encryption key is stored in EC2 instance memory only and never written to disk, for security and performance considerations.

EFS (Elastic File Storage)

EFS is a managed implementation of file storage that supports NFS 4.0 and 4.1, with strong data consistency and file locking. An EFS includes a single mount target in (one subnet of) each availability zone. EC2 instance, or on-premise client via Direct Connect, can mount EFS volumes using amazon-efs-utils yum package. EC2 instance can also be configured to automatic mount EFS volume in launch wizard. EFS also has a lifecycle management policy, and a storage class for infrequent access.

Performance wise, EFS has two performance modes and two throughput modes. The two performance modes are:

General Purpose: for latency-sensitive applications and general-purpose workloads. limit of 7k ops/sec, best choice for most workloads
Max I/O: for large-scale and data-heavy applications, with virtually unlimited ability to scale out throughput/IOPS, but with slightly higher latencies. consider this for large scale-out workloads

The two throughput modes are:

Bursting throughput: recommended for the majority of workload. Since file system workload is typically spiky, aws use credit system to determine when the file system throughput can burst. credit accumates idle time, and consumed in retrieval
Provisioned throughput: recommended for higher throughput to storage ratio workload, can increase the provisioned throughput afterwards. but it incurs separate throughput charge

Other ways to achieve higher performance, include parallelization of file operation (e.g. multiple threads, more instances); and increase I/O size for better throughput.

In terms of security, EFS encryption at rest must be selected at the time of file system creation. There is an TLS mount option to encrypt traffic in transit. EFS involves its own resource-based policy called file system policy to manage file-level POSIX permissions. IAM policy is used to manage NFS administration access and client access. EFS access points is also a means to enforce the use of a specific operating system user, and group to access EFS.

S3 (Simple Storage Service)

S3 is one of the earliest and maturest AWS services for object storage. It is very cheap and easy to use, and supports user-defined metadata on objects as well as many peripheral features. There is no limit to the number of objects in a bucket. As the object in bucket increases, S3 scales to request rate by automatically creating more partitions to meet the target number of request per partition. There used to be a performance trick, that requires client to make object key naming pattern distribute across multiple prefixes. It is not required any more as of July 2018.

Versioning can be enabled at bucket level, and suspended afterwards. New version of object is created on every upload, without performance penalty.

S3 integrate well with other event-driven AWS services, such as SNS, SQS, Lambda, etc. Event can fire on request such as PUT, POST, COPY.

Object tags (not to be confused with object metadata) can help categorize storage. It also facilitates access control (i.e. by being referenced in bucket policy or IAM policy), lifecycle policy, analysis and CloudWatch configurations.

S3 select is a way to retrieve only a subset of data from an object based on a SQL expression, to reduce amount of data and help with performance. The input can be json or CSV and output will be in CSV.

S3 Inventory is a tool to audit object replication status and encryption status. It generates CSV report with all objects in the given bucket name, including: key name, version id, islatest, size, last modified date, etag, storage class, multipart upload flag, delete marker, replication status, encryption status. For storage-class analysis, S3 inventory is much faster than list-object API call which parses through all objects.

S3 also has access point, similar to EFS, with unique hostnames that customers create to enforce distinct permissions and network controls for any request made through the access point.

S3 transfer acceleration take advantage of edge locations (at additional charge) to speed up transfer of large object over long distance, by providing a separate end point. It is also helpful for faster uploads over long distances. Apart from transfer acceleration, for faster uploads for large object, user may also consider multi-part upload API when the object reaches 100MB. Orphaned uploaded parts can be cleaned up in lifecycle configuration. For better download performance, take advantage of CloudFront and byte range request.

How RPC and NFS work

Wed, 15 Jul 2020 10:45:00 -0400

I touched on NFS in several previous postings, and here is a deeper dive of this particular protocol. NFS is built on top of Remote Procedure Call (RPC) and therefore it is important to understand RPC first. In fact NFS is one of the most prominent user of RPC and the best example for learning RPC.

RPC overview

According to Wikipedia, an RPC is when a computer program causes a procedure to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote. This is a form of client–server interaction (caller is client, executor is server), typically implemented via a request–response message-passing system. In the object-oriented programming paradigm, RPCs are represented by remote method invocation (RMI), such as Java RMI API.

RPCs are a form of inter-process communication (IPC), in that other processes have a different address spaces: if on the same host machine, they have distinct virtual address spaces, even though the physical address space is the same; while if they are on different hosts, the physical address space is different.

RPC is a request–response protocol, and therefore synchronous. An RPC is initiated by the client, which sends a request message to a known remote server to execute a specified procedure with supplied parameters. The remote server sends a response to the client, and the application continues its process. While the server is processing the call, the client is blocked (it waits until the server has finished processing before resuming execution), unless the client sends an asynchronous request to the server. There are many variations and subtleties in various implementations, resulting in a variety of different (incompatible) RPC protocols.

NFS overview

NFS is defined as a set of RPCs, including their arguments, results and effects. RPC makes NFS protocol transparent. RPC is also stateless so the server does not keep the state of RPCs once the request has been served. Each RPC contains the necessary information to complete the call. In the event of server failure, client will need to resubmit requests. NFS has several versions, with v3 and v4 most popular. We will focus on v3 in this posting and brief on v4.

Portmap

RPC makes a remote call appears to client application as a local call, with the help of portmap. The utility for portmap is rpcbind. In RedHat/CentOS 5 or prior, it was even simply called portmap but they are essentially the same service for RPC port mapper.

The rpcbind service is required on both NFS client and NFS server. On the client, it talks to client application, as well as its counterpart on the server. Its main function is query its counterpart on the server, providing a RPC program number, and expects a port number in return. On the server, the rpcbind utility listens at port 111, waiting for request with RPC program number(service), and return the TCP or UDP port number on which the requested service is hosted. RPC program number is reserved numeric indicator of services as outlined in RFC5531. For example, 100005 for mountd, 100021 for nlockmgr, and 100003 for nfs.

The port that rpcbind service itself listens on is always at port 111, which is known to both client and server. This is also referred to as portmapper daemon. Other than this fixed port, each NFS-related service (with their respect reserved program number) may be hosted on different ports on the server. The client rpcbind service must first look up for the port for the requested program number, then it directs the client to initiate connection to the specified port for the specific service.

With the following command rpcinfo command you may look up the program to port number mapping on destination server isilon.company.com

# rpcinfo -p isilon.company.com

The following command is commonly used for displaying mountpoint and troubleshooting mount. Under the hood it is using the information from rpcinfo.

showmount -e isilon.dcb.digitalhunch.com

On the NFS server side, rpcbind service must start before nfs service start. Otherwise nfs service cannot register ports to rpcbind. If you restart rpcbind, every service that had registered to rpcbind must restart to register themselves again. By default, NFS server can dynamically assign a port for mountd, nlockmgr within a port range. This makes firewall setting a pain. These dynamically assigned port can be fixed via configuration files.

File handle

NFS uses file handles (or fhandle) to represent files. It is a better mechanism to reference a file object than pathname for three reasons: 1. file handle has fixed length (32bytes); 2. If the file is renamed, the file handle reference remain the same; 3. If a file is deleted, and then a new file is created with the same path, a new file handle will be created. A file handle has three parts:

volume ID: to identify the mounted file system
inode #: to identify the file within the mounted file system
generation #: to detect when file handle refers to an older version of inode. Traditional Unix filesystems may reuse inode and thus NFS client could mistakenly use an old file handle and access a new file.

The file handle information is only meaningful to the server. New file handles are returned to client by certain procedures, such as LOOKUP, CREATE, and MKDIR. The file handle for the root of the file system, is obtained by the client when it mounts the file system, as permission allows.

Permission and Locking

When accessing a file on server, the client passes uid/gid info in RPCs, and the server performs permission checks as if the user was performing the operation locally. So users and groups are represented as integers. There are two security problems:

The mapping from uid/gid to user must be the same on all clients. This is not practical in large deployment, although can be solved via Network Information Service (NIS);
Whether the root user on the client has root access to files on the server, is a server policy configuration. This can be addressed by enabling “root squashing” on server, so that client’s uid 0 (root) is mapped to 65534 (nobody).

Unix has two locking mechanisms (fcntl and flock). NFS protocol supports fcntl but not flock. The flock function is managed by a separate service (nfslock) to allow NFS to lock files. The nfslock daemon provides the ability to lock regions of NFS files. NFS service itself is still completely stateless with locking managed separately. This is changed in NFSv4.

Procedures used in NFS service

NFS service defines a list of procedures. Here is a list with brief summary of activities. The bottom five RPCs are introduced in v3.

Procedure	Activity
GETATTR(fh)	Returns the attributes of a file, similar to stat syscall.
SETATTR(fh, attr)	Sets the attributes of a file (mode, uid, gid, size, atime, mtime); setting the size to 0 truncates the file
STATFS(fh)	Returns the status of a filesystem, such as block size, number of free blocks. e.g. df command.
LOOKUP (dirfh, name)	Returns fhandle and attributes for the named file in the directory specified by dirfh
READ (fh, offset, count)	Reads from a file, with offset and count specified. In v2, the length is up to 8192 bytes; v3 support more.
WRITE (fh, offset, count, data)	Writes to a file, with offset and count specified, as well as a separate field called data. Returns the new attributes of the file after the write.
CREATE (dirfh, name, attr)	Creates a file with the name, in directory, returns new fhandle and attributes
REMOVE (dirfh, name)	Deletes the named file in from directory dirfh and returns status.
RENAME (dirfh, name, tofh, toname)	Renames name in directory dirfh, to toname in directory tofh.
LINK (dirfh, name, tofh, toname)	Creates a hard link toname, in directory tofh, that points to name, indirectory dirfh.
SYMLINK (dirfh, name, string)	Creates a symbolic link name, in the directory dirfh, with value string.
READLINK (fh)	Reads a symbolic link and get file name of the target.
MKDIR (dirfh, name, attr)	Creates a directory name in the directory dirfh, and returns the new fh and attributes.
RMDIR(dirfh, name)	Removes a directory with the name, from parent directory dirfh.
READDIR (dirfh, cookie, count)	Reads a directory and returns up to count bytes of directory entries from the directory dirfh. The cookie is used in subsequent readdir calls to start reading at a specific entry in the directory. Cookie of zero get the server to start with the first entry in the directory.
NULL	No activity. Used for testing only.
ACCESS	Helps with client caching.
MKNOD	Makes a device special file.
FSINFO	Returns information about the server’s capabilities.
READDIRPLUS	Returns both file handle and attributes to eliminate LOOKUP calls when scanning a directory
COMMIT	In NFSv3, the server can reply to WRITE RPCs immediately without syncing to disk. When client wants to ensure that the data is on stable storage, it sends a COMMIT RPC. This is used in asynchronous writes for better performance, which is an option negotiated at mount time.

The addition of COMMIT procedure in v3 offers the option to improve write performance in place of synchronous write. However, asynchronous write requires more coordination to ensure data integrity during transmission, in the event of server crash. NFSv3 uses write verifier for this purpose. A write verifier is an 8-bye value that the server must change if it crashes.

After an asynchronous write, the reply from WRITE RPC includes a write verifier, the client must keep it for later use;
The client then sends a COMMIT RPC and the reply contains another write verifier;
The client compares the verifiers from the two returns for crash detection. If the verifiers don’t match, the client must rewrite all uncommitted data.
The client must keep all uncommitted data in case of a server crash.

Additional daemon processes

In addition to the three essential services, (nfs, rpcbind and nfslock), there are several auxiliary processes that facilitates NFS services. Their functions are listed here:

Process	Description
rpc.mountd	Used by NFS server to process MOUNT requests from NFSv3 client. It checks that the requested NFS share is currently exported by the NFS server, and that the client is allowed to access it. If the mount request is allowed, the rpc.mountd server replies with a Success status and provides the File-Handle for this NFS share back to the NFS client.
rpc.nfsd	Allows explicit NFS versions and protocols the server advertises to be defined. It works with the Linux kernel to meet the dynamic demands of NFS clients, such as providing server threads each time an NFS client connects. This process corresponds to the nfs service.
rpc.lockd	A kernel thread which runs on both clients and servers. It implements the Network Lock Manager (NLM) protocol, which allows NFSv3 clients to lock files on the server, using procedures such as NLM_NULL, NLM_TEST, NLM_LOCK, NLM_GRANTED, NLM_UNLOCK, NLM_FREE. The service is started automatically whenever the NFS server is run and whenever an NFS file system is mounted.
rpc.statd	This process implements the Network Status Monitor (NSM) RPC protocol, which notifies NFS clients when an NFS server is restarted without being gracefully brought down. rpc.statd is started automatically by the nfslock service, and does not require user configuration. This is not used with NFSv4.
rpc.rquotad	This process provides user quota information for remote users. rpc.rquotad is started automatically by the nfs service and does not require user configuration.
rpc.idmapd	provides NFSv4 client and server upcalls, which map between on-the-wire NFSv4 names (strings in the form of user@domain) and local UIDs and GIDs. For idmapd to function with NFSv4, the /etc/idmapd.conf file must be configured. At a minimum, the “Domain” parameter should be specified, which defines the NFSv4 mapping domain. If the NFSv4 mapping domain is the same as the DNS domain name, this parameter can be skipped. The client and server must agree on the NFSv4 mapping domain for ID mapping to function properly.

NFSv4

Even NFSv4 was introduced in 20 years ago, it improves access and performance of NFS on the Internet. It should be the default option for any new deployment.

NFSv4 is TCP only protocol and it is stateful.
NFSv4 combines mount and lock protocols into NFS so only one port is being used.
Users and groups are identified with strings (user@domain, or group@domain where domain represents a registered DNS domain or sub-domain), instead of integers. The access control policies are compatible with both Unix and Windows.
NFSv4 mandates strong RPC security built on cryptography, with negotiation at the time of mount
NFSv4 adopted a framework for authentication, integrity and privacy at RPC level
Introduced new RPC COMPOUND, which allows for several operations in one go. At the server, operations are evaluated in order, and each has a return value.

NFSv4.1 was release in 2010, and 4.2 in 2016. Both AWS EFS and Azure File storage supports 4.1.

EMC Isilon storage product

Wed, 08 Jul 2020 20:04:00 -0400

EMC has several product lines for different use cases in enterprise data storage. Like may other IT solutions, the website is clouded with marketing terms and slogans, and is purposefully not technical. This makes it difficult for technical staff to grasp the advantage of its product in a glimpse. I personally have to know their product (mostly with Isilon and ECS) well in order to make integration decisions. So I’m putting together this note (updated as of July 2020), with lots of details from their technical white paper.

Overview of EMC storage

At the highest level, the EMC enterprise data storage product lines are categorized into two groups: primary storage (along the lines of block-level storage) and unstructured storage (mostly file and object storage). The primary storage includes the following product:

PowerMax for OLTP database (Oracle, MicrosoftSQL and SAP)
PowerFlex: for Software defined storage, Oracle RAC, Elastic Stack, Kubernetes, Splunk
XtremIO for VMware, VDI, SAP
PowerStore for Database, VMware
PowerVault for Entry-level SAN and DAS environment

This post only expands on the unstructured storage product line, which mainly consists of PowerScale and ECS. ECS (elastic cloud storage) is EMC’s object storage. PowerScale (aka Isilon) is scale-out NAS platform for high-volume storage (up to 50 PB in a single file system), backup and archiving of unstructured data. For the rest of this post, I will still refer to PowerScale as Isilon.

Dell’s official support website is the most resourceful place to get information. For example, when I want to read about Isilon. I start with Dell support, then click on knowledgebase at the top, then go to “servers, storage and networking“, then “storage technical documents and videos“. There I can select a productline such as Isilon.

Overview of Isilon Family

Isilon is a clustered storage system consisting of three or more nodes. A node is a server with OneFS as its operating system. Based on FreeBSD, OneFS is EMC’s proprietary operating system to unify a cluster of nodes into a single shared resource. So OneFS is for Isilon only. It is the basis of Isilon. Isilon has three series of products:

F series: F200, F600, F800 and F810.
H series: typical models are H400, H500 and H600, which seeks to balance performance and capacity
A series: typical models are A200 and A2000 for active and deep archive storage

In June 2020, Dell decoupled OneFS software (with 9.0 released) from server hardware (referred to as PowerScale). Going forward EMC will refer to Isilon as PowerScale for OneFS version newer than 9.0 in spec sheets and white papers.

F200 is the cost-effective choice with SSD for remote office, small hospital, retail outlets, IOT or factory floor. F600 uses NVMe drives instead, and has more ECC memory and faster ethernet backend network. and is higher than F200 in its use case. Both F200 and F600 provide inline data compression and deduplication capabilities. F800 and F810 both use SSD and they are similar. F800 comes with InfiniBand backend network and F810 provides inline data compression and deduplication capabilities. H series tries to strike a balance between performance and capacity so they are pretty much everything in betwee. On the other end, A200 and A2000 are almost the same except for capacity difference.

Isilon’s advantage

Isilon has lots of intelligence built into its solution compared to a traditional NAS. Here are some aspects from its product white paper:

Aspects of Design	Isilon OneFS Scale-Out NAS	Traditional NAS
Network	Separation of front-end and back-end network to isolate node-to-node communication to a private low-latency network. Front-end traffic load balanced with SmartConnect	Single network for both external and internal traffic
File system structure and NameSpace	The storage is completely virtualized to users as a truly single file system with one namespace. There is no partitioning or volumes. The single file tree can grow organically without requiring planning or oversight about how the tree grows. SmartPool handles tiering of files to appropriate disk, without disrupting the single file tree.	An appearance of single namespace is typically achieved through namespace aggregation, where files are still managed in separate volumes, and a simple “veneer” layer glues individual directories to a “top-level” tree via symbolic links. LUNs and volumes, as well as volume limits are still present. Files have to be manually moved from volume-to-volume to load-balance.
Data Layout	OneFS controls the placement of file directly, down to the sector-level on any drive anywhere in the cluster. The addressing scheme for data and metadata is indexed at physical level by a tuple of {node, drive, offset}	Data are sent through RAID and volume management layers, introducing inefficiencies in data layout and providing non-optimized block access.
Redundancy Control	OneFS can flexibly control the type of striping as well as the redundancy level of the storage system at the system, directory and even file-levels.	The entire RAID volume is dedicated to a particular performance type and protection setting.

Isilon terms

The Isilon technology re-implemented the read and write path during file storage and introduced several terms along with its technology.

SmartPools – Job that runs and moves data between the tiers of nodes within the same cluster. Also executes the CloudPools functionality if licensed and configured. FilePolicy is changelist-based SmartPools file pool policy job. SmartPoolsTree enforces SmartPools file policies on a subtree.

Storage Pools – Storage pools provide the ability to define subsets of hardware within a single cluster, allowing file layout to be aligned with specific sets of nodes through the configuration of storage pool policies. The notion of Storage pools is an abstraction that encompasses disk pools, node pools, and tiers.

Disk Pools – Disk pools are the smallest unit within the storage pools hierarchy. OneFS provisioning works on the premise of dividing similar nodes’ drives into sets, or disk pools, with each pool representing a separate failure domain. Disk pools are laid out across all five sleds in each node.

Node Pools – groups of disk pools, spread across similar storage nodes (or equivalent classes). Multiple groups of different node types can work together in a single, heterogeneous cluster. For example, one node pool of all-flash F-Series anodes, one node pool of H-series, and one node pool of A-series. Each node pool only contains disk pools from the same type of storage nodes.

Tiers – groups of nodepools combined into a logical superset to optimize data storage, according to OneFS platform type. this allows customers who consistently purchase highest capacity nodes available to consolidate a variety of node styles within a single tier, and manage them as one logical group. SmartPools users typically deploy 2 to 4 tiers. different node pools under a tier needs to be compatible.

Global Namespace Acceleration (GNA)’s principal goal is to help accelerate metadata read operations by keeping a copy of a cluster’s metadata on high performance, low latency SSD media.

SmartConnect is a load balancer that works at the front-end Ethernet layer to evenly distribute client connections across the cluster. SmartConnect supports dynamic NFS failover and failback to ensure that when a node failure occurs, or preventative maintenance is performed, all in-flight reads and writes are handed off to another node in the cluster to finish its operation without any user or application interruption.

Auto Balance reallocates and rebalances data and make storage space more usable and efficient.

SmartQuotas is directory-level quota management. Note: there is no partitioning, and no need for volume creation in OneFS.

SmartRead creates a data “pipeline” from L2 cache, prefetching into a local “L1” cache, on the captain node, in order to greatly improve sequential-read performance. For high-sequential cases, SmartRead can very aggressively prefetch ahead. SmartRead can control how aggresive the pre-fetching is, and how long data stays in the cache, and optimizes where data is cached.

In-line Data Reduction – the write path involves zero block removal, in-line deduplication, and in-line compression. This is supported in some models only.

Smart Dedupe – post-process, asynchronous deduplication. Smart Dedupe scans the on-disk data for identical blcoks and then eliminate the duplicates. After duplicate blocks are discovered, SmartDedupe movees a single copy of those blocks to a special set of files known as shadow stored. With post-process deduplication, new data is first stored on the storage device and then a subsequent process analyzes the data looking for commonality. This means that initial file write or modify performance is not impacted, since no additional computation is required in the write path, as opposed to in-line deduplication. This is supported on some models only.

OneFS SSD strategy – How OneFS leverage the SSD for performance. It has these options:

L3 cache (implemented at nodepool level)
metadata read
metadata read/write
Global Namespace Acceleration (GNA)
Data on SSD

L3 cache consumes all the SSD in node pool. L3 cannot coexist with other SSD strategies, with the exception of GNA just because L3 cache node pool SSD cannot participate in GNA.

Isilon’s High Availability

The OneFS is distributed across all nodes in the cluster and is accessible by clients connecting to any node in the cluster. Metadata and locking tasks are managed by all nodes collectively and equally in a peer-to-peer architecture. This symmetry is key to the simplicity and resiliency of the architecture. There is no single metadata server, lock manager or gateway node.

The entire cluster forms a single file system with a single namespace that runs across every node equally. No one node controls or “masters” the cluster; all nodes are true peers.

During failover, clients are evenly redistributed across all remaining nodes in the cluster, ensuring minimal performance impact. If a node is brought down for any reason, including a failure, the virtual IP addresses on that node is seamlessly migrated to another node in the cluster. When the offline node is brought back online, SmartConnect automatically rebalances the NFS and SMB3 clients across the entire cluster to ensure maximum storage and performance utilization. This functionality allows for per-node rolling upgrades affording full-availability throughout the duration of the maintenance window.

There are two logical roles in processing an I/O request from client:

The initiator: the node that the client connects to with front-end protocol. The initiator acts as the ‘captain’ for the entire I/O operation.
The participant: Every node in the cluster is a participant for a particular I/O operation.

File Write in Isilon

OneFS employs a patented transaction system during write to eliminate single point of failure. In a write operation, the initiator “captains” or orchestrates the layout of data and metadata, the creation of erasure codes, and the normal operations of lock management and permission control.

When a client connects to a node to write a file, it is connecting to the Initiator. OneFS breaks the file down into atomic units. An atomic unit is a smaller logical chunk of data, also called stripe, or protection groups in the context of data protection. The size of each file chunk is referred to as the stripe unit size. After this division, OneFS then write the stripe individually to the Participant (with disks). This design ensures that data is protected at the specified level as soon as it is being written. Redundancy is built into protection groups, such that if every protection group of a file is safe, then the entire file is safe. In terms of protection mechanism, OneFS can use either Reed-Solomon erasure coding system, or simply mirroring for data protection. Erasure coding is the predominant mechanism with very high performance without sacrificing on-disk efficiency.

The initiator node uses a modified two-phase commit transaction to safely distribute writes to multiple NVRAMs across the cluster. As client initiates write to OneFS cluster, instead of immediately writing to disk, OneFS temporarily writes the data to an NVRAM-based journal cache on the initiator node and acknowledge the write the client. As outlined above, these writes are also mirrored to participant nodes’ NVRANM journals to satisfy the file’s protection requirement. Later, at a more convenient time, OneFS then flush these cached writes to disks asynchronously.

Since NVRADM journals all the transactions that are occurring across every node in the storage cluster. If a node fails mid-transaction, and then re-joins the cluster, the uncommitted cached writes are fully protected, and the only required actions for the node, are to replay its journal from NVRAM, and occasionally for AutoBalance to rebalance files that were involved in the transaction. Writes are never blocked due to a failure. There is no ‘fsck’ or ‘disk-check’ process.

OneFS file system block size is 8KB. A file smaller than 8KB will use a full 8KB block. For larger files, OneFS can maximize sequential performance by taking advantage of a stripe unit consisting of 16 contiguous blocks, for a total of 128KB per stripe unit.

Cache in Isilon

OneFS aggregates the cache present on each node in a cluster into one globally accessible pool of memory by using a messaging system similar to NUMA (non-uniform memory access). This allows all the nodes’ memory cache to be available to each and every node in the cluster. Remote memory is access over internal network with much lower latency than accessing hard disk drives. The internal network as distributed system bus, is a redundant, under-subscribed flat Ethernet up to 40Gb. The oneFS caching subsystem is coherent across the cluster, due to the use of MESI protocol to maintain cache coherency. If the same content exists in the private caches of multiple nodes, this cached data is consistent across all instances.
OneFS uses up to three levels of read cache, plus an NVRAM-backed write cache, or coalescer.

OneFS Caching Hierarchy

L1 cache – prefetches data from remote nodes. Data is prefetched per file, and this is optimized in order to reduce the latency associated with the nodes’ back-end network. The L1 cache refers to memory on the same node as the initiator. It is only accessible to the local node, and typically the cache is not the master copy of the data.

L1 is also known as remote cache because it contains data retrieved from other nodes in the cluster. It is coherent across the cluster but is used only by the node on which it resides and is not accessible by other nodes. Data in L1 cache on storage nodes is aggressively discarded after it is used. L1 cache uses file-based addressing, in which data is accessed via an offset into a file object.

OneFS also uses a dedicated inode cache in which recently requested inodes are kept. The inode cache frequently has a large impact on performance, because clients often cache data, and many network I/O activities are primarily requests for file attributes and metadata, which can be quickly returned from the cached inode.

L2 cache (backend cache) refers to local memory on the node on which a particular block of data is stored. L2 cache is globally accessible from any node in the cluster and is used to reduce the latency of a read operation by not requiring a seek directly from the disk drives.

L2 cache is also known as local cache because it contains data retrieved from disk drives located on that node and then made available for requests from remote nodes. Data in L2 cache is evicted according to a Least Recently Used (LRU) algorithm.

L3 cache, or Smart Flash, is configurable on nodes that contain solid state drives. Smart Flash (L3) is an eviction cache that is populated by L2 cache blocks as they are aged out from memory.

During I/O request, clients talk to L1 cache and write coalescer; L1 cache talks to L2 cache on all cluster nodes. L2 cache buffers to and from disks. L3 cache is optionally enabled per node pool, as an extension from L2. L3 and L2 communicate in backend network.

Name	Medium	Description
L1 Cache (aka front-end cache or remote cache)	RAM (volatile)	holds clean, cluster coherent copies of file system data and metadata blocks requested by clients via front-end network
L2 Cache (aka back-end cache or local cache)	RAM (volatile)	contains clean copies of file system data and metadata on a local node
SmartCache (Write Coalescer)	Battery-backed NVRAM (Persistent)	a persistent journal cache that buffers any pending writes to front-end files that have not been committed to disk
SmartFlash or L3 Cache	SSD (persistent)	contains file data and metadata blocks evicted from L2 cache, effectively increasing L2 cache capacity

File Read in Isilon

The high-level steps for fulfilling a read request with cache interaction involves:

Step 1 – on local node, determine whether part of the requested data is in the local L1 cache:

if so, return to client
if not, request data from remote nodes

Step 2 – on remote nodes, determine whether requested data is in the local L2 or L3 cache:

if so, return to the requesting node
if not, read from disk and return to requesting node

During a read operation, the “captain” node gathers all of the data from the various nodes in the cluster and presents it in a cohesive way to the requestor.

The cluster provides a high ratio of cache to disk (multiple GB per node) that is dynamically allocated for read and write operations as needed. This RAM-based cache is unified and coherent across all nodes in the cluster, allowing a client read request on one node to benefit from I/O already transacted on another node. As the cluster grows larger, the cache benefit increases. For this reason, the amount of I/O to disk on a cluster is generally substantially lower than it is on traditional platforms.

For files marked with an access pattern of concurrent or streaming, OneFS can take advantage of pre-fetching of data based on heuristics used by the SmartRead component

Conclusion

This post provided a high level introduction to EMC storage product line and expanded into some technical details in the read write operation in OneFS/Isilon. Some of the features can be seen in OneFS simulator which is a free tool from EMC.