fsx ontap on Digi Hunch

Kubernetes Storage on Azure 1 of 3 – built-in storage and NFS

Sun, 31 Jul 2022 15:22:00 -0400

In the previous post, we understand that to host stateful workload, we need to manage persistent storage to the Kubernetes platform. In this post, I will explore the different storage options. These options are specific to Azure Kubernetes service. However, the principals apply to any Kubernetes platform regardless of cloud vendor.

In another old post, I discussed in-tree and CSI storage classes, and from a developer’s perspective, how to mount volumes statically and dynamically once the storage class is available. Here in this post we are concerned with how to make storage classes available, from a platform specialist perspective.

Storage in Azure Kubernetes

As soon as the Azure Kubernetes cluster is launched, a number of built-in storage classes are available. Unlike third-party storage classes, they do not require kubelet identity to be contributor for node resource group. As discussed we shall use CSI based storage classes.

managed-csi
managed-csi-premium
azurefile-csi
azurefile-csi-premium

The main difference between them is the backing technology. However, form Kubernetes workload’s perspective, the Pods as storage consumers are concerned with the access mode instead of backing technology. Here are supported access modes:

ReadWriteOnce: read-write by a single node
ReadOnlyMany: read only by many nodes
ReadWriteMany: read-write by many nodes
ReadWriteOncePod: new in Kubernetes 1.22 to restrict volume access to a single Pod

The storage classes managed-csi and managed-csi-premium support ReadWriteOnce. The storage classes azurefile-csi and azurefile-csi-premium support ReadWriteMany.

Apart from these built-in options, Azure also suggests a few more options based on other types of Azure resources. For example, AKS can integrate with HPC cache and it for HPC. We can also self-manage a virtual machine configured as NFS server, and use the NFS subdir external provisioner to configure storage class. Despite of the overhead with managing a VM, you have more configurability. My previous client reports that they gain better performance than the built-in options.

Another alternative is Azure NetApp Files. However, being a full enterprise grade solution (similar to FSx ONTAP), Azure NetApp Files costs an arm and a leg. In this comparison, it cost 60 to 100 times as the cost by built-in options.

Another option is Azure Ultra Disk, which needs to be enabled at cluster level. You can provision performance target (DiskIOPSReadWrite and DiskMBpsReadWrite) in the storage class. Ultra Disk is a good middle ground between the pricey NetApp files and the less performant built-in options.

Terraform Template for AKS

To explore the storage options, I use my own terraform template to create an AKS cluster. The template is in the azure directory of the cloudkube repo. The template consists a few configurations with Azure Kubernetes to facilitate storage configuration. First, it configures an SSH key pair to use between the bastion host and the node. Users can SSH to Kubernetes nodes from bastion host as soon as terraform apply is completed. Second, the third party storage options installed after the cluster creation need their Pod to instruct Azure to create Azure disks. This requires that a Kubernetes node agent have the permission to provision resources in the node resource group. This is important to understand because there are a couple of managed identities at play (refer to this post) when building an AKS cluster.

In Azure Kubernetes, it is the managed identity of kubelet, that needs to have contributor permission over the resource group for the nodes (not the one for the AKS cluster itself). A managed identity is expressed by a client ID, an object ID (aka principal ID), and the identity ID. We can find them out with an AZ CLI command as below:

We can also tell that the kubelet managed identity represents node agent, by connecting to a node and looking at the argument (kubernetes.azure.com/kubelet-identity-client-id) of kubelet process:

In the template, I also chose to designate the same BYO identity for both the cluster and for kubelet (node agent), in order to minimize my requirement on permission. If I had left it with a system assigned identity for node agent, I would have to assign that identity as a contributor for the node resource group, either as a user, or via Terraform’s identity. Either way, it is beyond what a Contributor is allowed to do.

Benchmarking with kubestr

I used fio utility for storage benchmarking from virtual machines. However, fio utility is not for container. For fio testing on Kubernetes, I’d have to use a Docker image, and test with target volume attached. Fortunately, the Kasten team shared their initiative in the open source project Kubestr. The kubestr release is available as an executable on common platforms. It connects to the cluster the same way as kubectl and here is a demo. To begin with, download the utility to bastion host, and run it without any argument, which prints the storage classes and volume snapshot classes:

curl -L -o kubestr.tar.gz https://github.com/kastenhq/kubestr/releases/download/v0.4.31/kubestr_0.4.31_Linux_amd64.tar.gz
tar -xvf kubestr.tar.gz && rm kubestr.tar.gz && chmod +x kubestr

./kubestr # if kubectl is configured, this command will print out the details of storage classes and volume snapshot classes

In addition to outputting details, it is also very simple to perform storage benchmarking with kubestr. All we need to do is giving it the storage class name and it will run four tests by default with common global options (ioengine=libaio verify=0 direct=1 gtod_reduce=1). The four tests are:

JobName	block_size	filesize	iodepth	rw
read_iops	4k	2G	64	randread
write_iops	4k	2G	64	randwrite
read_bw	128k	2G	64	randread
write_bw	128k	2G	64	randwrite

During each test, it measures and reports IOPS and bandwidth (throughput). If your I/O profile falls out of the four jobs, you can even customize your test by supplying a fio config file. For example, you need a longer test duration, or you need a larger total size for the test. Before the test, kubestr automatically mount their test volumes using the storage class being tested.

./kubestr fio -s my-storage-class # benchmarking a storage class

For read_iops and write_iops, we mainly look at the IOPS. For read_bw and write_bw, we mainly look at the bandwidth. The iops and bw based on samples are reported as first line of result, followed by min, max and average.

Metrics

With kubestr I ran a performance test amongst the native storage classes with Azure Kubernetes Service, with results as below:

	read_iops	write_iops	read_bw	write_bw
managed-csi	IOPS=314.729797 BW(KiB/s)=1275	IOPS=297.071136 BW(KiB/s)=1204	IOPS=315.311188 BW(KiB/s)=40887	IOPS=261.048645 BW(KiB/s)=33941
managed-csi-premium	IOPS=493.395844 BW(KiB/s)=1990	IOPS=426.731812 BW(KiB/s)=1723	IOPS=455.950348 BW(KiB/s)=58894	IOPS=422.888855 BW(KiB/s)=54662
azurefile-csi	IOPS=259.333282 BW(KiB/s)=1053	IOPS=283.985779 BW(KiB/s)=1152	IOPS=240.447403 BW(KiB/s)=31298	IOPS=230.689804 BW(KiB/s)=30048
azurefile-csi-premium	IOPS=394.044739 BW(KiB/s)=1586	IOPS=371.181793 BW(KiB/s)=1494	IOPS=380.360535 BW(KiB/s)=49018	IOPS=491.313446 BW(KiB/s)=63310

These numbers are based on default test parameters (e.g. 100Gi PVC size). The numbers indicate that block storage generally performs better in default setting. So we should use managed disk instead of azure file unless multiple pods needs to access the same volume.

Validate Persistent Storage

We can use mysql as a quick and dirty test of storage classes. We can deploy the following manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: mysql-pvc
spec:
 storageClassName: managed-csi-premium # the storage class being tested
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi
---
apiVersion: v1
kind: Secret
metadata:
 name: mysql
data:
 password: eHl6  # base64 code of xyz
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
 labels:
 app: mysql
spec:
 replicas: 1
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - image: mysql:5.6
 name: mysql
 env:
 - name: MYSQL_ROOT_PASSWORD
 valueFrom:
 secretKeyRef:
 name: mysql
 key: password
 ports:
 - containerPort: 3306
 name: mysql
 volumeMounts:
 - name: mysql-persistent-storage
 mountPath: /var/lib/mysql
 volumes:
 - name: mysql-persistent-storage
 persistentVolumeClaim:
 claimName: mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
 name: mysql-service
 labels:
 service: mysql
spec:
 selector:
 app: mysql
 ports:
 - name: tcp-mysql
 protocol: TCP
 port: 3306
 targetPort: 3306

Once the Pod has been created, then we use a throw-away Pod to connect to mysql service and build some data:

$ kubectl run mysql-cli --rm -i --tty --image imega/mysql-client -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 9 Server version: 5.6.51 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]>

From the mysql shell, we create a test database with dummy data populated:

CREATE DATABASE `testdb`; 
USE testdb; 
CREATE TABLE IF NOT EXISTS tasks ( 
 task_id INT AUTO_INCREMENT PRIMARY KEY, 
 title VARCHAR(255) NOT NULL, 
 description TEXT, 
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP 
) ENGINE=INNODB; 
INSERT INTO tasks (title, description) VALUES ('Job A','Morning Standup');
INSERT INTO tasks (title, description) VALUES ('Job B','Latte with two shots of espresso');
INSERT INTO tasks (title, description) VALUES ('Job C','Coding coding and coding');
INSERT INTO tasks (title, description) VALUES ('Job D','git commit');

Each SQL command should return with number of rows affected and then we can exit the MySQL shell and Pod shell. Once we exit out of the Pod shell, the Pod is deleted. We can re-connect to validate the data are still present:

$ kubectl run mysql-tester --rm -i --tty --image imega/mysql-client -- mysql --host=mysql-service.default.svc.cluster.local --user=root --password=xyz --database testdb --execute='SELECT * FROM tasks;'
If you don't see a command prompt, try pressing enter.
Got error: Access denied for user 'root'@'147.206.3.15' (using password: NO)
+---------+-------+----------------------------------+---------------------+
| task_id | title | description | created_at |
+---------+-------+----------------------------------+---------------------+
| 1 | Job A | Morning Standup | 2022-06-22 20:29:12 |
| 2 | Job B | Latte with two shots of espresso | 2022-06-22 20:29:12 |
| 3 | Job C | Coding coding and coding | 2022-06-22 20:29:12 |
| 4 | Job D | git commit | 2022-06-22 20:29:13 |
+---------+-------+----------------------------------+---------------------+
Session ended, resume using 'kubectl attach mysql-tester -c mysql-tester -i -t' command when the pod is running
pod "mysql-tester" deleted

The output validates the persistent storage of the data. When testing a different storage class, simply start over with a different storage class specified in the PVC part of the manifest.

Summary

This post focuses on storage options for persistent volumes on Azure Kubernetes service. I use my own Terraform template with some custom configuration. I also covered kubestr as benchmarking tool and ran it against the built-in storage classes.

In some situations, the built-in options do not suit your needs. For example, you might want your application to use persistent volume in a consistent way across multiple cloud vendors. We will have to resort to third-party software defined storage layer. In the next post, I will explore a couple of SDS-based options, namely Portworx and Ceph. The custom configurations in the Terraform template will be helpful when we configure Portworx.

FSx ONTAP – Enterprise storage on AWS

Sat, 14 May 2022 22:28:00 -0400

Even though object storage has gained a lot of popularity, file storage is still prevalent. AWS has Elastic File System but the performance is insufficient for enterprise workload. The FSx product line has enterprise storage options and on Sept 2, 2021, AWS launched FSx ONTAP.

This post is my impression about FSx ONTAP. As previously discussed, FSx ONTAP is a managed NetApp storage service by AWS. Essentially, AWS installs NetApp arrays in their data centres, so that users can provision ONTAP volumes from AWS console, or using AWS CLI.

FSx ONTAP

ONTAP (or Data OONTAP) has been a very successful operating system to manage storage arrays. It was so successful that NetApp uses ONTAP as brandname for their storage arrays. This is similar to Isilon, a name of BSD based operating system to manage storage and later becomes brandname of EMC’s storage products. ONTAP has been in competition with other enterprise storage players such as EMC Isilon, HP 3PAR, etc and AWS landed on ONTAP as their partner for enterprise storage. It appears that NetApp still owns their ONTAP storage technology. AWS operates the data centre and provides capability via the CLI layer and console.

Sometimes, people confuses FSx ONTAP with NetApps offering Cloud volumes ONTAP. The two are fundamentally different. Cloud volumes ONTAP works with Cloud Manager (available as self-hosted or SaaS) as the management UI. It manages volumes provisioned from cloud vendor such as AWS, Azure, etc. Clients often configure these “Cloud volumes” as extension to an existing on-premise ONTAP storage deployment. When their on-prem ONTAP volumes fall short of space, the Cloud Manage is aware of a remotely available, cloud backed volume to move cold data off to.

Client Tool

FSx ONTAP is essentially an ONTAP storage cluster sitting in AWS data centre. Users have the options of using AWS CLI or ONTAP CLI to manage the cluster. Users with storage administrator background are likely to prefer the latter. In my professional service experience, I have taken some iterations to come to best practice to use the right tool to interact with ONTAP resources. In a nut shell, it depends on the level of resource that we are interacting with. I categorize those resources into two classes:

Category	Example	Identification
AWS-level resource	FSx ONTAP file system, Storage Virtual Machine	These resources are from ONTAP but they are identified as AWS resources (with ARN). They are also exposed to AWS SDK and can be managed by AWS CLI.
ONTAP-native resource	Volume, Snapshot policy, Schedule, Snapmirror relationship, Vserver	These resources come from ONTAP and can only be managed using ONTAP CLI. The AWS CLI cannot manage _these resources simply because they are not exposed to AWS SDK. Some type of resource such as Volume, may be managed by AWS CLI with very limited options. So we still prefer ONTAP CLI to manage resources.

The AWS CLI (v 2.2.37) has very limited options when creating volumes. For example, the create-volume documentation states that the OntapVolumeType section of output can display types of RW, DP (data protection), or LS. However, it doesn’t allow users to create a volume other than the default RW type. When we configure SnapMirror destination, we need a volume of DP type. We had to use ONTAP CLI to achieve that.

Our practice works out to be: use AWS CLI to create a file system and storage virtual machine. Then we use ONTAP CLI to create everything else. Even though AWS CLI intend to support volume creation, we prefer ONTAP CLI for full functionality support, and alignment with the ONTAP literature.

Administration Tasks

You should configure most of the administrative tasks with ONTAP CLI. We use the documentation by ONTAP as reference. For example, when a volume runs out of Inode, the AWS CLI reports that there is no space left. We need to increase the inode limit and this is, again, not something that AWS CLI can manage. We’d have to use the volume modify command from ONTAP CLI.

NFS version

The AWS document states that FSx ONTAP supports NFSv3.0, v4.0 and v4.1. However, FSx ONTAP is currently backed by NetApp ONTAP 9.10.0, which partially supports NFSv4.2, with basic protocol and Labled NFS feature. NetApp’s ONTAP Best Practices and Implementation Guide suggests a method for clients to mount as NFSv4.2. In the POC we mount as NFS v4.2 in all of our testings.

Clarity of terminology

The ONTAP storage system has been around for a while and many of its concepts are well known in the storage community. For example, A “Snapshot copy” is a read-only, point-in-time image of a volume. (ref: ONTAP 9 documentation -> ONTAP concepts -> Replication -> Snapshot copies) This concept becomes “Snapshot” in AWS literature. It took us some research to come to realize that “Snapshot” in AWS document, essentially maps to “Snapshot copy” in ONTAP literature.

This creates confusion, because we use ONTAP documentation for operation guidance because we can’t get enough help from AWS documentation. The terminology in AWS documentation should align with ONTAP.

The other example is the difference between “backup” and “snapshot” in AWS documentation. It is my understanding that they both use the same underlying Snapshot technology on the ONTAP side. I’m not exactly sure what their difference is.

ONTAP can create a Snapshot copy nearly instantaneously. However, when taking a snapshot using web console in AWS, it takes up to 10 minutes to update the status. This is confusing because it creates a perception that it takes 10 minutes to complete snapshot.

Cross Region Replication

There is a document page on AWS about using SnapMirror at a very high level. It points to two documents: using NetApp Cloud Manager and ONTAP CLI.

The former is not a viable option as we started natively on FSx ONTAP and do not have NetApp Cloud Manager. As to the latter, we managed to configure cross-region replication with ONTAP CLI following the document and identified some gaps in the documentation. Specifically, it would be helpful if AWS documentation calls out that:

inter-cluster network connectivity is a prerequisite (e.g. via VPC peering, transit gateway)
Port 10000, 11104-11105 must be added to security group for inter cluster communication.
The ONTAP CLI command to validate connectivity between clusters (using the ping command from ONTAP CLI).

With AWS CLI alone it is not possible to configure cross-region replication.

Final words

As someone who lived with enterprise storage for more than a decade, I’m glad to see that cloud vendors brings enterprise storage into their data centre, acknowledging that consumer grade file storage are just insufficient for heavy storage use cases such as medical imaging. FSx ONTAP seems to be in early maturity level. However, since AWS exposes ONTAP CLI access to users, ONTAP professionals are able to leverage its full potential.

File storage vs object storage in the cloud

Thu, 23 Sep 2021 22:54:00 -0400

File storage (e.g. NFS) used to be prevalent until object storage comes in for competition.

The competition

Traditionally, enterprise storage product lines are built around three capabilities, as listed in this table below:

Capability	Typical Implementation	Data served
T1 – Block stroage	DAS (e.g. SAS cable) or SAN (Fibre Cable for FCP protocol, or Ethernet for iSCSI protocol)	Mission critical data that are extremely sensitive to latency (e.g. database). Client has block-level access.
T2 – File storage	NAS (connect via CIFS or NFS protocols). Storage arrays are typically a mix of HDD and SSD. Storage servers are usually deployed in the same location over low latency network. DR location is usually in the same region.	Hot data. Multiple client access at file level. The size of each data request varies from small to medium (e.g. text document)
T3 – Object storage	Hardware agnostic, connect via layer-7 protocol (e.g. S3). Storage backend can be either on premise, or in the cloud, over WAN connection.	Warm and code data. Multiple client access at object level. Traditionally for backup but use cases are expanding. The size of each data request varies significantly, from small to very large (e.g. media content).

In the last couple decades, leading players for T2 have been enterprise storage vendors. They each have developed their secret sauces to tackle the challenges. For example, EMC has OneFS, a parallel distributed file system as the foundation of PowerScale (formerly Isilon) product line. NetApp develops ONTAP, featuring proprietary techniques for storage efficiency (deduplication, compaction and compression).

The leading players in T3 are mostly public cloud provider, such as Amazon’s S3. They might work with enterprise storage vendor behind the scene. But the T3 services appear to the end users as provided by the public cloud. Originally, the use case for T3 was archive only for its virtually unlimited capacity. This is not entirely true today. With the drastic improvement in modern network infrastructure, T3 can also brings satisfactory performance to serve hot data.

A competition between T2 and T3 arises. After all, both offer storage service over Ethernet, and both support multiple clients. Today when developers architect the storage layer of their applications, they need to weigh between supporting T2 and T3. Since NFS is the typical protocol for T2 storage (sorry Windows guys) and S3 is typical T3 storage. This competition essentially boils down to NFS versus S3.

For many, the fancy S3 is a no-brainer. While I have suffered from many NFS drawbacks, and there’s even a whole article by Linux folks about why NFS sucks, is it sentenced to death today? Does it beat S3 in some cases? Do so many organizations still stick to NFS just out of inertia?

To answer these questions, I examine four aspects to explore the differences between file storage via NFS protocol, and object storage in S3.

Data request size

Storage client can make request by byte range of a file. Therefore, data request size, instead of file size, is what ultimately matters. I pick a few data request sizes (1K, 4K, 16K, 64K, 246K, 1024K and 4096K) in my experiment, and want to see how much network traffic a write operation produces using NFS and using S3.

To emulate request size, I created files at each size (using dd command), and copy the entire file to each backend. In the mean time, I use tcpdump to write out traffic across the wire into capture files. The size of capture file gives me an idea of how much network traffic went through the network interface, which is closely related to latency.

For NFS, I mounted the target with sync option. This requires NFS client to write out to server synchronously on file copy. I’ve also set the wsize to be 1M. For S3, I simply use the following CLI command to copy file:

aws s3 cp 1kb.img s3://digihunch5ffafe32ab0fd40f

On the network interface, I use tcpdump to filter traffic through specific TCP port (443 for S3, or 2049 for NFS) and record the size of the capture file:

sudo tcpdump -s0 -pi eth0 dst port 443 or src port 443 -w /tmp/4096kb.cap

The key indicator is the payload size (file size) as a percentage of the capture size. I call it payload ratio. The closer it is to 1, the better. I have the following result from my experiment:

Request	Payload	S3 capture size (byte)	NFS capture size (byte)	S3 payload ratio	NFS payload ratio
1K	1024	9352	4332	0.11	0.24
4K	4096	12640	7404	0.32	0.55
16K	16384	25969	20472	0.63	0.80
64K	65536	79183	69746	0.83	0.94
256K	262144	290035	271408	0.90	0.97
1024K	1048576	1085366	1074076	0.97	0.98
4096K	4194304	4381547	4286910	0.96	0.98

This result indicates that NFS has a higher ratio in all groups. However, its advantage diminishes as the data request size grows. What it tells us is that if your applications workload issues most request in small chunks of data, such as 1K, 4K, then NFS will require much less traffic over the network, and thus less latency.

This essentially explains the use case of NFS against S3: workload with small data requests.

Client Support

NFS is natively supported by Linux operating system kernel. NFS client sits below the virtual file system (VFS) layer, which sits below the system call layer. The NFS client translate system calls into RPC (remote procedure calls). Communication between client and server is completed with RPC, on top of TCP.

NFS architecture

Because of the native support, in most cases, developer can treat NFS mounts as if they were local. For performance to be sustainable as file system grows, the directory structure on NFS should follow a certain naming conventions so that files are evenly distributed across directories. The client should also use list operation as sparse as it can because that operation is expensive across the network.

From developer’s perspective, NFS support is brought in by operating system and does not require much effort. On the other hand, S3 client support is not included by default in the operating system. S3 support requires special library, code changes, and integration effort to manage dependency and library version.

NFS has an advantage on client supportability. However, as we move applications to containers, and as container storage options mature, we will need an intermediary layer (storage class, storage provisioner, CSI driver, etc), NFS, or in general file storage, does not have this advantage any more.

Client-side Cache

The NFS support behind VFS layer also means it can leverage the I/O caching mechanism on the client side, that comes with operating system. Client operating system with sufficient memory can take advantage of this mechanism to give it a performance boost. Check out this guide for NFS cache tuning.

In comparison, S3 does not have a cache mechanism by itself. Either the application needs to implement its own cache mechanism, or a cache architecture needs to be introduced, such as CloudFront.

Consistency and concurrency

A common consistency problem is whether client can read the changes immediately after it writes the file. S3 and NFS make a tie in this round.

S3 originally came with eventual consistency model for read after write since 2006. As of Dec 2020 it introduced strong read-after-write consistency. For more information, refer to the guide here.

NFS has a similar consistency guarantee called close-to-open cache coherency. Any changes made by client are flushed to the server on closing the file, and a cache revalidation occurs when you re-open it.

There are more to consider in terms of consistency. For example, multiple clients tries to write the same file/object at the same time.

On the S3 side, there is a locking mechanism called S3 object lock at object level (no byte-range lock). Without an object lock, when two PUT requests are simultaneously made to an object, the request with the latest timestamp wins. Refer to the section Concurrent application on this page.

As far as NFS goes, managing this kind of consistency problem is not in the scope of the standard. Although there are some tinkers. For example, NFS v4 includes a file locking mechanism. Client can choose to lock the entire file, or a byte range within the file. Locking can be mandatory or advisory.

The convergence

NFS and S3 each has their respective advantage. Enterprise NAS customers have been looking for ways to expand into the cloud for lower storage cost. To combine the advantages of the two, solution providers started to converge file storage and object storage. There are two types of solutions that reflects this trend of convergence. In the first trend, enterprise NAS deployed on premise now have the ability to scale out into the cloud. In the second trend, public cloud just brought enterprise NAS into their product offerings.

Scale-out NAS

NAS is traditionally expensive to scale because it requires physical storage media. The idea of scale-out NAS allows NAS to connect to object storage in the public cloud, making it a hybrid architecture. This essentially makes T3 storage as a backend of T2 and it can be implemented with a virtual storage appliance (VSA). The VSA translate file system activities into API calls for object storage operations. One example is AWS storage gateway. EMC has a similar appliance called ECS and this white paper explains how it proxies file system calls and interact with object backends. NetApp, a vested enterprise NAS provider, also has a counterpart called Cloud Volumes ONTAP (CVO). It works well with NetApp on-premise deployment, but the architecture is similar. Here‘s NetApp’s take on how CVO is different than AWS Storage Gateway.

In the scale-out NAS architecture, the public cloud acts merely as extension to on-premise storage solution, to provide capacity. The NAS on premise serves the storage workload primarily.

Cloud hosted NAS

For applications hosted in public cloud, it makes sense for public cloud provider to operate enterprise NAS storage as a service. The underlying storage technology is provided by storage vendor. It is just installed in the data centre managed by the public cloud vendor, instead of customer’s own data centre.

One example is Azure NetApp Files (ANF). ANF is fully managed services, presented to users as storage volumes. The underlying storage technology is NetApp ONTAP. Because it is offered as a fully managed service, the customers are not able to manage the fine details of the storage, as they could with an ONTAP cluster on premise. This takes a lot of flexibility away from the user.

FSx ONTAP is a managed NetApp storage service by AWS, launched in September 2021. The NetApp arrays are installed in AWS data centre, ready for users to provision from AWS console, or using CLI. The Terraform provider support is not available as of yet. Unlike ANF, FSx ONTAP exposes the ONTAP CLI to users, allowing for advanced storage managed by storage gurus. They can use ONTAP CLI commands to configure custom policy for Snapshot, setup SnapMirror replication, and so forth.

Likewise, PowerScale landed on GCP as public cloud partner to launch Dell Cloud PowerScale for Google Cloud in 2020. However, it seems to require a purchase agreement before APIs are enabled.

Conclusion

Object storage has a great momentum and some sees that as a replacement of file storage in the long run. However file storage has its advantages for small data requests, OS-level cache support, and built-in POSIX compatibility. It will continue to be an option for customers with specific workload. Customer stickiness to file storage is so firm, that public cloud providers now install them in their data centres.

From competition to collaboration, it will be interesting to watch what happens next for enterprise storage.

Follow-up Reading

Tom Lyon’s presentation on why NFS must die.