asynchronous replication on Digi Hunch

EMC Isilon storage product

Wed, 08 Jul 2020 20:04:00 -0400

EMC has several product lines for different use cases in enterprise data storage. Like may other IT solutions, the website is clouded with marketing terms and slogans, and is purposefully not technical. This makes it difficult for technical staff to grasp the advantage of its product in a glimpse. I personally have to know their product (mostly with Isilon and ECS) well in order to make integration decisions. So I’m putting together this note (updated as of July 2020), with lots of details from their technical white paper.

Overview of EMC storage

At the highest level, the EMC enterprise data storage product lines are categorized into two groups: primary storage (along the lines of block-level storage) and unstructured storage (mostly file and object storage). The primary storage includes the following product:

PowerMax for OLTP database (Oracle, MicrosoftSQL and SAP)
PowerFlex: for Software defined storage, Oracle RAC, Elastic Stack, Kubernetes, Splunk
XtremIO for VMware, VDI, SAP
PowerStore for Database, VMware
PowerVault for Entry-level SAN and DAS environment

This post only expands on the unstructured storage product line, which mainly consists of PowerScale and ECS. ECS (elastic cloud storage) is EMC’s object storage. PowerScale (aka Isilon) is scale-out NAS platform for high-volume storage (up to 50 PB in a single file system), backup and archiving of unstructured data. For the rest of this post, I will still refer to PowerScale as Isilon.

Dell’s official support website is the most resourceful place to get information. For example, when I want to read about Isilon. I start with Dell support, then click on knowledgebase at the top, then go to “servers, storage and networking“, then “storage technical documents and videos“. There I can select a productline such as Isilon.

Overview of Isilon Family

Isilon is a clustered storage system consisting of three or more nodes. A node is a server with OneFS as its operating system. Based on FreeBSD, OneFS is EMC’s proprietary operating system to unify a cluster of nodes into a single shared resource. So OneFS is for Isilon only. It is the basis of Isilon. Isilon has three series of products:

F series: F200, F600, F800 and F810.
H series: typical models are H400, H500 and H600, which seeks to balance performance and capacity
A series: typical models are A200 and A2000 for active and deep archive storage

In June 2020, Dell decoupled OneFS software (with 9.0 released) from server hardware (referred to as PowerScale). Going forward EMC will refer to Isilon as PowerScale for OneFS version newer than 9.0 in spec sheets and white papers.

F200 is the cost-effective choice with SSD for remote office, small hospital, retail outlets, IOT or factory floor. F600 uses NVMe drives instead, and has more ECC memory and faster ethernet backend network. and is higher than F200 in its use case. Both F200 and F600 provide inline data compression and deduplication capabilities. F800 and F810 both use SSD and they are similar. F800 comes with InfiniBand backend network and F810 provides inline data compression and deduplication capabilities. H series tries to strike a balance between performance and capacity so they are pretty much everything in betwee. On the other end, A200 and A2000 are almost the same except for capacity difference.

Isilon’s advantage

Isilon has lots of intelligence built into its solution compared to a traditional NAS. Here are some aspects from its product white paper:

Aspects of Design	Isilon OneFS Scale-Out NAS	Traditional NAS
Network	Separation of front-end and back-end network to isolate node-to-node communication to a private low-latency network. Front-end traffic load balanced with SmartConnect	Single network for both external and internal traffic
File system structure and NameSpace	The storage is completely virtualized to users as a truly single file system with one namespace. There is no partitioning or volumes. The single file tree can grow organically without requiring planning or oversight about how the tree grows. SmartPool handles tiering of files to appropriate disk, without disrupting the single file tree.	An appearance of single namespace is typically achieved through namespace aggregation, where files are still managed in separate volumes, and a simple “veneer” layer glues individual directories to a “top-level” tree via symbolic links. LUNs and volumes, as well as volume limits are still present. Files have to be manually moved from volume-to-volume to load-balance.
Data Layout	OneFS controls the placement of file directly, down to the sector-level on any drive anywhere in the cluster. The addressing scheme for data and metadata is indexed at physical level by a tuple of {node, drive, offset}	Data are sent through RAID and volume management layers, introducing inefficiencies in data layout and providing non-optimized block access.
Redundancy Control	OneFS can flexibly control the type of striping as well as the redundancy level of the storage system at the system, directory and even file-levels.	The entire RAID volume is dedicated to a particular performance type and protection setting.

Isilon terms

The Isilon technology re-implemented the read and write path during file storage and introduced several terms along with its technology.

SmartPools – Job that runs and moves data between the tiers of nodes within the same cluster. Also executes the CloudPools functionality if licensed and configured. FilePolicy is changelist-based SmartPools file pool policy job. SmartPoolsTree enforces SmartPools file policies on a subtree.

Storage Pools – Storage pools provide the ability to define subsets of hardware within a single cluster, allowing file layout to be aligned with specific sets of nodes through the configuration of storage pool policies. The notion of Storage pools is an abstraction that encompasses disk pools, node pools, and tiers.

Disk Pools – Disk pools are the smallest unit within the storage pools hierarchy. OneFS provisioning works on the premise of dividing similar nodes’ drives into sets, or disk pools, with each pool representing a separate failure domain. Disk pools are laid out across all five sleds in each node.

Node Pools – groups of disk pools, spread across similar storage nodes (or equivalent classes). Multiple groups of different node types can work together in a single, heterogeneous cluster. For example, one node pool of all-flash F-Series anodes, one node pool of H-series, and one node pool of A-series. Each node pool only contains disk pools from the same type of storage nodes.

Tiers – groups of nodepools combined into a logical superset to optimize data storage, according to OneFS platform type. this allows customers who consistently purchase highest capacity nodes available to consolidate a variety of node styles within a single tier, and manage them as one logical group. SmartPools users typically deploy 2 to 4 tiers. different node pools under a tier needs to be compatible.

Global Namespace Acceleration (GNA)’s principal goal is to help accelerate metadata read operations by keeping a copy of a cluster’s metadata on high performance, low latency SSD media.

SmartConnect is a load balancer that works at the front-end Ethernet layer to evenly distribute client connections across the cluster. SmartConnect supports dynamic NFS failover and failback to ensure that when a node failure occurs, or preventative maintenance is performed, all in-flight reads and writes are handed off to another node in the cluster to finish its operation without any user or application interruption.

Auto Balance reallocates and rebalances data and make storage space more usable and efficient.

SmartQuotas is directory-level quota management. Note: there is no partitioning, and no need for volume creation in OneFS.

SmartRead creates a data “pipeline” from L2 cache, prefetching into a local “L1” cache, on the captain node, in order to greatly improve sequential-read performance. For high-sequential cases, SmartRead can very aggressively prefetch ahead. SmartRead can control how aggresive the pre-fetching is, and how long data stays in the cache, and optimizes where data is cached.

In-line Data Reduction – the write path involves zero block removal, in-line deduplication, and in-line compression. This is supported in some models only.

Smart Dedupe – post-process, asynchronous deduplication. Smart Dedupe scans the on-disk data for identical blcoks and then eliminate the duplicates. After duplicate blocks are discovered, SmartDedupe movees a single copy of those blocks to a special set of files known as shadow stored. With post-process deduplication, new data is first stored on the storage device and then a subsequent process analyzes the data looking for commonality. This means that initial file write or modify performance is not impacted, since no additional computation is required in the write path, as opposed to in-line deduplication. This is supported on some models only.

OneFS SSD strategy – How OneFS leverage the SSD for performance. It has these options:

L3 cache (implemented at nodepool level)
metadata read
metadata read/write
Global Namespace Acceleration (GNA)
Data on SSD

L3 cache consumes all the SSD in node pool. L3 cannot coexist with other SSD strategies, with the exception of GNA just because L3 cache node pool SSD cannot participate in GNA.

Isilon’s High Availability

The OneFS is distributed across all nodes in the cluster and is accessible by clients connecting to any node in the cluster. Metadata and locking tasks are managed by all nodes collectively and equally in a peer-to-peer architecture. This symmetry is key to the simplicity and resiliency of the architecture. There is no single metadata server, lock manager or gateway node.

The entire cluster forms a single file system with a single namespace that runs across every node equally. No one node controls or “masters” the cluster; all nodes are true peers.

During failover, clients are evenly redistributed across all remaining nodes in the cluster, ensuring minimal performance impact. If a node is brought down for any reason, including a failure, the virtual IP addresses on that node is seamlessly migrated to another node in the cluster. When the offline node is brought back online, SmartConnect automatically rebalances the NFS and SMB3 clients across the entire cluster to ensure maximum storage and performance utilization. This functionality allows for per-node rolling upgrades affording full-availability throughout the duration of the maintenance window.

There are two logical roles in processing an I/O request from client:

The initiator: the node that the client connects to with front-end protocol. The initiator acts as the ‘captain’ for the entire I/O operation.
The participant: Every node in the cluster is a participant for a particular I/O operation.

File Write in Isilon

OneFS employs a patented transaction system during write to eliminate single point of failure. In a write operation, the initiator “captains” or orchestrates the layout of data and metadata, the creation of erasure codes, and the normal operations of lock management and permission control.

When a client connects to a node to write a file, it is connecting to the Initiator. OneFS breaks the file down into atomic units. An atomic unit is a smaller logical chunk of data, also called stripe, or protection groups in the context of data protection. The size of each file chunk is referred to as the stripe unit size. After this division, OneFS then write the stripe individually to the Participant (with disks). This design ensures that data is protected at the specified level as soon as it is being written. Redundancy is built into protection groups, such that if every protection group of a file is safe, then the entire file is safe. In terms of protection mechanism, OneFS can use either Reed-Solomon erasure coding system, or simply mirroring for data protection. Erasure coding is the predominant mechanism with very high performance without sacrificing on-disk efficiency.

The initiator node uses a modified two-phase commit transaction to safely distribute writes to multiple NVRAMs across the cluster. As client initiates write to OneFS cluster, instead of immediately writing to disk, OneFS temporarily writes the data to an NVRAM-based journal cache on the initiator node and acknowledge the write the client. As outlined above, these writes are also mirrored to participant nodes’ NVRANM journals to satisfy the file’s protection requirement. Later, at a more convenient time, OneFS then flush these cached writes to disks asynchronously.

Since NVRADM journals all the transactions that are occurring across every node in the storage cluster. If a node fails mid-transaction, and then re-joins the cluster, the uncommitted cached writes are fully protected, and the only required actions for the node, are to replay its journal from NVRAM, and occasionally for AutoBalance to rebalance files that were involved in the transaction. Writes are never blocked due to a failure. There is no ‘fsck’ or ‘disk-check’ process.

OneFS file system block size is 8KB. A file smaller than 8KB will use a full 8KB block. For larger files, OneFS can maximize sequential performance by taking advantage of a stripe unit consisting of 16 contiguous blocks, for a total of 128KB per stripe unit.

Cache in Isilon

OneFS aggregates the cache present on each node in a cluster into one globally accessible pool of memory by using a messaging system similar to NUMA (non-uniform memory access). This allows all the nodes’ memory cache to be available to each and every node in the cluster. Remote memory is access over internal network with much lower latency than accessing hard disk drives. The internal network as distributed system bus, is a redundant, under-subscribed flat Ethernet up to 40Gb. The oneFS caching subsystem is coherent across the cluster, due to the use of MESI protocol to maintain cache coherency. If the same content exists in the private caches of multiple nodes, this cached data is consistent across all instances.
OneFS uses up to three levels of read cache, plus an NVRAM-backed write cache, or coalescer.

OneFS Caching Hierarchy

L1 cache – prefetches data from remote nodes. Data is prefetched per file, and this is optimized in order to reduce the latency associated with the nodes’ back-end network. The L1 cache refers to memory on the same node as the initiator. It is only accessible to the local node, and typically the cache is not the master copy of the data.

L1 is also known as remote cache because it contains data retrieved from other nodes in the cluster. It is coherent across the cluster but is used only by the node on which it resides and is not accessible by other nodes. Data in L1 cache on storage nodes is aggressively discarded after it is used. L1 cache uses file-based addressing, in which data is accessed via an offset into a file object.

OneFS also uses a dedicated inode cache in which recently requested inodes are kept. The inode cache frequently has a large impact on performance, because clients often cache data, and many network I/O activities are primarily requests for file attributes and metadata, which can be quickly returned from the cached inode.

L2 cache (backend cache) refers to local memory on the node on which a particular block of data is stored. L2 cache is globally accessible from any node in the cluster and is used to reduce the latency of a read operation by not requiring a seek directly from the disk drives.

L2 cache is also known as local cache because it contains data retrieved from disk drives located on that node and then made available for requests from remote nodes. Data in L2 cache is evicted according to a Least Recently Used (LRU) algorithm.

L3 cache, or Smart Flash, is configurable on nodes that contain solid state drives. Smart Flash (L3) is an eviction cache that is populated by L2 cache blocks as they are aged out from memory.

During I/O request, clients talk to L1 cache and write coalescer; L1 cache talks to L2 cache on all cluster nodes. L2 cache buffers to and from disks. L3 cache is optionally enabled per node pool, as an extension from L2. L3 and L2 communicate in backend network.

Name	Medium	Description
L1 Cache (aka front-end cache or remote cache)	RAM (volatile)	holds clean, cluster coherent copies of file system data and metadata blocks requested by clients via front-end network
L2 Cache (aka back-end cache or local cache)	RAM (volatile)	contains clean copies of file system data and metadata on a local node
SmartCache (Write Coalescer)	Battery-backed NVRAM (Persistent)	a persistent journal cache that buffers any pending writes to front-end files that have not been committed to disk
SmartFlash or L3 Cache	SSD (persistent)	contains file data and metadata blocks evicted from L2 cache, effectively increasing L2 cache capacity

File Read in Isilon

The high-level steps for fulfilling a read request with cache interaction involves:

Step 1 – on local node, determine whether part of the requested data is in the local L1 cache:

if so, return to client
if not, request data from remote nodes

Step 2 – on remote nodes, determine whether requested data is in the local L2 or L3 cache:

if so, return to the requesting node
if not, read from disk and return to requesting node

During a read operation, the “captain” node gathers all of the data from the various nodes in the cluster and presents it in a cohesive way to the requestor.

The cluster provides a high ratio of cache to disk (multiple GB per node) that is dynamically allocated for read and write operations as needed. This RAM-based cache is unified and coherent across all nodes in the cluster, allowing a client read request on one node to benefit from I/O already transacted on another node. As the cluster grows larger, the cache benefit increases. For this reason, the amount of I/O to disk on a cluster is generally substantially lower than it is on traditional platforms.

For files marked with an access pattern of concurrent or streaming, OneFS can take advantage of pre-fetching of data based on heuristics used by the SmartRead component

Conclusion

This post provided a high level introduction to EMC storage product line and expanded into some technical details in the read write operation in OneFS/Isilon. Some of the features can be seen in OneFS simulator which is a free tool from EMC.

Storage Nitty-Gritty 5 of 5 – Replication

Tue, 19 Nov 2019 00:10:23 -0400

Replication Terms

PIT (point in time) replica – snapshot of the source at some specific timestamp;
Continuous Replica – always in-sync with the production data;
Recoverability – enables restoration of data from the replica to the source if data loss or corruption occurs;
Restartability – enables restarting business operations using the replicas;

Local Replication

Use Case:

Alternative source for backup
Fast recovery
Decision-support activities such as data warehousing
Testing platform
Data migration

Consistency in file system replication
File systems buffer the data in the host memory to improve the application response time. The buffered data is periodically written to the disk. In UNIX operating systems, sync daemon is the process that flushes the buffers to the disk at set intervals. In some cases, the replica is created between the set intervals, which might result in the creation of an inconsistent replica. Therefore, host memory buffers must be flushed to ensure data consistency on the replica, prior to its creation.

Flushing the file system buffer

In the illustration above, If the host memory buffers are not flushed, the data on the replica will not contain the information that was buffered in the host. If the file system is unmounted before creating the replica, the buffers will be automatically flushed and the data will be consistent on the replica.

Consistency in database replication
When a database is replicated while it is online, changes made to the database at this time must be applied to the replica to make it consistent. A consistent replica of an online database is created by using the dependent write I/O principle or by holding I/Os momentarily to the source before creating the replica.

A dependent write I/O principle is inherent in many applications and database management systems (DBMS) to ensure consistency. According to this principle, a write I/O is not issued by an application until a prior related write I/O has completed. For example, a data write is dependent on the successful completion of the prior log write.

For a transaction to be deemed complete, databases require a series of writes to have occurred in a particular order. These writes will be recorded on the various devices or file systems.

Another way to ensure consistency is to make sure that the write I/O to all
source devices is held for the duration of creating the replica. This creates a
consistent image on the replica. However, databases and applications might time out if the I/O is held for too long.

Local Replication Technologies

Host-based Local Replication

LVM-based replication: logical volume manager (LVM) is responsible for creating and controlling the host-level logical volumes. Each logical block in a logical volume is mapped to two physical blocks on two different physical volumes. LVM-based replication is part of operating system and comes without additional license cost. However, every write generated by application translates into two writes on the disk, and thus, an additional burden is placed on the host CPU. This can degrade application performance. Presenting an LVM-based logical replica to another host is usually not possible because the replica will still be part of the volume group, which is accessed by one host at any given time. You can’t track changes on LVMs either so it does not support incremental resynchronization.

File system snapshot: a pointer-based replica that requires a fraction of the space used by the production FS. This snapshot can be implemented by either FS or by LVM. It uses the Copy on First Write (CoFW) principle to create snapshot. When a snapshot is created, a bitmap and blockmap are created in the metadata of the Snap FS. The bitmap is used to keep track of blocks that are changed on the production FS after the snap creation. The blockmap is used to indicate the exact address from which the data is to be read when the data is accessed from the Snap FS. Immediately after the creation of the FS snapshot, all reads from the snapshot are actually served by reading the production FS. In a CoFW mechanism, if a write I/O is issued to the production FS for the fi rst time after the creation of a snapshot, the I/O is held and the original data of production FS corresponding to that location is moved to the Snap FS. Then, the write is allowed to the production FS. The bitmap and blockmap are updated accordingly. Subsequent writes to the same location do not initiate the CoFW activity. To read from the Snap FS, the bitmap is consulted. If the bit is 0, then the read is directed to the production FS. If the bit is 1, then the block address is obtained from the blockmap, and the data is read from that address on the Snap FS. Read requests from the production FS work as normal.

File system snapshot

Storage Array-based local replication
the array-operating environment performs the local replication process. The host resources, such as the CPU and memory, are not used in the replication process. Consequently, the host is not burdened by the replication operations. The replica can be accessed by an alternative host for other business operations.

Full-Volume Mirroring – the target is attached to the source and established as a mirror of the source. After all the data is copied and both the source and the target contain identical data, the target can be considered as a mirror of the source. After the synchronization is complete, the target can be detached from the source and made available for other business operations. The target becomes a point-in-time (PIT) copy of the source. After detachment, changes made to both the source and replica can be tracked at some predefined granularity. This enables incremental resynchronization (source to target) or incremental restore (target to source). The granularity of the data change can range from 512 byte blocks to 64 KB blocks or higher.

Full volume mirroring

Pointer-based, Full-Volume Replication – the target is immediately accessible by the BC host after the replication session is activated. Therefore, data synchronization and detachment of the target is not required to access it.

Pointer-based, Virtual Replication – at the time of the replication session activation, the target contains pointers to the location of the data on the source. The target does not contain data at any time. Therefore, the target is known as a virtual replica. the target is immediately accessible after the replication session activation. A protection bitmap is created for all data blocks on the source device. Granularity of data blocks can range from 512 byte blocks to 64 KB blocks or greater.

Network-based local replication: the replication occurs at the network layer between host and storage arrays. By offloading replication from servers and arrays, network-based replication can work across a large number of server platforms and storage arrays, making it ideal for highly heterogeneous environments.

Continuous Data Protection: CDP provides the ability to restore data to any previous PIT. In CDP, data changes are continuously captured and stored in a separate location from the primary storage. With CDP, recovery from data corruption poses no problem because it allows going back to a PIT image prior to the data corruption incident. CDP uses a journal volume to store all data changes on the primary storage. The journal volume contains all the data that has changed from the time the replication session started. The amount of space that is configured for the journal determines how far back the recovery points can go. CDP appliance is an intelligent hardware platform that runs the CDP software and manages local and remote data replications. Write splitters intercept writes to the production volume from the host and split each write into two copies. Write splitting can be performed at the host, fabric, or storage array.

CDP Local Replication Operation: before the start of replication, the replica is synchronized with the source and then the replication process starts. After the replication starts, all the writes to the source are split into two copies. One of the copies is sent to the CDP appliance and the other to the production volume. When the CDP appliance receives a copy of a write, it is written to the journal volume along with its timestamp. As a next step, data from the journal volume is sent to the replica at predefi ned intervals.

Continuous Data Protection

Tracking Changes to Source and Replica
Changes can occur on the replica device if it is used for other business operations. To enable incremental resynchronization or restore operations, changes to both the source and replica devices after the PIT should be tracked.
This is typically done using bitmaps, where each bit represents a block of data. For example, if the block size is 32 KB, then a 1-GB device would require 32,768 bits (1 GB divided by 32 KB). The size of the bitmap would be 4 KB. If the data in any 32 KB block is changed, the corresponding bit in the bitmap is flagged. If the block size is reduced for tracking purposes, then the bitmap size increases correspondingly.
The bits in the source and target bitmaps are all set to 0 (zero) when the replica is created. Any changes to the source or replica are then fl agged by setting the appropriate bits to 1 in the bitmap. When resynchronization or restore is required, a logical OR operation between the source bitmap and the target bitmap is performed. The bitmap resulting from this operation references all blocks that have been modifi ed in either the source or replica.
This enables an optimized resynchronization or a restore operation because it eliminates the need to copy all the blocks between the source and the replica. The direction of data movement depends on whether a resynchronization or a restore operation is performed.
If resynchronization is required, changes to the replica are overwritten with the corresponding blocks from the source. If a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.
If a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.

Tracking Changes

Comparison of local replication technologies

Local Replication in a Virtualized Environment
Typically, local replication of VMs is performed by the hypervisor at the compute level. However, it can also be performed at the storage level using array-based local replication, similar to the physical environment. In the array-based method, the LUN on which the VMs reside is replicated to another LUN in the same array. VM Snapshot captures the state and data of a running virtual machine at a specifi c point in time. The VM state includes VM files, such as BIOS, network confi guration, and its power state (powered-on, powered-off, or suspended). The VM data includes all the files that make up the VM, including virtual disks and memory. A VM Snapshot uses a separate delta file to record all the changes to the virtual disk since the snapshot session is activated. Snapshots are useful when a VM needs to be reverted to the previous state in the event of logical corruptions. Reverting a VM to a previous state causes all settings confi gured in the guest OS to be reverted to that PIT when that snapshot was created. There are some challenges associated with the VM Snapshot technology. It does not support data replication if a virtual machine accesses the data by using raw disks. Also, using the hypervisor to perform snapshots increases the load on the compute and impacts the compute performance.

Remote Replication

Synchronous remote replication – writes must be committed to the source and remote replica (or target), prior to acknowledging “write complete” to the host. Additional writes on the source cannot occur until each preceding write has been completed and acknowledged. This ensures that data is identical on the source and replica at all times. Further, writes are transmitted to the remote site exactly in the order in which they are received at the source. Therefore, write ordering is maintained. If a source-site failure occurs, synchronous remote replication provides zero or near-zero RPO. However, application response time is increased with synchronous remote replication because writes must be committed on both the source and target before sending the “write complete” acknowledgment to the host. The degree of impact on response time depends primarily on the distance between sites, bandwidth, and quality of service (QOS) of the network connectivity infrastructure.

Synchronous replication

In asynchronous remote replication, a write is committed to the source and immediately acknowledged to the host. In this mode, data is buffered at the source and transmitted to the remote site later. Asynchronous replication eliminates the impact to the application’s response time because the writes are acknowledged immediately to the source host. This enables deployment of asynchronous replication over distances ranging from several hundred to several thousand kilometers between the primary and remote sites.

Asynchronous replication

Below are the bandwith requirement for both:

Bandwidth requirement for synchronous replication

Bandwidth requirement for asynchonous replication

Asynchronous replication implementation can also take advantage of locality of reference (repeated writes to the same location). If the same location is written multiple times in the buffer prior to transmission to the remote site, only the final version of the data is transmitted. This feature conserves link bandwidth.

Remote Replication Technologies

Host-Based Remote Replication

LVM-based remote replication: performed and managed at the volume group level. Writes to the source volumes are transmitted to the remote host by the LVM. The LVM on the remote host receives the writes and commits them to the remote volume group.
LVM-based remote replication supports both synchronous and asynchronous modes of replication. LVM-based remote replication is independent of the storage arrays and therefore supports replication between heterogeneous storage arrays.
The replication process adds overhead on the host CPUs. CPU resources on the source host are shared between replication tasks and applications. Because the remote host is also involved in the replication process, it must be continuously up and available.

LVM based remote replication

Host-Based Log Shipping
Database replication via log shipping is a host-based replication technology supported by most databases. Transactions to the source database are captured in logs, which are periodically transmitted by the source host to the remote host. The remote host receives the logs and applies them to the remote database.
RPO at the remote site is fi nite and depends on the size of the log and the frequency of log switching. Available network bandwidth, latency, rate of updates to the source database, and the frequency of log switching should be considered when determining the optimal size of the log file. Host-based log shipping requires low network bandwidth because it transmits only the log fi les at regular intervals.

Host based log shipping

Storage Array-Based Remote Replication

Synchronous replication mode
To optimize the replication process and to minimize the impact on application response time, the write is placed on cache of the two arrays. The intelligent storage arrays destage these writes to the appropriate disks later.
If the network links fail, replication is suspended; however, production work can continue uninterrupted on the source storage array. The array operating environment keeps track of the writes that are not transmitted to the remote storage array. When the network links are restored, the accumulated data is transmitted to the remote storage array. During the time of network link outage, if there is a failure at the source site, some data will be lost, and the RPO at the target will not be zero.

Array-based remote synchronous replication

Asynchronous replication mode Fig 12-8
Data is buffered at the source and transmitted to the remote site later. The source and the target devices do not contain identical data at all times. The data on the target device is behind that of the source, so the RPO in this case is not zero. Asynchronous replication writes are placed in cache on the two arrays and are later destaged to the appropriate disks. Some implementations of asynchronous remote replication maintain write ordering. A timestamp and sequence number are attached to each write when it is received by the source. Writes are then transmitted to the remote array, where they are committed to the remote replica in the exact order in which they were buffered at the source. This implicitly guarantees consistency of data on the remote replicas.

Array-based asynchronous replication

Disk-buffered replication mode: a combination of local and remote technologies. A consistent PIT local replica of the source device is fi rst created. This is then replicated to a remote replica on the target array.
At the beginning of the cycle, the network links between the two arrays are suspended, and there is no transmission of data. While production application runs on the source device, a consistent PIT local replica of the source device is created. The network links are enabled, and data on the local replica in the source array transmits to its remote replica in the target array.

Disk buffered remote replication

Network-based Remote Replication

CDP remote replication
Fig 12-10

Three site replication

Cascade/Multihop: data fl ows from the source to the intermediate storage array, known as a bunker, in the fi rst hop, and then from a bunker to a storage array at a remote site in the second hop. Replication between the source and the remote sites can be performed in two ways: synchronous + asynchronous or synchronous + disk buffered. Replication between the source and bunker occurs synchronously, but replication between the bunker and the remote site can be achieved either as disk-buffered mode or asynchronous mode.

Three-site remote replication cascade/multihop

Triangle/Multitarget: data at the source storage array is concurrently replicated to two different arrays at two different sites. The source-to-bunker site (target 1) replication is synchronous with a near-zero RPO. The source-to-remote site (target 2) replication is asynchronous with an RPO in the order of minutes. The distance between the source and the remote sites could be thousands of miles. The key benefit of three-site triangle/multitarget replication is the ability to failover to either of the two remote sites in the case of source-site failure, with disaster recovery (asynchronous) protection between the bunker and remote sites.

Three-site replication triangle/multitarget

Data migration solutions
Data mobility refers to moving data between heterogeneous storage arrays for cost, performance, or any other reason. It helps implement a tiered storage strategy.
Data migration refers to moving data from one storage array to other heterogeneous storage arrays for technology refresh, consolidation, or any other reason. The array performing the replication operations is called the control array.
Data migration solutions perform push and pull operations for data movement.
These terms are defined from the perspective of the control array. In the push operation, data is moved from the control array to the remote array.
The control device, therefore, acts like the source, while the remote device is the target.
In the pull operation, data is moved from the remote array to the control array.
The remote device is the source, and the control device is the target.

The push and pull operations can be either hot or cold. These terms apply to the control devices only. In a cold operation the control device is inaccessible to the host during replication. Cold operations guarantee data consistency because both the control and the remote devices are offl ine. In a hot operation the control device is online for host operations. During hot push and pull operations, changes can be made to the control device because the control array can keep track of all changes and thus ensure data integrity.

Remote replication and migration in a virtualized environment

In hypervisor-to-hypervisor VM migration, the entire active state of a VM is moved from one hypervisor to another. This method involves copying the contents of virtual machine memory from the source hypervisor to the target and then transferring the control of the VM’s disk fi les to the target hypervisor. Because the virtual disks of the VMs are not migrated, this technique requires both source and target hypervisor access to the same storage.

Hypervisor-to-hypervisor VM migration

In array-to-array VM migration, virtual disks are moved from the source
array to the remote array. This approach enables the administrator to move VMs across dissimilar storage arrays. Array-to-array migration starts by copying the metadata about the VM from the source array to the target. The metadata essentially consists of configuration, swap, and log files. After the metadata is copied, the VM disk file is replicated to the new location.

Array-to-array VM migration