Technical Report

ONTAP Select
Product Architecture and Best Practices
Tudor Pascu, NetApp
June 2017 | TR-4517
TABLE OF CONTENTS

1 Introduction ........................................................................................................................................... 5
1.1 Software-Defined Infrastructure ......................................................................................................................5

1.2 Running ONTAP as Software .........................................................................................................................5

1.3 ONTAP Select Versus Data ONTAP Edge .....................................................................................................6

1.4 ONTAP Select Standard versus ONTAP Select Premium ..............................................................................6

1.5 ONTAP Select Evaluation software versus running ONTAP Select in Evaluation mode ................................7

1.6 ONTAP Select Platform and Feature Support.................................................................................................7

2 Architecture Overview ....................................................................................................................... 10

2.1 Virtual Machine Properties ............................................................................................................................11

2.2 VSAN and External Array Configurations (vNAS) .........................................................................................12

2.3 RAID Services for local attached storage......................................................................................................14

2.4 vSphere VMFS Limits ...................................................................................................................................17

2.5 ONTAP Select Virtual Disks ..........................................................................................................................18

2.6 Virtualized NVRAM .......................................................................................................................................19

2.7 High Availability for local attached storage....................................................................................................20

3 Deployment and Management ........................................................................................................... 26

3.1 ONTAP Select Deploy ..................................................................................................................................26

3.2 ONTAP Select Licensing...............................................................................................................................29

3.3 ONTAP Management ....................................................................................................................................30

4 Network Design Considerations ....................................................................................................... 31

4.1 Network Configuration: Multi-node ................................................................................................................31

4.2 Network Configuration: Single Node .............................................................................................................34

4.3 Networking: Internal and External .................................................................................................................35

4.4 Supported Network Configurations ...............................................................................................................37

4.5 vSphere: vSwitch Configuration ....................................................................................................................38

4.6 Physical Switch Configuration .......................................................................................................................41

4.7 Data and Management Separation ...............................................................................................................43

4.8 Four-NIC Configuration .................................................................................................................................45

4.9 Two-NIC Configuration .................................................................................................................................47

5 Use Cases ............................................................................................................................................ 48

5.1 Remote/Branch Office ...................................................................................................................................48

5.2 Schedule-driven SnapMirror relationships periodically replicate the data from the remote office to a single
consolidated engineered storage array, located in the main data center. .....................................................49

5.3 Private Cloud (Data Center) ..........................................................................................................................49

2 TR-4517 Technical Report

6 Upgrading ............................................................................................................................................ 50
6.1 Increasing Capacity ......................................................................................................................................51

6.2 Increasing Capacity for ONTAP Select 9.0 ...................................................................................................52

6.3 Single-Node to Multinode Upgrade ...............................................................................................................54

7 Performance ........................................................................................................................................ 54

Version History ......................................................................................................................................... 59

LIST OF TABLES
Table 1) ONTAP Select versus Data ONTAP Edge. ......................................................................................................6
Table 2) ONTAP Select virtual machine properties. .......................................................................................................9
Table 3) ONTAP Select virtual machine properties. .....................................................................................................11
Table 4) ONTAP Select 9.0 versus ONTAP Select 9.1 versus ONTAP Select 9.2 .......................................................11
Table 5) Internal versus external network quick reference. ..........................................................................................36
Table 6) Network configuration support matrix. ............................................................................................................37
Table 7) ONTAP Deploy vs. ONTAP Select support matrix .........................................................................................50
Table 8) Performance results for a 4-node ONTAP Select Standard cluster and a 4-node ONTAP Select Premium
cluster. ..........................................................................................................................................................................55
Table 9) Performance results for a single-node ONTAP Select Standard cluster on an AF VSAN datastore. .............58

LIST OF FIGURES
Figure 1) Server LUN configuration with only RAID-managed spindles. ......................................................................16
Figure 2) Server LUN configuration on mixed RAID/non-RAID system. .......................................................................17
Figure 3) Virtual disk to physical disk mapping.............................................................................................................18
Figure 4) Incoming writes to ONTAP Select VM. ..........................................................................................................20
Figure 5) Two-node ONTAP Select cluster with remote Mediator and using local attached storage............................21
Figure 6) Four-node ONTAP Select cluster using local attached storage. ...................................................................21
Figure 7) ONTAP Select mirrored aggregate. ..............................................................................................................23
Figure 8) ONTAP Select write path workflow. ..............................................................................................................24
Figure 9) HA heart-beating in a 4-node cluster: steady state. ......................................................................................26
Figure 10) ONTAP Select installation VM placement. ..................................................................................................28
Figure 11) ONTAP Select multinode network configuration. ........................................................................................32
Figure 12) Network configuration of a multinode ONTAP Select VM. ..........................................................................33
Figure 13) Network configuration of single-node ONTAP Select VM. ..........................................................................35
Figure 14) Port group configurations using a standard vSwitch. ..................................................................................39
Figure 15) Link aggregation group properties when using LACP. ................................................................................40
Figure 16) Port group configurations using a distributed vSwitch with LACP enabled. ................................................40
Figure 17) Network configuration using shared physical switch. ..................................................................................42

3 TR-4517 Technical Report

Figure 18) Network configuration using multiple physical switches. .............................................................................43
Figure 19) Data and management separation using VST.............................................................................................44
Figure 20) Data and management separation using VGT. ...........................................................................................44
Figure 21) Four 10Gb NIC network configuration with LACP on a distributed vSwitch. ...............................................46
Figure 22) Four 10Gb NIC network configuration without LACP. .................................................................................46
Figure 23) Four NIC network configuration (2x10Gb + 2x1Gb). ...................................................................................47
Figure 24) Two-NIC network configuration. ..................................................................................................................48
Figure 25) Scheduled backup of remote office to corporate data center. .....................................................................49
Figure 26) Private cloud built on direct-attached storage. ............................................................................................50
Figure 27) Storage Add Operation ...............................................................................................................................51

4 TR-4517 Technical Report

1 Introduction
ONTAP® Select is NetApp’s solution for the software-defined storage (SDS) market. ONTAP Select brings
enterprise-class storage management features to the software-defined data center. ONTAP Select extends
the Data Fabric solution to the commodity server offerings likely already existing in the customer’s data
center.
This document describes the best practices that should be followed when building an ONTAP Select cluster,
from hardware selection to deployment and configuration. Additionally, it aims to answer the following
questions:
• How is ONTAP Select different from the engineered FAS storage platforms?
• Why were certain design choices made when creating the ONTAP Select architecture?
• What are the performance implications of the various configuration options?

1.1 Software-Defined Infrastructure

The implementation and delivery of IT services through software provide administrators with the ability to
rapidly provision resources with a level of speed and agility that was previously impossible.
Modern data centers are moving toward software-defined infrastructures as a mechanism to provide IT
services with greater agility and efficiency. Separating out IT value from the underlying physical
infrastructure allows them to react quickly to changing IT needs by dynamically shifting infrastructure
resources to where they are needed most.
Software-defined infrastructures are built on three tenets:
• Flexibility
• Scalability
• Programmability

Software-Defined Storage
The shift toward software-defined infrastructures may be having its greatest impact in an area that has
traditionally been one of the least affected by the virtualization movement: storage. Software-only solutions
that separate out storage management services from the physical hardware are becoming more
commonplace. This is especially evident within private cloud environments: enterprise-class service-
oriented architectures designed from the ground up with software defined in mind. Many of these
environments are being built on commodity hardware: white box servers with locally attached storage, with
software controlling the placement and management of user data.
This is also seen within the emergence of hyperconverged infrastructures (HCIs), a building-block style of
IT design based on the premise of bundling compute, storage, and networking services. The rapid adoption
of hyperconverged solutions over the past several years has highlighted the desire for simplicity and
flexibility. However, as companies make the decision to replace enterprise-class storage arrays with a more
customized, make your own model, by building storage management solutions on top of home-grown
components, a set of new problems emerges.
In a commodity world, where data lives fragmented across silos of direct-attached storage, data mobility
and data management become complex problems that need to be solved. This is where NetApp can help.

1.2 Running ONTAP as Software

There is a compelling value proposition in allowing customers to determine the physical characteristics of
their underlying hardware, while still giving them the ability to consume ONTAP and all of its storage

5 TR-4517 Technical Report

management services. Decoupling ONTAP from the underlying hardware allows us to provide enterprise-
class file and replication services within a software-defined storage environment.
Still, one question remains: Why do we require a hypervisor?
Running ONTAP as software on top of another software application allows us to leverage much of the
qualification work done by the hypervisor, critical in helping us rapidly expand our list of supported platforms.
Additionally, positioning ONTAP as a Virtual Machine (VM) allows customers to plug into existing
management and orchestration frameworks, allowing for rapid provisioning and end-to-end automation,
from deployment to sun-setting.
This is the goal of the ONTAP Select product.

1.3 ONTAP Select Versus Data ONTAP Edge

If you’re familiar with the past NetApp software-defined offering Data ONTAP® Edge, you may be wondering
how ONTAP Select is different. Although many of the differences are covered in additional detail in the
architecture overview section of this document, Table 1 highlights some of the major differences between
the two products.

Table 1) ONTAP Select versus Data ONTAP Edge.

Description Data ONTAP Edge ONTAP Select

Single node, 2 node HA and 4
Node count Single node
node HA

Virtual machine CPU/memory 2 vCPUs/8GB 4 vCPUs/16GB (Small Instance)

8 vCPUs/64GB (Medium Instance)

Hypervisor vSphere 5.1, 5.5 vSphere 5.5 update 3a (build

3116895 or greater) and 6.0 (build
2494585 or greater); VSAN 6.0,
6.1 and 6.2.

High availability No Yes

iSCSI/CIFS/NFS Yes Yes

SnapMirror® and SnapVault® Yes Yes

Compression No Yes

Capacity limit Up to 10TB, 25TB, or 50TB Up to 100TB/node

Hardware platform support Select families within qualified Wider support for major vendor
server vendors offerings that meet minimum
criteria

1.4 ONTAP Select Standard versus ONTAP Select Premium

ONTAP Select 9.1 adds support for Standard and Premium Select licenses. Only the Standard license is
available with ONTAP Select 9.0. The Premium license in 9.1 allows the user to configure either a Standard
Select VM (Small Instance) or a Premium Select VM (Medium Instance). The difference between the
Standard VM and Premium VM consists of the amount of resources reserved for each instance of Select.
For example, the Premium VM consumes 8 CPU cores and 64 GB of RAM. More information can be found
in section 2.1 Virtual Machine Properties.
As with the Standard VM, the number of cores and amount of memory cannot be further modified. In
addition, the Premium Select license allows the use of SSD drives for the Select datastore. This provides

6 TR-4517 Technical Report

a higher performance point that better matches the performance of SSD drives with the additional CPUs
and memory and allows Select to be positioned as a solution for more demanding workloads. Due to the
performance characteristics of the SSD drives, a minimum of 4 SSDs is required for Select Premium. The
RAID controller and a RAID group are still requirements.
The Select license is node specific, therefore in a 4 node cluster, it is possible to have a 2 node Premium
HA and a 2 node Standard HA. Within an HA pair however, the Select version and license type should be
identical.

1.5 ONTAP Select Evaluation software versus running ONTAP Select in

Evaluation mode
The ONTAP Select version available on the web portal (Downloads/Software) is the full version of the
product that can be run in EVALUATION mode. This means that the client can test the full solution including
ONTAP Deploy, the ONTAP Select setup product. Deploy will check and enforce all minimum requirements
for ONTAP Select which is useful for both documenting the procedure as well as vetting the environment
for suitability.
However, there are times when the test environment does not match the production environment or does
not meet the minimum requirements enforced by ONTAP Deploy. For a quick test of ONTAP Select only,
we are providing an OVF download of ONTAP Select only (Downloads / Product Evaluation). When using
this OVF, the ONTAP Deploy utility is not used but instead you directly install a single node ONTAP Select
cluster, which is capacity and time-limited, just as the single node cluster created using the Deploy tool in
EVALUATION model. The main benefit of the OVF setup is that it lowers the requirements for testing
ONTAP Select.

1.6 ONTAP Select Platform and Feature Support

The abstraction layer provided by the hypervisor allows ONTAP Select to run on a wide variety of
commodity platforms from virtually all the major server vendors, providing they meet minimum hardware
criteria. These specifications are detailed in the following sections.

Hardware Requirements
ONTAP Select requires that the hosting physical server meet the following minimum requirements:
• Intel Xeon E5-26xx v3 (Haswell) CPU or greater: 6 cores (4 for ONTAP Select, 2 for OS)
• 24GB RAM (16 GB for ONTAP Select, 8 GB for OS)
• Min. 2 1Gb NIC Ports for single node clusters, min 4 1Gb NIC Ports for two node clusters and 2 10GbE
NIC ports (4 recommended) for four node clusters.

ONTAP Select 9.1 Premium license supports both the Standard VM (minimum requirements above) as well
as the Premium VM. The ONTAP Select Premium VM will reserve 8 cores and 64 GB of RAM therefore the
server minimum requirements should be adjusted accordingly.

For locally attached storage (DAS), the following requirements also apply
• 8–24 internal disks (SAS)
• 8 – 24 internal disks (SAS, NL-SAS, or SATA – Select 9.1 and above)
• 4 – 24 SSD disks (Select 9.1 Premium and above)
• Hardware RAID controller with 512MB writeback cache and 12Gb/s of throughput

7 TR-4517 Technical Report

For shared storage (VSAN or external arrays), the RAID controller is no longer a requirement. However,
the following restrictions and best practices should be considered in selecting the type of datastore used
for hosting ONTAP Select.
• Support for VSAN and external arrays requires the following minimum versions: ONTAP Select 9.1 and
Deploy 2.3
• Support for VMware HA / vMotion / DRS requires the following minimum versions: ONTAP Select 9.2
and Deploy 2.4
• Only single node Select clusters are supported with VSAN or external array type datastores. For multi
node clusters, one must use local storage (DAS).
• The VSAN configuration or the external array must be supported by VMware as evidenced by the
configuration being present on the VMware HCL.

ONTAP Feature Support

The ONTAP Select 9.0 release offers full support for most of the ONTAP 9.0 functionality, with the exception
of those features that have hardware-specific dependencies such as MetroCluster™ and FCoE.
The supported functionality includes:
• NFS/CIFS/iSCSI
• SnapMirror and SnapVault
• FlexClone®
• SnapRestore®
• NetApp Volume Encryption (NVE)
• FlexGroups (Starting with ONTAP Select 9.2)

Additionally, support for the OnCommand management suite is included. This includes most tooling used
to manage NetApp FAS arrays, such as OnCommand Unified Manager (OCUM), OnCommand Insight
(OCI), Workflow Automation (WFA), and SnapCenter®. Use of SnapCenter, SnapManager or SnapDrive
with ONTAP Select requires server-based licenses
Consult the IMT for a complete list of supported management applications.
Note that the following ONTAP features are not supported by ONTAP Select:
• Interface groups (IFGRPs)
• Service Processor
• Hardware-centric features such as MetroCluster, Fibre Channel (FC/FCoE), and full disk encryption
(FDE)
• SnapLock®
• NSE Drives
• FabricPools

ONTAP Select 9.1 and 9.2 are providing Storage Efficiency options that are similar to the Storage Efficiency
options present on FAS and AFF arrays. Both ONTAP Select 9.1 and 9.2 support SSD media, however
there are significant differences in default behaviors between these releases, as well as between ONTAP
Select Premium with SSD media and AFF arrays.
Please note that ONTAP Select vNAS deployments using All Flash VSAN or all flash arrays should follow
the best practices for ONTAP Select with non-SSD DAS storage.

8 TR-4517 Technical Report

ONTAP Select 9.1 did not verify that the media under management was of type SSD, Therefore all Storage
Efficiency settings were available, even if some of these features were optimized for SSD storage. ONTAP
Select 9.2 upgraded from ONTAP Select 9.1 will have a similar behavior. The main difference between
ONTAP Select 9.1 Premium with SSD or ONTAP Select 9.2 upgraded from 9.1 Premium with SSD on one
side, and the ONTAP Select 9.2 Premium with SSD new installation on the other side, is the Inline Dedupe
functionality. In the case of the former, Inline Dedupe consist of Zero Detection only. In the case of the later,
the full Volume Level Inline Dedupe functionality is available. For ONTAP Select 9.2 Premium with SSD
systems that were upgraded from ONTAP Select 9.1, the following setting needs to be changed to take
advantage of the full Volume level Inline Dedupe functionality:
filer::*> run local options sis.idedup_allow_non_aff_hya on

2. Enable inline deduplication for each volume.

filer::> volume efficiency modify -vserver <vs> -volume <vol> -inline-deduplication true

ONTAP Deploy 2.4 adds an additional configuration check during the ONTAP Select cluster setup. This
configuration check asks the user to confirm the DAS storage is of SSD type. ONTAP Deploy will enforce
this check during setup, as well as during storage add operations. In other words, once an ONTAP Select
Premium VM is configured for SSD storage, only local (DAS) SSD media can be added to that VM. There
are a number of reasons for this, including the fact that ONTAP Select does not support multiple RAID
controllers, nor does it support mixing media types on the same RAID controller. However this enablement
enforcement insures that the SSD appropriate Storage Efficiency options cannot be enabled on HDD based
datastores.
Please note that unlike an AFF array which automatically enables its Inline Storage Efficiency policies,
configuring an ONTAP Select 9.2 Premium with the SSD feature during cluster setup does not automatically
enable Inline Storage Efficiencies inside ONTAP Select, but it simply makes this functionality available to
use later, at the time of volume creation. In other words, the client may enable Inline Storage Efficiencies,
on a volume per volume basis, for each volume provisioned on an ONTAP Select 9.2 Premium with SSD
media.

Table 2 summarizes the various Storage Efficiency options available and recommended depending on the
ONTAP Select version and media type:

Table 2) ONTAP Select virtual machine properties.

9.2 Premium 9.2 Premium OR 9.1 Premium 9.1 Premium
(SSD) Standard (HDD) (SSD) OR Standard
(HDD)
Inline Zero Yes, in case of Yes, in case of Yes, enabled by Yes, enabled by
Detection upgrade from 9.1 upgrade from 9.1 and user per volume user per volume
and enabled by enabled by user per basis. basis.
user per volume volume basis.
basis.
Volume Inline Yes, on new installs No No No
Dedupe of 9.2, enabled by
user per volume
basis.
NOTE: full Inline Dedupe
functionality can be enabled
on 9.1 -> 9.2 upgraded
systems by running node shell
command:
options
sis.idedup_allow_non_aff_hya
on and then enabling on a per
volume basis.

9 TR-4517 Technical Report

 32K Inline Yes (Default), Yes (Default and Yes (Default), Yes (Default and
Compression enabled by user per Recommended), enabled by user by Recommended),
(Secondary
volume basis. enabled by user by user per volume enabled by user
Compression) user per volume basis. basis. per volume basis.
8K Inline Yes Yes, enabled by user Yes Yes, enabled by
Compression (Recommended), per volume basis. (Recommended) user per volume
(Adaptive
enabled by user per enabled by user per basis.
Compression) volume basis. volume basis.
Background Not Supported. Yes, enabled by user Not Supported. Yes, enabled by
Compression per volume basis. user per volume
basis.
Compression Yes, enabled by Yes, enabled by user Yes, enabled by Yes, enabled by
Scanner user per volume per volume basis. user per volume user per volume
basis. basis. basis.
Inline Data Yes, enabled by Yes, enabled by user Yes, enabled by Yes, enabled by
Compaction user per volume per volume basis. user per volume user per volume
basis. basis. basis.
Compaction Yes, enabled by Yes, enabled by user Yes, enabled by Yes, enabled by
Scanner user per volume per volume basis. user per volume user per volume
basis. basis. basis.

Aggregate Inline Yes N/A N/A N/A

Dedupe
Background Yes Yes (Recommended) Yes Yes
Dedupe (Recommended) (Recommended) (Recommended)

2 Architecture Overview
ONTAP Select is clustered Data ONTAP deployed as a virtual machine and providing storage management
services on a virtualized commodity server.
The ONTAP Select product can be deployed two different ways:
• Non-HA (single node). The single-node version of ONTAP Select is well suited for storage
infrastructures that provide their own storage resiliency such as VSAN datastores or external arrays
which offer data protection at the array layer. The single node Select cluster can also be used for remote
and branch offices where the data is protected by replication to a core location.
• High availability (multi-node). The multi-node version of the solution uses two or four ONTAP Select
nodes and adds support for high availability and clustered Data ONTAP non-disruptive operations, all
within a shared-nothing environment.
When choosing a solution, resiliency requirements, environment restrictions, and cost factors should be
taken into consideration. Although both versions run clustered Data ONTAP and support many of the same
core features, the multi-node solution provides high availability and supports non-disruptive operations, a
core value proposition for clustered Data ONTAP.
Note: The single-node and multi-node versions of ONTAP Select are deployment options, not separate
products. Although the multi-node solution requires the purchase of additional node licenses, both
share the same product model, FDvM300.
This section provides a detailed analysis of the various aspects of the system architecture for both the
single-node and multi-node solutions while highlighting important differences between the two variants.

10 TR-4517 Technical Report

2.1 Virtual Machine Properties
The ONTAP Select virtual machine has a fixed set of properties, described in Table 3. Increasing or
decreasing the amount of resources allocated to the VM is not supported. Additionally, the ONTAP Select
instance hard reserves the CPU and memory resources, meaning the physical resources backed by the
virtual machine are unavailable to any other VMs hosted on the server.

Table 3) ONTAP Select virtual machine properties.

Description Single Node Multinode (per Node)

CPU / Memory 4 cores / 16 GB RAM OR 4 cores / 16 GB RAM OR
8 cores / 64 GB RAM* 8 cores / 64 GB RAM*
* ONTAP Select Premium (9.1 and newer) * ONTAP Select Premium (9.1 and newer)

Virtual network interfaces 2 6

SCSI controllers 4 4

System boot disk 10GB 10GB

System core dump disk 120GB 120GB

Mailbox disk 556MB 556MB

Cluster root disk 68GB 68GB x 2 (because disk is mirrored)

Serial ports 2 network serial ports (Select 9.0 and 2 network serial ports (Select 9.0 and
9.1 only) 9.1 only)

Note: The core dump disk partition is separate from the system boot disk. Because the core file size is
directly related to the amount of memory allocated to the ONTAP instance, this allows NetApp to
support larger-sized memory instances in the future without requiring a redesign of the system boot
disk.
Note: The serial ports were removed from the ONTAP Select 9.2 VM which allows ONTAP Select 9.2 to
support and install on any vSphere license. Prior to ONTAP Select 9.2, only the vSphere Enterprise
/ Enterprise + licenses were supported.
Starting with ONTAP Select 9.2, the ONTAP console will only be accessible via the Virtual Machine
video console tab in the vSphere client.
Table 4 show the differences between the ONTAP Select 9.0, 9.1 and 9.2 releases.

Table 4) ONTAP Select 9.0 versus ONTAP Select 9.1 versus ONTAP Select 9.2

Description ONTAP Select 9.0 ONTAP Select 9.1 ONTAP Select 9.2

ONTAP Select Standard Standard or Premium Standard or Premium

License

4 vCPUs/16GB OR 4 vCPUs/16GB OR
CPU/memory 4 vCPUs/16GB RAM 8 vCPUs/64GB* 8 vCPUs/64GB*
(*Requires Premium License) (*Requires Premium License)

Disk type SAS only SAS, NL-SAS, SATA or SSD* SAS, NL-SAS, SATA or
(* Requires Premium License) SSD*
(* Requires Premium License)

11 TR-4517 Technical Report

 Description ONTAP Select 9.0 ONTAP Select 9.1 ONTAP Select 9.2

Minimum number 8 SAS 8 SAS, NL-SAS, SATA OR 4 8 SAS, NL-SAS, SATA OR 4

of disks SSD* SSD*
(* Requires Premium License) (* Requires Premium License)

Maximum number 24 24 24
of disks

Network Serial 2 2 none

Ports

vSpere license Enterprise / Enterprise + Enterprise / Enterprise + All vSphere licenses are
requirements supported

VMware HA / No No vNAS only *

vMotion support (* Requires ONTAP Deploy 2.4
min)

Select Cluster single node / 4 node single node / 4 node single node / 2 node / 4
Size node*
(* Requires ONTAP Deploy 2.4
min)

When using local attached storage (DAS), ONTAP Select will make use of the hardware RAID controller
cache, to achieve a significant increase in write performance. Additionally, when using locally attached
storage (DAS), certain restrictions apply to the ONTAP Select virtual machine.
Specifically:
• Only one ONTAP Select VM can reside on a single server.
• ONTAP Select may not be migrated or vMotioned to another server. This includes storage vMotion of
the ONTAP Select VM.
• vSphere Fault Tolerance (FT) is not supported

2.2 VSAN and External Array Configurations (vNAS)

Starting with ONTAP Select 9.1 and Deploy 2.3, single node ONTAP Select clusters are supported on
VSAN or external array types of datastores. This deployment model is generally referred to as vNAS or
virtual NAS.
In these configurations, the datastore resiliency is assumed to be provided by the underlying infrastructure.
The minimum requirement is that the underlying configurations is supported by VMware and therefore
should be listed on the respective VMware compatibility lists (HCL).
ONTAP Select 9.2 and Deploy 2.4 extend the functionality of the vNAS solution in a number of ways,
including support for VMware HA, vMotion and DRS, as well as support for all vSphere license types.
The following best practices should be considered when installing a single node ONTAP Select cluster on
a VSAN type datastore:

12 TR-4517 Technical Report

 • VSAN 6.0, 6.1, 6.2 and 6.3 are supported; Enterprise License is required when creating clusters
with versions prior to ONTAP Deploy 2.4 / ONTAP Select 9.2. All vSphere licenses are supported
starting with ONTAP ONTAP Deploy 2.4 / ONTAP Select 9.2.
• HY and All Flash VSAN configurations are supported with both ONTAP Select Standard and
Premium.
• VSAN storage efficiency features are supported.
• There are no restrictions on the VSAN storage policy settings, including FTT (Failures to Tolerate)
and FTM (Failure Tolerance Method)
• Depending on the FTT / FTM settings, the ONTAP Select VM size can be significantly larger than
the capacity configured during its setup. ONTAP Select uses thick-eager zeroed VMDKs that are
created during the setup. In order to avoid impacting other VMs using the same shared datastore,
it is important to insure that there is sufficient free capacity in the datastore to accommodate the
true Select VM size as derived from the ONTAP Select capacity and the FTT/FTM settings.
• VMware HA, vMotion, and DRS are supported starting with ONTAP Select 9.2 and ONTAP Deploy
2.4. When the ONTAP Select VM changes its original ESX hosts as a result of a VMware HA or
vMotion operation, the ONTAP Deploy 2.4 instance managing this ONTAP Select instance will
temporarily lose connectivity to the ONTAP Select VM. The ONTAP Deploy 2.4 instance will
attempt to automatically discover the new ESX machine hosting the ONTAP Select VM the next
time a management operation is attempted. Therefore the first operation will fail with an error
message stating that the ONTAP Select VM no longer exists on host <hostname>. This is the
expected behavior, and it starts an asynchronous background task to locate the ONTAP Select VM
using the vCenter credentials provided during the cluster setup. The old VSAN host will be labeled
“re-hosting in progress” for all subsequent queries until the background task completes.

The following limitations should be considered when installing a single node ONTAP Select cluster on a
VSAN type datastore:
• Only one ONTAP Select node per VSAN / ESX host is supported. Multiple single node Select
clusters can share a VSAN datastore as long as they are installed on separate VSAN hosts.
• The ONTAP Deploy auto-discovery and re-host operations require that all ESX hosts be managed
by the same vCenter.
• A VMware HA or vMotion operation can result in a situation where two ONTAP Select VMs reside
on the same ESX host. This configuration is not currently supported and ONTAP Deploy 2.4 will
be unable to re-establish management connectivity to the ONTAP Select VM until that VM is moved
to another ESX host.

The following best practices should be considered when installing a single node Select cluster on an
external array type datastore:
• vSphere 6.0 Update 1, 2 or 3 are supported. Enterprise License is required for versions prior to
ONTAP Select 9.2 and ONTAP Deploy 2.4. All vSphere licenses are supported starting with
ONTAP Select 9.2 and ONTAP Deploy 2.4
• FC/FCoE/iSCSI and NFS are supported protocols for the connectivity between the ESX host and
the external array.
• Hybrid arrays and All Flash arrays are supported with both ONTAP Select Standard and Premium.
• Array side storage efficiency policies are supported.

13 TR-4517 Technical Report

 • Connectivity between the ESX host and the arrays should be via 10Gb with no single point of
failure. Jumbo frames are recommended.
• The ONTAP Select VM should have dedicated network ports for client traffic that do not overlap
with ports used for connectivity to the back end array.
• VMware HA, vMotion, and DRS are supported starting with ONTAP Select 9.2 and ONTAP Deploy
2.4. When the ONTAP Select VM changes its original ESX hosts as a result of a VMware HA or
vMotion operation, the ONTAP Deploy 2.4 instance managing this ONTAP Select instance will
temporarily lose connectivity to the ONTAP Select VM. The ONTAP Deploy 2.4 instance will
attempt to automatically discover the new ESX machine hosting the ONTAP Select VM the next
time a management operation is attempted. Therefore the first operation will fail with an error
message stating that the ONTAP Select VM no longer exists on host <hostname>. This is the
expected behavior, and it starts an asynchronous background task to locate the ONTAP Select VM
using the vCenter credentials provided during the cluster setup. The old ESX host will be labeled
“re-hosting in progress” for all subsequent queries until the background task completes.

The following limitations should be considered when installing a single node Select cluster on an external
array type datastore:
• VVOLS are not supported
• Only one ONTAP Select node per ESX host is supported. Multiple single node ONTAP Select
clusters can share an external array datastore as long as they are installed on separate ESX hosts.
• The ONTAP Deploy auto-discovery and re-host operations require that all ESX hosts be managed
by the same vCenter.
• A VMware HA or vMotion operation can result in a situation where two ONTAP Select VMs reside
on the same ESX host. This configuration is not currently supported and ONTAP Deploy 2.4 will
be unable to re-establish management connectivity to the ONTAP Select VM until that VM is moved
to another ESX host.

NetApp FAS, SolidFire and E-Series arrays are supported as long as they are on VMware HCL. We
recommend following the NetApp and VMware vSphere Storage Best Practices documentation for the
respective array.

2.3 RAID Services for local attached storage

Although some software-defined solutions require the presence of an SSD to act as a higher speed write-
staging device, ONTAP Select uses a hardware RAID controller to achieve both a write performance boost
and the added benefit of protection against physical drive failures by moving RAID services to the hardware
controller. As a result, RAID protection for all nodes within the ONTAP Select cluster are provided by the
locally attached RAID controller and not through Data ONTAP software RAID.
Note: ONTAP Select data aggregates are configured to use RAID 0, because the physical RAID controller
is providing RAID striping to the underlying drives. No other RAID levels are supported.

RAID Controller Configuration for local attached storage

All locally attached disks that provide ONTAP Select with backing storage must sit behind a RAID controller.
Most commodity servers come with multiple RAID controller options across multiple price points, and each
with varying levels of functionality. The intent is to support as many of these options as possible, providing
they meet certain minimum requirements placed on the controller.

14 TR-4517 Technical Report

The RAID controller that is managing the ONTAP Select disks must support the following:
• The HW RAID controller must have a battery backup unit (BBU) or flash-backed write cache (FBWC)
and support 12Gb/s of throughput
• The RAID controller must support a mode that can withstand at least one or two disk failures (RAID 5,
RAID 6).
• The drive cache should be set to disabled.
• The write policy should be configured for writeback mode with a fallback to writethrough upon BBU or
flash failure.
• The I/O policy for reads must be set to cached.
All locally attached disks that provide ONTAP Select with backing storage must be placed into RAID groups
running RAID 5 or RAID 6. For SAS and SSD drives, using a single RAID group of up to 24 drives allows
ONTAP to reap the benefits of spreading incoming read requests across a higher number of disks, providing
a significant gain in performance. With SAS/SSD configurations, performance testing was done against
single LUN vs. multi-LUN configurations. No significant differences were found, so for simplicity’s sake, we
recommend creating the fewest number of LUNs necessary to support your configuration needs.
NL-SAS and SATA drives require a different set of best practices. For performance reasons, the minimum
number of disks is still 8, but the RAID group size should not be larger than 12 drives. We also recommend
one spare per RAID group, though a global spare for all RAID groups can also be used. Please note that
the maximum ESX 5.5 / 6.0 extent and datastore size is 64 TB, which may impact the number of LUNs
necessary to support the total raw capacity provided by these large capacity drives.

RAID Mode
Many RAID controllers support up to three modes of operation, each representing a significant difference
in the data path taken by write requests. These are:
• Writethrough. All incoming I/O requests are written to the RAID controller cache and then
immediately flushed to disk before acknowledging the request back to the host.
• Writearound. All incoming I/O requests are written directly to disk, circumventing the RAID
controller cache.
• Writeback. All incoming I/O requests are written directly to the controller cache and immediately
acknowledged back to the host. Data blocks are flushed to disk asynchronously using the controller.
Writeback mode offers the shortest data path, with I/O acknowledgement occurring immediately after the
blocks enter cache, and thus lower latency and higher throughput for mixed read/write workloads. However,
without the presence of a BBU or nonvolatile flash technology, when operating in this mode, users run the
risk of losing data should the system incur a power failure.
Because ONTAP Select requires the presence of a battery backup or flash unit, we can be confident that
cached blocks are flushed to disk in the event of this type of failure. For this reason, it is a requirement that
the RAID controller be configured in writeback mode.

Best Practice

The server RAID controller should be configured to operate in writeback mode. If write workload
performance issues are seen, check the controller settings and make sure that writethrough or
writearound is not enabled.

15 TR-4517 Technical Report

Local Disks Shared Between ONTAP Select and OS
The most common server configuration is one where all locally attached spindles sit behind a single RAID
controller. A minimum of two LUNs should be provisioned: one for the hypervisor and another for the
ONTAP Select VM.
Note: The “one LUN for ONTAP Select” statement assumes the physical storage capacity of the system
doesn’t surpass the hypervisor-supported file system extent limits.
For example, let’s say a customer purchases an HP DL380 g8 with six internal drives and a single Smart
Array P420i RAID controller. All internal drives are managed by this RAID controller, and no other storage
is present on the system.
The following figure shows this style of configuration. In this example, no other storage is present on the
system, so the hypervisor needs to share storage with the ONTAP Select node.

Figure 1) Server LUN configuration with only RAID-managed spindles.

Provisioning the OS LUNs from the same RAID group as ONTAP Select allows the hypervisor OS (and any
client VMs that are also provisioned from that storage) to benefit from RAID protection, preventing a single-
drive failure from bringing down the entire system

Best Practice

If the physical server contains a single RAID controller managing all locally attached disks, we
recommend creating a separate LUN for the server OS and one or more LUNs for ONTAP Select. In the
event of boot disk corruption, this allows the administrator to recreate the OS LUN without affecting
ONTAP Select.

Local Disks Split Between ONTAP Select and OS

The other possible configuration provided by server vendors involves configuring the system with multiple
RAID or disk controllers. In this configuration, a set of disks is managed by one disk controller, which may
or may not offer RAID services, with a second set of disks being managed by a hardware RAID controller
that is able to offer RAID 5/6 services.

16 TR-4517 Technical Report

With this style of configuration, the set of spindles that sits behind the RAID controller that is able to provide
RAID 5/6 services should be used exclusively by the ONTAP Select VM. Depending on the total storage
capacity under management, the disk spindles should be configured into one or more RAID groups, and
one or more LUNs. These LUNs would then be used to create one or more datastores, with all datastores
being protected by the RAID controller.
The first set of disks is reserved for the hypervisor OS (and any client VMs not using ONTAP storage).
This is shown in further detail in the following figure.

Figure 2) Server LUN configuration on mixed RAID/non-RAID system.

Multiple LUNs
There are two cases where single–RAID group / single-LUN configurations must change. When using NL-
SAS or SATA drives, the RAID group size must not exceed 12 drives. Additionally, when a single LUN
becomes larger than the underlying hypervisor storage limits (either individual file system extent maximum
size or total storage pool maximum size), then the underlying physical storage must be broken up into
multiple LUNs to allow for successful file system creation.

Best Practice

ONTAP Select receives no performance benefits by increasing the number of LUNs within a RAID group.
Multiple LUNs should only be used in order to follow best practices for SATA / NL-SAS configurations or
to bypass hypervisor file system limitations.

2.4 vSphere VMFS Limits

The maximum extent size on a vSphere 5.5 / 6.0 server is up to 64TB. A VMFS file system cannot use
disks or LUNs that are larger than this size. The maximum size of an ESX 5.5 / 6.0 hosted datastore is also
64TB. This datastore can consist of one large extent or multiple smaller extents.
If a server has more than 64TB of storage attached, multiple LUNs must be provisioned for the host, each
smaller than 64TB. Creating multiple RAID groups in order to improve the RAID rebuild time for SATA / NL-
SAS drives will also result in multiple LUNs being provisioned.

17 TR-4517 Technical Report

When multiple LUNs are required, a major point for consideration is insuring that these LUNs have similar
and consistent performance. This is especially important if all the LUNs are to be used in a single ONTAP
aggregate. Alternatively, if a subset of one or more LUNs has a distinctly different performance profile, we
strongly recommend isolating these LUNs in a separate ONTAP aggregate
Multiple file system extents can be used to create a single datastore up to the maximum size of the
datastore. To restrict the amount of capacity that requires an ONTAP Select license, make sure to specify
a capacity cap during the cluster installation. This functionality allows ONTAP Select to utilize (and therefore
require a license for) only a subset of the space in a datastore.
Alternatively, one can start by creating a single datastore on a single LUN. When additional space (which
requires a larger ONTAP Select capacity license) is needed, that space can be added to the same datastore
as an extent, up to maximum size of the datastore. After the maximum size is reached, new datastores can
be created and added to ONTAP Select. Both types of capacity extension operations are supported and
can be achieved using the ONTAP Deploy storage add functionality. This functionality is further covered in
section 7.1.

2.5 ONTAP Select Virtual Disks

At its core, ONTAP Select presents Data ONTAP with a set of virtual disks, provisioned from one or more
storage pools. Data ONTAP is presented with a set of virtual disks, which it treats as physical, and the
remaining portion of the storage stack is abstracted by the hypervisor. The following figure shows this
relationship in more detail, highlighting the relationship between the physical RAID controller, the
hypervisor, and the ONTAP Select VM. Note the following:
• RAID group and LUN configuration occur from within the server's RAID controller software. This
configuration is not required when using VSAN or external arrays.
• Storage pool configuration happens from within the hypervisor.
• Virtual disks are created and owned by individual VMs, in this case, ONTAP Select.

Figure 3) Virtual disk to physical disk mapping.

Virtual Disk Provisioning

To provide for a more streamlined user experience, the ONTAP Select management tool, ONTAP Deploy,
automatically provisions virtual disks from the associated storage pool and attaches them to the ONTAP
Select virtual machine. This operation occurs automatically during both initial setup and during storage add
operations. If the ONTAP Select node is part of an HA pair, the virtual disks are automatically assigned to
a local and mirror storage pool.

18 TR-4517 Technical Report

Because all virtual disks on the ONTAP Select VM are striped across the underlying physical disks, there
is no performance gain in building configurations with a higher number of virtual disks. Additionally, shifting
the responsibility of virtual disk creation and assignment from the administrator to the management tool
prevents the user from inadvertently assigning a virtual disk to an incorrect storage pool.
ONTAP Select breaks up the underlying attached storage into equal-sized virtual disks, each not exceeding
8TB. If the ONTAP Select node is part of an HA pair, a minimum of two virtual disks are created on each
cluster node and assigned to the local and mirror plex to be used within a mirrored aggregate.
For example, if ONTAP Select is assigned a datastore or LUN that is 31TB (space remaining after VM is
deployed and system and root disks are provisioned), four ~7.75TB virtual disks are created and assigned
to the appropriate ONTAP local and mirror plex.
Please note that adding capacity to an ONTAP Select VM will likely result in having VMDKs of different
sizes. Unlike FAS systems, different size VMDKs can exist in the same aggregate. ONTAP Select uses a
RAID 0 stripe across these VMDKs which results in our ability to fully utilize all the space in each VMDK
regardless of its size.

Best Practice

Similar to creating multiple LUNs, ONTAP Select receives no performance benefits by increasing the
number of virtual disks used by the system.

2.6 Virtualized NVRAM

NetApp FAS systems are traditionally fitted with a physical NVRAM PCI card: a high-performing card
containing nonvolatile flash memory that provides a significant boost in write performance by granting Data
ONTAP with the ability to:
• Immediately acknowledge incoming writes back to the client
• Schedule the movement of modified data blocks back to the slower storage media (this process is
known as destaging)
Commodity systems are not traditionally fitted with this type of equipment. Therefore, the functionality of
the NVRAM card has been virtualized and placed into a partition on the ONTAP Select system boot disk. It
is for precisely this reason that placement of the system virtual disk of the instance is extremely important
and why the product requires the presence of a physical RAID controller with a resilient cache for local
attached storage configurations. When using VSAN or external arrays for hosting the datastore, the NVRAM
protection is assumed by the underlying storage infrastructure.

Data Path Explained: NVRAM and RAID Controller

The interaction between the virtualized NVRAM system partition and the RAID controller can be best
highlighted by walking through the data path taken by a write request as it enters the system.
Incoming write requests to the ONTAP Select virtual machine are targeted at the VM’s NVRAM partition.
At the virtualization layer, this partition exists within an ONTAP Select system disk: a VMDK attached to the
ONTAP Select VM. At the physical layer, these requests are cached in the local RAID controller, like all
block changes targeted at the underlying spindles. From here, the write is acknowledged back to the host.
At this point:
• Physically, the block resides in the RAID controller cache, waiting to be flushed to disk.
• Logically, the block resides in NVRAM, waiting for destaging to the appropriate user data disks.

19 TR-4517 Technical Report

Because changed blocks are automatically stored within the RAID controller’s local cache, incoming writes
to the NVRAM partition are automatically cached and periodically flushed to physical storage media. This
should not be confused with the periodic flushing of NVRAM contents back to Data ONTAP data disks.
These two events are unrelated and occur at different times and frequencies.
Figure 4 is intended to show the I/O path an incoming write takes, highlighting the difference between the
physical layer, represented by the RAID controller cache and disks, from the virtual layer, shown through
the virtual machine’s NVRAM and data virtual disks.
Note: Although blocks changed on the NVRAM VMDK are cached in the local RAID controller cache, the
cache is not aware of the VM construct or its virtual disks. It stores all changed blocks on the
system, of which NVRAM is only a part. This includes write requests bound for the hypervisor, if it
is provisioned from the same backing spindles.

Figure 4) Incoming writes to ONTAP Select VM.

Best Practice

Because the RAID controller cache is used to store all incoming block changes and not only those
targeted toward the NVRAM partition, when choosing a RAID controller, select one with the largest cache
available. A larger cache allows for less frequent disk flushing and an increase in performance of the
ONTAP Select VM, the hypervisor, and any compute VMs collocated on the server.

2.7 High Availability for local attached storage

Although customers are starting to move application workloads from enterprise-class storage appliances to
software-based solutions running on commodity hardware, the expectations and needs around resiliency
and fault tolerance have not changed. A high-availability solution providing a zero RPO is required, one that
protects the customer from data loss due to a failure from any component in the infrastructure stack.
A large portion of the SDS market is built on the notion of shared nothing storage, with software replication
providing data resiliency by storing multiple copies of user data across different storage silos. ONTAP
Select builds on this premise by using the synchronous replication features (RAID SyncMirror®) provided
by clustered Data ONTAP to store an additional copy of user data within the cluster. This occurs within the
context of an HA pair. Every HA pair stores two copies of user data: one on storage provided by the local
node and one on storage provided by the HA partner. Within an ONTAP Select cluster, HA and synchronous
replication are tied together, and the functionality of the two cannot be decoupled or used independently.
As a result, the synchronous replication functionality is only available in the multinode offering.

20 TR-4517 Technical Report

Note: In an ONTAP Select cluster, synchronous replication functionality is a function of the HA
implementation, not a replacement for the asynchronous SnapMirror or SnapVault replication
engines. Synchronous replication cannot be used independently from HA.
There are two ONTAP Select HA options: the 4-node cluster introduced with ONTAP Select 9.0, and the 2-
node cluster introduced with ONTAP Select 9.2 and ONTAP Deploy 2.4. The salient feature of a 2-node
Select cluster is the use of an external Mediator service in order to resolve split brain scenarios. The ONTAP
Deploy 2.4 VM will serve as the default Mediator for all the 2-node HA pairs that it will configure.

The two architectures are represented in Figure 5.

Figure 5) Two-node ONTAP Select cluster with remote Mediator and using local attached storage.

Figure 6) Four-node ONTAP Select cluster using local attached storage.

Note: The 4-node ONTAP Select cluster is composed of two HA pairs. The 2-node ONTAP Select cluster
is composed on one HA pair and a Mediator. Within each HA pair, data aggregates on each cluster
node are synchronously mirrored, and in the event of a failover there is no loss of data.

21 TR-4517 Technical Report

Note: Only one ONTAP Select instance may be present on a physical server. That instance is tied to the
server, meaning the VM may not be migrated off to another server. ONTAP Select requires
unshared access to the local RAID controller of the system and is designed to manage the locally
attached disks, which would be impossible without physical connectivity to the storage.

2-node HA vs. 4-node HA

Unlike FAS arrays, ONTAP Select nodes in an HA pair communicate exclusively over the IP network. That
means that the IP network is a single point of failure, and protecting against network partitions and split
brain scenarios becomes a real and important design aspect. The 4-node cluster can sustain single node
failures because the cluster quorum can be established by the three surviving nodes. The 2-node cluster
relies on the Mediator service hosted by the ONTAP Deploy VM to achieve the same result. The minimum
version of the ONTAP Deploy VM required to support a 2-node cluster with the Mediator service is 2.4.
The heart beat network traffic between the ONTAP Select nodes and the ONTAP Deploy Mediator service
is minimal and resilient so that the ONTAP Deploy VM can be hosted in a different datacenter than the
ONTAP Select 2-node cluster. Please note that the ONTAP Deploy VM becomes an integral part of a 2-
node cluster when serving as the Mediator for that cluster. If the Mediator service is unavailable, the 2-node
cluster will continue serving data, but the storage fail-over capabilities of the ONTAP Select cluster will be
disabled. Therefore the ONTAP Deploy Mediator service needs to maintain constant communication with
each ONTAP Select node in the HA pair. A minimum bandwidth of 5Mb/s and maximum latency of 500 ms
RTT are required to insure proper functioning of the cluster quorum.

In the situation in which the ONTAP Deploy VM acting as a Mediator is temporarily or potentially
permanently unavailable, a secondary ONTAP Deploy VM (minimum version 2.4) can be used to restore
the 2-node cluster quorum. This will result in a configuration in which the new ONTAP Deploy VM is unable
to manage the ONTAP Select nodes but it will successfully participate in the cluster quorum algorithm. The
communication between the ONTAP Select nodes and the ONTAP Deploy VM will be done using the iSCSI
protocol. The ONTAP Select node management IP address is the initiator and the ONTAP Deploy VM IP
address is the target. The ONTAP Deploy hosted mailbox disks are automatically created and masked to
the proper ONTAP Select node management IP addresses at the time of the 2-node cluster creation. The
entire configuration is automatically done during setup and no further administrative action is required. The
ONTAP Deploy instance creating the cluster is the default Mediator for that cluster.
An administrative action is required if the original Mediator location needs to be changed. It is possible to
recover a cluster quorum even if the original ONTAP Deploy VM is completely lost. However, we
recommend that a backup of the ONTAP Deploy database be done after every 2-node cluster is
instantiated.
For a complete list of steps required to configure a new Mediator location, please refer to the “ONTAP
Select 9 Installation and Cluster Deployment Guide”.

Synchronous Replication
The Data ONTAP HA model is built on the notion of HA partners. As explained earlier, ONTAP Select
extends this architecture into the non-shared commodity server world by using the RAID SyncMirror
functionality that is present in clustered Data ONTAP to replicate data blocks between cluster nodes,
providing two copies of user data spread across an HA pair.
Note: This product is not intended to be an MCC-style disaster recovery replacement and cannot be used
as a stretch cluster. Cluster network and replication traffic occurs using link-local IP addresses and
requires a low-latency, high-throughput network. As a result, spreading out cluster nodes across
long distances is not supported.

22 TR-4517 Technical Report

Mirrored Aggregates
An ONTAP Select cluster is composed of two or four nodes and contains two copies of user data,
synchronously mirrored across HA pairs over an IP network. This mirroring is transparent to the user and
is a property of the aggregate assigned at the time of creation.
Note: All aggregates in an ONTAP Select cluster must be mirrored in order to make sure of data
availability in the case of a node failover and avoid a single point of failure in case of hardware
failure. Aggregates in an ONTAP Select cluster are built from virtual disks provided from each node
in the HA pair and use:
• A local set of disks, contributed by the current ONTAP Select node
• A mirror set of disks, contributed by the HA partner of the current node
Note: Both the local and mirror disks used to build a mirrored aggregate must be of the same size. We
refer to these aggregates as plex 0 and plex 1 to indicate the local and remote mirror pairs,
respectively. The actual plex numbers may be different in your installation.
This is an important point and fundamentally different from the way standard ONTAP clusters work. This
applies to all root and data disks within the ONTAP Select cluster. Because the aggregate contains both
local and mirror copies of data, an aggregate that contains N virtual disks actually offers N/2 disks’ worth
of unique storage, because the second copy of data resides on its own unique disks.
Figure 6 depicts an HA pair within a four-node ONTAP Select cluster. Within this cluster is a single
aggregate, “test,” which uses storage from both HA partners. This data aggregate is composed of two sets
of virtual disks: a local set, contributed by the ONTAP Select owning cluster node (plex 0), and a remote
set, contributed by the failover partner (plex 1).
Plex 0 is the bucket that holds all local disks. Plex 1 is the bucket that holds mirror disks, or disks
responsible for storing a second replicated copy of user data. The node that owns the aggregate contributes
disks to plex 0, and the HA partner of that node contributes disks to plex 1.
In our figure, we have a mirrored aggregate with two disks. The contents of this aggregate are mirrored
across our two cluster nodes, with local disk NET-1.1 placed into the plex 0 bucket and remote disk NET-
2.1 placed into plex 1. In this example, aggregate “test” is owned by the cluster node to the left and uses
local disk NET-1.1 and HA partner mirror disk NET-2.1.

Figure 7) ONTAP Select mirrored aggregate.

Note: When an ONTAP Select cluster is deployed, all virtual disks present on the system are auto assigned
to the correct plex, requiring no additional step from the user with respect to disk assignment. This prevents
the accidental assignment of disks to an incorrect plex and makes sure of optimal mirror disk configuration.

23 TR-4517 Technical Report

 Best Practice

Although the existence of the mirrored aggregate is needed to provide an up to date (RPO 0) copy of the
primary aggregate, care should be taken that the primary aggregate does not run low on free space. A
low-space condition in the primary aggregate may cause ONTAP to delete the common Snapshot® copy
used as the baseline for storage giveback. Although this works as designed in order to accommodate
client writes, the lack of a common Snapshot copy on failback requires the ONTAP Select node to do a
full base line from the mirrored aggregate. This operation can take a significant amount of time in a
shared-nothing environment.
A good baseline for monitoring aggregate space utilization is up to 85%.

Write Path Explained

Synchronous mirroring of data blocks between cluster nodes and the requirement of no data loss in the
event of a system failure have a significant impact on the path an incoming write takes as it propagates
through an ONTAP Select cluster. This process consists of two stages:
1. Acknowledgement
2. Destaging
Writes to a target volume occur over a data LIF and are committed to the virtualized NVRAM partition,
present on a system disk of the ONTAP Select node, before being acknowledged back to the client. On an
HA configuration, an additional step occurs, because these NVRAM writes are immediately mirrored to the
HA partner of the target volume’s owner before being acknowledged. This makes sure of the file system
consistency on the HA partner node, in case of a hardware failure on the original node.
After the write has been committed to NVRAM, Data ONTAP periodically moves the contents of this partition
to the appropriate virtual disk, a process known as “destaging.” This process only happens once, on the
cluster node owning the target volume, and does not happen on the HA partner.
The following figure shows the write path of an incoming write request to an ONTAP Select node.

Figure 8) ONTAP Select write path workflow.

Incoming write acknowledgement:

1. Writes enter the system through a logical interface owned by Select Node A.
2. Writes are committed to the NVRAM of Node A and mirrored to the HA partner, Node B.

24 TR-4517 Technical Report

3. After the I/O request is present on both HA nodes, it is then acknowledged back to the client.
ONTAP Select destaging from NVRAM to the data aggregate (ONTAP CP):
4. Writes are destaged from virtual NVRAM to virtual data aggregate.
5. Mirror engine synchronously replicates blocks to both plexes.

Disk Heart-beating
Although the ONTAP Select HA architecture leverages many of the code paths used by the traditional FAS
arrays, some exceptions exist. One of these exceptions is in the implementation of disk-based heart-
beating, a non network-based method of communication used by cluster nodes to prevent network isolation
from causing split-brain behavior. Split brain is the result of cluster partitioning, typically caused by network
failures, whereby each side believes the other is down and attempts to take over cluster resources.
Enterprise-class HA implementations must gracefully handle this type of scenario, and Data ONTAP does
this through a customized disk-based method of heart-beating. This is the job of the HA mailbox, a location
on physical storage that is used by cluster nodes to pass heart-beat messages. This helps the cluster
determine connectivity and therefore define quorum in the event of a failover.
On FAS arrays, which use a shared-storage HA architecture, Data ONTAP resolves split-brain issues
through:
• SCSI persistent reservations
• Persistent HA metadata
• HA state sent over HA interconnect
However, within the shared-nothing architecture of an ONTAP Select cluster, a node is only able to “see”
its own local storage and not that of the HA partner. Therefore, when network partitioning isolates each side
of an HA pair, the preceding methods of determining cluster quorum and failover behavior are unavailable.
Although the existing method of split-brain detection and avoidance cannot be used, a method of mediation
is still required, one that fits within the constraints of a shared-nothing environment. ONTAP Select extends
the existing mailbox infrastructure further, allowing it to act as a method of mediation in the event of network
partitioning. Because shared storage is unavailable, mediation is accomplished through access to the
mailbox disks over network-attached storage. These disks are spread throughout the cluster, including the
Mediator in a 2-node cluster, using the iSCSI protocol, so intelligent failover decisions can be made by a
cluster node based on access to these disks. If a node is able to access the mailbox disks of other nodes
outside of its HA partner, it is likely up and healthy.
Note: The mailbox architecture and disk-based heart-beating method of resolving cluster quorum and
split-brain issues are the reasons the multi-node variant of ONTAP Select requires either four
separate nodes or a mediator for a 2-node cluster.

HA Mailbox Posting
The HA mailbox architecture uses a message “post” model. At repeated intervals, cluster nodes post
messages to all other mailbox disks across the cluster, including the Mediator, stating that the node is up
and running. Within a healthy cluster, at any given point in time, a single mailbox disk on a cluster node has
messages posted from all other cluster nodes.
Attached to each Select cluster node is a virtual disk that is used specifically for shared mailbox access.
This disk is referred to as the mediator mailbox disk, because its main function is to act as a method of
cluster mediation in the event of node failures or network partitioning. This mailbox disk contains partitions
for each cluster node and is mounted over an iSCSI network by other Select cluster nodes. Periodically,
these nodes post health status to the appropriate partition of the mailbox disk. Using network-accessible
mailbox disks spread throughout the cluster allows us to infer node health through a reachability matrix. For
example, if cluster nodes A and B can post to the mailbox of cluster node D, but not node C, and cluster

25 TR-4517 Technical Report

node D cannot post to the mailbox of node C, it’s likely that node C is either down or network isolated and
should be taken over.

HA Heart-beating
Like NetApp’s FAS platforms, ONTAP Select periodically sends HA heartbeat messages over the HA
interconnect. Within the ONTAP Select cluster, this is done over a TCP/IP network connection that exists
between HA partners. Additionally, disk-based heartbeat messages are passed to all HA mailbox disks,
including mediator mailbox disks. These messages are passed every few seconds and read back
periodically. The frequency with which these are sent/received allows the ONTAP Select cluster to detect
HA failure events within approximately 15 seconds, the same window available on FAS platforms. When
heartbeat messages are no longer being read, a failover event is triggered.
Figure 8 illustrates the process of sending and receiving heartbeat messages over the HA interconnect and
mediator disks from the perspective of a single ONTAP Select cluster node, node C. Note that network
heartbeats are sent over the HA interconnect to the HA partner, node D, while disk heartbeats use mailbox
disks across all cluster nodes, A, B, C, and D.

Figure 9) HA heart-beating in a 4-node cluster: steady state.

3 Deployment and Management

This section covers the deployment and management aspects of the ONTAP Select product.

3.1 ONTAP Select Deploy

The ONTAP Select cluster is deployed using specialized tooling that provides the administrator with the
ability to build the ONTAP cluster as well as manage various aspects of the virtualized server. This utility,
called ONTAP Select Deploy, comes packaged inside of an installation VM along with the ONTAP Select
OS image. Bundling the deployment utility and ONTAP Select bits inside of a single virtual machine allows
NetApp to include all the necessary support libraries and modules while helping reduce the complexity of
the interoperability matrix between various versions of ONTAP Select and the hypervisor.
The ONTAP Deploy application can be accessed via the following methods:
• CLI
• REST API

26 TR-4517 Technical Report

• GUI
The ONTAP Deploy CLI is shell-based and immediately accessible upon connecting to the installation VM
using SSH. Navigation of the shell is similar to that of the ONTAP shell, with commands bundled into
groupings that provide related functionality (for example, “network create,” “network show,” “network
delete”).
For automated deployments and integration into existing orchestration frameworks, ONTAP Deploy can
also be invoked programmatically, through a REST API. All functionality available through the shell-based
CLI is available through the API. The entire list of API calls is documented using the OpenAPI Specification
(originally known as Swagger Specification) and can be accessed via https://{IPaddress of Deploy}/api/v2/ui

Deploy Upgrades
The Deploy utility can be upgraded separately from the Select cluster. Similarly, the Select cluster can be
upgraded separately from the Deploy utility. Please see the Upgrade section for the Deploy x Select
interoperability matrix.

Server Preparation
Although ONTAP Deploy provides the user with functionality that allows for configuration of portions of the
underlying physical server, there are several requirements that must be met before attempting to manage
the server. This can be thought of as a manual preparation phase, because many of the steps are difficult
to orchestrate through automation. This preparation phase involves the following:
• For local storage, the RAID controller and attached local storage are configured.
 RAID groups and LUNs have been provisioned.
• For VSAN or external array hosted datastores, insure that the configurations are supported by VMware
HCL and follow the specific vendor best practices.
• Physical network connectivity to server is verified.
- For external arrays, the network resiliency, speed and throughput are critical to the performance
of the ONTAP Select VM.
• Hypervisor is installed.
• Virtual networking constructs (vSwitches/port groups) are configured.
Note: After the ONTAP Select cluster has been deployed, the appropriate ONTAP management tooling
should be used to configure SVMs, LIFs, volumes, and so on. ONTAP Deploy does not provide
this functionality.
The ONTAP Deploy utility and ONTAP Select software are bundled together into a single virtual machine,
which is then made available as a .OVA file for vSphere. The bits are available from the NetApp Support
site, from this link:
http://mysupport.netapp.com/NOW/cgi-bin/software
This installation VM runs the Debian Linux OS and has the following properties:
• 2 vCPUs
• 4GB RAM
• 40GB virtual disk

27 TR-4517 Technical Report

ONTAP Select Deploy Placement in the Environment
Careful consideration should be given to the placement of the ONTAP Deploy installation VM, because the
Deploy VM will be used to verify hypervisor minimum requirements, deploy ONTAP Select clusters, apply
the license and optionally troubleshoot network connectivity between Select nodes during setup.

VM Placement
The ONTAP Select installation VM can be placed on any virtualized server in the customer environment.
For 4-node clusters, the ONTAP Deploy VM can be collocated on the same host as an ONTAP Select
instance or on a separate virtualized server. For 2-node clusters, where the ONTAP Deploy VM is also the
cluster Mediator, the collocation model is NOT supported as a it would become a cluster single point of
failure (SPOF).
The ONTAP Deploy VM can be installed in the same datacenter as the ONTAP Select cluster, or it can be
centrally deployed in a core datacenter. The only requirement is that there exists network connectivity
between the ONTAP Deploy VM and the targeted ESX host as well as the future ONTAP Select cluster
management IP address. Note that creating an ONTAP Select cluster over the WAN may take a
considerably longer amount of time since the copying of the ONTAP Select binary files depends on the
latency and bandwidth available between datacenters. Deploying a 2-node ONTAP Select cluster is
supported on a WAN network whose maximum latency and minimum bandwidth can support the Mediator
service traffic (min throughput 5Mb/s, max latency 500 ms RTT).
The following figure shows these deployment options.

Figure 10) ONTAP Select installation VM placement.

Note: Collocating the ONTAP Deploy VM and one of the ONTAP Select instances is NOT supported for
2-node clusters.

28 TR-4517 Technical Report

Multiple ONTAP Select Deploy Instances
Depending on the complexity of the environment, it may be beneficial to have more than one ONTAP Deploy
instance managing the ONTAP Select environment. When this is desired, make sure that each ONTAP
Select cluster is managed by a single ONTAP Deploy instance. ONTAP Deploy stores cluster metadata
within an internal database, so managing an ONTAP Select cluster using multiple ONTAP Deploy instances
is not recommended.
When deciding whether to use multiple installation VMs, keep in mind that while ONTAP Deploy attempts
to create unique MAC addresses by using a numeric hash based on the IP address of the installation VM,
the uniqueness of the MAC address can only occur within that Deploy instance. Because there is no
communication across Deploy instances, it is theoretically possible for two separate instances to assign
multiple ONTAP Select network adapters with the same MAC address.

Best Practice

To eliminate the possibility of having multiple Deploy instances assign duplicate MAC addresses, one
Deploy instance per L2 network should be used to manage existing or create new Select clusters /nodes.

Note: Each ONTAP Deploy instance can generate up to 64,000 unique MAC addresses. Each ONTAP
Select node consumes four MAC addresses for its internal communication network schema. Each
Deploy instance is also limited to managing 100 Select clusters and 400 hosts (a host is equivalent
to one hypervisor server).
For 2-node clusters, the ONTAP Deploy VM that creates the cluster is also the default Mediator and it
requires no further configuration. However it is absolutely critical that the Mediator service is continuously
available in order to insure proper functioning of the storage failover capabilities. For configurations where
the network latency, bandwidth or other infrastructure issues require the re-positioning of the Mediator
service closer to the ONTAP Select 2-node cluster, another ONTAP Deploy VM can be used to host the
Mediator mailboxes temporarily or permanently.

Best Practice

The ONTAP Select 2-node cluster should be carefully monitored for EMS messages indicating that the
storage failover is disabled. These messages indicate a loss of connectivity to the Mediator service and
should be rectified immediately.

3.2 ONTAP Select Licensing

ONTAP Deploy must be used to apply capacity licenses to the ONTAP Select nodes deployed by that
instance of Deploy. The ONTAP Select license allows for a flexible, consumption-based licensing model,
specifically designed to allow customers to only pay for the storage that they need. Capacity licenses are
sold in 1TB increments and must be applied to each node in the ONTAP Select cluster within 30 days of
deployment. Failure to apply a valid capacity license to each cluster node results in the ONTAP Select VM
being shut down until a valid license is applied.
The current ONTAP Select licensing model is on a per node basis and there is no concept of a cluster level
license. The per node minimum license capacity is 2TB for single node clusters and 3TB per node in a
multi-node cluster. Both maximums are 100TB. The capacity license relates to the total size of the virtual
data disks attached to the ONTAP Select virtual machine. In other words, the capacity license controls the

29 TR-4517 Technical Report

total data that a customer is entitled to store on a given ONTAP Select virtual machine, including the sync
mirror copy of the data from the HA partner.
Starting with ONTAP Select 9.0 GA and Deploy 2.2, the user has the option to consume only a portion of a
datastore. This functionality can be useful when the server capacity exceeds the desired Select license.
The capacity license will generally have to be larger than the desired active capacity under management
because of the ONTAP Select overhead and in the case of a multi-node cluster, the sync mirror copy of
active data.
Please note that the actual amount of data stored on ONTAP Select is not relevant in the capacity license
conversation, and it can vary depending on data type and storage efficiency ratios. The amount of raw
storage (defined as physical spindles inside the server) is also irrelevant because the datastore in which
Select is installed may consume only a portion of the total space. For VSAN and external storage arrays,
the total space consumed by the ONTAP Select VM will vary depending on FTT/FTM and storage efficiency
settings enabled at the VSAN / external storage array level. The ONTAP Select capacity license is not an
indication of how much space the ONTAP Select VM will consume.

3.3 ONTAP Management

Because ONTAP Select runs Data ONTAP, it supports all common NetApp management tools. As a result,
after the product is deployed and Data ONTAP is configured, it can be administered using the same set of
applications that a system administrator would use to manage FAS storage arrays. There is no special
procedure required to build out an ONTAP configuration, such as creating SVMs, volumes, LIFs, and so
on.
There are however a number of ONTAP Select management tasks that require the use of ONTAP Deploy.
ONTAP Deploy is the only method to create Select clusters. Therefore issues encountered during the
cluster creation can only be investigated using Deploy. ONTAP Deploy communicates with the ONTAP
Select clusters it created using the information configured at the time of deployment. That includes the ESX
host name or IP address as well as the ONTAP Select cluster management IP address. For 2-node ONTAP
Select clusters, the node management IP addresses are used for the iSCSI Mediator traffic.
Changing the ONTAP Select cluster management IP address from inside ONTAP Select after deployment
will not impact clients accessing the ONTAP Select data LIFs, but it will prevent ONTAP Deploy from
managing that ONTAP Select cluster.
Changing the ONTAP Select node management IP addresses for 2-node clusters after deployment will
result in an immediate loss of storage-failover capabilities for that ONTAP Select cluster. A new Mediator
location on the same or different ONTAP Deploy VM must be configured immediately.
Changing the ESX host name or IP address is not supported with the exception of a VMware HA or vMotion
operation of the single node cluster running on VSAN or external array datastores. ONTAP Deploy will
attempt to re-host the ONTAP Select VM, as long as the new ESX host is managed by the same vCenter
server.

After the installation, ONTAP Deploy can be used to complement the other NetApp management tools for
troubleshooting purposes.

The ONTAP Deploy command line interface provides options for troubleshooting that are not available in
the GUI. Most commands include a “show” option. This allows you to gather information about the
environment.

The ONTAP Deploy logs can contain valuable information to help troubleshooting cluster setup issues. The
ONTAP Deploy GUI and command line interfaces allow you generate an AutoSupport bundle containing
the ONTAP Deploy logs. The GUI also allows you to download the bundle for immediate inspection.

Finally, the Deploy GUI can be used to invoke node specific AutoSupport bundles.

30 TR-4517 Technical Report

Given that ONTAP Deploy plays an important role in the quorum service for 2-node clusters as well as
troubleshooting of the environment, the ONTAP Deploy database should be backed up regularly and after
every change in the environment. It is not currently possible to re-discover an ONTAP Select cluster that
was created by a different instance of ONTAP Deploy, and having an un-managed cluster will result in the
loss of some important troubleshooting functionality. The ONTAP Deploy configuration database can be
backed up using the ONTAP Deploy command line interface and issuing the ‘configuration backup’
command.

4 Network Design Considerations

This section covers the various network configurations and best practices that should be taken into
consideration when building an ONTAP Select cluster. Like the design and implementation of the underlying
storage, care should be taken when making network design decisions because these choices have a
significant impact on both the performance and resiliency of the ONTAP Select cluster.
In traditional FAS systems, interface groups are used to provide aggregate throughput and fault tolerance
using a single, logical, virtualized network interface configured on top of multiple physical network
interfaces. ONTAP Select leverages the underlying hypervisor’s virtualization of multiple physical network
interfaces to achieve the same goals of throughput aggregation and resiliency. The network interface cards
that ONTAP Select manages are therefore logical constructs, and configuring additional interface groups
does not achieve the goals of throughput aggregation or recovering from hardware failures.

4.1 Network Configuration: Multi-node

The multi-node ONTAP Select network configuration consists of two networks: an internal network,
responsible for providing cluster and internal replication services, and an external network, responsible for
providing data access and management services. End-to-end isolation of traffic that flows within these two
networks is extremely important in allowing us to build an environment that is suitable for the cluster
resiliency.
These networks are represented in figure 10, which shows a four-node ONTAP Select cluster running on a
VMware vSphere platform. Note that each ONTAP Select instance resides on a separate physical server,
and internal and external traffic is isolated through the use of separate network port groups, which are
assigned to each virtual network interface and allow the cluster nodes to share the same physical switch
infrastructure.

31 TR-4517 Technical Report

Figure 11) ONTAP Select multinode network configuration.

Each ONTAP Select virtual machine contains six virtual network adapters, presented to Data ONTAP as a
set of six network ports, e0a through e0f. Although ONTAP treats these adapters as physical NICs, they
are in fact virtual and map to a set of physical interfaces through a virtualized network layer. As a result,
each hosting server does not require six physical network ports.
Note: Adding virtual network adapters to the ONTAP Select VM is not supported.
These ports are preconfigured to provide the following services:
• e0a, e0b. Data and management LIFs
• e0c, e0d. Cluster network LIFs
• e0e. RAID SyncMirror (RSM)
• e0f. HA interconnect
Ports e0a and e0b reside on the external network. Although ports e0c–e0f perform several different
functions, collectively they compose the internal Select network. When making network design decisions,
these ports should be placed on a single L2 network. There is no need to separate these virtual adapters
across different networks.
The relationship between these ports and the underlying physical adapters can be seen in Figure 11, which
depicts one ONTAP Select cluster node on the ESX hypervisor.

32 TR-4517 Technical Report

Figure 12) Network configuration of a multinode ONTAP Select VM.

Segregating internal and external traffic across different physical NICs ensures that we are not introducing
latencies into the system due to insufficient access to network resources. Additionally, aggregation through
NIC teaming makes sure that failure of a single network adapter does not prevent the ONTAP Select cluster
node from accessing the respective network.

LIF Assignment
With the introduction of IPspaces, Data ONTAP port roles have been deprecated. Similar to FAS arrays,
ONTAP Select clusters contain both a default and cluster IPspace. By placing network ports e0a and e0b
into the default IPspace and ports e0c and e0d into the cluster IPspace, we have essentially walled off
those ports from hosting LIFs that do not belong. The remaining ports within the ONTAP Select cluster are
consumed through the automatic assignment of interfaces providing internal services and not exposed
through the ONTAP shell, as is the case with the RSM and HA interconnect interfaces.
Note: Not all LIFs are visible through the ONTAP command shell. The HA interconnect and RSM
interfaces are hidden from ONTAP and used internally to provide their respective services.
The network ports/LIFs are explained in further detail in the following sections.

Data and Management LIFs (e0a, e0b)

Data ONTAP ports e0a and e0b have been delegated as candidate ports for logical interfaces that carry
the following types of traffic:
• SAN/NAS protocol traffic (CIFS, NFS, iSCSI)
• Cluster, node, and SVM management traffic
• Intercluster traffic (SnapMirror, SnapVault)
Note: Cluster and node management LIFs are automatically created during ONTAP Select cluster setup.
The remaining LIFs may be created post deployment.

Cluster Network LIFs (e0c, e0d)

Data ONTAP ports e0c and e0d have been delegated as home ports for cluster interfaces. Within each
ONTAP Select cluster node, two cluster interfaces are automatically generated during Data ONTAP setup
using link-local IP addresses (169.254.x.x).
Note: These interfaces cannot be assigned static IP addresses, and additional cluster interfaces should
not be created.
Cluster network traffic must flow through a low-latency, nonrouted layer 2 network. Due to cluster
throughput and latency requirements, the ONTAP Select cluster is expected to be physically located within

33 TR-4517 Technical Report

proximity (for example, multipack, single data center). Building a stretch cluster configuration by separating
HA nodes across a wide area network or across significant geographical distances is not supported.
Note: To make sure of maximum throughput for cluster network traffic, this network port is configured to
use jumbo frames (9000 MTU). This is not configurable, so to make sure of proper cluster operation,
verify that jumbo frames are enabled on all upstream virtual and physical switches providing internal
network services to ONTAP Select cluster nodes.

RAID SyncMirror Traffic (e0e)

Synchronous replication of blocks across HA partner nodes occurs using an internal network interface
residing on network port e0e. This functionality happens automatically, using network interfaces configured
by Data ONTAP during cluster setup, and requires no configuration by the administrator.
Because this port is reserved by Data ONTAP for internal replication traffic, neither the port nor the hosted
LIF is visible in the Data ONTAP CLI or management tooling. This interface is configured to use an
automatically generated link-local IP address, and the reassignment of an alternate IP address is not
supported.
Note: This network port requires the use of jumbo frames (9000 MTU).
Throughput and latency requirements that are critical to the proper behavior of the replication network
dictate that ONTAP Select nodes be located within close physical proximity, so building a “hot” disaster
recovery solution is not supported.

HA Interconnect (e0f)
NetApp FAS arrays use specialized hardware to pass information between HA pairs in an ONTAP cluster.
Software-defined environments, however, do not tend to have this type of equipment available (such as
Infiniband or iWARP devices), so an alternate solution is needed. Although several possibilities were
considered, ONTAP requirements placed on the interconnect transport required that this functionality be
emulated in software. As a result, within an ONTAP Select cluster, the functionality of the HA interconnect
(traditionally provided by hardware) has been designed into the OS, using Ethernet as a transport
mechanism.
Each ONTAP Select node is configured with an HA interconnect port, e0f. This port hosts the HA
interconnect network interface, which is responsible for two primary functions:
• Mirroring the contents of NVRAM between HA pairs
• Sending/receiving HA status information and network heartbeat messages between HA pairs
HA interconnect traffic flows through this network port using a single network interface by layering RDMA
frames within Ethernet packets. Similar to RSM, neither the physical port nor the hosted network interface
is visible to users from either the ONTAP CLI or management tooling. As a result, the IP address of this
interface cannot be modified, and the state of the port cannot be changed.
Note: This network port requires the use of jumbo frames (9000 MTU).

4.2 Network Configuration: Single Node

Single-node ONTAP Select configurations do not require the ONTAP internal network, because there is no
cluster, HA, or mirror traffic. Unlike the multi-node version of the ONTAP Select product, which contains six
virtual network adapters, each ONTAP Select virtual machine contains two virtual network adapters,
presented to Data ONTAP network ports e0a and e0b.
These ports are used to provide all the following services: data, management, and intercluster LIFs.
The relationship between these ports and the underlying physical adapters can be seen in figure 12, which
depicts one ONTAP Select cluster node on the ESX hypervisor.

34 TR-4517 Technical Report

Figure 13) Network configuration of single-node ONTAP Select VM.

Note that NIC teaming is still required, though two adapters are sufficient for a single-node cluster.

LIF Assignment
As explained in the multinode LIF assignment section of this document, IPspaces are used by ONTAP
Select to keep cluster network traffic separate from data and management traffic. The single-node variant
of this platform does not contain a cluster network; therefore, no ports are present in the cluster IPspace.
Note: Cluster and node management LIFs are automatically created during ONTAP Select cluster setup.
The remaining LIFs may be created post deployment.

4.3 Networking: Internal and External

ONTAP Select Internal Network

The internal ONTAP Select network, which is only present in the multinode variant of the product, is
responsible for providing the ONTAP Select cluster with cluster communication, HA interconnect, and
synchronous replication services. This network includes the following ports and interfaces:
• e0c, e0d. Hosting cluster network LIFs
• e0e. Hosting the RAID SyncMirror (RSM) interface
• e0f. Hosting the HA interconnect
The throughput and latency of this network are critical in determining the performance and resiliency of the
ONTAP Select cluster. Network isolation is required for cluster security and to make sure that system
interfaces are kept separate from other network traffic. Therefore, this network must be used exclusively by
the ONTAP Select cluster.

Note: Using the Select internal network for traffic other than Select cluster traffic, such as application or
management traffic, is not supported. There can be no other VMs or hosts on the ONTAP internal
VLAN.
Network packets traversing the internal network must be on a dedicated VLAN tagged layer-2 network.
This can be accomplished by one of the following:
• Assigning a VLAN-tagged port group to the internal virtual NICs (e0c–e0f)
• Using the native VLAN provided by the upstream switch where the native VLAN is not used for any
other traffic

35 TR-4517 Technical Report

ONTAP Select External Network
The ONTAP Select external network is responsible for all outbound communications by the cluster and
therefore is present on both the single-node and multinode configurations. Although this network does not
have the tightly defined throughput requirements of the internal network, the administrator should be careful
not to create network bottlenecks between the client and ONTAP VM, because performance issues could
be mischaracterized as ONTAP Select problems.

Internal Versus External Network

Table 5 highlights the major differences between the ONTAP Select internal and external networks.

Table 5) Internal versus external network quick reference.

Description Internal Network External Network

Cluster, HA/IC, RAID SyncMirror Data, management, intercluster
Network services
(RSM) (SnapMirror and SnapVault)

Network isolation Required Optional

Frame size (MTU) 9,000 1,500 (default)/9,000 (supported)

NIC aggregation Required Required

IP address assignment Autogenerated User defined

DHCP support No No

Internal Network Validation and Troubleshooting

Starting with Deploy 2.2, the internal network in a multi node cluster can be validated using the network
connectivity checker functionality, which can be invoked from the Deploy command line interface using
the ‘network connectivity-check start’ command.
The output of the test can be viewed using the ‘network connectivity-check show --run-id X’ command,
This tool is only useful for troubleshooting the internal network in a multi node Select cluster. It should
not be used to troubleshoot single node clusters or client side connectivity issues.

NIC Aggregation
To make sure that the internal and external networks have both the necessary bandwidth and resiliency
characteristics required to provide high performance and fault tolerance, physical network adapter
aggregation is used. This is a requirement on both the internal and external networks of the ONTAP Select
cluster and provides the ONTAP Select cluster with two major benefits:
• Isolation from a single physical port failure
• Increased throughput
NIC aggregation allows the ONTAP Select instance to balance network traffic across two physical ports.
LACP-enabled port channels are only supported with distributed vSwitches.

Best Practice

In the event that a NIC has multiple ASICs, select one network port from each ASIC when building
network aggregation constructs through NIC teaming for the internal and external networks.

36 TR-4517 Technical Report

MAC Address Generation
The MAC addresses assigned to all ONTAP Select network ports are generated automatically by the
included deployment utility, using a platform-specific organizationally unique identifier (OUI) specific to
NetApp to make sure there is no conflict with FAS systems. A copy of this address is then stored in an
internal database, within the ONTAP Select installation VM (ONTAP Deploy), to prevent accidental
reassignment during future node deployments. At no point should the administrator modify the assigned
MAC address of a network port.

4.4 Supported Network Configurations

Server vendors understand that customers have different needs, and choice is critical. As a result, when
purchasing a physical server, there are numerous options available when making network connectivity
decisions. Most commodity systems ship with a variety of NIC choices, offering single-port and multiport
options with varying permutations of 1Gb and 10Gb ports. Care should be taken when selecting server
NICs, because the choices provided by server vendors can have a significant impact on the overall
performance of the ONTAP Select cluster.
Link aggregation is a core construct used to provide sufficient bandwidth to both the external and internal
ONTAP Select networks. Link Aggregation Control Protocol (LACP) is a vendor-neutral standard providing
an open protocol for network endpoints to use to bundle groupings of physical network ports into a single
logical channel.
When choosing an ONTAP Select network configuration, use of LACP, which requires specialized hardware
support, may be a primary consideration. Although LACP requires support from both the software virtual
switch and the upstream physical switch, it can provide a significant throughput benefit to incoming client
protocol traffic.
Table 6 shows the various supported configurations. The use of LACP is called out, because environmental
and hypervisor-specific dependencies prevent all combinations from being supported.

Table 6) Network configuration support matrix.

Client Environment Select Configuration Best Practices

• 2 or more x 10GB physical ports • Single LACP channel with all ports. • Load-balancing policy at the port
• Distributed vSwitch • Internal network uses a port group with group level is “route based on IP
• Physical uplink switch supports VST to add VLAN tagging. hash” and “source and destination IP
LACP and 9,000 MTU size on all • External network uses a separate port address and TCP/UDP port and
VLAN” on the link aggregation group
ports group; VST and VGT are supported.
(LAG).
• LACP mode set to ACTIVE on both
the ESX and the physical switches;
LACP timer should be set to FAST (1
second) on the port channel
interfaces and on the VMNICs.
• VMware recommends that STP be
set to Portfast on the switch ports
connected to the ESXi hosts.

37 TR-4517 Technical Report

 Client Environment Select Configuration Best Practices

• 2 x 10Gb ports and 2 x 1 Gb ports • Do not use any LACP channels. • Load-balancing policy at the port
OR • Internal network must use a port group group level is “route based on
• 9,000 MTU is not supported on all with at least 2 10Gb ports and MTU originating virtual port ID.”
physical ports or switch ports 9,000. 1Gb ports and ports that do not • VMware recommends that STP be
OR support 9000 MTU should be used for set to Portfast on the switch ports
the external network. connected to the ESXi hosts.
• Using a standard vSwitch
• External network uses a separate port
group containing all the ports. The
ACTIVE ports are ports that are not
used for the internal network. The
STANDBY ports are the internal
network ports.
• All the ports must be owned by the
same vSwitch. The MTU setting on the
vSwitch must be set to 9,000.

Because the performance of the ONTAP Select VM is tied directly to the characteristics of the underlying
hardware, increasing the throughput to the VM by selecting 10Gb-capable NICs results in a higher
performing cluster and a better overall user experience. When cost or form factor prevents the user from
designing a system with four 10Gb NICs, two 10Gb NICs can be used.
See figure 20 for an example of a configuration where LACP is used and figure 21 for a configuration without
LACP.

4.5 vSphere: vSwitch Configuration

ONTAP Select supports the use of both standard and distributed vSwitch configurations. This section
describes the vSwitch configuration and load-balancing policies that should be used in both two-NIC and
four-NIC configurations.

vSphere: Standard vSwitch

All vSwitch configurations require a minimum of two physical network adapters bundled into a single link
aggregation group (referred to as “NIC teaming”). On a vSphere server, NIC teams are the aggregation
construct used to bundle multiple physical network adapters into a single logical channel, allowing the
network load to be shared across all member ports. It’s important to remember that NIC teams can be
created without support from the physical switch. Load-balancing and failover policies can be applied
directly to a NIC team, which is unaware of the upstream switch configuration. In this case, policies are only
applied to outbound traffic. In order to balance inbound traffic, the physical switch must be properly
configured. Port channels are the primary way this is accomplished.
Note: Static port channels are not supported with ONTAP Select. LACP enabled channels are only
supported with Distributed vSwitches

Best Practice

To make sure of optimal load balancing across both the internal and the external ONTAP Select
networks, the load-balancing policy of “Route based on originating virtual port” should be used.

Figure 13 shows the configuration of a standard vSwitch and the two port groups responsible for handling
internal and external communication services for the ONTAP Select cluster.
Note that the external network may use the internal network vmnics in the case of a network outage, but
the opposite may not always be the case, depending on the vmnic properties for speed and MTU size.

38 TR-4517 Technical Report

Figure 14) Port group configurations using a standard vSwitch.

vSphere: Distributed vSwitch

When using distributed vSwitches in your configuration, LACP can be used in order to increase the
throughput and resiliency of the network construct. The only supported LACP configuration requires that
all the vmnics are in a single LAG. The uplink physical switch must support 9,000 MTU on all the ports in

39 TR-4517 Technical Report

the channel. The internal and external Select networks should be isolated at the port group level. The
internal network should use a nonroutable (isolated) VLAN. The external network can use either VST or
VGT.

Figure 15) Link aggregation group properties when using LACP.

Figure 16) Port group configurations using a distributed vSwitch with LACP enabled.

40 TR-4517 Technical Report

Note: LACP requires the upstream switch ports to be configured as a port channel. Prior to enabling this
on the distributed vSwitch, make sure that an LACP-enabled port channel is properly configured.

Best Practice

NetApp recommends that the LACP mode be set to ACTIVE on both the ESX and the physical switches.
Furthermore, the LACP timer should be set to FAST (1 second) on the portchannel interfaces and on the
VMNICs.
When using a Distributed vSwitch with LACP, we recommend configuring the load-balancing policy to
“Route based on IP Hash” on the port group and “Source and Destination IP Address and TCP/UDP port
and VLAN” on the link aggregation group (LAG)

4.6 Physical Switch Configuration

Careful consideration should be taken when making connectivity decisions from the virtual switch layer to
physical switches. Separation of internal cluster traffic from external data services should extend to the
upstream physical networking layer through isolation provided by L2 VLANs.
This section covers upstream physical switch configurations based on single-switch and multi-switch
environments.
Physical switch ports may be configured as trunk or access ports, depending on the VLAN configuration of
the internal and external ONTAP Select networks. ONTAP Select external traffic can be separated across
multiple layer 2 networks, through either the use of ONTAP VLAN tagged virtual ports or by assigning
separate port groups to management port e0a and data port e0b. If this is the case, the uplink physical
switch ports should be configured in trunk mode, because each is tagged using a separate VLAN tag.
Otherwise, if all traffic flowing in to the upstream physical switch port is part of the same VLAN, access
ports may be used.
ONTAP Select internal network traffic occurs using virtual interfaces defined with link-local IP addresses.
Because these IP addresses are non-routable, internal traffic between cluster nodes must flow across a
single layer 2 network. Route hops between ONTAP Select cluster nodes are unsupported.

41 TR-4517 Technical Report

 Best Practice

VMware recommends that STP be set to Portfast on the switch ports connected to the ESXi hosts. Not
setting STP to Portfast on the switch ports may affect ONTAP Select's ability to tolerate uplink failures.

Shared Physical Switch

Figure 16 depicts a possible switch configuration used by one node in a multi node ONTAP Select cluster.
In this example, the physical NICs used by the vSwitches hosting both the internal and external network
port groups are cabled to the same upstream switch. Switch traffic is kept isolated through the use of
broadcast domains contained within separate VLANs. Note that for the ONTAP Select internal network,
tagging is done at the port group level. While the following example uses VGT for the external network,
both VGT and VST are supported on that port group.

Figure 17) Network configuration using shared physical switch.

Note: In this configuration, the shared switch becomes a single point of failure. If possible, multiple
switches should be used to prevent a physical hardware failure from causing a cluster network
outage.

Multiple Physical Switches

When redundancy is needed, multiple physical network switches should be used. Figure 17 depicts a
recommended configuration used by one node in a multi node ONTAP Select cluster. NICs from both the
internal and external port groups are cabled in to different physical switches, protecting the user from a
single hardware switch failure. A virtual port channel is configured between switches to prevent spanning
tree issues.

Best Practice

When sufficient hardware is available, NetApp recommends using the following multi-switch
configuration, due to the added protection against physical switch failures.

42 TR-4517 Technical Report

Figure 18) Network configuration using multiple physical switches.

4.7 Data and Management Separation

ONTAP Select external network traffic is defined as data (CIFS, NFS, iSCSI), management, and replication
(SnapMirror) traffic. Within an ONTAP cluster, each style of traffic uses a separate logical interface that
must be hosted on a virtual network port. On the multinode version of ONTAP Select, these are designated
as ports e0a and e0b, because the remaining ports are reserved for internal cluster services.
NetApp recommends isolating data traffic and management traffic into separate L2 networks. In the ONTAP
Select environment, this is done using VLAN tags. This can be achieved by assigning a VLAN-tagged port
group to select “network adapter 1” (port e0a) for management traffic and a separate port group to select
“network adapter 2” (port e0b) for data traffic.
Because there are only two virtual ports available to host all data and management LIFs, the virtual switch
tagging (VST) solution described earlier may not be sufficient, because it requires that all egress traffic from
virtual ports be tagged with the same VLAN ID. With limited ports available, collocating both data and
management LIFs on the same virtual port may be required, with VLAN tagging performed by the virtual
machine, a process known as virtual guest tagging (VGT).
Note: Data and management network separation through VGT is not available when using the ONTAP
Deploy utility. This must be performed after cluster setup has completed.
Both configuration options are supported. Figure 18 shows the first scenario, where traffic is tagged at the
vSwitch layer through the assigned port group. In this configuration, cluster and node management LIFs
are assigned to ONTAP port e0a and tagged with VLAN ID 10 through the assigned port group. Data LIFs
are assigned to port e0b and given VLAN ID 20 using a second port group, while the cluster ports are using
a third port group and are on VLAN ID 30.

43 TR-4517 Technical Report

Figure 19) Data and management separation using VST.

Figure 19 shows the second scenario, where traffic is tagged by the ONTAP VM using virtual VLAN ports
that are placed into separate broadcast domains. In this example, virtual ports e0a-10/e0b-10 and e0a-
20/e0b-20 are placed on top of VM ports e0a and e0b, allowing the network tagging to be done directly
within ONTAP, rather than at the vSwitch layer. Management and data LIFs are placed on these virtual
ports, allowing further L2 subdivision within a single VM port. The cluster VLAN (VLAN ID 30) is still tagged
at the port group.
Note: This style of configuration is especially desirable when using multiple IPspaces. Group VLAN ports
into separate custom IPspaces if further logical isolation and multitenancy are desired.

Figure 20) Data and management separation using VGT.

44 TR-4517 Technical Report

 Best Practice

If data traffic spans multiple layer 2 networks (and the use of VLAN ports is required) or when using
multiple IPspaces, VGT should be used.

4.8 Four-NIC Configuration

As described earlier, supported network configurations involve permutations based on two and four physical
NIC ports. For optimum performance and resiliency, NetApp strongly recommends that the ONTAP Select
instance reside on a physical server with four 10Gb NIC ports. NIC teaming is a requirement on both two-
NIC and four-NIC configurations, and having four NIC ports present on the system allows for the physical
separation of traffic and reduces the potential for network-based bottlenecks between the internal and
external networks.
Within an ONTAP Select cluster, internal and external traffic are separated through the use of virtual layer
2 network objects known as port groups. Proper vSwitch assignment of these port groups is extremely
important, especially for the internal network, which is responsible for providing cluster, HA interconnect,
and mirror replication services. Insufficient network bandwidth to these network ports can cause
performance degradation and even affect the stability of the cluster node.
Therefore, for a 4-node cluster, the internal ONTAP network requires 10Gb connectivity; 1Gb NICs are not
supported. Tradeoffs can be made to the external network, however, because limiting the flow of incoming
data to an ONTAP Select cluster does not affect its ability to operate reliably.
A 2-node cluster may use 4 1Gb ports for internal traffic instead of the 2 10Gb ports required by the 4-node
cluster.

Best Practice

In an environment where conditions prevent the server from being fit with 4 10Gb NIC cards, 2 1Gb NICs
can be used for the external ONTAP network.
4 1Gb ports can used for internal traffic in 2-node ONTAP Select clusters.

Figures 20 through 22 depicts various ways in which to configure the network on a physical server with four
physical NIC ports, depending on the whether a distributed switch is used or whether all four ports are
10Gb.
For 2-node ONTAP Select clusters, Figures 20 and 21 are also supported with 4 1Gb ports.

45 TR-4517 Technical Report

Figure 21) Four 10Gb NIC network configuration with LACP on a distributed vSwitch.

Figure 22) Four 10Gb NIC network configuration without LACP.

46 TR-4517 Technical Report

Figure 23) Four NIC network configuration (2x10Gb + 2x1Gb).

Note that in all cases VLAN tagging for internal network traffic is done by the port group (VLAN 10). External
traffic, however, is untagged by the port group and instead is tagged by the upstream switch, using the
native VLAN tag (VLAN 20). This is only intended to highlight one possible way of implementing layer 2
tagging within an ONTAP Select cluster. Like the ONTAP internal port group, a static VLAN ID could also
be assigned to the external network. Implementing tagging at the VM layer and not at the vSwitch does
have one added benefit, however. Similar to FAS systems, ONTAP Select allows the use of multiple
IPspaces and VLAN tagging in its support for multitenancy implementations. In order for this functionality
to be available to the ONTAP Select administrator, VLAN tagging should be done at the VM level.

Implementing the tagging within a virtual machine is a process known as virtual guest tagging (VGT). Using
VGT with ONTAP Select, rather than implementing VLAN tagging through the port group or physical switch,
allows data, management, and replication traffic to be further split across multiple layer 2 networks.

4.9 Two-NIC Configuration

When four physical NIC ports are unavailable, two NICs may be used as an alternative. Like the four-NIC
configuration described earlier, NIC teaming of the physical NIC ports is required, providing the cluster with
increased throughput and resiliency in the event of a NIC failure. Two-NIC configurations require the use
of 10Gbps NICs. Running ONTAP Select on a system with only two 1Gbps NICs is only supported for single
node Select clusters.

47 TR-4517 Technical Report

Figure 24) Two-NIC network configuration.

5 Use Cases

5.1 Remote/Branch Office

The ONTAP Select VM can be collocated with application VMs, making it an ideal solution for remote and
branch offices. Using ONTAP Select to provide enterprise-class file services while allowing for bidirectional
replication to other ONTAP Select or FAS clusters allows for resilient solutions to be built in low-touch or
low-cost environments. Because ONTAP Select comes prepopulated with feature licenses for CIFS, NFS,
and iSCSI protocol services as well as both SnapMirror and SnapVault replication technologies, all these
features are available immediately upon deployment.
Starting with ONTAP Select 9.2 and ONTAP Deploy 2.4, all vSphere and VSAN licenses are now
supported.
The ONTAP Select 2-node cluster with a remote Mediator is an attractive solution for small datacenters. In
this configuration, the HA functionality is provided by ONTAP Select.
The vNAS ONTAP Select solution running on VSAN (including the 2-node VSAN ROBO configuration) is
another option. In this configuration, the HA functionality is provided by VSAN.
Finally, a single node ONTAP Select cluster replicating its data to a core location can provide a set of robust
enterprise data management tools on top of a commodity server.
Figure 24 depicts a common remote office configuration using ONTAP Select.

48 TR-4517 Technical Report

5.2 Schedule-driven SnapMirror relationships periodically replicate the data from
the remote office to a single consolidated engineered storage array, located
in the main data center.
Figure 25) Scheduled backup of remote office to corporate data center.

5.3 Private Cloud (Data Center)

Another common use case for ONTAP Select is providing storage services for private clouds built on
commodity servers. In Figure 25, a storage farm provides compute and locally attached storage to the
ONTAP Select VM, which provides storage services upstream to an application stack. The entire workflow,
from the provisioning of storage virtual machines (SVMs) to the deployment and configuration of application
VMs, is automated through a private cloud orchestration framework.
This is the service-oriented private cloud model, and using the HA version of ONTAP Select allows for the
same Data ONTAP experience one would expect on higher cost FAS arrays. Storage server resources are
consumed exclusively by the ONTAP Select VM, with application VMs hosted on separate physical
infrastructure.

49 TR-4517 Technical Report

Figure 26) Private cloud built on direct-attached storage.

6 Upgrading
This section contains important information concerning the maintenance of various aspects of an ONTAP
Select cluster. It is possible to upgrade ONTAP Select and ONTAP Deploy independent of each other. The
following table describes the support matrix for Select and Deploy:

Table 7) ONTAP Deploy vs. ONTAP Select support matrix

Select 9.0 Select 9.1 Select 9.2

Deploy 2.1. Not Supported Not Supported Not Supported

Deploy 2.2.2 Supported Supported Not Supported

Deploy 2.3 Supported Supported Not Supported

Deploy 2.4 Not Supported Supported Supported

ONTAP Deploy will only manage Select clusters that it has deployed. Currently there is no functionality to
“discover” Select clusters installed using another instance of Deploy. We recommend backing up the Deploy
configuration every time a new Select cluster has been deployed. Restoring the Deploy database allows a
new Deploy instance to manage Select clusters installed using another Deploy VM. However, care should
be taken to ensure that one cluster is not managed by multiple Deploy instances.

Best Practice

NetApp recommends that the Deploy database be backed up on a regular basis as well as every time a
configuration change is made and before any upgrade.

50 TR-4517 Technical Report

6.1 Increasing Capacity
The storage add functionality can be used to increase the space assigned to an ONTAP Select node. This
functionality is available starting with Deploy 2.3 GUI / CLI / API. Prior versions of Deploy do not support
this functionality, but Deploy can be upgraded independently of ONTAP Select. Additionally, the storage
add functionality is supported by ONTAP Select starting with version 9.1. For adding capacity to ONTAP
Select version 9.0 (regardless of the Deploy version), please see section 7.2.
The following considerations are important for the success of the capacity expansion operation. Adding
capacity requires that the existing license covers the total amount of space (existing plus new). A storage
add operation that will result in the node exceeding its licensed capacity will fail. A new license with sufficient
capacity should be installed first.
Deploy 2.3 supports creating single node Select 9.1 clusters using VSAN, external arrays or local storage
(DAS) for its storage pool (datastore). If the extra capacity is to be added to the existing Select aggregate,
then the new storage pool (datastore) should have a similar performance profile to the existing storage pool
(datastore). For example, capacity from an external type datastore should never be added to the same
aggregate as capacity from a DAS type datastore. Instead, the new capacity should be used to create a
new aggregate.
In the event that locally attached storage is added to a system to provide for additional local (DAS) storage
pools, it is necessary to build an additional RAID group and LUN or LUNs. Just as with FAS systems, care
should be taken to ensure that the new RAID group performance is similar to the original RAID group, if the
new space is to be added to the same aggregate. If a new aggregate is to be created, the new RAID group
layout could be different as long as the performance implications for the new aggregate are well understood.
The new space can be added to same datastore as an extent, as long as the total size of the datastore
does not exceed the ESX supported maximum datastore size. Adding a datastore extent to the datastore
where ONTAP Select is already installed can be done dynamically and does not affect the operations of
the ONTAP Select node.
If the ONTAP Select node is part of an HA pair, some additional considerations are in order. VSAN and
external arrays are not supported types of storage pools for HA pairs. Therefore increasing capacity in an
HA pair requires adding local storage to both nodes in the pair.
In an HA pair, each node contains a mirror copy of the data from its partner. Adding space to node 1 requires
that an identical amount of space is added to its partner (node 2) in order to insure that all the data from
node 1 is replicated to node 2. That is to say, the space added to node 2 as part of the capacity add
operation for node 1 is not visible or accessible on node 2. The space is added to node 2 to insure that the
node 1 data is fully protected during an HA event. There is an additional consideration with regards to
performance. The data on node 1 is synchronously replicated to node 2. Therefore the performance of the
new space (datastore) on node 1, needs to match the performance of the new space (datastore) on node
2. In other words, adding space on both nodes but using different drive technologies or different RAID group
sizes can lead to performance issues due to the RAID sync mirror used to maintain a copy of the data on
the partner node.
To increase user accessible capacity on both nodes in an HA pair, two storage add operations need to be
performed, one for each node. Each storage add operation will require additional space on both nodes. The
total space required on each node will be equal to the space required on node 1 plus the space required
on node 2.
Figure 26 shows the steps required to add space to a Select node that is part on an HA Pair.

Figure 27) Storage Add Operation

Initial setup with two nodes having 60TB of space each. ONTAP Select utilizes 15TB on each node. There is free
space left in the Datastore 1 and Datastore 2 is completely free.

51 TR-4517 Technical Report

The storage add operations on Node 1 will consume of the rest of Datastore 1 as well as a part of Datastore
2 (using capacity cap).

6.2 Increasing Capacity for ONTAP Select 9.0

This section describes the process used to add capacity to a Select 9.0 cluster.
ONTAP Select 9.0 does not support external storage. This section will only cover the process of adding
additional local storage to the ONTAP Select VM.
ONTAP Select 9.0 does not support multiple storage pools (datastores). This section will only cover the
process of adding additional local storage as a new extent to the datastore where the Select node is already
installed. This operation can be done dynamically using vSphere tools and does not affect the operations
of the ONTP Select node.
After the storage is added to the ESX server, the new RAID group is created and the new LUN is imported
as a datastore extent, virtual disks must be created and attached to the ONTAP Select VM. This process
must be done using the native vSphere tooling.

52 TR-4517 Technical Report

Note: Nodes in an HA pair must have the same total capacity. Increasing capacity for node 1 by 32TB
implies a similar and simultaneous capacity expansion on its HA partner (node 2).
Within each ONTAP Select node, the newly assigned storage should be split into a number of equal-sized
virtual disks, with no virtual disk exceeding 8TB.
For example:
If 32TB of storage is added to the ONTAP Select cluster node, configure four 8TB virtual disks.
If 7TB of storage is added to the ONTAP Select node, configure one 7TB virtual disk.
After the virtual disks have been provisioned, use the mirrored aggregate creation workflow below for details
on assigning and configuring newly attached storage.

Our first step is to assign the disks to the proper cluster node and plex. To accomplish this, use the following
steps (in this example, we’re using a newly installed ONTAP Select cluster with two 100GB data disks per
node):
From the ONTAP CLI, run the following command:
disk show –fields location,aggregate,owner

mycluster::> disk show -fields location,aggregate,owner

disk owner aggregate location
------- ----- ------------- --------
NET-1.1 sdotb aggr0_sdotb_0 sdota
NET-1.2 - - sdota
NET-1.3 - - sdota
NET-1.4 sdota aggr0 sdota
NET-2.1 sdotb aggr0_sdotb_0 sdotb
NET-2.2 sdota aggr0 sdotb
NET-2.3 - - sdotb
NET-2.4 - - sdotb
NET-3.1 - - sdotc
NET-3.2 - - sdotc
NET-3.3 sdotc aggr0_sdotc_0 sdotc
NET-3.4 sdotd aggr0_sdotd_0 sdotc
NET-4.1 - - sdotd
NET-4.2 - - sdotd
NET-4.3 sdotc aggr0_sdotc_0 sdotd
NET-4.4 sdotd aggr0_sdotd_0 sdotd
16 entries were displayed.

The “location” field lists the ONTAP Select cluster node that has a physical connection to the backing
VMDK. This is the owning node.
a. From here we can see that node “sdota” has two unassigned data disks physically connected: NET-
1.2 and NET-1.3.
b. We can also see that node “sdotb” has two unassigned data disks physically connected: NET-2.3
and NET-2.4.
To create an aggregate on node sdota, assign a local disk to the storage pool 0 (another term for plex) and
a mirror disk to storage pool 1. Remember that the mirror disk must be contributed by the HA partner,
in this case sdotb, so we’ll use disk NET-2.4.

mycluster::> disk assign -disk NET-1.2 -owner sdota -pool 0

mycluster::> disk assign -disk NET-2.3 -owner sdota -pool 1

53 TR-4517 Technical Report

Our aggregate uses these two disks: NET-1.2 and NET-2.3. Although both disks have been assigned to
ONTAP Select node sdota:
c. NET-1.2 is physically connected to ONTAP Select VM “sdota.”
d. NET-2.3 is physically connected to ONTAP Select VM “sdotb.”

Now that disks have been assigned to the correct plex (pool), our next step is to create the aggregate.
Note: This step may also be performed using System Manager.
To build the aggregate, issue the following command:
aggregate create -aggregate <aggr-name> -diskcount 2 -mirror true -node
<ontap-node>

mycluster::> aggregate create -aggregate data_aggr1 -diskcount 2 -mirror true -node sdota
(storage aggregate create)

Info: The layout for aggregate "data_aggr1" on node "sdota" would be:

First Plex

RAID Group rg0, 1 disks (advanced_zoned checksum, raid0)

Position Disk Type Size
---------- ------------------------- ---------- ---------------
data NET-1.2 VMDISK 98.41GB

Second Plex

RAID Group rg0, 1 disks (advanced_zoned checksum, raid0)

Position Disk Type Size
---------- ------------------------- ---------- ---------------
data NET-2.3 VMDISK 98.41GB

Aggregate capacity available for volume use would be 84.14GB.

Do you want to continue? {y|n}: y

[Job 41] Job succeeded: DONE. Creation of aggregate "data_aggr1" has been initiated. 2 disks need
to be zeroed before they can be added to the aggregate. The process has been initiated. Once
zeroing completes on these disks, all disks will be added at once. Note that if the system reboots
before the disk zeroing is complete, the aggregate will not exist.

Note: From this point, SVMs, volumes, LIFs, and protocol configuration can be done through System
Manager (or the ONTAP CLI) using the same set of procedures you would use to configure these
on a FAS.

6.3 Single-Node to Multinode Upgrade

Upgrading from the single-node, non-HA version of ONTAP Select to the multinode scale-out version is not
supported. Migrating from the single-node to multinode version requires the provisioning of a new ONTAP
Select cluster and using SnapMirror to copy existing data from the single-node cluster.

7 Performance
The following performance numbers are intended to be used as a rough estimate of the performance of a
Select cluster and are not a performance guarantee. The performance of an ONTAP Select cluster can
vary considerably due to the characteristics of the underlying hardware and configuration. The following
numbers should be used solely as a guide.

54 TR-4517 Technical Report

4-node with Direct Attached Storage

Reference Platform

ONTAP Select 9.0 (Standard) hardware (per node):

• Dell R530
 8-core 2.4GHz Haswell
 24GB RAM
 ESX 5.5u3
• 1 MD1420 Dell drive enclosure
 23 600GB 10K RPM SAS drives (22 in use, 1 hot spare)
 PERC H830 RAID controller
 2GB NV cache

ONTAP Select 9.1 (Premium) hardware (per node):

• CISCO C240 UCS
 14-cores 2.6 GHz E5-2697
 128GB RAM
 ESX 5.6
 24 400GB SSD drives
 CISCO RAID controller
 2GB NV cache

Client hardware:
• 4 NFSv3 IBM 3650 clients

Config info:
• 1500 MTU for data path between clients and Select cluster
• No storage efficiency features in use (compression, dedupe, Snapshot copies, SnapMirror, and so on)

Results
Table 8 shows the throughput measured against read/write workloads on a four-node ONTAP Select
Standard and Premium clusters. The ONTAP Select Premium cluster used SSD media. Performance
measurements were taken using the SIO load-generating tool using the configuration defined earlier.
For each test scenario, further details are provided later.

Table 8) Performance results for a 4-node ONTAP Select Standard cluster and a 4-node ONTAP Select
Premium cluster.

Description Sequential Read Sequential Write Random Read Random Write

64 KiB 64KiB 4KiB 4KiB
549MBps 155MBps 19MBps 54MBps
ONTAP Select Standard
(Version 9.0)
8,784 IOPs 2,480 IOPs 4,864 IOPs 13,824 IOPs
SAS disks

55 TR-4517 Technical Report

 Description Sequential Read Sequential Write Random Read Random Write
64 KiB 64KiB 4KiB 4KiB
1151MBps 233MBps 158MBps 89MBps
ONTAP Select Premium
(Version 9.1)
18,416 IOPs 3,728 IOPs 40,448 IOPs 22,784 IOPs
SSD disks

4-Core Standard vs 8-Core Premium

1400

1200 +110%
Throughput per Node (MB/s)

1000

800

600

400
+50%
200 +749%
+64%

0
Seq Wr Rnd Wr Seq Rd Rnd Rd
4-Core 16GB 8-Core 64GB

Sequential Read
Details:
• SIO direct I/O enabled
• 1 data NIC
• 1 data aggregate (1TB)
 64 volumes, 64 SIO procs/threads
 32 volumes per node (64 total)
 1 SIO proc per volume, 1 SIO thread per file
 1 file per volume; files 12GB each
 Files pre-created using mkfile

56 TR-4517 Technical Report

Using 100% sequential 64KiB I/Os, each thread reads through each file sequentially from beginning to end.
Each measurement lasts for 300 seconds. Tests are purposefully sized so that the I/O never wraps within
a given file. Performance measurements are designed to force I/O from disk.

Sequential Write
Details:
• SIO direct I/O enabled
• 1 data NIC
• 1 data aggregate (1TB)
 64 volumes, 128 SIO procs/threads
 32 volumes per node (64 total)
 2 SIO procs per volume, 1 SIO thread per file
 2 files per volume; files are 30720MB each
Using 100% sequential 64KiB I/Os, each thread writes through each file sequentially from beginning to end.
Each measurement lasts for 300 seconds. Tests are purposefully sized so that the I/O never wraps within
a given file. Performance measurements are designed to force I/O to disk.

Random Read
Details:
• SIO direct I/O enabled
• 1 data NIC
• 1 data aggregate (1TB)
 64 volumes, 64 SIO procs, 512 threads
 32 volumes per node (64 total)
 64 SIO procs per volume, each with 8 threads
 1 SIO proc per volume, 8 threads per file
 1 file per volume; files are 8192MB each
 Files precreated using mkfile
Using 100% random 4KiB I/Os, each thread randomly reads through each file. Each measurement lasts for
300 seconds. Performance measurements are designed to force I/O from disk.

Random Write
Details:
• SIO direct I/O enabled
• 1 data NIC
• 1 data aggregate (1TB)
 64 volumes, 128 SIO procs, 512 threads
 32 volumes per node (64 total)
 64 SIO procs, each with 8 threads
 1 SIO proc per volume, 8 threads per file
 1 file per volume; files are 8192MB each
Using 100% random 4KiB I/Os, each thread randomly writes through each file. Each measurement lasts
for 300 seconds. Performance measurements are designed to force I/O to disk.

57 TR-4517 Technical Report

7.2 Single-node with VSAN Storage

Reference Platform

ONTAP Select 9.2 (Standard) hardware (per node / 4 node AF VSAN cluster):
• Dell R630
 Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz.
 2 Sockets, 14 CPUs per socket.
 56 logical CPUs (HT enabled).
 256 GB RAM.
 ESXi version: VMware ESXi 6.0.0 build-3620759.
• VSAN data-store
 Drives per host.
 INTEL SSDSC2BX40 - 372 GB for Cache tier.
 4 INTEL SSDSC2BX01 – 1.46 TB for Capacity tier.

Client hardware:
- 1 NFSv3 - Debian Linux VM deployed on the same vSAN cluster.
- 80 GB workload distributed equally across 4 NFS volumes / mounts.
- No storage efficiency features in use.
- Separate 10Gig networks for NFS data traffic and vSAN internal traffic.
- 1500 MTU for NFS interfaces and 9000 MTU for VSAN interface.
- Block size: random work-load – 4k and sequential work-load – 64k.

Results
Table 9 shows the throughput measured against the read/write workloads on a single-node Select Standard
cluster running on an All Flash VSAN datastore. Performance measurements were taken using the FIO
load-generating tool.

Table 9) Performance results for a single-node ONTAP Select Standard cluster on an AF VSAN datastore.

Description Sequential Read Sequential Write Random Read Random Write

64 KiB 64KiB 4KiB 4KiB
527MBps 63MBps 129MBps 34MBps
ONTAP Select Standard
(version 9.2)
8,427 IOPs 1,005 IOPs 32,899 IOPs 8,626 IOPs
All Flash VSAN

58 TR-4517 Technical Report

ONTAP Select using an AF VSAN datastore vs. ONTAP Select using DAS datastore

1400
ONTAP Select 9.0 Standard with DAS
1200 1151 (SAS)
Throughput per Node (MB/s)

ONTAP Select 9.1 Premium with DAS

1000 (SSD)
ONTAP Select 9.2 Standard with AF
VSAN
800

600 549 527

400
233
200 155 158 129
63
19 54 89 34
0
Seq Read Seq Write Random Read Random Write

Version History
Version Date Document Version History
Version 1.0 June 15, 2016 Initial version

Version 1.1 August 15, 2016 Updated the networking sections 2.5 and 5

Version 1.2 December 22, 2016 Added support for ONTAP Select 9.1 and OVF evaluation
method.
Consolidated the Networking Section.
Consolidated the Deploy Section.

Version 1.3 March 20, 2017 Added support for ONTAP Deploy 2.3, external array and VSAN.
Added support for SATA and NL-SAS along with datastore size
considerations for larger capacity media.
Added IOPS metric to performance table.
Added network checker for Internal Network troubleshooting.

Version 1.41 June, 2017 Added support for ONTAP Deploy 2.4, ONTAP Select 9.2, and 2-
node clusters.
Added VSAN performance information.

59 TR-4517 Technical Report

Refer to the ONTAP Select Release Notes and the Interoperability Matrix Tool (IMT) on the
NetApp Support site to validate that the exact product and feature versions described in this
document are supported for your specific environment. The NetApp IMT defines the product
components and versions that can be used to construct configurations that are supported by
NetApp. Specific results depend on each customer's installation in accordance with published
specifications.
Copyright Information
Copyright © 1994–2016 NetApp, Inc. All rights reserved. Printed in the U.S. No part of this
document covered by copyright may be reproduced in any form or by any means—graphic,
electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic
retrieval system—without prior written permission of the copyright owner.
Software derived from copyrighted NetApp material is subject to the following license and
disclaimer:
THIS SOFTWARE IS PROVIDED BY NETAPP "AS IS" AND WITHOUT ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
NetApp reserves the right to change any products described herein at any time, and without
notice. NetApp assumes no responsibility or liability arising from the use of products described
herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product
does not convey a license under any patent rights, trademark rights, or any other intellectual
property rights of NetApp.
The product described in this manual may be protected by one or more U.S. patents, foreign
patents, or pending applications.
RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject
Trademark Information
to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and
NetApp, the NetApp logo, Go Further, Faster, AltaVault, ASUP, AutoSupport, Campaign Express, Cloud
Computer Software
ONTAP, Clustered clause
Data at DFARS
ONTAP, 252.277-7103
Customer Fitness, Data(October 1988) and FAR
ONTAP, DataMotion, 52-227-19
Fitness, (June
Flash Accel,
1987).
Flash Cache, Flash Pool, FlashRay, FlexArray, FlexCache, FlexClone, FlexPod, FlexScale, FlexShare,
FlexVol, FPolicy, GetSuccessful, LockVault, Manage ONTAP, Mars, MetroCluster, MultiStore, NetApp
Insight, OnCommand, ONTAP, ONTAPI, RAID DP, RAID-TEC, SANtricity, SecureShare, Simplicity,
Simulate ONTAP, SnapCenter, Snap Creator, SnapCopy, SnapDrive, SnapIntegrator, SnapLock,
SnapManager, SnapMirror, SnapMover, SnapProtect, SnapRestore, Snapshot, SnapValidator,
SnapVault, SolidFire, StorageGRID, Tech OnTap, Unbound Cloud, WAFL, and other names are
trademarks or registered trademarks of NetApp Inc., in the United States and/or other countries. All other
brands or products are trademarks or registered trademarks of their respective holders and should be
treated as such. A current list of NetApp trademarks is available on the web at
http://www.netapp.com/us/legal/netapptmlist.aspx.

60 TR-4517 Technical Report