HA Deepdive

My posts on VMware High Availability(HA) has historically been my best read post. I decided to rewrite the posts and create a page which is easier to maintain when functionality changes and a lot easier to find via Google and or the menu. This section is always under construction so check it every once in a while! Everybody probably knows the basics of VMware HA so Im not going to explain how to set it up or that is uses a heartbeat for monitoring outages or isolation. if you want to know more about HA and DRS I recommend reading our HA and DRS technical deepdive book. Keep in mind that a lot of this info is derived from the availability guide, VMworld presentations and by diving into HA in my lab. Ive divided the article into several sections and each section will contain a basic design principle:

Node Types Node role Isolation response

o

Isolation response gotchas

Admission control Host failures

o o

Whats a slot? How does HA calculate how many slots are available per host?

Percentage of cluster resources reserved Specify a failover host My Admission Control Policy Recommendation

Flattening shares Advanced settings

Node Types
A VMware HA Cluster consists of nodes, primary and secondary nodes. Primary nodes hold cluster settings and all node states which are synchronized between primaries. Node states hold for instance resource usage information. In case that vCenter is not available the primary nodes will have a rough estimate of the resource occupation and can take this into account when a fail-over needs to occur. Secondary nodes send their state info to the primary nodes. Nodes send a heartbeat to each other, which is the mechanism to detect possible outages. Primary nodes send heartbeats to primary nodes and secondary nodes. Secondary nodes send their heartbeats to primary nodes only. Nodes send out these heartbeats every second by default. However this is a changeable value: [Link]. (Advanced Settings on your HA-Cluster) The first 5 hosts that join the VMware HA cluster are automatically selected as primary nodes. All the others are automatically selected as secondary nodes. When you do a reconfigure for HA the primary nodes and secondary nodes are selected again, this is at random. The vCenter client does not show which host is a primary and which is not. This however can be revealed from the Service Console: cat /var/log/vmware/aam/aam_config_util_listnodes.log Another method of showing the primary nodes (including the Master Primary aka failover coordinator) is is shown in the following screenshot where /opt/vmware/aam/bin/Cli is used:

Of course this is also possible with the latest version of PowerCLI: Get-Cluster clustername | Get-HAPrimaryVMHost As of vSphere 4.1 a third option has been added. It must be noted that this option will only show results when there is an error. If an error occurs you can easily check what the issue is by going to your cluster and clicking the Cluster Operational Issues line on the Summary tab. If there are no issues the screen will be completely gray. I forced an issue though so you can see what is shown.

Now that youve seen that it is possible that you can list all node wi th the CLI you probably wonder what else is possible Lets start with a warning, this is not supported. Also keep in mind that the supported limit of primaries is 5, I repeat 5. This is a soft limit, so you can manually add a 6th, but this is not supported. Now heres the magic Using Cli you can also promote nodes from secondary to primary and vice versa. This is shown in the following screenshots: To promote a node:

To demote a node:

I cant say this enough, it is unsupported but it does work. With vSphere 4.1 a new advanced setting has been introduced. This setting is not even experimental, it is also unsupported. I dont recommend anyone using this in a production environment, if you do want to play around with it use your test environment. Here it is: [Link] = hostname1 hostname2 hostname3 or [Link] = [Link],[Link],[Link] The list of hosts that are preferred as primary can either be space or comma separated. You dont need to specify 5 hosts, you can specify any number of hosts. If you specify 5 and all 5 are available they will be the primary nodes in your cluster. If you specify more than 5, the first 5 of your list will become primary. When primaries are selected by using promoteNode or by pow ering up hosts in the right order you will need to verify occasionally if your hosts are still primary or not as HA has a re-election mechanism. Election time! Now when does a re-election occur? It is a common misconception that a re-election occurs when a primary node fails. This is not the case. The promotion of a secondary host only occurs

when a primary host is either put in Maintenance Mode, disconnected from the cluster, removed from the cluster or when you do a reconfigure for HA. If all primary hosts fail simultaneously no HA initiated restart of the VMs will take place. HA needs at least one primary host to restart VMs. This is why you can only take four host failures in account when configuring the host failures HA admission control policy. (Remember 5 primaries) However, when you select the Percentage admission control policy you can set it to 50% even when you have 32 hosts in a cluster. That means that the amount of failover capacity being reserved equals 16 hosts. Although this is fully supported but there is a caveat of course. The amount of primary nodes is still limited to five. Even if you have the ability to reserve over 5 hosts as spare capacity that does not guarantee a restart. If, for what ever reason, half of your 32 hosts cluster fails and those 5 primaries happen to be part of the failed hosts your VMs will not restart. (One of the primary nodes coordinates the fail-over!) Although the percentage option enables you to save additional spare capacity theres always the chance all primaries fail.

Basic design principle: In blade environments, divide hosts over all blade chassis and never exceed four hosts per chassis to avoid having all primary nodes in a single chassis.

Node Role
You will need at least one primary because the fail-over coordinator role will be assigned to this primary, this role is also described as active primary or master primary. I will use fail over coordinator for now, but if you use the HA Cli is is listed as the Master Primary. The fail over coordinator coordinates the restart of VMs on the remaining primary and secondary hosts. The coordinator takes restart priorities in account. Keep in mind, when two hosts fail at the same time it will handle the restart sequentially. In other words, restart the VMs of the first failed host (taking restart priorities in account) and then restart the VMs of the host that failed as second (again taking restart priorities in account). If the fail-over coordinator fails one of the other primaries will take over.

Isolation Response
Talking about HA initiated fail-overs; one of the settings everyone has looked into is the isolation response. The isolation response refers to the action that HA takes when the heartbeat network is isolated. Today there are three isolation responses, power off, leave powered on and shut down. Up to ESX 3.5 U2 / vCenter 2.5U2 the default isolation response when creating a new cluster was Power off. As of ESX 3.5 U3 / vCenter 2.5 U3 the default isolation response is leave powered on. For vSphere ESX / vCenter 4.0 this has been changed to Shut down. Keep this in mind when installing a new environment, you might want to change the default depending on customer requirements.

Power off When a network isolation occurs all VMs are powered off. It is a hard stop. Shut down When a network isolation occurs all VMs running on that host are shut down via

VMware Tools. If this is not successful within 5 minutes a power off will be executed.
Leave powered on When a network isolation occurs on the host the state of the VMs remains

unchanged.
The question remains, which setting should I use? It depends. I personally prefer Shut down because I do not want to use a deprecated host and it will shut down your VMs in a clean manner. Many people prefer to use Leave powered on because it reduces the chances of a false positive. A false positive in this case is an isolated heartbeat network but a non-isolated VM network and a non-isolated iSCSI / NFS network. I guess most of you would like to know how HA knows if the host is isolated or completely unavailable when you have selected leave powered on. HA actually does not know the difference. HA will try to restart the affected VMs in both cases. When the host has failed a restart will take place, but if a host is merely isolated the nonisolated hosts will not be able to restart the affected VMs. This is because of the VMDK file lock; no other host will be able to boot a VM when the files are locked. When a host fails this lock starves and a restart can occur. The amount of retries is configurable as of vCenter 2.5 U4 with the advanced option [Link]. The default value is 5. Pre vCenter 2.5 U4 HA would keep retrying forever which could lead to serious problems as described in the KB article. The isolation response is a setting that needs to be taken into account when you create your design. For instance when using an iSCSI array or NFS choosing leave powered on as your default isolation response might lead to a split-brain situation depending on the version of ESX used. The reason for this being that the disk lock times out if the iSCSI network is also unavailable. In this case the VM is being restarted on a different host while it is not being powered off on the original host. In a normal situation this should not lead to problems as the VM is restarted and the host on which it runs owns the lock on the VMDK, but for some weird reason when disaster strikes you will not end up in a normal situation but you might end up in an exceptional situation. Do you want to take that risk? As of vSphere 4 Update 2 a new mechanism has been introduced which will recover VMs that are in a split brain situation. First let me explain what a split brain scenario is, lets start with describing the situation which is most commonly encountered: 4 Hosts iSCSI / NFS based storage Isolation response: leave powered on

When one of the hosts is completely isolated, including the Storage Network, the following will happen: 1. Host ESX001 is completely isolated including the storage network(remember iSCSI/NFS based storage!) but the VMs will not be powered off because the isolation response is set to leave powered on. 2. 3. 4. After 15 seconds the remaining, non isolated, hosts will try to restart the VMs. Because of the fact that the iSCSI/NFS network is also isolated the lock on the VMDK will time out and the remaining hosts will be able to boot up the VMs. When ESX001 returns from isolation it will still have the VMX Processes running in memory and this is when you will see a ping-pong effect within vCenter, in other words VMs flipping back and forth between ESX001 and any of the other hosts. As of version 4.0 Update 2 ESX(i) detects that the lock on the VMDK has been lost and issues a question which is automatically answered. The VM will be powered off to recover from the splitbrain scenario and to avoid the ping-pong effect. The following screenshot shows the event that HA will generate for this auto-answer mechanism which is viewable within vCenter.

Basic design principle 1: Isolation response should be chosen based on the version of ESX used. For pre-vSphere 4 Update 2 environment with iSCSI/NFS Storage I recommend to set the isolation response to Power off to avoid a possible split brain scenario. I also recommend to have a secondary service console running on the same vSwitch as the iSCSI network to detect an iSCSI outage and avoid false positives. Basic design principle 2: Base your isolation response on your SLA. If your SLA dictates that hosts with degraded hardware should not be used, make sure to select shutdown or power off.

Isolation response gotchas

I thought this issue was something that was common knowledge but a recent blog article by Mike Laverick proved me wrong. I think I can safely assume that if Mike doesnt know this its not common knowledge.

The default value for isolation/failure detection is 15 seconds. In other words the failed or isolated host will be declared dead by the other hosts in the HA cluster on the fifteenth second and a restart will be initiated by one of the primary hosts. For now lets assume the isolation response is power off. The power off(isolation response) will be initiated by the isolated host 1 second before the [Link]. A power off will be initiated on the fourteenth second and a restart will be initiated on the fifteenth second. Does this mean that you can end up with your VMs being down and HA not restarting them? Yes, when the heartbeat returns between the 14th and 15th second the power off could already have been initiated. The restart however will not be initiated because the heartbeat indicates that the host is not isolated anymore. How can you avoid this? Pick Leave VM powered on as an isolation response. Increasing the [Link] will also decrease the chances of running in to issues like these.

Basic design principle: Increase [Link] to 30 seconds (30000) to decrease the likely-hood of a false positive.

Admission control
This has always been a hot topic, HA and Slot sizes/Admission Control. One of the most extensive (Non-VMware) articles is by Chad Sakac aka Virtual Geek, but of course since then a couple of things has changed. Chad commented on this article and asked if I could address this topic, here you go Chad. Lets start with the basics. Whats HA admission control about? Why is it there? The Availability Guide states the following: vCenter Server uses admission control to ensure that sufficient resources are available in a cluster to provide failover protection and to ensure that virtual machine resource reservations are respected. Admission Control ensures available capacity for HA initiated fail-overs by reserving capacity. Keep in mind that admission control calculates the capacity required for a fail-over based on available resources. In other words if a host is placed into maintenance mode, or disconnected for that matter, it is taken out of the equation. The same goes for DPM, if if admission control is set to strict DPM in no way will violate availability constraints. To calculate available resources and needed resources for a fail-over HA uses different concepts based on the chosen admission control policy. Currently there are three admission control policies: 1. 2. 3. Host failures cluster tolerates Percentage of cluster resources reserved as failover spare capacity Specify a failover host As stated each of these uses a different mechanism for reserving resources for a failover. Host failures uses a mechanism called slots. Slots dictate how many VMs can be started up before vCenter starts yelling Out Of Resources!! Normally each slot represents one VM.

Percentage based admission control uses a more flexible mechanism. It accumulates all reservations and subtracts it from the total amount of available resources while making sure the specified spare capacity is always available. A failover host doesnt use any of those mechanisms, this host is dedicated for failover. It will not be used. All three policies and concept will be explained in-depth below.

Host failures
Now what happens if you set the number of allowed host failures to 1? The host with the most slots will be taken out of the equation. (Slots are explained in more detail below) If you have 8 hosts with 90 slots in total but 7 hosts each have 10 slots and one host 20 this single host will not be taken into account. Worst case scenario! In other words the 7 hosts should be able to provide enough resources for the cluster when a failure of the 20 slot host occurs. And of course if you set it to 2 the next host that will be taken out of the equation is the host with the second most slots and so on. One thing worth mentioning, as Chad stated with vCenter 2.5 the number of vCPUs for any given VM was also taken in to account. This led to a very conservative and restrictive admission control. This behavior has been modified with vCenter 2.5 U2, the amount of vCPUs is not taken into account.

Basic design principle: Think about maintenance mode. If a single host needs mai ntenance it will be taken out of the equation and this means you might not be able to boot up new VMs when admission control is set to strict.

What is a Slot?
A slot is a logical representation of the memory and CPU resources that satisfy the requirements for any powered-on virtual machine in the cluster. In other words a slot size is the worst case CPU and Memory reservation scenario in a cluster. This directly leads to the first gotcha: HA uses the highest CPU reservation of any given VM and the highest memory reservation of any given VM. If no reservations of higher than 256Mhz are set HA will use a default of 256Mhz for CPU and a default of 0MB+memory overhead for memory. If VM1 has 2GHZ and 1024GB reserved and VM2 has 1GHZ and 2048GB reserved the slot size for memory will be 2048MB+memory overhead and the slot size for CPU will be 2GHZ.

Basic design principle: Be really careful with reservations, if theres no need to have them on a per VM basis dont configure them.

How does HA calculate how many slots are available per host?
Of course we need to know what the slot size for memory and CPU is first. Then we divide the total available CPU resources of a host by the CPU slot size and the total available Memory Resources of a host by the memory slot size. This leaves us with a slot size for both memory and CPU. The most restrictive number is the amount of slots for this host. If you have 25 CPU slots but only 5 memory slots the amount of available slots for this host will be 5. As you can see this can lead to very conservative consolidation ratios. With vSphere this is something thats configurable. If you have just one VM with a really high reservation you can set the following advanced settings to lower the slot size being used during these calculations: [Link] or [Link]. The advanced setting [Link] and [Link] will allow you to specify an upper boundary for your slot size. When one of your VMs has an 8GB reservation this setting can be used to define for instance an upper boundary of 1GB to avoid resource wastage and an overly conservative slot size. However when for instance [Link] is configured to 2048MB and the lowest reservation is 500MB then the slotsize for memory will be 500MB+memory overhead. If a lower boundary needs to be specified the advanced setting [Link] or [Link] can be used. To avoid not being able to power on the VM with high reservations these VM will take up multiple slots. Keep in mind that pre-vSphere 4.1 when you were low on resources this could mean that you were not able to power-on this high reservation VM as resources would be fragmented throughout the cluster instead of located on a single host. As of vSphere 4.1 HA is closely integrated with DRS. When a failover occurs HA will first check if there are resources available on that host for the failover. If resources are not available HA will ask DRS to accommodate for these where possible. HA, as of 4.1, will be able to request a defragmentation of resources to accommodate for this VMs resource requirements. How cool is that?! One thing to note though is that HA will request it, but a guarantee can still not be given so you should be cautious when it comes to resource fragmentation. The following is an example of where resource fragmentation could lead to issues: If you need to use a high reservation for either CPU or Memory these options ([Link] or [Link]) could definitely be useful, there is however something that you need to know. Check this diagram and see if you spot the problem, the [Link] has been set to 1024MB.

Notice that the memory slot size has been set to 1024MB. VM24 has a 4GB reservation set. Because of this VM24 spans 4 slots. As you might have noticed none of the hosts has 4 slots left. Although in total there are enough slots available; they are fragmented and HA might not be able to actually boot VM24. Keep in mind that admission control does not take fragmentation of slots into account when slot sizes are manually defined with advanced settings. It does count 4 slots for VM24, but it will not verify the amount of available slots per host. As explained, as of vSphere 4.1 it will request defragmentation, but as stated it can not be guaranteed.

Basic design principle: Avoid using advanced settings to decrease slot size as it might lead to more down time.
Another issue that needs to be discussed is Unbalanced clusters. Unbalanced would for instance be a cluster with 5 hosts of which one contains substantially more memory than the others. What would happen to the total amount of slots in a cluster of the following specs: Five hosts, each host has 16GB of memory except for one host(esx5) which has recently been added and has 32GB of memory. One of the VMs in this cluster has 4CPUs and 4GB of memory, because there are no reservations set the memory overhead of 325MB is being used to calculate the memory slot sizes. (Its more restrictive than the CPU slot size.)

This results in 50 slots for esx01, esx02, esx03 and esx04. However, esx05 will have 100 slots available. Although this sounds great admission control rules the host out with the most slots as it takes the worst case scenario into account. In other words; end result: 200 slot cluster. With 5 hosts of 16GB, (5 x 50) (1 x 50), the result would have been exactly the same. (Please keep in mind that this is just an example, this also goes for a CPU unbalanced cluster when CPU is most restrictive!)

Basic design principle: Balance your clusters when using admission control and be conservative with reservations as it leads to decreased consolidation ratios.

Percentage of cluster resources reserved

Can I avoid large HA slot sizes due to reservations without resorting to advanced settings? Yes there is. The simplest way, without using advanced settings is selecting Percentage of cluster resources reserved as your admission control policy. With vSphere VMware introduced a percentage next to an amount of host failures. The percentage avoids the slot size issue as it does not use slots for admission control. So what does it use? When you select a specific percentage that percentage of the total amount of resources will stay unused for HA purposes. First of all VMware HA will add up all available resources to see how much it has available. Then VMware HA will calculate how much resources are currently consumed by adding up all reservations of both memory and CPU for powered on virtual machines. For those virtual machines that do not have a reservation larger than 256Mhz a

default of 256Mhz will be used for CPU and a default of 0MB+memory overhead will be used for Memory. (Amount of overhead per config type can be found on page 28 of theresource management guide.) In other words: ((total amount of available resources total reserved VM resources)/total amount of available resources) Where total reserved VM resources include the default reservation of 256Mhz and the memory overhead of the VM. Lets use a diagram to make it a bit more clear:

Total cluster resources are 24Ghz(CPU) and 96GB(MEM). This would lead to the following calculations: ((24Ghz-(2Gz+1Ghz+256Mhz+4Ghz))/24Ghz) = 69 % available ((96GB-(1,1GB+114MB+626MB+3,2GB)/96GB= 85 % available As you can see the amount of memory differs from the diagram. Even if a reservation has been set the amount of memory overhead is added to the reservation. For both metrics HA admission control will constantly check if the policy has been violated or not. When one of either two thresholds are reached, memory or CPU, admission control will disallow powering on any additional virtual machines. Please keep in mind that if you have an unbalanced cluster(host with different CPU or memory resources) your percentage is equal or preferably larger than the percentage of resources provided by the largest host. This way you ensure that all virtual machines residing on this host can be restarted in case of a host failure. Another thing to keep in mind is as there are no slots which HA uses resources might be fragmented throughout the cluster. As explained earlier, HA will request DRS to defragment resource to cater for that specific VM, but it is not a guarantee. I recommend making sure you have at least a host with enough available capacity to boot the largest VM (reservation CPU/MEM). Also make sure you select the highestrestart priority for this VM(of course depending on the SLA) to ensure it will be able to boot.)

I created a diagram which makes it more obvious I think. So you have 5 hosts, each with roughly 76% memory usage. A host fails and all VMs will need to failover. One of those VMs has a 4GB memory reservation, as you can imagine failing over this particular VM will be difficult due to the fact that none of the hosts has enough memory available to guarantee it. Although HA will request DRS to free up resources it is not guaranteed DRS can actually do this.

Basic design principle: Do the math, verify that a single host has enough resources to boot your largest VM. Also take restart priority into account for this/these VM(s).
Specify a failover host With the Specify a Failover Host admission control policy, when a host fails, HA will attempt to restart all virtual machines on the designated failover host. The designated failover host is essentially a hot standby. In other words DRS will not migrate VMs to this host when resources are scarce or the cluster is imbalanced.

My Admission Control Policy Recommendation

It depends. Yes I know, that is the obvious answer but it actually does. There are three options and each have its own advantages and disadvantages. Here you go: 1. Amount of host failures Pros: a. Fully automated, when a host is added to a cluster HA calculates how many slots are available.

b. Ensures fail-over by calculating slotsizes. Cons: c. Can be very conservative and inflexible when reservations are used as the largest reservation dictates slot sizes. d. Unbalanced clusters leads to waste of resources. 2. Percentage reserved Pros: a. Flexible as it considers actual reservation per VM. b. Cluster dynamically adjusts number of host failure capacity when resources are added. Cons: c. Manual calculations need to be done when adding additional hosts in a cluster and amount of host failures need to remain unchanged. d. Unbalanced clusters can be a problem when chosen percentage is too low. 3. Designated failover host Pros: a. What you see is what you get. b. High resource utilization as dedicated fail-over host is unused. Cons: c. What you see is what you get. d. Maximum of one fail-over host. e. Dedicated fail-over host not utilized during normal operations. 4. Basic design principle: Do the math, and take customer requirements into account. if

you need flexibility a Percentage is the way to go.

Flattening Shares
Prior to vSphere 4.1, an HA failed over virtual machine could be granted more resource shares then it should causing resource starvation until DRS balanced the load. As of vSphere 4.1 HA calculates normalized shares for a virtual machine when it is powered on after an isolation event! Pre-vSphere 4.1 an issue could arise when shares had been set custom on a virtual machine. When HA fails over a virtual machine it will power-on the virtual machine in the Root Resource Pool. However, the virtual machines shares were scaled for its appropriate place in the resource pool hierarchy, not for the Root Resource Pool. This could cause the virtual machine to receive either too many or too few resources relative to its entitlement. A scenario where and when this can occur would be the following: VM1 has a 1000 shares and Resource Pool A has 2000 shares. However Resource Pool A has 2 VMs and both will have 50% of those 2000 shares.

When the host would fail both VM2 and VM3 will end up on the same level as VM1. However as a custom shares value of 10000 was specified on both VM2 and VM3 they will completely blow away VM1 in times of contention. This is depicted in the following diagram:

This situation would persist until the next invocation of DRS would re-parent the virtual machine to its original Resource Pool. To address this issue as of vSphere 4.1 DRS will flatten the virtual machines shares and limits before fail -over. This flattening process ensures that the VM will get the resources it would have received if it would have been failed over to the correct Resource Pool. This scenario is depicted in the following diagram. Note that both VM2 and VM3 are placed under the Root Resource Pool with a shares value of 1000.

Of course when DRS is invoked both VM2 and VM3 will be re-parented under Resource Pool A and will receive the amount of shares they had originally assigned again. I hope this makes it a bit more clear what this flattened shares mechanism actually does.

Advanced Settings
VMware HA is probably the feature with the most advanced settings. Although many of them are rarely used some of them are needed in specific situations or included in best practices documents. The most used and valuable advanced settings are described below: [Link] Amount of milliseconds, timeout time, for isolation response action (with a default of 15000 milliseconds). Its a best practice to increase the value to 60000 when an active/standby Service Console setup is used. For a host with two Service Consoles and a secondary isolation address its a best practice to increase it to at least 20000. I would recommend to always increase it to at least 30000 [Link][x] IP address the ESX hosts uses to check on isolation when no heartbeats are received, where [x] = 110. (see screenshot below for an example) VMware HA will use the default gateway as an isolation address and the provided value as an additional checkpoint. I recommend to add an isolation address when a secondary service console is being used for redundancy purposes.

[Link] Value can be true or false and needs to be set in case the default gateway, which is the default isolation address, should not or cannot be used for this purpose. In other words, if the default gateway is a non-pingable address set the [Link] to a pingable address and disable the usage of the default gateway by setting this to false. [Link] Allows a NIC that is used for VMotion networks to be considered for VMware HA heartbeat usage. This permits a host (ESXi only) to have only one NIC configured for management and VMotion combined. [Link][x] Enables the use of port group names to control the networks used for VMware HA, where [x] = 0 ?. You can set the value to be Service Console 2 or Management Network to use (only) the networks associated with those port group names in the networking configuration. These networks need to be compatible for HA to configure successful. [Link] Configure the maximum amount of retries for a restart of a virtual machine. Select the value 0 for no restarts or -1 for indefinite. The default is 5 and has been chosen to avoid the issues described in KB 1009625. [Link] Disable the compatible network check for HA that was introduced with Update 2. Default value is false, setting it to true disables the check. This setting can be useful when nodes in a cluster are not in the same subnet. [Link] Remove the error icon/message from your vCenter when you dont have a redundant Service Console connection. Default value is false, setting it to true will disable the warning. [Link] - When multiple VMs are restarted on one host, up to 32 VMs will be powered on concurrently by default. This is to avoid resource contention on the host. This limit can be changed through the HA advanced option: [Link]. Setting a larger value will allow more VMs to be restarted concurrently and might reduce the overall VM recovery time, but the average latency to recover individual VMs might increase. We recommend using the default value. [Link] - The [Link] option controls the HA polling interval. HA polls the system periodically to update the cluster state with such information as how many VMs are powered on, and so on. The polling interval was 1 second in vSphere 4.0. A smaller value leads to faster VM power on, and a larger value leads to better scalability if a lot of concurrent power operations need to be performed in a large cluster. The default is 10 seconds in vSphere 4.1, and it can be set to a value between 1 and 30 seconds.

Basic design principle: Avoid using advanced settings as much as possible as it leads to increased complexity