Microsoft Network Load Balancing (NLB) on VMware ESX

Microsoft Network Load Balancing (NLB) on VMware ESX

The other day I went into a Microsoft Network Load Balancing issue. The customer had configured a Microsoft NLB cluster in Unicast mode with 4 nodes (VMs) in a ESX farm. One could say it works out of the box, just go for the default configuration and voila. Well not true in that particular ESX and physical network environments.
So let me explain the 2 main cast modes when deploying a NLB cluster:

1. UNICAST mode

In Unicast mode, NLB reassigns the station MAC (media access control) address of the network adapter for which it is enabled and all cluster hosts are assigned the same MAC address. Unicast mode induces switch flooding, where all switch ports are flooded with NLB traffic, even ports to which non-NLB servers are attached. Since all hosts in the cluster have the same IP Address and the same MAC Address, there is no inter-host communication possible between the hosts configured in Unicast mode therefore a second NIC needed for other host communication. UNICAST requires you to modify the vSwitches in an ugly way. For more info check this VMware KB: Sample Configuration – Network Load Balancing (NLB) UNICAST Mode Configuration

1. MULTICAST mode (prefered)

In multicast mode, NLB assigns a layer-2 multicast address to the cluster adapter instead of changing the adapter’s station address. Multicast allows inter-host communication because it adds a layer two multicast address to the cluster instead of changing it. This makes inter-host communication possible as the hosts retain their original unique MAC addresses and already have unique dedicated IP addresses. However, in multicast mode, the ARP reply sent out by a host in the cluster, in response to an ARP request, maps the clusters Unicast IP Address to its multicast MAC Address. Such a mapping in an ARP reply is rejected by some routers so administrators must add a static ARP entry in the router mapping the Cluster IP Address to its MAC Address. MULTICAST mode is by far the best one in my customer’s case, and in most scenarios as well IMO.
Here are the following steps to configure NLB in MULTICATS mode:

1. Install Microsoft NLB and set MULTICAST mode (more at VMware KB 1006558)
2. Disable DDNS/WINS. Network Load Balancing does not support dynamic Domain Name System (DNS) resolution, where the name of the cluster is automatically registered by the host when the host starts. This functionality must be disabled on the Network Load Balancing interface for both DNS and Windows Internet Name Service (WINS); otherwise, each host’s computer name will be registered with the cluster IP address. When using Network Load Balancing with DNS, you will need to directly configure the DNS server to register the name.
3. Add a static ARP entry in your default router (more at VMware KB 1006525)
4. Turn on MULTICAST support on your physical switches. If your switches do not support MULTICAST, you will have to setup Microsoft NLB in UNICAST mode.