Thursday, 18 October 2012

Load Balancing Exchange 2010 Client Access Servers using an Hardware Load Balancer Solution (Part 1)


With Exchange 2010, Outlook MAPI clients use the Client Access Server (CAS) role in the middle tier as the RPC endpoint, which has resulted in this role being even more critical than in previous versions of the product. Because of this, all organizations (big and small) should consider making this role highly available by introducing multiple CAS servers in each Active Directory site as well as load balance the protocols and services provided by this role.
In this previous multipart article of mine I, among other things, explained how you load balance the RPC CA service using Windows NLB and HLB technology, but I did not go into the details on how you configure load balancing for protocols and services such as Outlook Web Access (OWA), Exchange ActiveSync (EAS), Exchange Control Panel (ECP), Offline Address Book (OAB), Post Office Protocol (POP), Internet Message Access Protocol (IMAP), Exchange Web Services (EWS), and AutoDiscover (AutoD).
In this multipart article, I will show you how you load-balance the different protocols and services on an Exchange 2010 CAS role using a redundant external hardware load Balancer (HLB) solution. By implementing a load balancer solution, you distribute client workload among multiple servers and thereby increase performance and decrease downtime by eliminating the single point of failure that exists in a topology with only one single CAS server or when you have multiple CAS servers where the internal URL for the miscellaneous services point to the server FQDN.

Why use a Hardware Load Balancing solution over Windows NLB?

With the architectural changes in Exchange 2010 that amongst other things, introduces the new RPC Client Access service (which moves Outlook MAPI mailbox connections from the back-end Mailbox servers in the data tier to the Client Access servers (CAS) in the middle tier) providing both a load balanced and highly available Client Access Server (CAS) solution is even more important than was the case with previous versions of Exchange.
Windows Network Load Balancing (WNLB) technology may be a fine choice for organizations that do not plan to deploy multi-role Exchange 2010 servers with both DAG protected mailbox databases and load balanced/highly available CAS clients and services. In addition, using WNLB can be the right fit for organizations that do not have:
  • More than 8 nodes in a WNLB based array (the Exchange Product group does not recommend more than 8 nodes in a WNLB based cluster due to scalability and functionality limitations).
  • Requirements for the LB solution to be application-aware (check state of application and not just check for IP connectivity like WNLB does).
  • The need for affinity methods other than source IP address based affinity which is the only method provided by WNLB (a HLB solution provides other affinity methods such as cookie and SSL ID based affinity).
However, if you plan to deploy multi-role Exchange 2010 servers with both DAG protected mailbox databases and load balanced/highly available CAS server service, you cannot use WNLB due to Windows Failover Cluster (WFC) and WNLB hardware sharing conflicts (see this KB article for more information). Also, depending on your environment and network topology, the persistence (affinity) settings provided by WNLB may not be sufficient. This may especially be true if you have clients that look like they are coming from the same source IP address etc.
When a hardware load balancer based CAS array has been properly configured, all servers in the array are represented by a single virtual IP (VIP) address and a fully qualified domain name (FQDN). When a client request comes in, it will be sent to an Exchange 2010 CAS server in the CAS array using DNS round robin distribution method. Of course we have options to prefer one or more CAS servers over other via features such as weighted round robin, least connection and so on.

But my organization cannot afford a hardware-based load balancer solution

This could definitely be true if you go with one of the big players on the market (such as F5 BIG-IP, Cisco ACE, Citrix NetScaler etc.), but you know what? A hardware based load balancer solution is not just an expensive luxury of LORGs (large organizations) with just as large IT budgets at their disposal. A hardware load balancer solution does not necessarily need to cost many thousands of dollars. You can actually get sophisticated, high performance devices at a very affordable price (you just need to find the right vendor). This means that even though you work for an organization with a limited IT budget, it does not mean they cannot afford to invest in a hardware load balancer solution.
Personally, I have recommended different hardware load balancer solutions from different vendors to my customers over the years, but for Exchange 2010, I really like the low cost devices from KEMP Technologies. Their smallest device (LoadMaster 2000) has a price tag of $1,590 dollars which even includes one year of support. This means that you can get a redundant hardware load balancer solution for approximately $3,000 dollars if you are a SMORG (small or medium organization)! On top of that, the LoadMaster 2000 device has the same rich feature set as the LoadMaster 2200 (this one gives you a lot more bang for the buck that the LM 2000 model although the price difference is very small!), 2500, 3500, and 5500 models (which are minded for the LORGS (large organizations). That is it has full support for premium features such as load balancing using layer 4 and 7, automatic failover cluster (active/hot standby with failover time of less than 3 seconds in my test environment), SSL offloading, layer 7 persistence (stickiness), up to 256 virtual services (with a total of up to 1000 real servers!) and server/application health checking etc. These are features you typically only see listed when looking at expensive load balancer devices from the aforementioned more well-known vendors on the market.
By the way, if you are on the virtualization bandwagon (who isn’t?), KEMP Technologies also has a virtual appliance with a feature set identical to the hardware based devices.
Note:LORGs with lots of users or SMORGs that will use the HLB solution for purposes other than Exchange may need to use one of the larger KEMP models. To help you decide, check out the product matrix here.
Because I have very good experience with the devices from KEMP Technologies and because they are affordable even for the SMORGs that typically are planning to deploy a fully redundant Exchange solution consisting of two multi-role Exchange 2010 servers, I have used two LoadMaster 2000 devices configured in a cluster (one active and one hot standby) as the basis for this article. The setup is illustrated in Figure 1 below.

Figure 1: Topology used in this lab environment
Note:It is important to stress that I am in no way affiliated with KEMP Technologies. In addition, I am not being paid to point readers at hardware load balancer devices provided by this company. I simply do so as I have good experience with their devices.

What about reverse proxies such as TMG/ISA/IAG/UAG?

Can’t I use one of these solutions to load balance the miscellaneous protocols and services on a CAS server? You definitely could! At least you can load balance everything that’s HTTP or HTTPS protocol. However, none of these products are capable of load balancing RPC traffic. Read more in this newsletter I wrote a few months ago. In addition, you may not want to send traffic from internal clients to the reverse proxy solution in your perimeter network and back again.

Finally if you load balance HTTP/HTTPS traffic using one of the above mentioned solutions as well as an internal HLB solution, it’s also important to mention that you shouldn’t point them at the VIP/FQDN of the HLB, but instead have the reverse proxy itself distribute the traffic across the CAS servers in the CAS array.

What Persistence (affinity) type should I use?

Persistence (aka affinity, stickiness etc.) is the ability of a load balancer to maintain a connection between a client and a server. Persistence can make sure that all requests from a client are sent to the same server in a NLB array or server farm (in case of Exchange CAS array).
So depending on the Exchange client or service, there are different recommendations in regards to what persistence settings to use. Below I highlight which are the preferred ones for each client and service.
Exchange Clients:
  • Outlook Web App (OWA) - For OWA the recommended persistence methods are Client IP (source IP address) or Cookie (either existing cookie or one created by hardware load balancer aka LB-cookie). Both methods works fine in most deployments, but if you’re working with environments where client’s looks like them come from the same source IP address, you should avoid using Client IP and instead go with one of the cookie based persistence methods. It is recommend to not use SSL ID based persistence with OWA as this can result in users required to re-authenticate because browsers like Internet Explorer 8 create new separate worker processes when for instance creating a new message in OWA. The issue here is that with each new worker process a new SSL ID is used.

  • Exchange Control Panel (ECP) - Same recommendation as above.

  • Exchange ActiveSync (EAS) - For Exchange ActiveSync the recommended persistence methods are Client IP (source IP address) or Authorization header. If your organization uses the same mobile provider/cellular carrier network for all users that connect to Exchange using EAS, then chances are they appear to come from the same source IP address as NAT are often used in a cellular carrier network. This means that you may not see optimal distribution of EAS traffic among the CAS servers behind the NLB array. So for EAS it’s often a good idea to use the Authorization HTTP header as a key for persistence. Again, it is not recommended to use SSL ID based persistence for EAS as some mobile devices renegotiates SSL security parameters on a frequent basis.

  • Outlook Anywhere (OA) - For Outlook Anywhere (aka RPC over HTTP), the recommended persistence methods are Client IP (source IP address), Authorization header or “OutlookSession” cookie based persistence. If OA clients appear to come from the same Client IP, then you should consider using Authorization header or “OutlookSession” cookie persistence. Bear in mind though that “OutlookSession” persistence only is supported by Outlook 2010.

  • IMAP and POP3 - IMAP and POP3 do not require any special persistence settings, so the recommendation is to set it to no persistence.
Exchange Services:
  • Autodiscover- The Autodiscover service doesn’t require any special persistence settings, so the recommendation is to set it to no persistence.

  • RPC Client Access Service (RPC CA)- For the RPC CA service used as endpoint for internal Outlook clients, the recommended persistence method is Client IP.

  • Exchange Address Book Service- Same recommendation as for RPC CA service.

  • Exchange Web Services (EWS)- For EWS the recommended persistence methods are cookie or SSL ID.
Now since many of the above clients and services use the same port, you can often only specify one persistence method for all clients and services that use the same port/IP address. If you want to use a different persistence method for let’s say OWA and OA, depending on your HLB solution, this may be possible (by using split-persistence etc.) but is outside the scope of this multipart article. Instead, I suggest you contact the vendor of the HLB solution you plan on using.

Timeout Settings for each Protocol and Service


For each virtual service you can set time out values for the sessions that are established from the miscellaneous clients to the HLB solution (memory, CPU etc.).
In order to make optimal use of your HLB solution you should not set these timeout values to high, but also be careful not to set them too low as this could result in clients needing to reestablish as session which may or may not mean the end user will be informed to re-authenticate.
Needless to say you would want to set timeout values for protocols and services such as OWA, ECP, EAS, Outlook Anywhere, and RPC CA relatively high (several hours such as hours in a workday) while IMAP, POP, AutoD, EWS, OAB should have low values set (typically few minutes). To be on the safe side contact the vendor of the HLB solution for details on what makes most sense with their solution.

Okay we have reached the end of part 1. But what we covered so far should make you well prepared for part 2 where we dive into how the virtual services for each protocol and service is created in a LoadMaster based HLB solution. If you have questions in regards to what has been covered so far, let me know.
If you would like to read the other parts in this article series please go to: