Saturday, September 1, 2012

HA Storage Cost Comparison : NetApp MetroCluster vs DRBD

One of the cost saving techniques being used by Google is that they used commodity SATA drives in their GFS Clusters, as opposed to a specialized storage cluster. That makes me wonder just how much saving that could be realized by using SATA drives.
For our comparison case, lets build a highly availably MySQL server. This MySQL is not clustered, but will be failover onto the second host upon first host's failure. Our core solution primarily depends on two servers, one server runs mysqld and the other will run mysqld upon failure. Both mysqld is configured to save data to the same storage, so there only can be one mysqld server allowed to run at a time. The storage solution being used is NetApp storage solution described in http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001783, that will supply storage redundancy and availability zone redundancy. NetApp's MetroCluster will do synchronous mirroring between two separate NetApp disk shelves. For comparison we will use DRBD synchronous storage replication on commodity SATA drives (http://dev.mysql.com/doc/refman/5.5/en/ha-drbd.html) with similar effect. Heartbeat open source cluster software will be our means to failover in our DRBD solution.

Requirement 

High Available MySQL server with 5-6 TB storage.
Modes of failure that need to be supported :

HA Scenario :

  • Host Failure -> Hardware problem on one of the server
  • Storage Failure -> Hardware problem on one of database storage  drives
  • Availability zone Failure -> Entire zone disabled by some problem, such as total network loss or total power disruption. We assume we have two availability zone, maybe one zone in one building and the other is in the building across the street.

Note: Network failure not discussed here because network redundancy is a blog topic for another day :), And disaster recovery not being a requirement because cannot be fulfilled using MetroCluster nor DRBD. MySQL master slave replication is a better solution for disaster recovery.

Expected failover time is in one minute or less. In practice this will be achievable depending on InnoDB Log size configured in the MySQL server, the larger the log size will result longer failover time.

Solution Design

Virtual Machine or physical servers could be used interchangeably. Performance implications of the choice of physical or virtual server might be significant but lets assume its already decided by IT organization's policy. VM servers notably easier to configure onto HA because we could configure Fault Tolerant feature into VMs, but in the open source solution we deliberately choose not to because Heartbeat cluster software installed into two machine will provide fault tolerance onto our solution.
This solution requires two servers. If we require to support availability zone failure, then the server will be deployed across two availability zone, with the consequence of some performance impact.  If we are not required to support such failure, then deploying the two servers in the same room connected to the same switch will give us better performance.

A. NetApp solution

NetApp disk shelves with one controller and configured in a MetroCluster is enough to provide storage failure and site failure protection. It still has local SPOF on the controller but lets ignore it for now because NetApp stuffs are pricey :). MetroCluster configuration ensure that each disk is mirrored in the other availability zone. 
Host failure will prompt VMWare to restart the VM, and if not possible, start the MySQL VM in the other host. Storage mirroring and storage failover in event of failure will be taken care of by NetApp MetroCluster. Availability zone failure requires manual intervention to do a forced takeover in the surviving VM site (see http://communities.vmware.com/message/1394567)
Product required :
Two disk shelf with 5.4 TB total capacity, Est price @$32,002.00 (ref : http://storagemojo.com/storagemojos-pricing-guide/netapp-price-guide/)
(DSK SHLF,12x450GB,10K,6Gb SAS,IOM6,-C,R5).
Two Brocade 200E Distance Bundle @ $750
Two FAS 3170 FAS3170A-CL-BASE-R5 @ $51,224
Note: For the Brocade I don't know which one to buy so I choose the lower price one.
Total  : $167,952,00 (Not included : VMWare licenses and VM server nodes)

Storage specific price: Price per redundant GB is 2x$32002 / 5400 = $11.8/GB

B. DRBD solution

Our DRBD solution uses commodity SATA drives in place of NetApp disk shelves. Server's price (A HP Proliant DL 180) will be counted with storage price because 4 drive bay limitation for the server. If we need more storage we need to buy another couple of servers and share the additional storage via NFS or iSCSI (and also mirror them via DRBD).
Product required : (ref: HP Proliant 100 product line)
2x HP Proliant DL 180 G6  with 4 GB memory and Xeon E5606 quad core: $1,824.00
8 x  HP 2TB 3G SATA 7.2K Hot Plug 3.5 MDL Hard Drive - 1-year warranty  @ $ 509.00
(4 drive in each server)
2 TB configured as root filesystem
6 TB configured as mysql data, giving 6TB mirrored storage
Total : $18,664.00
Upper bound Storage specific cost, price per redundant GB :  $18664 / 6000 = $3.11

The DRBD solution typically failover in under 1 minutes, this is mostly the time taken to start MySQL because the Heartbeat system have < 1 second response. (see note above regarding InnoDB Log size impact). When Heartbeat detects that filesystem failure occured on the primary disk, it will initate failover. This also occurs when it detects primary host's no longer has a heartbeat, that could be caused by host failure or site failure. Heartbeat will start the MySQL server in the secondary node and could also take over the primary's IP if necessary.
HA MySQL configuration with DRBD



Short Summary

Comparing upper bound storage cost of our DRBD with SATA solution with NetApp solution (disk shelf only), we have comparison of $3.11 / Gb versus $11.8 / GB. Tremendous cost savings indeed when choosing SATA, even if we take the whole system cost of DRBD solution into account.
Comparing total system cost for similar capacity (6 TB vs 5.4 TB redundant storage), we have $18,664.00 vs $167,952.00 .. a difference of almost 1 : 9 between NetApp solution and our DRBD solution. This primarily caused by costs of synchronization support devices.

What does the large amount price difference buy us for the NetApp? 12 disk performance vs 4 disk performance for one. But that many disk could give us no performance gain if the condition not right (such as when striping and data spread not good). Vendor support for another. We could buy linbit's Support for DRBD solutions but I haven't found a price list yet. But from the point of view of High Availability features seems that the DRBD solution is adequate by giving us failover time under 1 minute.

Other Scenarios

This configuration could be modified to support disaster recovery scenario by adding another Mysql server, with the same redundant storage, located in remote location. These two Mysql server is configured as slave to the original server, and by giving this server the same capability as the first, it will be ready to perform with full performance and availability when the need arise.

More IO performance could be given by configuring the DRBD solution with smaller capacity (but more performant) disks and more servers, sharing storage using iSCSI or NFS (maybe following HA NFS configuration in my previous post). Thus we trade cost with performance, but keeping the system cost below NetApp's.

Another idea for next article is for implementing HA on Oracle database server .. while I look for solutions I found these reference docs :  http://eval.symantec.com/mktginfo/enterprise/white_papers/b-ha_for_oracle_db_vcs_hadr_WP_14216725.en-us.pdf and http://www.vmware.com/files/pdf/partners/oracle/Oracle_Databases_on_VMware_-_High_Availability_Guidelines.pdf. Seems Symantec got two product for lower cost oracle HA : Symantec ApplicationHA and Veritas Cluster Server.

2 comments:

Greg Eckert said...
This comment has been removed by the author.
Greg Eckert said...

LINBIT support pricing is on their website, here: http://www.linbit.com/en/products-and-services/drbd-support/pricing/pricing-usd. Even with Enterprise Support, your cost is still well below proprietary storage. Installation is included in the support, so they will even set up DRBD with Pacemaker via SSH or VPN with/for you. Source- I work there. Thanks for the article Yudhi :) .