ESX White Box

“The machinery that makes our blog work”

 

Our web infrastructure is built on a redundant topology which includes both onsite and offsite backups of a virtual infrastructure based on VMWare ESX and Nexentastor. We have virtualized our storage units so that they are portable between hosts in case one host fails it can be restarted on another host. This is a more advanced iteration of my original setup where I had the virtualized storage unit on the same host as running the VMs. We utilize ISCSI and flash backed up storage for speed purposes in a multi-tiered configuration for maximizing storage capability while providing redundancy.

 

In my collection I have two SuperMicro X7DVL- E motherboards fully populated with 24 gigs of RAM and CPU’s of the Xeon 5405 series.  1 Motherboard

 

I have two redundant 1500 watt power supplies in each of the servers that are capable of handling 10 SATA drives that are in a RAID 10 configuration that has a battery backed up cache. (That is two mirrored RAID 5’s with 5 drives each in each of the two hosts each RAID 5 is on a separate controller)

 

Each server has two Xeon 5405 CPU’s which natively support FT/HA in VMWare but do not support VMDirectPath (which is not necessary and causes too many failures anyways.)

 

The cases of the two servers support the drives a, half dozen fans, keyboard, mouse, monitor, both have IPMI cards for offline video access which has come to be very handy.

The IPMI protocol leverages an out-of-band network for access when a video isn’t available. In my case, I access the servers strictly from this and do not even attach a monitor.

One SATA drive holds VMWare ESXi host OS along with a flash drive with the same.

Nexenta is virtualized on each of two other ESX servers that only run the Nexentastor appliance and it utilizes RAW disk mapping techniques.

Nexentastor is a software SAN and by far the best of all the choices that I looked at. It is very professional in its web GUI access to nearly the full set of capabilities of the software SAN but it can be manipulated at a bash shell using techniques such as found here. It is extremely stable in this virtual configuration once one has properly mapped the raw disks to the Nexentastor guest.

I have only one domain controller (This eliminates a lot of sync issues with Active Directory). It is backed up each day with SAN snapshots during a brief file system sequester (since it does not change that often it works great like this. I would even recommend this configuration for medium sized businesses. I keep backups in a 1-year rotation of 5 daily, 4 weekly 12 monthly. I run a Windows 2008 domain controller in native mode with only the highest security communication protocols allowed.3 SAN to SAN replication

 

I utilize  SAN to SAN replication inside the Nexentastor VM’s. Every VM has 4 levels of backups:

 

  1. data and databases
  2. offline and remote backups of VMs
  3. VM snapshots
  4. Storage Level Snapshots

 

ESX has been installed on flash drives and each server has a spare flash drive attached incase the flash fails as well as the copy on the internal drive. Licenses for ESX and for Nexentastor cost nearly 20K but won’t need to be upgraded any time in the near future.

 

On the physical network, I have two switches connected to two pfSense boxes which run CARP between them and provide me redundancy at this level.

 

All machines are on UPS’s with a 2-hour backup life.

 

On the internet domain level, I can restore capability within 24 hours by remapping my internet DNS to a backup server at a sister site. This is all the redundancy I will need for the foreseeable future.

 

I hope this article has been informative for the bloggers out there interested in going professional. This setup is nearly 40K so it is a little expensive for the home hobbyist. A single server version could be built for testing for under $1000 dollars that demonstrate the basic capability of this infrastructure. Good luck!