Introduction to Scaling and Redundancy
We’ve received a number of inquiries lately asking about scaling and redundancy (load-balancing and high-availability & Online Backup Architecture). The WholesaleBackup Server Manual covers these topics, but we felt a high-level overview would be a good topic for a blog entry.
Hardware Selection for Performance
A decent quad-core server should be able to handle several hundred backup clients. Disk I/O performance is the single most important factor in choosing hardware, in particular the server’s storage devices (both RAID controller cards and the physical disk drives) ability to handle multiple simultaneous readers and writers is critical. We suggest you think ahead when making hardware purchases and configuring it. For example, RAID10 is typically much quicker to rebuild multi-terabyte arrays then RAID5 or RAID6, so in the event of a disk failure (which is inevitable) you’ll be much happier with RAID10. In addition, SAS drives typically handle multiple simultaneous readers better than SATA drives and SCSI and eSATA interfaces are much faster then USB. If your server’s disk controller card and chassis will support it, you can add additional disk drives and make them available in Linux at any time. If you run out of local storage capacity, it is straightforward to setup a Linux NAS or SAN and remote mount that network storage via NFS to your backup server (the file systems need to be Linux formatted).
Metadata Backup and Replication
Our backup server will automatically backup its key meta-data to the /backup/ directory, which you can then backup or sync to another device. We also provide a script that automatically replicates this data to an Amazon EC2 Cloud server (the script will fire up the server,replicate the data, and then automatically stop the server to save you $s). It is straightforward to modify this script to replicate to a server of your choosing.
Components of the WholesaleBackup Server
- A MySQL database containing meta-data about every account’s backups and restores.
- A web-portal consisting of PHP and HTML pages which connect to the MySQL database.
- The storage device(s) that hold the actual customer’s data, logs, and meta-data.
- The actual backup server(s) running WholesaleBackup Server software.
Scaling With WholesaleBackup Server
A typical scenario then is for new WholesaleBackup Server customers to setup a single backup server with all four components on it, and then be sure that the /backup/ directory is safely backed up (or synced) to a separate device. As one adds more users one may add more storage (either local or remote) and when the server’s processing ability (RAM and CPU) become maxed out after several hundred clients, one then adds a second server, and a third and so on.
Support for Multiple Backup Servers
- Unique backup server implementations with distinct software client installers.
- Load-balanced backup servers with storage-specific client backups.
- Universal backup servers with cross-mounted storage for all clients.
Professional Services for Load-Balanced Configurations
Given the intricacies of configuring and testing enterprise-level load-balancing and high-availability schemes with multiple backup servers, you’ll need to enter into a professional services engagement with a trained WholesaleBackup engineer to configure and support the online backup architecture for load-balanced configurations #2 and #3 above.
Getting Started and Scaling Up
To get started, we suggest you setup a single backup server, knowing that our solution will scale as your business scales. When you’ve hit several hundred users and exhausted the resources of your first server then it’s time to add a second server and make some decisions as to whether to go with option #2 or #3 above (you can change between them so no decision is permanent)…
Case Study: Load-Balancing in Action
The following graph shows four under-utilized backup servers (e03 – e06) plus a MySQL/web-server (e01) in production in the Amazon EC2 Cloud using load-balancing scheme #2 listed above.