How to automate Amazon AWS Cloud Backups with cron and snapshot script, including deletions
One of our partners asked me today about how to backup the EBS volumes running on their Amazon AWS instance by automating snapshots, like in this discussion: http://aws.amazon.com/articles/1663?_encoding=UTF8&jiveRedirect=1.
In our case we’ve got to backup a MySQL database, plus some other directories, and we want to snapshot those. Since MySQL’s files are not necessarily going to be in a consistent state through the snapshot window, we should also backup the database. Let’s tackle that first.
Backing up the MySQL database with a cron job, and setup crontab lines for daily, monthly, and weekly snapshots:
- use crontab -e to edit your crontab
- Add a line to backup the database every day at 10:12am (this is usually a low load time for online backup), and a line for daily, weekly, and monthly snapshots:
12 10 * * * /usr/bin/mysqldump --add-drop-table -uroot -pSHHHHHHHH divinsamgmtdb > /var/backup/divinsamgmdb.sql 17 13 * * * /root/snapshotscripts/managesnapshots daily >/var/log/dailysnapshots 2>&1 20 13 * * 0 /root/snapshotscripts/managesnapshots weekly >/var/log/weeklysnapshots 2>&1 23 13 1 * * /root/snapshotscripts/managesnapshots monthly >/var/log/monthlysnapshots 2>&1
- Make sure you have a directory called /var/backup and that the drive has enough space.
- Verify that /var/backup is actually on your / (root) partition as that’s the one we’re going to backup using Amazon snapshots.
Install Amazon EC2 API Command line tools on your Ubuntu server
This is covered on the web in many places, so here is a great link to walk you through it (scroll down a bit): https://help.ubuntu.com/community/EC2StartersGuide
Deploy and customize the following script to create snapshots:
- Create a file called /root/snapshotscripts/managesnapshots and paste the following script into it:
#!/bin/bash # backups and and deletes given a dailysnapshotlist (or monthly or weekly - make sure you run the cron accordingly) # Set env variables for Amazon as cron may run without them export EC2_PRIVATE_KEY=/root/.ec2/YOURFILE.pem export EC2_CERT=/root/.ec2/YOURCERT.pem export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ #if no $1, exit if
- Now Create a file to list the volumes you want to snapshot, called /root/snapshotscripts/snapshotlist, and past the following contents, then modify them so that each line lists a volumeid of the volumes you want to backup, which instance they’re on and which device(for tagging purposes so that you have a note attached to each snapshot), and how many daily, weekly, and monthly images you wish to keep:
#volume_id description keepdays_daily weekly monthly [don't_use_spaces_in_values_-->_they_are_the_delimiter!] vol-dcsdssa6 WSBU_Serv01:/dev/sda1 3 3 3 vol-32fja92r WSBU_Serv02:/dev/sda1 6 6 6 vol-89asd0fe WSBU_Serv03:/dev/sda1 3 3 3
- make sure these are readable only by root, and that they’re executable only by root:
chown -R root:root /root/snapshotscripts/ chmod -R 700 /root/snapshotscripts/
OK – you’re now done!
Some notes and caveats: you should check to see that the script works in your environment, as even a small typo or configuration change can wreak havoc. Verify that you have snapshots by restoring one and testing it!
There will be logfiles in /var/log/ so you can see errors if they were logged and progress notes as they were logged.
You can also manually run the commands to test them, like: