Call us (425) 818-8252 | (800) 624-9561 x106 |sales@wholesalebackup.com

How to automate Amazon AWS Cloud Backups with cron and snapshot script, including deletions

How to automate Amazon AWS Cloud Backups with cron and snapshot script, including deletions

One of our partners asked me today about how to backup the EBS volumes running on their Amazon AWS instance by automating snapshots, like in this discussion: http://aws.amazon.com/articles/1663?_encoding=UTF8&jiveRedirect=1.

In our case we’ve got to backup a MySQL database, plus some other directories, and we want to snapshot those. Since MySQL’s files are not necessarily going to be in a consistent state through the snapshot window, we should also backup the database. Let’s tackle that first.

Backing up the MySQL database with a cron job, and setup crontab lines for daily, monthly, and weekly snapshots:

  1. use crontab -e to edit your crontab
  2. Add a line to backup the database every day at 10:12am (this is usually a low load time for online backup), and a line for daily, weekly, and monthly snapshots:
    12 10 * * * /usr/bin/mysqldump --add-drop-table -uroot -pSHHHHHHHH divinsamgmtdb > /var/backup/divinsamgmdb.sql
    
    17 13 * * * /root/snapshotscripts/managesnapshots daily >/var/log/dailysnapshots 2>&1
    
    20 13 * * 0 /root/snapshotscripts/managesnapshots weekly >/var/log/weeklysnapshots 2>&1
    
    23 13 1 * * /root/snapshotscripts/managesnapshots monthly >/var/log/monthlysnapshots 2>&1
  3. Make sure you have a directory called /var/backup and that the drive has enough space.
  4. Verify that /var/backup is actually on your / (root) partition as that’s the one we’re going to backup using Amazon snapshots.

Install Amazon EC2 API Command line tools on your Ubuntu server

This is covered on the web in many places, so here is a great link to walk you through it (scroll down a bit): https://help.ubuntu.com/community/EC2StartersGuide

Deploy and customize the following script to create snapshots:

  1. Create a file called /root/snapshotscripts/managesnapshots and paste the following script into it:
    #!/bin/bash
    
    		# backups and and deletes given a dailysnapshotlist (or monthly or weekly - make sure you run the cron accordingly)
    
    		
    
    		# Set env variables for Amazon as cron may run without them
    
    		export EC2_PRIVATE_KEY=/root/.ec2/YOURFILE.pem
    
    		export EC2_CERT=/root/.ec2/YOURCERT.pem
    
    		export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
    
    		
    
    		#if no $1, exit
    
    		if 
    [[ $1 == "daily" ]] || [[ $1 == "monthly" ]] || [[ $1 == "weekly" ]] then   export runtype="$1" else   printf "Usage: managesnapshots daily|weekly|monthly\n"   exit 1 fi #set date time stamp for temp files export dts=$(date +"%m-%d-%Y") while read volume_id description daily weekly monthly do   #exit loop if no values - otherwise get infinite loop   [ -z $volume_id ] && break;   # skip lines with comments in them   if [[ $volume_id =~ ^#.* ]]   then     true   else     #add runtype into the ec2 description so we can search for snapshot types for deletions later     export description=$(echo "$description $runtype")     #add 1 to the keepdays so the tail command get s right number of rows ($$ uses variable substitution)     case $runtype in      daily)        export keepdays=$daily        ;;      monthly)        export keepdays=$monthly        ;;      weekly)        export keepdays=$weekly        ;;      *)        exit 2        ;;     esac     export tailnumber=$(($keepdays + 1))     if [ $keepdays != 0 ]       then       # create the snapshot for today       echo "snapshotting $volume_id -- $description"       /usr/bin/ec2-create-snapshot "$volume_id" -d "$description"       #delete the older shapshots now       #get the snapshots that are older than the ones we want to keep       printf "checking for snapshots to delete from $volume_id\n"       /usr/bin/ec2-describe-snapshots --filter volume-id="$volume_id" | grep $runtype | sort -r -k 5 | tail -n +${tailnumber} > /var/tmp/delsnapshots$dts       while read deleteline       do         #get snapshot_id         export snapshot_id=$(echo $deleteline | awk '{print $2}')         #delete the older snapshot         printf "deleting snapshot $snapshot_id for $volume_id -- $description\n"         /usr/bin/ec2-delete-snapshot "$snapshot_id"       done < /var/tmp/delsnapshots$dts       rm /var/tmp/delsnapshots$dts     fi   fi done < /root/snapshotscripts/snapshotlist echo "end" exit 0
  2. Now Create a file to list the volumes you want to snapshot, called /root/snapshotscripts/snapshotlist, and past the following contents, then modify them so that each line lists a volumeid of the volumes you want to backup, which instance they’re on and which device(for tagging purposes so that you have a note attached to each snapshot), and how many daily, weekly, and monthly images you wish to keep:
    #volume_id description keepdays_daily weekly monthly [don't_use_spaces_in_values_-->_they_are_the_delimiter!]
    
    		vol-dcsdssa6 WSBU_Serv01:/dev/sda1 3 3 3
    
    		vol-32fja92r WSBU_Serv02:/dev/sda1 6 6 6
    
    		vol-89asd0fe WSBU_Serv03:/dev/sda1 3 3 3
    
  3. make sure these are readable only by root, and that they’re executable only by root:
    chown -R root:root /root/snapshotscripts/
    
    		chmod -R 700 /root/snapshotscripts/

OK – you’re now done!

Some notes and caveats: you should check to see that the script works in your environment, as even a small typo or configuration change can wreak havoc. Verify that you have snapshots by restoring one and testing it!

There will be logfiles in /var/log/ so you can see errors if they were logged and progress notes as they were logged.

You can also manually run the commands to test them, like:

 

/root/snapshotscripts/managesnapshots daily
Blog Headline: How to automate Amazon AWS Cloud Backups with cron and snapshot script, including deletions Blog Description: WholesaleBackup Online backup blog, white label backup software, backup strategy, tip and tricks, news. Published Date: Feb 1, 2015 Modified Date: Oct 25, 2017 Image URL: https://wholesalebackup.com/wp-content/uploads/2017/01/msp-backup-management-console.png Image Width: 250px Image Height: 163px Publisher Name: WholesaleBackup maker of white label backup software Logo URL: https://wholesalebackup.com/wp-content/uploads/2017/01/wholesalebackup-logo-adwords.jpg Logo Width: 288px Logo Height: 146px
By |2018-02-02T08:50:06+00:00February 1st, 2015|