HowTo: Backup My Server

This time I would like to explain how I backup my server with the help of some build in tools and a script written by myself. As I’m running my blogs and my mail server, which is Scalix, on a Linux server I need to make sure, that no data is lost during an outage and that I will be back online in a short period of time. Therefore I have two strategies to backup my stuff.

1. Using Proxmox, it is very easy to do an online backup of my virtual machine, as described here:
   How To: Proxmox Live Backup
   The drawback with this method is, that it takes some time to do the backup, it consumes a lot of system resources (I/O) and the backup is very large in size.
2. Using a file based backup from within the virtual machine, backing up only the real important stuff.
   The drawback here, you cannot restore the system from this backup. But in combination with method one, it works perfectly.

I will not describe the first method, as this is already described in the mentioned post. I do this kind of full backup once a week and use method two to get the increment data from every day between two full backups. As method two will not work out of the box, you have to make sure the data is consistent, I will cover this method in this post.

Requirements for the Backup

First of all, you need to have an external backup server to make sure the data is protected against loss. I will use an external rsync server which is included in my server subscription. This is still in the same data center, but finger crossed, it is in a different fire compartment. The full backup, produced with Proxmox is copied to my NAS system at home. This will make sure, that at least one of the backups will survive.

In addition to an external storage, you should also have LVM (Logical Volume Manager) installed on your system. This is the only option to make online backups of your system and keep data consistent using LVM snapshots. Instead of LVM, some hardware raid systems support a similar feature.

LVM (Logical Volume Manager)

I will not describe how to configure LVM, this is outside of the scope of this post, but I will explain shortly, what LVM is. LVM is included in the Linux kernel and provides a layer between the file system and the block device. LVM has features like encryption, easy expanding the volume and many more, but we would like to use only the snapshot function in the first place.

To use the snapshot function of LVM you need to make sure, that you still have some free unallocated space on your drive. This space will be used during the snapshot phase to write new changes to the disk. Make sure to leave enough free space on the disk, as you can expand you volume any time. There is no reason to oversize your volume in the first place. Keep in mind, if the space is not enough to hold all the data changes during the snapshot phase, you can be in big trouble.

Consistent LVM Snapshot

Creating a LVM snapshot is very fast but not instantaneous. There is a very little chance that some applications can be in an inconsistent state. MySQL such as could be in the middle of a transaction. Doing a snapshot in this moment will lead to an inconsistent MySQL backup. The same issue can happen with Scalix. As those are the only applications which needs spacial handling in my environment, I will only cover them. But keep this in mind and check if you have applications on your system which needs special handling as well.

Scalix

To get a consistent backup of the Scalix message store, you need to suspend Scalix before creating the backup. This could be accomplished using omsuspend like this:

omsuspend -s 120 & 
sleep 2 
sync; sync; sync 
 
omsuspend -r

This first line will suspend Scalix and no user interaction with the Scalix system is possible. The last line will resume the Scalix system again and Scalix will run normal.

MySQL

To avoid the situation, the you create the snapshot, when MySQL is in the middle of a transaction, you can lock MySQL. This will somehow pause MySQL. You need to issue this command to the MySQL server:

FLUSH TABLES WITH READ LOCK;

This will write the database to disk and lock the tables. Do this with a permanent session, as the lock is removed as soon as the session is terminated.

To unlock the tables again, after the snapshot is created, use this command:

UNLOCK TABLES;

You should now have a consistent snapshot and you can start the backup process… wait, I didn’t told you, how to create the snapshot.

Create the Snapshot

The snapshot is created very easy and needs only one command, two if you actually would like to get access to the snapshot data. It takes some more effort to get all the information needed, before creating the snapshot.

First of all, you need to know, the name of the volume for which you would like to create the snapshot of. In a very simple environment, you will have only one volume for all data. If you have split your data across multiple volumes,  you can check the specific path using this command:

df /var/opt/scalix/sx/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/scalix-root 96816868 6806528 85092276 8% /

This will tell me, that the Scalix message store is on the volume group “scalix” on the volume “root”. To get the correct path to the volume use this command:

lvscan
 ACTIVE '/dev/scalix/swap_1' [4.09 GiB] inherit
 ACTIVE '/dev/scalix/root' [93.80 GiB] inherit

As you can see, there are two volumes, but as the command from above told us to use the volume “root” we have now the complete path to the volume, which is

/dev/scalix/root

You also need a name for the snapshot volume. I chose “backup-snapshot”. Using this two pieces of information we can create the snapshot using this command:

lvcreate -l100%FREE -s -n backup-snapshot /dev/scalix/root

This will use 100% of the free space on the disk and create the snapshot. If you issue lvscan again it should look like this:

lvscan
 ACTIVE '/dev/scalix/swap_1' [4.09 GiB] inherit
 ACTIVE Original '/dev/scalix/root' [93.80 GiB] inherit
 ACTIVE Snapshot '/dev/scalix/backup-snapshot' [1.87 GiB] inherit

The last step would be to mount the snapshot to get access to the files. We can use a simple mount command here:

mount /dev/scalix/backup-snapshot /mnt/backup

Make sure, to create the mount folder before using this command. After backing up everything you need, you can simply unmount and drop the snapshot:

unmount /mnt/backup
lvremove -f /dev/scalix/backup-snapshot

Backup

As we now know, how to create the snapshot and remove it, I would like to share my thoughts about the backup itself. I run this kind of backup every day, except the day, when I run the full VM backup. I use rsync and create a backup folder for every day. This works fast and reliable and I can benefit from just backing up the changes.

My backup includes all configuration files within “/etc” and all logs in “/var/log”. To include the Scalix messages store I also backup “/var/opt” and to get the MySQL files I include “/var/lib/mysql”. “/var/www” is also included, as I have a Photoblog running, which stores the pictures there.
From my point of view, this includes all files needed to recover from a crash, using the full VM backup plus the file backup.

The Backup Script

To do all the above steps automatically I wrote a script for my usage. Feel free to use it and adapt the script to your own needs.

#!/bin/bash
#set -x

#is mysql present
#0=no
#1=yes
mysql=1

#is scalix present
#0=no
#1=yes
scalix=1

#mysql user name
mysqlusername="mysql"

#mysql user file
mysql_user_file=/etc/mysql/debian.cnf

#Path to the Volume to make the snapshot of
lvmpath="/dev/scalix/root"

#Logical Volume Name for the snapshot
lvmname="backup-snapshot"

#Path to the mountfolder
mountfolder="/mnt/snapshot"

#create path to snapshot volume
lvmsnapshotpath=${lvmpath%/*}"/"$lvmname

#list of folders to backup, separate with space
backupfolder="/etc /var/log /var/opt /var/lib/mysql /var/www"

#backup destination.
backupdestination="rsync://backup-server/folder/"

#get day of week
dow=$(date +%a)

#clean up if something goes wrong
function clean_up {

 #unlock mysql
 unlock_mysql
 
 #resume scalix
 activate_scalix
 
 #remove the snapshot
 remove_snapshot

 #exit the script
 exit
}

trap clean_up SIGHUP SIGINT SIGTERM

#lock mysql
function lock_mysql {

 if [ $mysql -eq 0 ]; then
 return
 fi
 #create a screen session to lock mysql during snapshot
 screen -S "mysql" -d -m

 #pause for 1sec to get the session created
 sleep 1

 #connect to mysql
 screen -r "mysql" -X stuff "/usr/bin/mysql --defaults-extra-file=$mysql_user_file -u $mysqlusername
 "

 #lock mysql
 screen -r "mysql" -X stuff "FLUSH TABLES WITH READ LOCK;
 "

 #wait for 1sec to fully lock mysql
 sleep 1
 
}

#unlock mysql
function unlock_mysql {
 
 if [ $mysql -eq 0 ]; then
 return
 fi
 
 #check if the screen session is still tunning
 if ! screen -list | grep -q "mysql"; then
 return
 fi
 
 #unlock mysql again
 screen -r "mysql" -X stuff "UNLOCK TABLES;
 "

 #quit from mysql
 screen -r "mysql" -X stuff "quit
 "

 #end the screen session
 screen -r "mysql" -X stuff "exit
 "
 
}

#create the snapshot
function create_snapshot {

 #create the snapshot
 lvcreate -l100%FREE -s -n $lvmname $lvmpath

}

#mount the snapshot
function mount_snapshot {

 #create the mountfolder
 mkdir -p $mountfolder

 #moun the snapshot
 mount $lvmsnapshotpath $mountfolder
 
}

#do the backup
function backup_everything {

 #do some backup
 rsync -apE --delete --stats $backupfolder $backupdestination$dow

}

#remove the snapshot
function remove_snapshot {

 #unmount the snapshot folder
 umount $mountfolder

 #remove the snapshot
 lvremove -f $lvmsnapshotpath

}

#suspend scalix
function suspend_scalix {

 if [ $scalix -eq 0 ]; then
 return
 fi
 
 /opt/scalix/bin/omsuspend -s 120 & 
 sleep 2 
 sync; sync; sync 

}

#activate scalix again
function activate_scalix {
 
 if [ $scalix -eq 0 ]; then
 return
 fi
 
 /opt/scalix/bin/omsuspend -r

}


lock_mysql

suspend_scalix

create_snapshot

activate_scalix

unlock_mysql

mount_snapshot

backup_everything

remove_snapshot

exit

You need to modify some of the variables at the top of the script, to make it work for you environment.

You can run the script as a cron job to run it automatically. I run my script every night at 04:30am.

If you have any questions, regarding this post or if you would like to provide feedback, please use the comment function below.