Problem to solve:
If you have a Proxmox Server and you like tinkering around with it like I do, then you probably know that feeling that all of a sudden nothing seems to be working as it used to be and you want to re-roll to a certain point and just start all over again. And yes, your containers and VMs are probably all save and in a backup, but what about the hours of work you put into your perfect qemu kernel sysctl tuning file?
The solution:
My solution is a Synology NFS share volume (can also be any other Server with a NFS share volume) and a backup script from github by DerDanilo that is located on that share, and is available and executable to all Proxmox Servers in the network. The server mounts that volume on startup and execute the script as a cronjob every day at night. The script creates a full *.gzip file from the entire etc*, var* and root* folder of the Proxmox instance. So if something goes wrong you can just copy back the zip backup file and extract the files back to the system and restart (see below: Restore process). The Proxmox Server than just starts in the state it was when the image was created.
And to avoid that our volume gets overfilled with .gzip files, his creator DerDanilo added the option "MAX_BACKUPS=5", to automatically delete the oldest *.gzip after 5 backups. I increased this number to 30.
Prepare the Synology share volume:
I already have a proxmox folder on my Synology so I am going to use this one. Its the same share proxmox uses for storing snapshots, isos and vm disks. If you did not have any nfs-volume mounted yet, please follow the official proxmox guidelines to add a nfs storage location to your proxmox server.
Setup your Proxmox Server Node:
ssh into your running Proxmox Server you want to setup for backup schedule. Make sure you login as user root (default). Now find the name of your mounted proxmox nfs storage with:
The result of this should look something like this and lists all your mounted options:
total 0
drwxrwxrwx 1 root root 118 Sep 14 03:44 nfs-homeserver
root@proxmox-node1-master:~#
So in my case my proxmox mount is called nfs-homeserver. Now we know the volume we want to store our backup files on, lets create a subfolder so we have all of the *gzip backup files in one folder later:
mkdir -m 777 /mnt/pve/nfs-homeserver/config-zips/Now we have our folder for the script ready, awesome! Lets create the backup script there now:
nano /mnt/pve/nfs-homeserver/config-zips/prox_config_backup.shThis will create and open and empty .sh file. Now we copy and paste our script that sets the backup command and the timing (feel free to alternate to your folder structure and needs):
#!/bin/bash
# Version 0.2.2 - BETA ! !
# Date 02.20.2020
# Author DerDanilo
# Contributors aboutte, xmirakulix, bootsie123
# set vars
# always exit on error
set -e
# permanent backups directory
# default value can be overridden by setting environment variable before running prox_config_backup.sh
# example: export BACK_DIR="/mnt/pve/media/backup
_bdir=${BACK_DIR:-/mnt/pve/nfs-homeserver/config-zips/}
# number of backups to keep before overriding the oldest one
MAX_BACKUPS=30
# temporary storage directory
_tdir=${TMP_DIR:-/var/tmp}
_tdir=$(mktemp -d $_tdir/proxmox-XXXXXXXX)
function clean_up {
echo "Cleaning up"
rm -rf $_tdir
}
# register the cleanup function to be called on the EXIT signal
trap clean_up EXIT
# Don't change if not required
_now=$(date +%Y-%m-%d.%H.%M.%S)
_HOSTNAME=$(hostname -f)
_filename1="$_tdir/proxmoxetc.$_now.tar"
_filename2="$_tdir/proxmoxpve.$_now.tar"
_filename3="$_tdir/proxmoxroot.$_now.tar"
_filename4="$_tdir/proxmoxcron.$_now.tar"
_filename5="$_tdir/proxmoxvbios.$_now.tar"
_filename6="$_tdir/proxmoxpackages.$_now.list"
_filename7="$_tdir/proxmoxreport.$_now.txt"
_filename_final="$_tdir/proxmox_backup_"$_HOSTNAME"_"$_now".tar.gz"
##########
function description {
clear
cat <<EOF
Proxmox Server Config Backup
Hostname: "$_HOSTNAME"
Timestamp: "$_now"
Files to be saved:
"/etc/*, /var/lib/pve-cluster/*, /root/*, /var/spool/cron/*, /usr/share/kvm/*.vbios"
Backup target:
"$_bdir"
-----------------------------------------------------------------
This script is supposed to backup your node config and not VM
or LXC container data. To backup your instances please use the
built in backup feature or a backup solution that runs within
your instances.
For questions or suggestions please contact me at
https://github.com/DerDanilo/proxmox-stuff
-----------------------------------------------------------------
Hit return to proceed or CTRL-C to abort.
EOF
read dummy
clear
}
function are-we-root-abort-if-not {
if [[ ${EUID} -ne 0 ]] ; then
echo "Aborting because you are not root" ; exit 1
fi
}
function check-num-backups {
if [[ $(ls ${_bdir}/*${_HOSTNAME}*.tar.gz -l | grep ^- | wc -l) -ge $MAX_BACKUPS ]]; then
local oldbackup="$(basename $(ls ${_bdir}/*${_HOSTNAME}*.tar.gz -t | tail -1))"
echo "${_bdir}/${oldbackup}"
rm "${_bdir}/${oldbackup}"
fi
}
function copyfilesystem {
echo "Tar files"
# copy key system files
tar --warning='no-file-ignored' -cvPf "$_filename1" /etc/.
tar --warning='no-file-ignored' -cvPf "$_filename2" /var/lib/pve-cluster/.
tar --warning='no-file-ignored' -cvPf "$_filename3" /root/.
tar --warning='no-file-ignored' -cvPf "$_filename4" /var/spool/cron/.
if [ "$(ls /usr/share/kvm/*.vbios 2>/dev/null)" != "" ] ; then
echo backing up custom video bios...
tar --warning='no-file-ignored' -cvPf "$_filename5" /usr/share/kvm/*.vbios
fi
# copy installed packages list
echo "Copying installed packages list from APT"
apt-mark showmanual | tee "$_filename6"
# copy pvereport output
echo "Copying pvereport output"
pvereport | tee "$_filename7"
}
function compressandarchive {
echo "Compressing files"
# archive the copied system files
tar -cvzPf "$_filename_final" $_tdir/*.{tar,list,txt}
# copy config archive to backup folder
# this may be replaced by scp command to place in remote location
cp $_filename_final $_bdir/
}
function stopservices {
# stop host services
for i in pve-cluster pvedaemon vz qemu-server; do systemctl stop $i ; done
# give them a moment to finish
sleep 10s
}
function startservices {
# restart services
for i in qemu-server vz pvedaemon pve-cluster; do systemctl start $i ; done
# Make sure that all VMs + LXC containers are running
qm startall
}
##########
description
are-we-root-abort-if-not
check-num-backups
# We don't need to stop services, but you can do that if you wish
#stopservices
copyfilesystem
# We don't need to start services if we did not stop them
#startservices
compressandarchiveClose and save the file with CTRL + X and Y for override file.
Make the script executable:
chmod +x /mnt/pve/nfs-homeserver/config-zips/prox_config_backup.shLast but not least create a cronjob to run the the backup task automatically. We will set it in this example to daily, at 0am and 12am. Feel free to alternate that to your needs as well, I recommend crontab guru for easy time set:
If you open crontab the first time it will ask you for the option to start with, I recommend option 1 for nano.
crontab -eand add in the last line of the file:
0 0,6,12,18 * * * /mnt/pve/nfs-homeserver/config-zips/prox_config_backup.shNow we are set and the Proxmox Server Node will backup itself every day at midnight (0am), morning (6am), noon (12am) and evening (6pm) to our Synology share!
If you don't want to wait for the cronjob to start, you can also test it in with this command:
/mnt/pve/nfs-homeserver/config-zips/prox_config_backup.shCongratulations, you now have also all your precious Proxmox server node configs backed up!
Restore process
If you need to restore from one of the zip files, you just do this:
To restore, move the file back to proxmox with cp, scp, webmin, a thumb drive, whatever. I place it back into the /var/tmp directory from where it came.
# Unpack the original backup
tar -zxvf proxmox_backup_proxmoxhostname_2017-12-02.15.48.10.tar.gz
# unpack the tared contents
tar -xvf proxmoxpve.2017-12-02.15.48.10.tar
tar -xvf proxmoxetc.2017-12-02.15.48.10.tar
tar -xvf proxmoxroot.2017-12-02.15.48.10.tar
# If the services are running, stop them:
for i in pve-cluster pvedaemon vz qemu-server; do systemctl stop $i ; done
# Copy the old content to the original directory:
cp -avr /var/tmp/var/tmp/etc /etc
cp -avr /var/tmp/var/tmp/var /var
cp -avr /var/tmp/var/tmp/root /root
# And, finally, restart services:
for i in qemu-server vz pvedaemon pve-cluster; do systemctl start $i ; doneIf nothing goes wrong, and you have separately restored the VM images using the default ProxMox process.
You should be back where you started. But let's hope it never comes to that.