This is a quick tutorial about a case I recently encountered. After a upgrade of a clients Proxmox cluster to the latest version, two very important VMs were not able to power on again. As it turned out fixing the root of the issue would take much longer than the downtime would allow for, as the VMs are using quite some complicated dedicated hardware passthroughs. So the quick fix to get back to production mode would be a reroll. And it turned out proxmox is very forgiving and allow for easy rerolls. I think that is a very useful procedure and I quickly want to share that here:

Step 1:

First you want to get the exact list of changed packages on the server. For this ssh into your proxmox server or cluster and type

cat /var/log/apt/history.log

In my case this returned:

Start-Date: 2020-10-17 15:39:01
Commandline: apt upgrade
Upgrade: proxmox-widget-toolkit:amd64 (2.2-12, 2.3-1), libpve-access-control:amd64 (6.1-2, 6.1-3), libpve-storage-perl:amd64 (6.2-8, 6.2-9), pve-qemu-kvm:amd64 (5.1.0-2, 5.1.0-3), libpve-cluster-api-perl:amd64 (6.1-8, 6.2-1), libpve-cluster-perl:amd64 (6.1-8, 6.2-1), zfs-initramfs:amd64 (0.8.4-pve1, 0.8.4-pve2), zfs-zed:amd64 (0.8.4-pve1, 0.8.4-pve2), proxmox-backup-client:amd64 (0.9.0-2, 0.9.1-1), pve-cluster:amd64 (6.1-8, 6.2-1), zfsutils-linux:amd64 (0.8.4-pve1, 0.8.4-pve2), spl:amd64 (0.8.4-pve1, 0.8.4-pve2), libzfs2linux:amd64 (0.8.4-pve1, 0.8.4-pve2), qemu-server:amd64 (6.2-14, 6.2-15), libzpool2linux:amd64 (0.8.4-pve1, 0.8.4-pve2), libnvpair1linux:amd64 (0.8.4-pve1, 0.8.4-pve2), libuutil1linux:amd64 (0.8.4-pve1, 0.8.4-pve2)
End-Date: 2020-10-17 15:39:28

Start-Date: 2020-10-30 15:47:06
Commandline: apt-get dist-upgrade
Upgrade: proxmox-widget-toolkit:amd64 (2.3-1, 2.3-6), linux-libc-dev:amd64 (4.19.146-1, 4.19.152-1), proxmox-backup-client:amd64 (0.9.1-1, 0.9.4-1), pve-i18n:amd64 (2.2-1, 2.2-2), pve-manager:amd64 (6.2-12, 6.2-14), qemu-server:amd64 (6.2-15, 6.2-17), libfreetype6:amd64 (2.9.1-3+deb10u1, 2.9.1-3+deb10u2), tzdata:amd64 (2020a-0+deb10u1, 2020d-0+deb10u1)
End-Date: 2020-10-30 15:47:13

NOTICE: the important part for us is the apt-get dist-upgrade command we did and that we want to undo.

Step 2:

Now from this list we forge our reroll command. We take every single package, for example proxmox-widget-toolkit:amd64 (2.3-1, 2.3-6)There we see the package name and in brackets behind the first version we had, and then the version that it upgraded to. With this information we rewrite the package name with specific version instructions, like in our example: proxmox-widget-toolkit=2.3-1

When applying this scheme to all done upgrades above we end up with a new list like this:

proxmox-widget-toolkit=2.3-1 
linux-libc-dev=4.19.146-1 
proxmox-backup-client=0.9.1-1 
pve-i18n=2.2-1 
pve-manager=6.2-12 
qemu-server=6.2-15 
libfreetype6=2.9.1-3+deb10u1 
tzdata=2020a-0+deb10u1

Step 3:

Now lets go back to our proxmox server cluster shell and reroll to the prior version with this:

apt-get install proxmox-widget-toolkit=2.3-1 linux-libc-dev=4.19.146-1 proxmox-backup-client=0.9.1-1 pve-i18n=2.2-1 pve-manager=6.2-12 qemu-server=6.2-15 libfreetype6=2.9.1-3+deb10u1 tzdata=2020a-0+deb10u1

After successful installation reboot your server:

reboot now

If you are running a cluster with multiple instances, repeat the above three steps on all servers of that cluster.

Voila, you are back to your prior working version of Proxmox server. In my case all VMs that had issues after the upgrade booted up immediately.

Hope this helps you as well in case of a faulty upgrade and prevents you form reinstalling or fully restoring a server from a backup instance. Saves a lot of time! 🙂