About two years ago, I rented a dedicated server to host a few virtual machines. It’s been running fine ever since, but at some point, faster and more energy-efficient hardware is available for the same price. I decided to rent a new server mostly for the additional RAM, which is very valuable when hosting virtual machines — other resources such as CPU, disk and network are usually not a bottleneck.
This article documents the different steps I used to migrate my (KVM and libvirt) virtual machines from my old server (called OLD with IP address 192.168.2.2 from now on) to my new server (called NEW with IP address 192.168.9.9 from now on).
Note that I did not do a live migration because I don’t use network storage which is accessible from both machines. I needed to transfer the data from one machine to the other.
- Transfer the virtual machine’s data to NEW.
- Stop the virtual machine on OLD, boot it up on NEW.
- Redirect traffic from OLD to NEW (the IPs are transferred to the new server later on).
The obvious problem is that the virtual machine is still running between step 1 and 2 and therefore it might modify the data. Normally, to avoid inconsistencies, you use an LVM snapshot, but that doesn’t help you with additional, consistent data which the virtual machine may write. The solution is described below, but let’s start with the tunnel setup first.
Traffic redirection: preparation
To be able to boot up the virtual machine on NEW and use it immediately (without contacting the hoster to change the routing table), we set up a tunnel from OLD to NEW. Beware: Depending on the routing setup at your hoster, there might be a reverse-path filter in our way. Such a filter blocks packets which come from the wrong path, in our case it might only allow packets originating from OLD, but not from NEW. Therefore, double-check first if you can use a tunnel like this at your hoster before you follow this article.
On OLD, add the following entries to
auto legacy iface legacy inet tunnel mode gre local 192.168.2.2 endpoint 192.168.9.9 address 10.0.1.1 netmask 255.255.255.0 ttl 255 auto legacy6 iface legacy6 inet tunnel mode sit local 192.168.2.2 endpoint 192.168.9.9 address fd26:a975:9d12::1 netmask 48
On NEW, add the same entries, but swap the "local" and "endpoint" fields and use a different IP address (e.g. 10.0.1.2 and fd26:a975:9d12::2). The IPv6 address is an automatically generated RFC4193 (Unique local address) address, generated with Holger Zuleger’s generate-rfc4193-addr.sh.
Afterwards, bring up the networks with
ifup legacy and
legacy6 on OLD and NEW and check that they can ping each other.
Transferring the data
We will transfer the data in two steps: First, we just copy the whole block device over, then we shut down the virtual machine and copy only the differences. Copying the differences is usually done in a few seconds, while a block device transfer of a 10 GiB device takes about 20 minutes, so this technique reduces the total down time for the virtual machine to a minimum. If you tend to chose too big disk sizes for your VMs, waiting for a long time to transfer them might be a lesson to choose smaller sizes in the future… :-)
First, create the new logical volume and take a snapshot of the old one (the snapshot size determines how much data the virtual machine can write as long as it’s still running):
NEW # lvcreate -L 20G -n domu-web newhost OLD # lvcreate --snapshot -L10G -n web-lvmsync oldhost/domu-web
Then, transfer the data, compressed, with a progress bar. I just love the power of pipes when seeing such a command :-).
OLD # dd if=/dev/oldhost/domu-web bs=1M | pv -ptrb -s 20G | \ gzip -3c | ssh new '(gunzip -c - | dd of=/dev/newhost/domu-web)'
Transferring the differences
Afterwards, shut down the VM, use lvmsync to transfer the differences and boot the VM on NEW:
web # shutdown -h now OLD # lvmsync /dev/oldhost/web-lvmsync new:/dev/newhost/domu-weg NEW # virsh create /etc/libvirt/qemu/web.xml
You can copy the file
web.xml from OLD and modify it, or, if you
have many changes in your setup, start with a fresh one (pay attention to
keeping the same MAC address).
Redirecting the traffic
The final step is easy: just add the appropriate route(s) on OLD (don’t forget
to make them persistent in
OLD # ip -4 route add 18.104.22.168/32 via 10.0.1.1 dev legacy OLD # ip -6 route add 2001:4d88:100e:2::/64 via fd26:a975:9d12::1 dev legacy6
And we’re done! The total downtime of the VM is a few minutes — the time it needs to shut down, transfer differences and boot up again. Booting might take longer than normal if a file system check is necessary, thus it’s not done in a few seconds probably.