How to move a 125G DOMU from one DOM0 to another without shared storage in 25 minutes

NOTE: this assumes your DOM0's are secure. It involves opening root access (via ssh) from the source DOM0 to the target. Obviously, you don't want to do this if you think your source is compromised. Target does NOT have access to source.

NOTE: One step gives a running DOMU access to the target. There is an alternate way of doing it (described at bottom) you should use if you are not absolutely confident of your security.

Description of system

  • DOM0 1 and DOM0 2 can ssh to each other as root via a 1G internal network
  • DOMU has the following partitions:
    • 10G /
    • 148G /home
    • 2G /var/lib/mysql
    • 1G swap


  1. On target DOM0; can be done days in advance if desired
    1. Give root access from source DOM0
    2. recreate all partitions on target DOM0
    3. mkswap the swap partition (ie, mkswap /dev/vg0/target-swap or something like that)
    4. format /home. Be sure to use same label (if any) from source
    5. mount /home partition on /mnt of DOM0
    6. copy xen DOMU configuration from source DOM0. Verify kernel, disk names, vifs, etc...
  2. Give DOMU access to target DOM0 via ssh as root -- WARNING, Major Security Breach, see below
  3. rsync /home partition to target DOM0/mnt, ie rsync -avz --delete --stats /home/* TARGET_DOM0:/mnt/. Since you will be using this over and over, it is easiest to create a script in root's directory (chmod 700). I generally run the script ( as nohup background, ie nohup /root/ &. The first run will take the longest. In the case of my machine, /home had e-mail, web logs (for clients) and web sites, so there were a lot of changes.
  4. Actual move
    1. With DOMU still up, perform one last via script above. The clients may notice some slowness, but they can still access it.
    2. Log into source DOM0
    3. Shut down DOMU. Server is now unavailable
    4. Mount DOM0/home on /mnt, then run offline rsync, ie rsync -avz --delete --stats /mnt/* TARGET_DOM0:/mnt/
    5. on target machine, umount /mnt
    6. From source machine, use dd over ssh to copy / and mysql partitions, ie:
      1. dd if=/dev/vg0/DOM0-disk | pv -petrs 10G | ssh TARGET_DOM0 'dd of=/dev/vg0/DOM0-disk'
      2. dd if=/dev/vg0/DOM0-mysql | pv -petrs 2G | ssh TARGET_DOM0 'dd of=/dev/vg0/DOM0-mysql'
    7. On target machine, perform a sync, then if desired, create snapshots for backing up later
    8. Start DOMU on target target DOM0
  5. Clean Up
    1. On source DOM0, remove configuration file and LV partitions for DOMU (don't forget any entries in /etc/xen/auto)
    2. On target DOM0, remove configurations allowing ssh by root
    3. If you created snapshots on target DOM0, back them up (ie, dd if=/dev/vg0/snap of=/home/xen-store/DOMU.img) and remove the snapshots

Alternate copy if DOMU is not 100% unquestioned secure.
If you have a nightly backup of the DOMU someplace, copy /home from it. Your backup server is secure, right?
OR (NOTE, I have not actually done this one, but it should work)

  1. take a snapshot of the partition you want to copy (ie, home in this case). This can be done hot, though the result is similar to unplugging a computer so you may have some filesystem errors.
  2. mount snapshot on /mnt of source
  3. rsync to mounted partition on target DOM0

NOTE: the reason for using rsync on /home is that is it large. Copying it via dd would, in this case, result in increasing the downtime of the DOMU to over 2 hours. By using rsync, you are priming the target with some information that will not change, then the finaly rsync simply copies the changed files. In my case, out of 825k files, totalling 117G, the final copy only transferred/updated 139 files and copied 39M.

< 1min  - final rsync
< 10min - dd over ssh of 10G / partition
< 2min  - dd over ssh of 2G mysql partition
Total time from xm stop on source DOM0 to xm create on target, 24 minutes

Last update:
2012-12-26 00:55
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.