How to set up an iSCSI backed Xen server

this is a work in progress.

First, if you don't have an iSCSI target, build one as in Building iSCSI target device

Now, build a standard Xen DOM0, then set it up as an iSCSI initiator as in Creating an iSCSI Initiator

I'd suggest your iSCSI target have NFS exports also. Very handy for stuff like image files (Debian ISO's, Windows ISO's, and some emergency repair ISO's). I also use NFS for my Xen DOMU configuration files; I mount it at /etc/xen/iscsi_configs, and then store my configs there.

The other reason for an NFS mount would be to build quick and dirty virtuals; something that can use file backed devices (FBD's). These would be machines for testing, or something that does not use a lot of disk I/O. I generally create three NFS exports on my iSCSI target;

  • xen-configs - mounted at /etc/xen/iscsi_configs, stores all shared DOMU config files
  • xen-images - mounted at /media/xen-images, used for file backed device Xen DOMU's. Much easier to create a single 10G file and do the install than to create an iSCSI target, export it from the target device, import it to the initiators, then use it. But it is very slow.
  • xen-store - mounted at /home/xen-store, it contains a bunch of ISO files that are used by all machines.

NFS allows you to use the same image on multiple machines, and has file locking so two people will not simultaneously edit the same config file.

Diagram of iSCSI backed Xen groupImage to left shows the basic layout of a robust Xen farm. The Dia source for this file is attached to this article.

There are two iSCSI targets, and they replicate themselves via DRBD.

The three DOM0's use these targets to store their virtual images. Since they are all connected to the same source (whichever of the iSCSI targets is currently primary), you can use the xen migrate utility to move a running image from one DOM0 to another.

If you are set up where any single machine can fail, this allows you to take a DOM0 down for maintenance with no visible loss to the end user. Xen migrate only needs to move memory from one machine to another, so it will take less than a second in most cases to do so. The original machine releases the DOMU and the new one picks it up.

I have migrated an IPFire firewall DOMU from one DOM0 to another, while connected over a VPN through that virtual to the DOM0's in question and barely noticed the move (yes, I just had to try it).

So, for maintenance, you migrate all virtuals off of one machine, balancing by placing on the other two. Once that is done, you perform your maintenance, then balance the virtuals back onto the newly up to date machine.

In case of catastrophic failure of one DOM0, you can recover simply by logging into one of the other DOM0's and bringing up the necessary virtuals.

By using DRBD between the iSCSI targets, you can also perform maintenance on your iSCSI targets using the same procedure; simply shift the load off of the machine you want to maintain, work on it, then bring it back on line as a slave. Heartbeat allows you to do this automatically in case of catastrophic failure.

Note, DRBD replication (and heartbeat cluster control) are not covered here.


Attached files: xen-iscsi.dia

Tags: iscsi, xen
Last update:
2015-08-05 09:10
Average rating:0 (0 Votes)

You can comment this FAQ

Chuck Norris has counted to infinity. Twice.