Unix Server Tech Knowledge Base

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Xen DOM0 ======

===== Base Install =====

<code bash>
apt -y install xen-system
# if you don't do this, it will boot into non-xen kernel
dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
# now, limit DOM0 memory to 4G
echo '# Xen boot parameters for all Xen boots' >> /etc/default/grub
echo 'GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M"' >> /etc/default/grub
echo '# do not probe anything but the boot drive for os' >> /etc/default/grub
echo 'GRUB_DISABLE_OS_PROBER=true' >> /etc/default/grub
echo 'autoballoon="0"' >> /etc/xen/xl.conf
# and tell grub to update itself
update-grub
# tell Xen to use the xl tool stack. It appears with newer Debian releases, this is the default
# so probably not necessary
# NOT TESTED, you may have to hand edit.
echo 'TOOLSTACK=xl' >> /etc/default/xen
# at this point, we are ready to go, so reboot the system into the Xen kernel
reboot
</code>

Note: There is an option to limit the number of vcpu's DOM0 can have, but my notes are possibly incorrect. I had:
<code>GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M dom0_max_vcpus=4"</code>

==== Limit DOM0 memory ====
At times, the DOM0 will attempt to grab all RAM and make it where no virtuals can exist. There may be other reasons to limit the amount of memory available to the DOM0, though I generally like to give it a bunch so it can cache things.

==== DOM0 and UEFI ====

I have run into some serious issues trying to get a DOM0 to boot with UEFI. A fresh Devuan (Debian) install works just fine, then when the Xen kernel is brought up, the boot fails. I don't remember the exact issue, but it has been around for several years.

The article at [[https://wiki.xenproject.org/wiki/Xen_EFI]] describes how to build a custom kernel.

However, I my fix is to simply set the server to boot into legacy BIOS mode. As long as that is an option, it appears to be better for the time being.

===== Adding iSCSI and NFS =====

For some of our systems, we are using an iSCSI target for the device back ends (vbd's). We also put common configuration files on an NFS share, then mount it at /etc/xen/iscsi_configs. This allows us to migrate virtuals (DOMU's) rapidly between the Xen Hypervisors.

Install the packages for the client (aka initiator for iSCSI)
<code bash>
apt -y install open-iscsi nfs-common
</code>

And, change /etc/iscsi/iscsid.conf to automatically start on boot. Default is to do manual startup, so uncomment automatic and comment node.startup = manual
<file - iscsid.conf>
# To request that the iscsi initd scripts startup a session set to "automatic".
node.startup = automatic
#
# To manually startup the session set to "manual". The default is manual.
# node.startup = manual
</file>

==== set up iSCSI targets ====

This is a manual operation, but there is a script that will do it automagically for you, if you like. See
[[unix:linux:iscsi_tricks_and_techniques|]]

==== set up NFS mounts ====
Now, create an entry in /etc/fstab to automount /etc/xen/iscsi_configs
<code bash>
mkdir -p /etc/xen/iscsi_configs
echo 'ip.of.iscsi.target:/media/nfs_root/xen/configs /etc/xen/iscsi_configs nfs defaults 0 0' >> /etc/fstab
mount /etc/xen/iscsi_configs
</code>

=== Real Life ===

Honestly, what we do is a little more complex on the nfs mounts. We have configurations that may be different depending on which hypervisor they are one. Sometimes, when upgrading a Hypervisor, the configuration will need to be changed.

We also like to have several utilities shared between all of our DOM0's. Installers for various operating systems, qparted to grow a partition, and systemrescuecd for just about everything.

Finally, sometimes we want to use an FBD (File Backed Device) off an nfs mount for some DOMU's. Not as efficient, but a decent quick and dirty if we need it for some reason.

For this, we actually create three nfs exports:
xen-configs - stores configs for virtuals in a tree, depending on the type needed.
xen-images - holds FBD's
xen-store - ISO's, scripts anything that we may need for managing our work

We mount those in /media/whatever (I just call it the name of the export) on each DOM0. For the configs, I then create a symbolic link from /media/xen-configs/4.9 to /etc/xen/iscsi_configs, with the 4.9 being the version of Xen the configuration is correct for.


===== Links =====

  - https://wiki.xen.org/wiki/Xen_Project_Best_Practices
  - https://wiki.debian.org/Xen