Boot Linux from USB Thumbdrive


Problem:

You want to use a USB drive to boot your Xen system, but USB drives do not have long lifespans (around a million writes). In order to do this, you need to make the file system as static as possible.

The good side is, USB drives are inexpensive, and they are quite easy to replicate. It also removes the operating system from the use of the server, namely to host virtual machines.

One 8G USB pendrive is more than sufficient and, if it has wear leveling, will give you a good lifetime (SWAG of 5 years). Once the device is set up the way you want it, shut down the machine and make a copy of it, which you can then copy to another pendrive in case of failure.

We will store static files on the pendrive itself, and put as many dynamic files into RAM disks (uses less than 512M of RAM)

Preformat USB drive

Most installers do not give us enough options, so we will preformat the drive with EXT4, but with no journaling.

# zero MBR to get rid of previous boot issues
dd if=/dev/zero of=/dev/sdx bs=2048 count=1
# partion the drive
# use Linux (83) as the type and make it bootable
fdisk /dev/sdx
# format partition as ext4 but without journaling
mkfs.ext4 -O ^has_journal -L dom0_root /dev/sdx1
  • Do Not install any additional media in your machine at this point. Don't install the other disk drives (if any). The Debian installer can be really stupid at times, and it will put grub on the other media (instead of the USB)
  • now, do the installation as per your distro. Select /dev/sdx1 as your installation location, but do not allow the installer reformat the drive. You want the EXT4 to be as is, without journaling. Also, don't create a swap space. Debian will warn you, but allow it.
  • Insert whatever additional media you may want to use.
  • Most likely you did not install mdadm or lvm2 (both of which I use with xen). You can add them with
apt-get install mdadm lvm2
  • do the full install, including xen, before continuing. Some of the following steps assume Xen is installed
  • Set up RAID and LVM if you are going to use them.
  • If you want swap space (good idea), put it on some other media. Do NOT put swap on the USB.
    • Execute the following commands. The examples assumes you are creating a 2 Gigabyte Logical Volume in the volume group virtuals and using it for your swap space. Adjust for however you are setting up the system.
      lvcreate -L 2G -n swap virtuals
      mkswap /dev/virtuals/swap
      swapon /dev/virtuals/swap
      
    • edit /etc/fstab and add the following line:
      /dev/virtuals/swap  none swap sw 0 0
      
    • reboot the machine, then issue the command
      swapon -s
      
      You should see the swap space listed if everything went well.
  • reboot into new system when done

Tweak directories to minimize writes

# Remove atime from root partition by editing /etc/fstab and finding the root entry. Add noatime,nodiratime to mount parameters. Example:

UUID=<some_long_uuid_number> / ext4 errors=remount-ro,noatime,nodiratime 0 1

# while still in fstab, add ramdisk entries for var/log and /var/tmp and /tmp

tmpfs /var/log.new tmpfs rw,noexec,size=64M 0 0
aufs /var/log aufs br=/var/log.new:/var/log.setup=ro 0 0
tmpfs /tmp tmpfs rw,size=256M 0 0
tmpfs /var/tmp tmpfs rw,size=64M 0 0
tmpfs /var/lib/xen rw,size=64M 0 0

# this creates /var/log as a AnotherUnionFS mount doubly mounted to /var/log.setup and /var/log.new

# exit fstab

# use the new ramfs for /tmp and /var/tmp

rm -r /tmp
mkdir /tmp
mount /tmp
rm -r /var/tmp
mkdir /var/tmp
mount /var/tmp
# set up on /var/log is slightly different since it needs some subdirectories
# pre-defined (especially for apt) and create /var/log.setup and /var/log/new
# as follows
# copy directory structure of /var/log
cd /var/log
mkdir /var/log.setup
find . -type d | cpio -pdvm /var/log.setup
mkdir /var/log.new
mount /var/log.new
# move old var/log and mount new setup in it
mv /var/log /var/log.orig
mkdir /var/log
mount /var/log
/etc/init.d/rsyslog restart
# fix permission on /var/log (it is mounted as rwxrwxrwt which will not work) # Since it takes its permissions from /var/log.new, change those also chmod go-w /var/log.new chmod go-w /var/log

for now, we'll tweak logrotate to only keep two days worth of logs. edit /etc/logrotate.d/rsyslog and make the following changes, basically changing all rotates to 2 and changing var/log/messages to rotate daily:

rotate 2
...
/var/log/messages
{
   rotate 2
   daily

Now, your logs will rotate daily and "fall off" after the second one.

Note: logrotate will not work if the permissions on /var/log are off; it must not be writeable by anyone but root. However, when you mount the tmpfs (ramfs, or ramdisk), it sets it up as rwxrwxrwt, which is bad. To fix this, do the chmod go-w /var/log above. I do not know if that will survive reboot (haven't tested yet).

# Xen uses something else, and I'm not sure what. However, I use the following script in /etc/cron.daily to remove some of the older files

#! /bin/bash
if ( /var/log/xen/*.[0123456789] )
then
   rm /var/log/xen/*.[0123456789] ;
fi

# Running this will remove the older qemu files.

 

Centralized Loggging

Basically, you now have a volatile /var/log, which is ok except that on reboot you lose all of it. The optimum solution here is if you have a dedicated syslog server that you can point your syslogs to you can make that a virtual on this machine or, better still, on another machine (or a dedicated server someplace).

Here is how to set up a server for that task (works for both virtual and stand alone)

NOTE: I will speak of client (the machine booting off the pendrive) and server (a physical or virtual rsyslog server)

# On Server, edit /etc/rsyslog.conf and add the following lines

# ====================================================================
# provides support for local system logging
$ModLoad imuxsock 

# provides kernel logging support (previously done by rklogd)
$ModLoad imklog

# provides UDP syslog reception. For TCP, load imtcp.
$ModLoad imudp

# For TCP, InputServerRun 514
$UDPServerRun 514

# This one is the template to generate the log filename dynamically, depending on the client's IP address.
$template FILENAME,"/var/log/%fromhost-ip%/syslog.log"

# Log all messages to the dynamically formed file. Now each clients log
# (192.168.1.2, 192.168.1.3,etc...), will be under a separate directory which is formed by
# the template FILENAME.
# ====================================================================
# restart rsyslog
service rsyslog restart

# On the client, add the following logs to /etc/rsyslog.conf.

# CHANGE 192.168.1.1 to the IP of the central logging server

# ====================================================================
$ModLoad imuxsock

$ModLoad imklog

# Provides UDP forwarding. The IP is the server's IP address
*.* @192.168.1.1:514 

# Provides TCP forwarding. But the current server runs on UDP
# *.* @@192.168.1.1:514
# ====================================================================
service rsyslog restart

 

# at this point, you should begin seeing systems logs replicated on the central server. They will be in /var/log/IP_ADDRESS/syslog.log

 

Links

Tags: boot, linux, usb
Last update:
2015-08-05 07:49
Author:
Rod
Revision:
1.7
Average rating: 5 (1 Vote)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.

Records in this category

Tags