Building an iSCSI target is a pain. It took me quite a while to figure it out, but I wanted total control over it. I made some mistakes along the way, but this document is how I ended up doing it.
Actually, not quite. I am adding DRBD to it so I can have a little cluster, so one machine can fail and I still have everything. You've heard of Redundant Array of Inexpensive Disks (RAID)? I like Redundant Array of Inexpensive Servers (RAIS). As I told one client in early 2015, why spend $5k on a server when you can get one a generation old for $1000. At that price, buy two and keep one for backup.
This article does not cover DRBD installation, which I want for redundancy. However, it does cover a basic iSCSI target (the "server") and only in one place would things need to be changed to set it up for DRBD.
I use the machines described as backing for a small Xen farm. I can have multiple DOM0's, and with all of the DOMU's iSCSI backed, I can migrate them from one machine to another with sub-second downtime. If that doesn't mean anything to you, ignore it. But, I see a lot of people using NFS exports for that, but NFS just doesn't handle as much disk access as we do at the NOC, so I chose iSCSI.
We do use NFS also. NFS is used for things that are shared between machines; a storage for the Xen configuration files, a quick and dirty place to build a low resource machine temporarily, etc... So, you will see this iSCSI Target is actually an iSCSI Target + NFS server. And, it could have other things added to it.
Once you have your target built, it will be time to build an initiator so you can use some of this. See Creating an iSCSI Initiator for information on that (and, how to attach from your workstation or whatever). NOTE: Microsoft windows, Apple OSX and Linux all support iSCSI as described here.
First, you should plan. How much space do you need, and what kind of underlying disk structure do you want. An IET iSCSI target device can use many things for its underlying structure; basically anything you can point to with in the /dev tree that is a block device. From the current list I see in webmin, you can use:
- Local Disk Partition
- RAID Device
- LVM logical volume
- Temporary RAM disk
- Other file or device
My choice, reflected in this document, is to build a RAID set, then put LVM2 on top of it.
Your disks and your memory are very important (cpu not so much). Remember, what you want is the fastest access to the hard drives as you can have. A RAID-0 would be optimum, but there is no redundancy, so I use RAID-6, which allows me to lose any two drives and still have my raid set readable. It is slower, but reliability is very important also. Remember, in theory, the more drives you read from, the faster things are. So, a 4 drive RAID-6 with 4T drives is going to be slower than an 8 drive RAID-6 with 1T drives, assuming the drives are on separate controllers and they are both the same "speed". So, more, smaller drives in a RAID-5 or 6 is faster than just one or two big drives.
I realize putting LVM2 on top of it, especially since I actually put two layers, slows things down. However, having the ability to dynamically resize my targets is worth it.
Memory is very important also. When you are reading from or writing to your targets, your target device can store things in memory temporarily (caching) and greatly speed up the appearant access to the disk. I have been told "4G is more than enough, 8G is overkill" and things like that. However, one of my iSCSI servers has 32G of RAM and it appears to be faster than its 16G brother. RAM is always full, most of it dedicated to caching. So, get as much as you can reasonably afford.
This is a real world system. I had some old HP DL-380G5's that I wanted to retire from actually serving things like web sites and stuff. Not as much RAM, cpu, or NIC's as I wanted. So, I pulled one in, replaced the BIOS battery, cleaned it out real good (remember the fans, remember the fans), and it made an excellent little iSCSI server.
I built it to boot from a thumbdrive (see Boot Linux from USB Thumbdrive). And, I build it minimal. Nothing that doesn't need to be there.
Now, purchase 8 1T hard drives. Using RAID-6, that will give us slightly less than 6T of usable space. I like the 7200 rpm drives from hitachi (see NewEgg) as they are pretty fast and very cheap. Cheap enough I can buy some extras to leave on top of the server for rapid replacement. The Western Digital "Intelli Drives" just seem to slow down too late in the game. Of course, if you have the money, just buy 8 SSD's and be done with it.
Oh, the DL380G5's have only hardware RAID. This really stinks. I just make all 8 individual RAID-0's. Total pain in the rear, but I have a backup machine if I really need to rebuild the whole thing.
Now, you have a very basic Linux installed on something (I use Debian Wheezy, not sure I'll go to Jessie as System-d really is causing problems on the few machines I built), and you have your 8 hard drives set up and ready to go. Do the following to get them set up.
# get the two packages we need right now apt-get install mdadm lvm2 # create RAID-6 software RAID consisting of your eight hard drives. # HP decided to do their own thing, so they are called /dev/cciss/c#d#, # where the first pound sign is the controller and the second is the # device on that controller. So, the first drive is cciss/c0d0, second # is cciss/c0d1, etc... mdadm --create /dev/md0 --raid-devices=8 --spare-devices=0 --level=6 --run /dev/cciss/c0d # now, add it (and all others) to the mdadm configuration file cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.save /usr/share/mdadm/mkconf --generate > /etc/mdadm/mdadm.conf # add LVM volume group on top of it pvcreate /dev/md0 vgcreate vg-md0 /dev/md0
At this point, we have a very large Volume Group, with nothing in it. We could simply create LV's in that and export them, but I actually have other things I'd like in here also, so I'm going to build another LVM layer into this. My reasons are simple; I want to use DRBD to sync only the iSCSI area to another machine, and I want it to be magic; I don't want to have to remember to do this every time I make a modification.
You definitely do NOT have to do this, and it will remove a whole layer between you and the hard drive if you don't. Your system will be faster if you do not do this step. If you do not, simply create all your iSCSI exports in the volume group vg-md0.
What I'll do, however, is create a new LV named iscsi-exports, make it a Physical Volume for LVM2, then create a new Volume Group named iscsi-export-pool and all my exports will come from there.
lvcreate -L 2T -n iscsi-exports vg-md0
vgcreate iscsi-export-pool /dev/vg-md0/iscsi-exports
At this point, we have a 2 Terabyte place to create iSCSI targets from. Adjust to suit your needs. Remember, however, it is easier to grow than it is to shrink.
Now, I also want to export an NFS volume, so I'll create it out of vg-md0 and set it up to be exported. I mount it in /media/nfs
# this is even smaller, as all we want to do is have some NFS space # available to the machines for needs that are not so efficient lvcreate -L 150G -n nfs-exports vg-md0 mkfs.ext4 -m0 -L nfs-exports /dev/vg-md0/nfs-exports
cp /etc/fstab /etc/fstab.save
cat '/dev/vg-md0/nfs-exports /media/nfs ext4 defaults 0 2' >> /etc/fstab
I want three packages now. The first, webmin (http://www.webmin.com/) is not in the standard Debian repository so I have to get it from them. A note about webmin. It is not very secure. It is a web based management tool for your machine, and anyone that can crack it owns your server. I set the iSCSI server on it's own vlan (and LAN), and that is behind about 4 layers of security, so I take the risk. However, even so, I stop it (/etc/init.d/webmin stop) when I'm not using it, and I start it back up when I want to do something with it (/etc/init.d/webmin start). If a black hat can make it through all those levels, I'm pretty well screwed anyway.
It is a nice, clean, simple interface to some complex packages. It knows IET, so I use it as a crutch so I don't have to learn all the intricacies. I'm pretty good with LVM2, so I do that from the CLI, but webmin can help you there also if you want. It can even build your RAID sets for you.
So, first, I'll install webmin. Read the instructions at http://www.webmin.com/ for the latest information. The following is modified from http://www.webmin.com/deb.html. I have not tested it on jessie with it's System-D abortion.
echo 'deb http://download.webmin.com/download/repository sarge contrib' > /etc/apt/sources.list.d/webmin.list echo 'deb http://webmin.mirror.somersettechsolutions.co.uk/repository sarge contrib' >> /etc/apt/sources.list.d/webmin.list cd /tmp wget http://www.webmin.com/jcameron-key.asc apt-key add jcameron-key.asc apt-get update apt-get install webmin
Now, install IET and NFS
# install iscsitarget # this will automatically build the kernel module and install it also apt-get install iscsitarget iscsitarget-dkms nfs-common nfs-kernel-server
Let's do the easy part first. We'll create an NFS export and let set it up. We'll call it temp
mkdir /media/nfs/temp echo '/media/nfs/temp 10.19.209.0/24(rw,sync,no_subtree_check,no_root_squash)'>>/etc/exports /etc/init.d/nfs-kernel-server reload
You can now mount this on a remote machine in your internal network with the fstab entry
# do this on the client machine, NOT on your iSCSI server
echo '10.19.209.2:/media/nfs/temp /media/temp nfs defaults 0 0' >> /etc/fstab mkdir /media/temp mount /media/temp
Now, the harder one. Lets do an iSCSI export. Remember, I'm using the "lvm within an lvm" but you do not have to. Adjust as you desire.
# First, edit /etc/default/iscsitarget to enable !!!
# Just set ISCITARGET_ENABLE= true
# if you have multiple IP ranges on this machine, you can also add
# OPTIONS='-a 10.19.209.2' so it will only listen on that NIC
/etc/init.d/iscsitarget start # verify iscsi working
grep -i iscsi /var/log/messages
# OUTPUT Jan 1 15:26:58 serenity kernel: [11767.027654] iSCSI Enterprise Target Software - version 126.96.36.199 # OUTPUT Jan 1 15:26:58 serenity kernel: [11767.027719] iscsi_trgt: Registered io type fileio # OUTPUT Jan 1 15:26:58 serenity kernel: [11767.027721] iscsi_trgt: Registered io type blockio # OUTPUT Jan 1 15:26:58 serenity kernel: [11767.027723] iscsi_trgt: Registered io type nullio #
# Create your first iSCSI export
lvcreate -L 10G -n test iscsi_export_pool # look at the iet conf directory to see the files created
# OUTPUT ietd.conf initiators.allow targets.allow
# build a target # see http://en.wikipedia.org/wiki/ISCSI#Addressing for naming # create new target, id=1 name=iqn.2009-02.com.example:test ietadm --op new --tid=1 --params Name=iqn.2009-02.com.example:test # add LUN ID 0 to target and assign to LVM volume as backing store. Set access as fileio ietadm --op new --tid=1 --lun=0 --params Type=fileio,Path=/dev/iscsi-export-pool/test # verify creation cat /proc/net/iet/volume # OUTPUT tid:1 name:iqn.2009-02.com.example:test # OUTPUT lun:0 state:0 iotype:fileio iomode:wt path:/dev/iscsi-export-pool/test # Add to /etc/iet/ietd.conf so it will be there on restart
echo 'Target iqn.2009-02.com.example:test' >> /etc/iet/ietd.conf
echo ' Lun 0 Path=/dev/iscsi-export-pool/test,Type=fileio' >> /etc/iet/ietd.conf
echo ' Alias test' >> /etc/iet/ietd.conf # restart daemon and show that target re-appears /etc/init.d/iscsitarget restart cat /proc/net/iet/volume
You should now see it as an export with the following command
Well, you need something to connect to this. For Linux, read Creating an iSCSI Initiator
iSCSI appliance distros
I don't use these because I'm a control freak. However, if you want a way to quickly create an iSCSI, Samba, NFS, AFS and whatever else in the world you want file server, check out the following.
- openmediavault http://openmediavault.org/
- freenas http://www.freenas.org/
- openfiler http://www.openfiler.com/