Linux Server Tech (provided by Daily Data, Inc)

Build a Bridging Firewall NOTE: This is a work in progress as I build my first bridging router. Today is 3 Oct 2010. I have to deploy this thing soon, so it should get finalized fairly soon.

Summary

Requirements

I wanted a bridging (transparent) router for my servers at a local NOC. The main purpose of this would be:

Traffic Shaping
1. Keep within my leased line speeds. If one of my servers was cracked, I would still be able to limit the amount of bandwidth via the router.
2. Allocate priorities to different services. On a scale of 1 (highest) to 10 (lowest)
  1. ssh access: Priority 1 (always allow me to manage the servers even if the network is compromised)
  2. http and https: Priority 2 (main time sensitive service)
  3. imap and pop: Priority 3 (people want to read their mail)
  4. smtp: Priority 5 (keep the mail moving, but not at the expense of web and/or reading e-mail)
  5. ssh file services: Priority 7 (I provide "network drives" but do not want this to impact the real time services)
  6. ftp: Priority 8 (clients uploading large updates to web sites can not interfere with the other services)
  7. backup service: Priority 10 (We provide batched backup service via ssh but it can happen when the network is not busy)
Firewall Protection
1. Stop attacks before they can even get to the servers
2. Block remote sites that consistently misbehave
3. In case a server is compromised, limit amount of damage that can be done by it
Statistics Generation
1. Overall statistics for servers and services
  1. Determine if we are consistently saturating the leased line speeds
  2. Determine if an IP/Port combination is using exceptional resources
  3. View overall trends in bandwidth usage
2. Client Usage -- Determine client machine bandwidth usage and trends

Network

The network this is installed in is described as follows. The IPs and ports have been changed.

Servers are all Xen virtual servers, with multiple DOMU's on each. DOMU's may be moved around for maintenance as needed
Each server has a public IP address, and at least one additional NIC for a private network. The private network allows large file transfers without impacting the public facing interfaces, and also allows some minor security for monitoring functions.
Virtuals have specific purposes. For example, a virtual may be for web hosting, or an SMTP Proxy. Each has a limited number of ports which must be open.
Some virtuals provide services to the rest of the network, ie SMTP services for the web hosting virtuals is handled through dedicated SMTP virtuals. The same with name services.
All appliances (routers, switches, etc...) which have management functions via IP have these interfaces set up on the private network.

Router Definition

Three network interfaces. Two for the bridge, one for management.
Low Power
Secure
Stable
Convenient interface

Hardware

I chose to build this from an Atom based system system with decent memory and an SSD for long term storage. I also wanted an excellent dual NIC for the bridge. I chose the following:

FoxConn R10-S4 Dual Core Aom 330 (http://www.newegg.com/Product/Product.aspx?Item=N82E16856119012)
- .25 amp at max load
- CPU at 1.1% initially
- CPU at xxx% running at 2Mb/s with normal rule set
Intel Pro/1000 MT Dual Port gigabit Server, low profile bracket
- Excellent throughput
- Small internet based operations are under 10Mb/s, so this is running at 1% rated load
1G RAM
- uses less than 100M initially
- xxxM when running under load
Super Talent 16G SSD

Total cost of the system in 2010 was $120 (Foxconn) + $60 (Intel NIC) + $40 (SSD) + $30 (RAM), or $250 plus shipping.

Initial Configuration

I built the machine. Following caveats:

Used the internal NIC for control. It is a RealTech, which is not as good as Intel, for sure. It is also easy to determine which card it is during install (ie, not the Intels). The Intel ports are going to be the bridge, so we don't even have to identify them. They don't care which way traffic flows.
The Intel card is a 64bit, which means it is much longer than the 32bit PCI port in the FoxConn. But, it fits.
FoxConn is a decent little machine, but taking the front cover off ALMOST guarantees breaking a tab. Be careful.
SSD does not fit in the drive bay built into the FoxConn. So, I attached it to the grill behind the front cover. Placed it close to where a standard 3.5" drive would go, but put the screws through the grill to lock it in place. Don't tighten too much; you will strip the holes out of the SSD

Installed Debian Lenny (stable as of this writing). Set up everything via the managment port. Did not do anything with the other ports.

Chose to put everything in one partition. I see no reason to separate the partitions out in this case.
Do a base Debian Lenny install. I chose "base system" at the virtual package, but unchecked everything else.
When done, less than 700M of disk space used, leaving 13G available

Install the extra packages

apt-get install joe sudo ssh ebtables iptables bridge-utils

joe -- my favorite editor, since I'm OLD and remember WordStar
sudo -- keep from logging in as root, and forget the root password
ssh -- Installs the ssh server
ebtables -- filter based on mac addresses
iptables -- filter based on IP
bridge-utils -- bridging utilities

Secure the machine

Secure /tmp

Change /tmp to a ramdisk, and give it good permissions.

joe /etc/fstab

Add the following two lines at the bottom of the file

# 100M /tmp in a ram disk
none /tmp tmpfs rw,noexec,nosuid,nodev,size=100000000 0 0

and restart the computer. This will give you a 100M RAM Disk for /tmp, setting it up securely.

Secure ssh

This is how you will access the router, but if you can access this way, so can black hats. So, we do some things to make it more difficult. Note: if you mess up doing this, you will lock yourself out. I do this via ssh, but I have messed up and had to actually go log in via the keyboard when I have locked my self out. A little "too secure".

We will secure ssh by changing the port from the default to something random, setting up the listen address to our management port only, and shutting down all access except to non-root users with public keys. If you have not created a public/private key, do so now:

 ssh-keygen -t rsa

This will create a public/private key pair on your machine (in ~/.ssh) named id_rsa (private) and id_rsa.pub (public). Copy the public file to your router BEFORE the configuration changes below:

scp .ssh/id_rsa.pub yourname@router_ip_address:
ssh yourname@router_ip_address
# create a directory for your ssh files if one does not exist
mkdir .ssh
# secure that directory
chmod 700 .ssh
# append your public key to the authorized_keys files
cat id_rsa.pub >> .ssh/authorized_keys
# secure that file
chmod 600 .ssh/authorized_keys
# change ownership (it should already be set, but just make sure)
chown -fRv yourname:yourgroup .ssh
# you can now remove the public key from your directory
rm id_rsa.pub

Now, you are ready to test this. Open another terminal and ssh to the router control IP. You will be asked for a password. That is the password you used when you create the rsa key! Enter it, and you will be logged into the router without having to enter your password for the account there.

Change the way ssh works

joe /etc/ssh/sshd_config

Port: change to something other than 22. Make it a high number like 54211 (don't use that, though. make up your own) ListenAddress: Make your private IP address ONLY PermitRootLogin no PasswordAuthentication no (you will need to uncomment) X11Forwarding no

Restart ssh. DO NOT LOG OUT!

/etc/init.d/ssh reload

Open another terminal and try to do an ssh with the new port number:

ssh -p 54211 username@router_ip_address

If you successfully log in, your configuration is golden.

Set up sudo

  visudo

enter the following line:

username ALL=(ALL) ALL

You now can put a really funky password that you can attempt to forget for root. To become root in the future, log in as you, then issue the command sudo su. Enter your account password (hopefully different from your ssh public key password) and you will become root.

Begin building Router

We are now ready to set up the bridge. Your "management port" should become static. In my case, I have a private network that I will be using to access it, so I set it up to be a static on that network. My configuration is as follows, and relies heavily on the article at http://www.annahegedus.com/index.php/Tutorials/bridge-firewall

 eth0  management port -- 192.168.150.1/24
eth1  external port for bridge
eth2  internal port for bridge

joe /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback
# management port. For max security, not in the subnet that bridge will handle
iface eth0 inet static
address 192.168.150.1
netmask 255.255.255.0
# Turn off any attempts to auto start eth1 and eth2
iface eth1 inet manual
iface eth2 inet manual
# create the bridge. Setting it to manual means no IP is assigned
iface br0 inet manual
bridge_ports eth1 eth2
auto br0

Now, either restart the network (/etc/init.d/network reload) or simply reboot the machine. Watch closely for any errors.

If you have made it this far, you now have a functional bridging router. No firewalls, not traffic shaping, and no statistics, but you can plug it into your network and have a transparent router. Plug the management interface into your private range, and bridge your external network to your internal one by connecting your internal to one of the other ports, and your internal to the other.

Note: the first time the bridge comes up, ie the first time a machine tries to use the bridge, it will be slow! The bridge is learning where everything is. After that inital run, things go just fine (<1ms). You can see this by issuing the following commands.

brctl show br0 # shows the bridge, and which NIC's actually make it up
brctl showmacs br0 # shows all the local and remote MAC addresses the bridge knows about.

Feel free to install a monitoring package so you can see traffic actually moving through. I use vnstat (very low resources) and set it up as follows:

apt-get install vnstat
vnstat -u -i eth1
vnstat -u -i eth2

which will begin monitoring the network traffic on both NIC's. At a later date, you can issue the command

vnstat

and see both NIC's traffic, reversed (ie, what is incoming for eth1 is outgoing for eth2).

After you are through testing, you can remove vnstat via

apt-get purge vnstat
rm -fRv /var/lib/vnstat

Alternate two NIC version

You can build this with only two NIC's, and have an administrative interface that is available only after you have logged into one of the servers you are protecting. To do this, you simply assign a public IP address to the bridge. You would then ssh to one of the other IP's, then ssh from there to the router IP address.

Basically, you remove the definition for eth0 above, then create your bridge as follows:

# Turn off any attempts to auto start eth1 and eth2
iface eth1 inet manual
iface eth2 inet manual
# create the bridge. Setting it to manual means no IP is assigned
iface br0 inet static
address some_public_ip
netmask the_public_range_netmask
# you will insert firewall/traffic shaping start/stop scripts here, see next sections
bridge_ports eth1 eth2
auto br0

Configuring Firewall and Traffic Shaping

There is a 2003 document that is the "Bible" of IPTables and Traffic Shaping, "Linux Advanced Routing & Traffic Control HOWTO" available from http://lartc.org/. Download this 158 page document and read it before you try this section. You do not ned to understand the whole thing, but you should familiarize yourself with its contents so you are not totally lost.

There are people who can sit and write IPTables firewalls in a text editor, but I am not one of those. I went to fwBuilder http://www.fwbuilder.org to create my firewall. This is available at no charge to Linux workstations, and at a reasonable price for Windows and OSX. Most Linux distributions keep it in their repositories. However, two additional Firewall tools of interest are Firestarter (http://www.fs-security.com/) and Mason (http://www.stearns.org/mason/). Firestarter is actually designed to build a firewall on your workstation or server via a GUI, not to design a firewall for another system. Mason, is interesting in that it will watch network traffic and build a script from it. I will likely install Mason on the firewall after the fact so I can tune the firewall I have built with fwBuilder, but decided not to do it at this point. Mason is a command line tool which operates by reading a log file.

Of course, there is always shorewall (http://www.shorewall.net/). In my case, I decided not to use it, but it is widely used, and very powerful with constant updates. It will require a decent learning curve, but it will (from what I've read) handle the firewall and traffic shaping, and there are tons of documentation.

You still need to understand a little about IPTables to use any of these tools, but they definitely help you create the rules. fwBuilder generates a script, suitable for installing in /etc/init.d. This script should be executed whenever br0 comes up. There are several possibilities for this, but I will mention only two. Under Debian, I simply add pre-up and post-down commands that turn on and off the firewall when br0 is brought up or down. In the interfaces example above, you would put them in the section that states you will insert firewall/traffic shaping start/stop scripts here, see next sections

The following assumes you either hand built your firewall (I bow to you, guru) or used fwBuilder. Shorewall builders are on their own until I try it and write something up.

One Note: vuurmuur ([http://vuurmuur.org is a very, very nice curses based iptables front end, and it will even do traffic shaping. However, it does not know about bridges :( However, it is definitely worth knowing about.

Firewall

Create Rules

Build your firewall. This is not necessary, but why waste a perfectly good firewall/router with no rules. I have always built my servers with no firewalls, relying on ensuring the only services running are the required ones, and weekly maintenance to ensure I have security fixes in place (plus, monitoring the appropriate mailing lists). But, a firewall with good rules gives you that added layer for those times when you forget and leave mySQL available to the outside world.

My firewall setup is pretty basic. I have about a dozen "servers" (mixes of Xen virtuals and stand alone machines) in my Colo, and none are general purpose. I will use a smaller example below, though. The IP's are made up; I'll use the private range 192.168.168.1-16 for my example. Also, I always change my ssh ports to something other than the default.

Everything needs access via ssh. I set them all up to use port 22222 for ssh, my first rule is to allow 22222 on all addresses in the net, ie 192.168.168.1-16
Mail proxy server needs ports 25 (SMTP), 465 (SSMPT), and port 25252 for my web interface to the proxy. Additionally, I opened alternate port 2525 so clients who are on networks blocked for port 25 can us it to connect.
DNS requires port 53
Web Server needs ports 80 (http), 443 (https), and 20-21 (ftp, if you allow ftp connections)
Mail servers require 143 (imap), 993 (imaps), 110 (pop) and 995 (pops) to be open. Since they will communicate with the Mail Proxy over local addresses, SMTP does not need to be opened. However, webmail requires 443 (https) to be open also.
I have an scp server that allows files to be transferred over port 22221, so it needs that port open.

Here are my sample servers

Xen DOM0 Servers at 192.168.168.15 and 16
1. nothing but the default ssh access. If I want vnc, I do port forwarding on the ssh, ie ssh -L localhost:5901:localhost:5900 192.168.168.15
Mail Proxy Server at 192.168.168.2
1. Rule #2 above, ie SMTP and management ports only
DNS server at 192.168.168.3
1. Open the DNS rules
Router "temporary" IP at 192.168.168.1
1. Nothing but the default
Web Server at 192.168.168.4
1. Rule 4 above, ie http, https, and ftp
Web/e-mail server at 192.168.168.5
1. Rule 4 provides web services
2. Rule 5 provides mail services
e-mail only server at 192.168.168.6
1. Rule 5 only, ie imap, POP and https
"File" server at 192.168.168.7
1. scp "special" port, 22221. Note that this requires a separate Port 22221 entry in /etc/ssh/sshd_config (you include both lines) so ssh listens on both ports.

Deploy Firewall

Create the rules, and copy them to your router.

The simplest way to do this is to copy the output of fwBuilder to /etc/init.d/firewall, and replace the line with the following two commands.

pre-up /etc/init.d/firewall start
post-down /etc/init.d/firewall stop

This is good, but if you will be doing traffic shaping also, it might be simpler to start your firewall, then take the rules and store them in a file using iptables-save. You can then load the rules necessary for both traffic shaping and your firewall with one simple command.

Note: REJECT will not work unless there is an IP address, so I use DROP when creating my rules.

"While it is not a requirement to give the bridge an IP address, doing so allows the bridge/firewall to access other systems and allows the bridge/firewall to be managed remotely. The bridge must also have an IP address for REJECT rules and policies to work correctly â€” otherwise REJECT behaves the same as DROP. It is also a requirement for bridges to have an IP address if they are part of a bridge/router." (http://www.shorewall.net/bridge-Shorewall-perl.html)

First, get your firewall set up (probably by executing the fwbuilder script), then execute the following command:

./firewall start
iptables-save > /etc/network/iptables.up.rules

Now, create an "accept all" rule set by issuing the following commands

./firewall stop
iptables-save > /etc/network/iptables.down.rules

Your entries in /etc/network/interfaces then become the following two lines

pre-up iptables-restore < /etc/network/iptables.up.rules
post-down iptables-restore < /etc/network/iptables.down.rules

You should now be able to see your firewall go up and down with your interface. Verify this with:

ifdown br0
iptables -L

and you will see an accept all policy. Conversely, issuing:

ifup br0 # will take a few seconds; will seem to hang
iptables -L

would show you a fully populated firewall

Caution Messing with a firewall remotely is dangerous. I have successfully locked myself out of servers and routers by creating rule sets that kill my network connection, necessitating a panic trip to the NOC to get the machine/network back online. Do the following before messing with an untested firewall.

Create a file /etc/cron.d/testFirewall with the following contents

# /etc/cron.d/testFirewall: Uncomment following before adding a new firewall
# Will clear the firewall every five minutes.
# uncomment if you have a clean (accept all) iptables save file in /etc/network/iptables.down.rules
#0-55/5 *	* * *	root	iptables-restore < /etc/network/iptables.down.rules
# uncomment for a script 'firewall' in /etc/init.d
#0-55/5 *	* * *	root	/etc/init.d/firewall stop

Before installing the new firewall rules, uncomment the appropriate line in the script above. This will drop the firewall every 5 minutes, giving it an Accept All status (ie, no firewall). This will give you a few minutes to test the firewall, but if you lock yourself out, you can wait five minutes, log back in, then fix whatever you messed up. Much, much better than a 30 minute drive through the snow at 3am.

Traffic Shaping

Traffic shaping allows you to define traffic limits through the main interface and, optionally, traffic based on target and port. This is the main reason I decided to build a router, though. My servers are located at a Colocation facility, and I have a certain bandwidth built into my contract. If I exceed that bandwidth, I have to pay extra money. Of course, if my clients require the additional bandwidth, I will purchase it, but I would rather have no surprises; I would like it to be planned.

The basic configuration is rather straight forward, using only the command tc. In this case, we actually deal with the true interfaces, the ones that make up our bridge. And, we need to know which one is internal and which one is external

  #!/bin/bash
# script used to set up and down speeds on a bridge
# much of this was taken from 
# http://atmail.com/kb/2009/throttling-bandwidth/
# and modified by RWR. See original for single interface
# version.
# Set up for Debian. Does no checks for modules installed
# correctly.
#
#  tc uses the following units when passed as a parameter.
#  kbps: Kilobytes per second
#  mbps: Megabytes per second
#  kbit: Kilobits per second
#  mbit: Megabits per second
#  bps: Bytes per second
#
# NOTE: this is set up to handle a bridge. In most cases, we
# don't care which eth device is on which side of the bridge.
# However, for traffic shaping, we are at a lower level than
# the bridge and must connect directly to the interfaces themselves.
####################################################################
# Edit the following for your system
####################################################################
# TC command
TC=/sbin/tc
# interface that is connected to the 'net.
EXTERNAL_IF=eth1
# interface connected to your servers
INTERNAL_IF=eth2
# limit download speed (incoming) to this value
INCOMING_SPEED=1mbit
# limit upload speed (outgoing) to this value
OUTGOING_SPEED=256kbit
# define the network we are dealing with
NETWORK=10.111.111.0/24
####################################################################
# End of editable values
####################################################################
start() {
# We'll use Hierarchical Token Bucket (HTB) to shape bandwidth.
# In each section, The first line creates the root qdisc, and 
# the next line creates a child qdisc used to shape bandwidth 
#
# The last line creates the filter to match the interface.
# The 'dst' IP address is used to limit download speed, and the  
# 'src' IP address is used to limit upload speed.
# first, set up upload speeds
$TC qdisc add dev $EXTERNAL_IF root handle 1:0 htb default 30
$TC class add dev $EXTERNAL_IF parent 1: classid 1:1 htb rate $OUTGOING_SPEED
$TC filter add dev $EXTERNAL_IF protocol ip parent 1:0 prio 1 u32 match ip src $NETWORK flowid 1:1
# now, the download speed. 
$TC qdisc add dev $INTERNAL_IF root handle 1:0 htb default 30
$TC class add dev $INTERNAL_IF parent 1: classid 1:1 htb rate $INCOMING_SPEED
$TC filter add dev $INTERNAL_IF protocol ip parent 1:0 prio 1 u32 match ip dst $NETWORK flowid 1:1
}
stop() {
# Stop the bandwidth shaping.
$TC qdisc del dev $EXTERNAL_IF root
$TC qdisc del dev $INTERNAL_IF root
}
restart() {
# Self-explanatory.
stop
sleep 1
start  
}
show() {
# Display status of traffic control status.
echo $EXTERNAL_IF
$TC -s qdisc ls dev $EXTERNAL_IF
echo $INTERNAL_IF
$TC -s qdisc ls dev $INTERNAL_IF
}
case "$1" in
start)
echo -n "Starting bandwidth shaping: "
start
echo "done"
;;
stop)
echo -n "Stopping bandwidth shaping: "
stop
echo "done"
;;
restart)
echo -n "Restarting bandwidth shaping: "
restart
echo "done"
;;
show)
echo "Bandwidth shaping status:"
show
echo ""
;;
*)
pwd=$(pwd)
echo "Usage: $0 {start|stop|restart|show}"
;;
esac
exit 0

Bringing up an external connection temporarily

Since the bridge has no IP address, it is too secure to do some common tasks, like updating the operating system! You can give it an IP and route temporarily, then remove the connection when done. Here is a short script to update the operating system (assumes Debian)

#! /bin/bash
INTERFACE=br0            # the bridge we built
IP=10.111.111.254        # the IP we want to give it
MASK=255.255.255.0       # network mask
NETWORK=10.111.111.0     # the "local" network
GW=10.111.111.1          # the gateway
# create an address and route for the interface
ifconfig $INTERFACE $IP netmask $MASK
route add -net $NETWORK netmask $MASK $INTERFACE
route add -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE
# Do the work
apt-get update  # get new packages from apt
# download the packages, but do not install. Remove --download-only if you want the updates performed automagically
apt-get --download-only --assume-yes upgrade
# now, shut down the interface. Remove routing entries
route del -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE
route del -net $NETWORK netmask $MASK $INTERFACE
# and unconfigure the interface
ifconfig $INTERFACE 0.0.0.0
echo $INTERFACE unconfigured.

When you have done this, you can then perform

apt-get --no-download upgrade

from a command line to do the actual upgrades

Obviously, this will not work for interactive updates. For that, simply populate the interface manually. The following script allows you to populate/depopulate the bridge:

  #! /bin/bash
INTERFACE=br0            # the bridge we built
IP=10.111.111.254        # the IP we want to give it
MASK=255.255.255.0       # network mask
NETWORK=10.111.111.0     # the "local" network
GW=10.111.111.1          # the gateway
CMD=$1
case "$CMD" in
start)
# create an address and route for the interface
ifconfig $INTERFACE $IP netmask $MASK
route add -net $NETWORK netmask $MASK $INTERFACE
route add -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE
RETVAL=$?
echo $INTERFACE now has an IP of $IP
;;
stop)
route del -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE
route del -net $NETWORK netmask $MASK $INTERFACE
# and unconfigure the interface
ifconfig $INTERFACE 0.0.0.0
RETVAL=$?
echo $INTERFACE unconfigured.
;;
status)
ifconfig $INTERFACE
route -n
RETVAL=$?
;;
*)
echo Usage $0 start stop status
;;
esac
exit $RETVAL

References

I read a lot of information on the net, and benefited greatly from the research of others. Following is a list of web sites I used in researching and building this router. Also, thanks to Loren Schweitzer and David North for their invaluable assistance, mainly making me aware when I was venturing down a fruitless path.