Build a Bridging Firewall NOTE: This is a work in progress as I build my first bridging router. Today is 3 Oct 2010. I have to deploy this thing soon, so it should get finalized fairly soon.
Summary
Requirements
I wanted a bridging (transparent) router for my servers at a local NOC. The main purpose of this would be:
- Traffic Shaping
- Keep within my leased line speeds. If one of my servers was cracked, I would still be able to limit the amount of bandwidth via the router.
- Allocate priorities to different services. On a scale of 1 (highest) to 10 (lowest)
- ssh access: Priority 1 (always allow me to manage the servers even if the network is compromised)
- http and https: Priority 2 (main time sensitive service)
- imap and pop: Priority 3 (people want to read their mail)
- smtp: Priority 5 (keep the mail moving, but not at the expense of web and/or reading e-mail)
- ssh file services: Priority 7 (I provide "network drives" but do not want this to impact the real time services)
- ftp: Priority 8 (clients uploading large updates to web sites can not interfere with the other services)
- backup service: Priority 10 (We provide batched backup service via ssh but it can happen when the network is not busy)
- Firewall Protection
- Stop attacks before they can even get to the servers
- Block remote sites that consistently misbehave
- In case a server is compromised, limit amount of damage that can be done by it
- Statistics Generation
- Overall statistics for servers and services
- Determine if we are consistently saturating the leased line speeds
- Determine if an IP/Port combination is using exceptional resources
- View overall trends in bandwidth usage
- Client Usage -- Determine client machine bandwidth usage and trends
- Overall statistics for servers and services
Network
The network this is installed in is described as follows. The IPs and ports have been changed.
- Servers are all Xen virtual servers, with multiple DOMU's on each. DOMU's may be moved around for maintenance as needed
- Each server has a public IP address, and at least one additional NIC for a private network. The private network allows large file transfers without impacting the public facing interfaces, and also allows some minor security for monitoring functions.
- Virtuals have specific purposes. For example, a virtual may be for web hosting, or an SMTP Proxy. Each has a limited number of ports which must be open.
- Some virtuals provide services to the rest of the network, ie SMTP services for the web hosting virtuals is handled through dedicated SMTP virtuals. The same with name services.
- All appliances (routers, switches, etc...) which have management functions via IP have these interfaces set up on the private network.
Router Definition
- Three network interfaces. Two for the bridge, one for management.
- Low Power
- Secure
- Stable
- Convenient interface
Hardware
I chose to build this from an Atom based system system with decent memory and an SSD for long term storage. I also wanted an excellent dual NIC for the bridge. I chose the following:
- FoxConn R10-S4 Dual Core Aom 330 (http://www.newegg.com/Product/Product.aspx?Item=N82E16856119012)
- .25 amp at max load
- CPU at 1.1% initially
- CPU at xxx% running at 2Mb/s with normal rule set
- Intel Pro/1000 MT Dual Port gigabit Server, low profile bracket
- Excellent throughput
- Small internet based operations are under 10Mb/s, so this is running at 1% rated load
- 1G RAM
- uses less than 100M initially
- xxxM when running under load
- Super Talent 16G SSD
Total cost of the system in 2010 was $120 (Foxconn) + $60 (Intel NIC) + $40 (SSD) + $30 (RAM), or $250 plus shipping.
Initial Configuration
I built the machine. Following caveats:
- Used the internal NIC for control. It is a RealTech, which is not as good as Intel, for sure. It is also easy to determine which card it is during install (ie, not the Intels). The Intel ports are going to be the bridge, so we don't even have to identify them. They don't care which way traffic flows.
- The Intel card is a 64bit, which means it is much longer than the 32bit PCI port in the FoxConn. But, it fits.
- FoxConn is a decent little machine, but taking the front cover off ALMOST guarantees breaking a tab. Be careful.
- SSD does not fit in the drive bay built into the FoxConn. So, I attached it to the grill behind the front cover. Placed it close to where a standard 3.5" drive would go, but put the screws through the grill to lock it in place. Don't tighten too much; you will strip the holes out of the SSD
Installed Debian Lenny (stable as of this writing). Set up everything via the managment port. Did not do anything with the other ports.
- Chose to put everything in one partition. I see no reason to separate the partitions out in this case.
- Do a base Debian Lenny install. I chose "base system" at the virtual package, but unchecked everything else.
- When done, less than 700M of disk space used, leaving 13G available
Install the extra packages
apt-get install joe sudo ssh ebtables iptables bridge-utils
- joe -- my favorite editor, since I'm OLD and remember WordStar
- sudo -- keep from logging in as root, and forget the root password
- ssh -- Installs the ssh server
- ebtables -- filter based on mac addresses
- iptables -- filter based on IP
- bridge-utils -- bridging utilities
Secure the machine
Secure /tmp
Change /tmp to a ramdisk, and give it good permissions.
joe /etc/fstab
Add the following two lines at the bottom of the file
# 100M /tmp in a ram disk none /tmp tmpfs rw,noexec,nosuid,nodev,size=100000000 0 0
and restart the computer. This will give you a 100M RAM Disk for /tmp, setting it up securely.
Secure ssh
This is how you will access the router, but if you can access this way, so can black hats. So, we do some things to make it more difficult. Note: if you mess up doing this, you will lock yourself out. I do this via ssh, but I have messed up and had to actually go log in via the keyboard when I have locked my self out. A little "too secure".
We will secure ssh by changing the port from the default to something random, setting up the listen address to our management port only, and shutting down all access except to non-root users with public keys. If you have not created a public/private key, do so now:
ssh-keygen -t rsa
This will create a public/private key pair on your machine (in ~/.ssh) named id_rsa (private) and id_rsa.pub (public). Copy the public file to your router BEFORE the configuration changes below:
scp .ssh/id_rsa.pub yourname@router_ip_address: ssh yourname@router_ip_address # create a directory for your ssh files if one does not exist mkdir .ssh # secure that directory chmod 700 .ssh # append your public key to the authorized_keys files cat id_rsa.pub >> .ssh/authorized_keys # secure that file chmod 600 .ssh/authorized_keys # change ownership (it should already be set, but just make sure) chown -fRv yourname:yourgroup .ssh # you can now remove the public key from your directory rm id_rsa.pub
Now, you are ready to test this. Open another terminal and ssh to the router control IP. You will be asked for a password. That is the password you used when you create the rsa key! Enter it, and you will be logged into the router without having to enter your password for the account there.
Change the way ssh works
joe /etc/ssh/sshd_config
Port: change to something other than 22. Make it a high number like 54211 (don't use that, though. make up your own) ListenAddress: Make your private IP address ONLY PermitRootLogin no PasswordAuthentication no (you will need to uncomment) X11Forwarding no
Restart ssh. DO NOT LOG OUT!
/etc/init.d/ssh reload
Open another terminal and try to do an ssh with the new port number:
ssh -p 54211 username@router_ip_address
If you successfully log in, your configuration is golden.
Set up sudo
visudo
enter the following line:
username ALL=(ALL) ALL
You now can put a really funky password that you can attempt to forget for root. To become root in the future, log in as you, then issue the command sudo su. Enter your account password (hopefully different from your ssh public key password) and you will become root.
Begin building Router
We are now ready to set up the bridge. Your "management port" should become static. In my case, I have a private network that I will be using to access it, so I set it up to be a static on that network. My configuration is as follows, and relies heavily on the article at http://www.annahegedus.com/index.php/Tutorials/bridge-firewall
eth0 management port -- 192.168.150.1/24 eth1 external port for bridge eth2 internal port for bridge
joe /etc/network/interfaces
# The loopback network interface auto lo iface lo inet loopback # management port. For max security, not in the subnet that bridge will handle iface eth0 inet static address 192.168.150.1 netmask 255.255.255.0 # Turn off any attempts to auto start eth1 and eth2 iface eth1 inet manual iface eth2 inet manual # create the bridge. Setting it to manual means no IP is assigned iface br0 inet manual bridge_ports eth1 eth2 auto br0
Now, either restart the network (/etc/init.d/network reload) or simply reboot the machine. Watch closely for any errors.
If you have made it this far, you now have a functional bridging router. No firewalls, not traffic shaping, and no statistics, but you can plug it into your network and have a transparent router. Plug the management interface into your private range, and bridge your external network to your internal one by connecting your internal to one of the other ports, and your internal to the other.
Note: the first time the bridge comes up, ie the first time a machine tries to use the bridge, it will be slow! The bridge is learning where everything is. After that inital run, things go just fine (<1ms). You can see this by issuing the following commands.
- brctl show br0 # shows the bridge, and which NIC's actually make it up
- brctl showmacs br0 # shows all the local and remote MAC addresses the bridge knows about.
Feel free to install a monitoring package so you can see traffic actually moving through. I use vnstat (very low resources) and set it up as follows:
apt-get install vnstat vnstat -u -i eth1 vnstat -u -i eth2
which will begin monitoring the network traffic on both NIC's. At a later date, you can issue the command
vnstat
and see both NIC's traffic, reversed (ie, what is incoming for eth1 is outgoing for eth2).
After you are through testing, you can remove vnstat via
apt-get purge vnstat rm -fRv /var/lib/vnstat
Alternate two NIC version
You can build this with only two NIC's, and have an administrative interface that is available only after you have logged into one of the servers you are protecting. To do this, you simply assign a public IP address to the bridge. You would then ssh to one of the other IP's, then ssh from there to the router IP address.
Basically, you remove the definition for eth0 above, then create your bridge as follows:
# Turn off any attempts to auto start eth1 and eth2 iface eth1 inet manual iface eth2 inet manual # create the bridge. Setting it to manual means no IP is assigned iface br0 inet static address some_public_ip netmask the_public_range_netmask # you will insert firewall/traffic shaping start/stop scripts here, see next sections bridge_ports eth1 eth2 auto br0
Configuring Firewall and Traffic Shaping
There is a 2003 document that is the "Bible" of IPTables and Traffic Shaping, "Linux Advanced Routing & Traffic Control HOWTO" available from http://lartc.org/. Download this 158 page document and read it before you try this section. You do not ned to understand the whole thing, but you should familiarize yourself with its contents so you are not totally lost.
There are people who can sit and write IPTables firewalls in a text editor, but I am not one of those. I went to fwBuilder http://www.fwbuilder.org to create my firewall. This is available at no charge to Linux workstations, and at a reasonable price for Windows and OSX. Most Linux distributions keep it in their repositories. However, two additional Firewall tools of interest are Firestarter (http://www.fs-security.com/) and Mason (http://www.stearns.org/mason/). Firestarter is actually designed to build a firewall on your workstation or server via a GUI, not to design a firewall for another system. Mason, is interesting in that it will watch network traffic and build a script from it. I will likely install Mason on the firewall after the fact so I can tune the firewall I have built with fwBuilder, but decided not to do it at this point. Mason is a command line tool which operates by reading a log file.
Of course, there is always shorewall (http://www.shorewall.net/). In my case, I decided not to use it, but it is widely used, and very powerful with constant updates. It will require a decent learning curve, but it will (from what I've read) handle the firewall and traffic shaping, and there are tons of documentation.
You still need to understand a little about IPTables to use any of these tools, but they definitely help you create the rules. fwBuilder generates a script, suitable for installing in /etc/init.d. This script should be executed whenever br0 comes up. There are several possibilities for this, but I will mention only two. Under Debian, I simply add pre-up and post-down commands that turn on and off the firewall when br0 is brought up or down. In the interfaces example above, you would put them in the section that states you will insert firewall/traffic shaping start/stop scripts here, see next sections
The following assumes you either hand built your firewall (I bow to you, guru) or used fwBuilder. Shorewall builders are on their own until I try it and write something up.
One Note: vuurmuur ([http://vuurmuur.org is a very, very nice curses based iptables front end, and it will even do traffic shaping. However, it does not know about bridges :( However, it is definitely worth knowing about.
Firewall
Create Rules
Build your firewall. This is not necessary, but why waste a perfectly good firewall/router with no rules. I have always built my servers with no firewalls, relying on ensuring the only services running are the required ones, and weekly maintenance to ensure I have security fixes in place (plus, monitoring the appropriate mailing lists). But, a firewall with good rules gives you that added layer for those times when you forget and leave mySQL available to the outside world.
My firewall setup is pretty basic. I have about a dozen "servers" (mixes of Xen virtuals and stand alone machines) in my Colo, and none are general purpose. I will use a smaller example below, though. The IP's are made up; I'll use the private range 192.168.168.1-16 for my example. Also, I always change my ssh ports to something other than the default.
- Everything needs access via ssh. I set them all up to use port 22222 for ssh, my first rule is to allow 22222 on all addresses in the net, ie 192.168.168.1-16
- Mail proxy server needs ports 25 (SMTP), 465 (SSMPT), and port 25252 for my web interface to the proxy. Additionally, I opened alternate port 2525 so clients who are on networks blocked for port 25 can us it to connect.
- DNS requires port 53
- Web Server needs ports 80 (http), 443 (https), and 20-21 (ftp, if you allow ftp connections)
- Mail servers require 143 (imap), 993 (imaps), 110 (pop) and 995 (pops) to be open. Since they will communicate with the Mail Proxy over local addresses, SMTP does not need to be opened. However, webmail requires 443 (https) to be open also.
- I have an scp server that allows files to be transferred over port 22221, so it needs that port open.
Here are my sample servers
- Xen DOM0 Servers at 192.168.168.15 and 16
- nothing but the default ssh access. If I want vnc, I do port forwarding on the ssh, ie ssh -L localhost:5901:localhost:5900 192.168.168.15
- Mail Proxy Server at 192.168.168.2
- Rule #2 above, ie SMTP and management ports only
- DNS server at 192.168.168.3
- Open the DNS rules
- Router "temporary" IP at 192.168.168.1
- Nothing but the default
- Web Server at 192.168.168.4
- Rule 4 above, ie http, https, and ftp
- Web/e-mail server at 192.168.168.5
- Rule 4 provides web services
- Rule 5 provides mail services
- e-mail only server at 192.168.168.6
- Rule 5 only, ie imap, POP and https
- "File" server at 192.168.168.7
- scp "special" port, 22221. Note that this requires a separate Port 22221 entry in /etc/ssh/sshd_config (you include both lines) so ssh listens on both ports.
Deploy Firewall
Create the rules, and copy them to your router.
The simplest way to do this is to copy the output of fwBuilder to /etc/init.d/firewall, and replace the line with the following two commands.
pre-up /etc/init.d/firewall start post-down /etc/init.d/firewall stop
This is good, but if you will be doing traffic shaping also, it might be simpler to start your firewall, then take the rules and store them in a file using iptables-save. You can then load the rules necessary for both traffic shaping and your firewall with one simple command.
Note: REJECT will not work unless there is an IP address, so I use DROP when creating my rules.
"While it is not a requirement to give the bridge an IP address, doing so allows the bridge/firewall to access other systems and allows the bridge/firewall to be managed remotely. The bridge must also have an IP address for REJECT rules and policies to work correctly — otherwise REJECT behaves the same as DROP. It is also a requirement for bridges to have an IP address if they are part of a bridge/router." (http://www.shorewall.net/bridge-Shorewall-perl.html)
First, get your firewall set up (probably by executing the fwbuilder script), then execute the following command:
./firewall start iptables-save > /etc/network/iptables.up.rules
Now, create an "accept all" rule set by issuing the following commands
./firewall stop iptables-save > /etc/network/iptables.down.rules
Your entries in /etc/network/interfaces then become the following two lines
pre-up iptables-restore < /etc/network/iptables.up.rules post-down iptables-restore < /etc/network/iptables.down.rules
You should now be able to see your firewall go up and down with your interface. Verify this with:
ifdown br0 iptables -L
and you will see an accept all policy. Conversely, issuing:
ifup br0 # will take a few seconds; will seem to hang iptables -L
would show you a fully populated firewall
Caution Messing with a firewall remotely is dangerous. I have successfully locked myself out of servers and routers by creating rule sets that kill my network connection, necessitating a panic trip to the NOC to get the machine/network back online. Do the following before messing with an untested firewall.
- Create a file /etc/cron.d/testFirewall with the following contents
# /etc/cron.d/testFirewall: Uncomment following before adding a new firewall # Will clear the firewall every five minutes. # uncomment if you have a clean (accept all) iptables save file in /etc/network/iptables.down.rules #0-55/5 * * * * root iptables-restore < /etc/network/iptables.down.rules # uncomment for a script 'firewall' in /etc/init.d #0-55/5 * * * * root /etc/init.d/firewall stop
- Before installing the new firewall rules, uncomment the appropriate line in the script above. This will drop the firewall every 5 minutes, giving it an Accept All status (ie, no firewall). This will give you a few minutes to test the firewall, but if you lock yourself out, you can wait five minutes, log back in, then fix whatever you messed up. Much, much better than a 30 minute drive through the snow at 3am.
Traffic Shaping
Traffic shaping allows you to define traffic limits through the main interface and, optionally, traffic based on target and port. This is the main reason I decided to build a router, though. My servers are located at a Colocation facility, and I have a certain bandwidth built into my contract. If I exceed that bandwidth, I have to pay extra money. Of course, if my clients require the additional bandwidth, I will purchase it, but I would rather have no surprises; I would like it to be planned.
The basic configuration is rather straight forward, using only the command tc. In this case, we actually deal with the true interfaces, the ones that make up our bridge. And, we need to know which one is internal and which one is external
#!/bin/bash # script used to set up and down speeds on a bridge # much of this was taken from # http://atmail.com/kb/2009/throttling-bandwidth/ # and modified by RWR. See original for single interface # version. # Set up for Debian. Does no checks for modules installed # correctly. # # tc uses the following units when passed as a parameter. # kbps: Kilobytes per second # mbps: Megabytes per second # kbit: Kilobits per second # mbit: Megabits per second # bps: Bytes per second # # NOTE: this is set up to handle a bridge. In most cases, we # don't care which eth device is on which side of the bridge. # However, for traffic shaping, we are at a lower level than # the bridge and must connect directly to the interfaces themselves. #################################################################### # Edit the following for your system #################################################################### # TC command TC=/sbin/tc # interface that is connected to the 'net. EXTERNAL_IF=eth1 # interface connected to your servers INTERNAL_IF=eth2 # limit download speed (incoming) to this value INCOMING_SPEED=1mbit # limit upload speed (outgoing) to this value OUTGOING_SPEED=256kbit # define the network we are dealing with NETWORK=10.111.111.0/24 #################################################################### # End of editable values #################################################################### start() { # We'll use Hierarchical Token Bucket (HTB) to shape bandwidth. # In each section, The first line creates the root qdisc, and # the next line creates a child qdisc used to shape bandwidth # # The last line creates the filter to match the interface. # The 'dst' IP address is used to limit download speed, and the # 'src' IP address is used to limit upload speed. # first, set up upload speeds $TC qdisc add dev $EXTERNAL_IF root handle 1:0 htb default 30 $TC class add dev $EXTERNAL_IF parent 1: classid 1:1 htb rate $OUTGOING_SPEED $TC filter add dev $EXTERNAL_IF protocol ip parent 1:0 prio 1 u32 match ip src $NETWORK flowid 1:1 # now, the download speed. $TC qdisc add dev $INTERNAL_IF root handle 1:0 htb default 30 $TC class add dev $INTERNAL_IF parent 1: classid 1:1 htb rate $INCOMING_SPEED $TC filter add dev $INTERNAL_IF protocol ip parent 1:0 prio 1 u32 match ip dst $NETWORK flowid 1:1 } stop() { # Stop the bandwidth shaping. $TC qdisc del dev $EXTERNAL_IF root $TC qdisc del dev $INTERNAL_IF root } restart() { # Self-explanatory. stop sleep 1 start } show() { # Display status of traffic control status. echo $EXTERNAL_IF $TC -s qdisc ls dev $EXTERNAL_IF echo $INTERNAL_IF $TC -s qdisc ls dev $INTERNAL_IF } case "$1" in start) echo -n "Starting bandwidth shaping: " start echo "done" ;; stop) echo -n "Stopping bandwidth shaping: " stop echo "done" ;; restart) echo -n "Restarting bandwidth shaping: " restart echo "done" ;; show) echo "Bandwidth shaping status:" show echo "" ;; *) pwd=$(pwd) echo "Usage: $0 {start|stop|restart|show}" ;; esac exit 0
Bringing up an external connection temporarily
Since the bridge has no IP address, it is too secure to do some common tasks, like updating the operating system! You can give it an IP and route temporarily, then remove the connection when done. Here is a short script to update the operating system (assumes Debian)
#! /bin/bash INTERFACE=br0 # the bridge we built IP=10.111.111.254 # the IP we want to give it MASK=255.255.255.0 # network mask NETWORK=10.111.111.0 # the "local" network GW=10.111.111.1 # the gateway # create an address and route for the interface ifconfig $INTERFACE $IP netmask $MASK route add -net $NETWORK netmask $MASK $INTERFACE route add -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE # Do the work apt-get update # get new packages from apt # download the packages, but do not install. Remove --download-only if you want the updates performed automagically apt-get --download-only --assume-yes upgrade # now, shut down the interface. Remove routing entries route del -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE route del -net $NETWORK netmask $MASK $INTERFACE # and unconfigure the interface ifconfig $INTERFACE 0.0.0.0 echo $INTERFACE unconfigured.
When you have done this, you can then perform
apt-get --no-download upgrade
from a command line to do the actual upgrades
Obviously, this will not work for interactive updates. For that, simply populate the interface manually. The following script allows you to populate/depopulate the bridge:
#! /bin/bash INTERFACE=br0 # the bridge we built IP=10.111.111.254 # the IP we want to give it MASK=255.255.255.0 # network mask NETWORK=10.111.111.0 # the "local" network GW=10.111.111.1 # the gateway CMD=$1 case "$CMD" in start) # create an address and route for the interface ifconfig $INTERFACE $IP netmask $MASK route add -net $NETWORK netmask $MASK $INTERFACE route add -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE RETVAL=$? echo $INTERFACE now has an IP of $IP ;; stop) route del -net 0.0.0.0 netmask 0.0.0.0 gw $GW $INTERFACE route del -net $NETWORK netmask $MASK $INTERFACE # and unconfigure the interface ifconfig $INTERFACE 0.0.0.0 RETVAL=$? echo $INTERFACE unconfigured. ;; status) ifconfig $INTERFACE route -n RETVAL=$? ;; *) echo Usage $0 start stop status ;; esac exit $RETVAL
References
I read a lot of information on the net, and benefited greatly from the research of others. Following is a list of web sites I used in researching and building this router. Also, thanks to Loren Schweitzer and David North for their invaluable assistance, mainly making me aware when I was venturing down a fruitless path.
- http://www.annahegedus.com/index.php/Tutorials/bridge-firewall
- http://www.debian.org/doc/manuals/securing-debian-howto/ap-bridge-fw.en.html
- http://www.sjdjweis.com/linux/bridging/
- http://wiki.debian.org/BridgeNetworkConnections#A.2BAC8-etc.2BAC8-network.2BAC8-interfacesandbridging
- http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge
- http://wiki.archlinux.org/index.php/Firewalls
- http://www.linuxquestions.org/questions/linux-networking-3/traffic-shaping-using-tc-and-iptables-fw-mark-798630/