=^.^=

Geofence with iptables: Blocking Countries at the Firewall

In some situations one may find it useful to block entire countries or restrict access to only one or a few. This is a technique known as geofencing, and if you've ever tried to watch a video only to be told that it's not available in your region you have been the victim of it. Geofencing, like geolocation, is possible because blocks of IP address space are handed out to specific countries, and additional details such as the province or city of the address holder may be obtained through reverse-whois. Data collected below the country level can be unreliable, often the location of a head office for a national ISP will appear to be the source of all if its users.

ahorli on the Clear forums just posted their geofencing solution for ClearOS at http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,7/func,view/id,10382/. It is intended to block specific countries that tend to produce a high volume of spam and automated attacks (in this case, Russia and China). I thought it would be neat to reverse the script so I could block every country except a specific one or two. Obviously this kind of tactic isn't going to stop someone who really wants into your box from outside the geofence - there's everything from proxies to VPNs to exploit. My interest here is in reducing automated attacks to those originating in the motherland, because that's the only place I expect to be connecting to our hypothetical server.

Download this script and put it somewhere appropriate, I would suggest /sbin or /usr/sbin. In order to work this requires that your default INPUT policy is DROP or REJECT. As mentioned above, geofencing is more art than science and when I ran this script my own subnet was not unblocked, I strongly recommend including your headquarters in the ALLOWSUBNET variable or you may find yourself one day without access. As you can see MAXZONEAGE is set to 6, so if we pop this in cron.weekly it should refresh its fence list every week. You should add the script to your firewall or local init scripts, on ClearOS use /etc/rc.d/rc.firewall.local.

Blocking ICMP Echo Requests (Pings) to your Linux Firewall with iptables

It is generally considered poor form and a violation of some arcane RFC for a host to ignore ICMP echo requests (common "pings") and turning them off does not afford you any additional "security" per se. That being said there are a number of very good reasons you might want to ignore pings in the wild. Due to the amount of time it takes to accurately port scan a host, bulk scanning operations generally ping a host to determine if it is worth spending the time and resources needed to scan the address. If your host is configured to drop pings you instantly take yourself off the radar of such robots, sparing your resources for say combating directed attacks rather than the automated attacks that follow such scans.

If you're dealing with a single host it isn't necessary to specify the IP or interface but on a firewall you probably want to be able to ping its internal interface from the internal network. We're going to assume that eth0 represents the external interface:

# iptables -A INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j DROP

To specify an IP or subnet use the -s flag in place of -i. The --icmp-type 8 flag specifies that only ICMP echo requests are to be blocked, we want to leave type 0 replies alone so hosts behind and including the firewall can ping and receive responses from hosts beyond the router/firewall.

You may have existing chains that accept pings, you must delete these. For example:

# iptables-save | grep icmp -A INPUT -i eth0 -p icmp -m icmp --icmp-type 0 -j ACCEPT -A INPUT -i eth0 -p icmp -m icmp --icmp-type 3 -j ACCEPT -A INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j ACCEPT -A INPUT -i eth0 -p icmp -m icmp --icmp-type 11 -j ACCEPT -A INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j DROP

You can see our rule at the bottom. The third rule from the top conflicts with this so let's remove it:

# iptables -D INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j ACCEPT

As you can see, it's as simple as switching the add (-A) flag to delete (-D) and now our rule works. To automate this process you should add these lines to your firewall startup script or your "local" init script where available.

To save these rules on gentoo make sure you have the iptables init script in the default runlevel and run:

# /etc/init.d/iptables save

if there is no conflicting firewall script that adds an ACCEPT rule for ICMP requests. Otherwise you may wish to use /etc/conf.d/local.start.

ClearOS users should add something like this to /etc/rc.d/rc.firewall.local:

/sbin/iptables -D INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j ACCEPT /sbin/iptables -A INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j DROP

Gentoo Xen 4 Migration

I was stoked to find out Xen 4 had finally made it into portage a couple weeks ago, The improvements are so sweet I had been checking gentoo-portage.com every few days or so. Let's take a look at a few of the advances Xen has made since 3x:

  • Better performance and scalability: 128 vcpus per guest, 1 TB of RAM per host, 128 physical CPUs per host (as a default, can be compile-time increased to lots more).
  • Blktap2 for VHD image support, including high-performance snapshots and cloning.
  • Improved IOMMU PCI passthru using hardware accelerated IO virtualization techniques (Intel VT-d and AMD IOMMU).
  • VGA primary graphics card passthru support to an HVM guest for high performance graphics using direct access to the graphics card GPU from the guest OS.
  • TMEM allows improved utilization of unused (for example page cache) PV guest memory. more information: http://oss.oracle.com/projects/tmem/
  • Memory page sharing and page-to-disc for HVM guests: Copy-on-Write sharing of identical memory pages between VMs.This is an initial implementation and will be improved in upcoming releases.
  • New Linux pvops dom0 kernel 2.6.31.x as a default, 2.6.32.x also available. You can also use linux-2.6.18-xen dom0 kernel with Xen 4.0 hypervisor if you want.
  • Netchannel2 for improved networking acceleration features and performance, smart NICs, multi-queue support and SR-IOV functionality.
  • Online resize of guest disks without reboot/shutdown.
  • Remus Fault Tolerance: Live transactional synchronization of VM state between physical servers. run guests synchronized on multiple hosts simultaneously for preventing downtime from hardware failures.
  • RAS features: physical cpu/memory hotplug.
  • Libxenlight (libxl): a new C library providing higher-level control of Xen that can be shared between various Xen management toolstacks.
  • PV-USB: Paravirtual high-performance USB passthru to both PV and HVM guests, supporting USB 2.0 devices.
  • gdbsx: debugger to debug ELF guests
  • Support for Citrix WHQL-certified Windows PV drivers, included in XCP (Xen Cloud Platform).
  • Pygrub improvements: Support for PV guests using GRUB2, Support for guest /boot on ext4 filesystem, Support for bzip2- and lzma-compressed bzImage kernels

What tickles me the most is the Remus Fault Tolerance, it basically lets you run a standby instance of a VM on a different physical server and it constantly updates that VM of the master's status, I/O etc. If the master VM dies, the standby kicks in so fast there may be no perceivable downtime. I've been dying for something that provides solid HA that's well supported and works out of the box - not to mention does its job transparently for years. Now that it's a core feature 0f Xen, competing technologies will be compelled to introduce their own easy-to-use HA solutions which I hope could usher in a golden age of reliability.

The original intent was to migrate 9 physical 32-bit servers the week it came out, most of them running kernel 2.6.21 on Xen 3.2.1. This was not to be, I was determined to make the new 2.6.32 kernel work (I had heard that .32 was going to be the new .18 in terms of adoption/support) and the thing just won't work with megaraid. I haven't tried it with cciss yet and I don't intend to, for the time being I have downgraded all of the dom0 kernels to 2.6.18. Things seem to be very stable now and much faster.

PAE, or physical address extension allows 32-bit processors to address up to 64GB of memory. When I first started working with Xen I had no idea that I had omitted PAE from my hypervisor build (it is not a default USE flag) nor that every shrinkwrapped Xen kernel out there required it. To make matters worse, the 2.6.21 dom0 kernel I was using on all of the servers for the sake of consistency lacked the ability to enable PAE at all - even by manually editing the .config, something I still haven't figured out. That kernel was eventually hard masked, then removed from portage. This situation cost me a lot of time because every new image I wanted to import required special preparation to work with my "foregin" domU kernel and without pleasantries like initramfs and pygrub.

I'm going to start off by showing you the make.conf that will be used in this article:

CFLAGS="-O2 -march=pentium4 -pipe -fomit-frame-pointer -mno-tls-direct-seg-refs -fstack-protector-all" #CFLAGS="-O2 -march=pentium4 -pipe -fomit-frame-pointer -mno-tls-direct-seg-refs" CXXFLAGS="${CFLAGS}" CHOST="i686-pc-linux-gnu" MAKEOPTS="-j4" GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://distro.ibiblio.org/pub/linux/distributions/gentoo/ http://www.gtlib.gatech.edu/pub/gentoo " SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="-alsa cracklib curlwrappers -gnome -gtk -kde -X -qt sse png snmp cgi usb cli berkdb bzip2 crypt curl ftp ncurses snmp xml zip zlib sse2 offensive geoip nptl nptlonly acm flask xsm pae pygrub xen" FEATURES="parallel-fetch -collision-protect" LINGUAS="en_CA en"

If you're upgrading from an existing Xen installation you may want to disable collision protection, past experience with 3x upgrades has been that sometimes portage will see every Xen-related file in /boot as a potential conflict. Note the second CFLAGs line that's commented; some packages don't compile well (or at all) with Stack Smashing Protection (particularly glibc) so I update them individually with the second CFLAGS enabled before any sort of emerge world or deep update. SSP is enabled by default if you are using the hardened profile. I choose not to use the hardened profile because it can be needlessly problematic/inflexible and in this application the term "hardened" is misleading. If your dom0 isn't going to be exposed to the Internet (and it certainly should not be) it might be safe to omit -fstack-protector-all altogether but there is no such thing as paranoia. Also bear in mind that an attacker gaining access to a dom0 can be more devastating than an attacker gaining access to all of the VMs running on it individually.

Depending on the circumstances at the time you read this, one may or may not have to add the following to /etc/portage/package.keywords:

app-emulation/xen app-emulation/xen-tools sys-kernel/xen-sources

Sync portage and run an emerge --update --deep --newuse world --ask, if you see xen 4 in the package list you're on the right track. Compile away. You might be interested in this article on global updates with gentoo.

Once Xen has been upgraded it's time to build the new kernels. Follow the usual routine, making sure to enable xen backend drivers in the dom0 and frontend drivers in the domU. I like to make a monolithic domU kernel so there's no mess with installing or updating modules to the VMs. Make sure you have IP KVM/Console redirection if you're going to be booting this machine remotely and a non-xen fallback kernel configured in /etc/grub/grub.conf in case the hypervisor fails. Xen and some x86 BIOSes can be configured to use a serial console; a null modem to another server in the rack is often all you need.

I got all sorts of shit from the 2.6.32/34 kernels, for instance the kernel won't build properly if you enable Export Xen atributes in sysfs (on by default in 2.6.32-xen-r1). I got this message at the end of make and was not particularly successful at tracking down solutions:

WARNING: vmlinux.o (__xen_guest): unexpected non-allocatable section. Did you forget to use "ax"/"aw" in a .S file? Note that for example <linux/init.h> contains section definitions for use in .S files.

I don't know what - if anything - needs the Xen /sys interface to work, so it's probably no big deal.

When trying to compile at least versions 2.6.32 and 2.6.34 if the Xen compatibility code is set to 3.0.2 you can expect this error at the end of building the kernel:

  MODPOST vmlinux.o
WARNING: vmlinux.o (__xen_guest): unexpected non-allocatable section.
Did you forget to use "ax"/"aw" in a .S file?
Note that for example
 contains
section definitions for use in .S files.                                                                          

  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
ld: kernel image bigger than KERNEL_IMAGE_SIZE
ld: kernel image bigger than KERNEL_IMAGE_SIZE
make: *** [.tmp_vmlinux1] Error 1

This seems to be fixable by upping the lowest version to at least 3.0.4.

In all cases 2.6.29, 2.6.32 and the yet-un-portaged 2.6.34 kernels panicked on bootup if the megaraid driver was compiled in or made available in an initrd. After four days of dusk-until-dawn tinkering I got tired of fucking with it and decided to go with 2.6.18, which compiled and booted without a hitch.

Previously I had been doing all sorts of contorted things to the networking configuration but since I was dealing with a clean slate anyway I decided to set things up the Gentoo way. The Gentoo way of Xen networking is to abandon Xen networking. Suddenly life's great. In the set of four servers I migrated this week all of them have one physical interface on an external-facing VLAN and another interface on an internal VLAN as depicted in the diagram (left). I wanted to make it so I could take a router VM and move it from physical server to physical server as quickly as possible, and this is how I did it (thanks xming on the Gentoo forums):

  1. Edit /etc/conf.d/rc and change RC_PLUG_SERVICES to look like this: RC_PLUG_SERVICES="!net.*" this will prevent Gentoo's hotplug script from automatically starting your interfaces on bootup
  2. Remove existing interfaces from default runlevel, i.e. rc-update del net.eth0 default
  3. Configure one bridge that connects to the external VLAN and one bridge that connects to the internal VLAN
    config_extbr0=("null") bridge_extbr0="eth0"

    config_xenbr0=("x.x.x.x/24")
    bridge_xenbr0="eth1"
    routes_xenbr0=("default via x.x.x.y")

  4. Create init scripts for the new bridges, i.e cd /etc/init.d; ln -s net.lo net.extbr0; ln -s net.lo net.xenbr0
  5. Add the bridges to the default runlevel: rc-update add net.extbr0 default; rc-update add net.xenbr0
  6. Edit /etc/xen/xend-config.sxp and comment out (network-script network-bridge) and add (vif-script vif-bridge bridge=xenbr0)
  7. Edit VM configuration files, edit VIFs to connect to the appropriate bridge, i.e: vif = ['mac=00:16:3e:XX:XX:XX,bridge=xenbr0' ]

I found that my Gentoo VMs needed this line added to their config in order to get any connection to xenconsole at all:

extra="xencons=tty console=tty1"

My ClearOS VMs, for the first time running the kernel that ships with them, needed a more dramatic approach. I added this line to their config files:

extra="xencons=ttyS"

and then in the VM's /etc/inittab I added this line to make it talk on what would be its serial port:

s0:12345:respawn:/sbin/mingetty ttyS0

I had some minor complaints from init about the dom0 kernel being too old for some udev feature so I added this line to /etc/portage/package.mask and rebuilt it:

>sys-fs/udev-124-r2

Bad Snort Rules

Protecting a busy web platform or ISP with snort is made challenging due to the large number of false positives it generates in these environments. This post will serve as a compilation of snort rules I think should be disabled or tuned out of the box, it will be updated as I find new ones. The only way to tell if snort is interfering with legitimate traffic is to monitor its logs over a long period and look up the rules which appear most often. Make sure these rules don't do anything that could interfere with your regular operations - for example a rule to block the slammer worm that hits a dozen times a day probably has nothing to do with your e-mail troubles but a rule for an outdated MTA might explain why Blackberry's servers are being blocked every time a client tries to check their mail.

On ClearOS systems snort rules are located in /var/lib/suva/services/intrusion-protection/rules/ (formerly /var/lib/suva/services/snort/rules/).  To find a rule change to that directory then

# grep sid:number *

and the file name and line number will be printed next to the rule's line. Disabling a rule is as simple as prefixing it with a hash mark (#) and restarting snort:

# /etc/init.d/snort restart

I've started a thread in the Clear forums so everyone can share and discuss their false positives: http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,7/func,view/id,12205/

SID 1091:
False positive on any request containing ?????????? in the URL

SID 1233:
"I noticed, what I believe to be a false positive in rule 1233 (Outlook EML access).: a user was accessing hotmail and the GET was for /i.p.emlips.gif. Since the rule is looking for uricontent:".eml", an alert was triggered. "
http://www.networksecurityarchive.org/html/Snort-Signatures/2005-06/msg00008.html

SID 1390:
False positive on payloads containing CCCCCCCCCCCCCCCCCCCCCCCC, brought to my attenti0n on the ClearOS forums (thanks Kevin and Tim)
http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,8/func,view/id,9967

SID 1394:
False positive on payloads containing 'aaaaaaaaaaaaaaaaaaaaa'

SID 1807:
False positive on any request where the string "chunked" preceeds the Transfer-Encoding header. A fix is available at:
http://osdir.com/ml/security.ids.snort.sigs/2004-07/msg00018.html

SID  2109:
False positive on any attempt to retrieve e-mail from BlackBerry servers. An interesting note about this, I had dozens of users with blackberries on the network running just fine for years then all at once a few months ago - without even touching the software - snort started blocking the BlackBerry pop3 IP pool and the complaints came flooding in. When I researched it at the time it seemed as though this false positive was reported on at least one mailing list between 2004-6, perhaps RIM switched "back" to the would-be offending platform.

SID 2229:
This rule is false positive for ANY request to viewtopic.php, taking out any visitors to a phpBB installation. There is a fix for this rule outlined in this e-mail from 2004:
http://www.webservertalk.com/archive253-2004-11-452217.html

SID 2000540:
False positive on SSL connections with Google Gmail web servers
It appears to be generating false positives for fragmented ACK packets intended to match a form of Nmap scan
http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,8/func,view/id,19544/#19551

SIDs 1000000211, 1000000212, 1000000213, 1000000214:
"These are known to cause issues with Gallery2 (part of the web PHP rulesets)"
Tim Burgess supplied these on Clear's forum, I think the 1000000000-range rules may be unique to ClearOS.
http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,7/func,view/id,12205/#12227

SID 966
This rule has caused me a lot of problems when uploading large binary files, though I'm not sure why since only the URI should be scanned and not entire POST variables. May be a bad one for me only.

Foolproof Gentoo World Update Build Order

Users new to Gentoo often end up breaking their system after their first global package update - particularly if they stopped reading the manual as soon as they got their system booting (I'm usually one of those). Over the years I have run into every problem that portage can imaginably throw at a person and from that I have formulated a fool-proof update process which, sometimes with a little intervention, always works and - assuming you take proper care of configuration - leaves you with a working system.

The first thing you need to do is think of any software you have installed where a version change might break something, for example: some fglrx driver versions will work with old cards and Xinerama, and some will not - in no particular order. I had the unfortunate experience of blindly upgrading my fglrx implementation to one that didn't work only to find the version I had been using was no longer available in portage. Use quickpkg to make a quick standalone package of the existing files for a given atom:

# quickpkg cate-gory/package-name

The package will be available for later use in /usr/portage/packages.

Now let's update portage:

# emerge --sync

If you are given a message instructing you to update the portage package it is important that you do so before continuing.

# emerge portage

Before we begin compiling anything I need to point out that if you use certain CFLAGS some packages may fail to compile. I always include -fstack-protector which adds buffer overflow protection to software at compile-time. Glibc in particular tends to have a problem with this feature. Updating world with this flag in would break the long update process when emerge gets to glibc and we want the newest GCC suite available to compile the rest of our system anyway - so let's update glibc and gcc separately from world, and if we encounter any problems we'll temporarily strip down our CFLAGS:

# emerge --update gcc glibc

Now we want to make sure your profile is set to use the latest version of GCC available on your system, it might look something like this:

# gcc-config -l
[1] x86_64-pc-linux-gnu-4.1.2 *
[2] x86_64-pc-linux-gnu-4.3.4

The asterisk indicates which version of GCC you are using. You can set the version by running gcc-config number where number is in the leftmost position of the output above, thus:

# gcc-config 2

You will be instructed to update your profile environment variables:

# source /etc/profile

It should be noted here that some software may not compile correctly with a given GCC version, requiring you to move up or down. At the time of writing Quemu requires GCC 3x and I have had a few isolated incidents of Xen being picky.

Before we get started let's launch a screen instance (emerge screen if you have not already done so) so we can move from terminal to terminal without interrupting the build process:

# emerge screen
# screen -S update

You can detatch from the session with the key combination control+a, d. Returning to the session is as simple as logging in from anywhere as the user who spawned the session and typing:

# screen -x update

Multiple clients may attach to the same screen session at once. This is particularly useful if you want to be able to orchestrate some or all of the upgrade over secure shell.

Now we're ready for the magic line:

# emerge --update --newuse --deep world --ask

Recent versions of portage have excellent automatic blocked package resolution, but it doesn't work all the time. If you can't continue due to blocked packages try to emerge either package one at a time. This often isn't enough - particularly where one package has taken over "duties" for the same files an existing package used to provide. In these cases you will have to unmerge one or both of the conflicting packages - but sometimes unmerging the wrong one(s) may leave your system unusable. Google the package names and version numbers; without fail I have always found someone on the Gentoo bugtraq and elsewhere who has had the same problem and someone has usually posted the safest order of removal. Of course, nothing can't be fixed by booting with the livecd.

If you get strange errors about file collision protection but you're sure it's safe to overwrite the listed files, you may add this line to your make.conf or precede your emerge command similarly:

FEATURES="-collision-protect"

If you will be unable to attend the update process (going home for the weekend etc) you may find it useful to add --keep-going to the emerge flags; this will skip over packages that fail to build and continue emerging. If you can't tell from the error messages repeated at the end of the build simply re-run the emerge command, making sure to use either --ask or --pretend and the missing packages will be neatly listed.

Often enough these packages fail to compile due to a missing reverse-dependency. The next stop on the line is revdep-rebuild, run simply:

# revdep-rebuild

This tool will scour your system for missing reverse dependencies, build a list then start compiling them if it finds any. This process can fail if it attempts to rebuild a package for which there is no longer a matching version in portage, in which case you may have to do some emerging/unmerging/updating of the given package's dependants. If the builds themselves fail it is safe to move on to the next step then return to revdep-rebuild later.

Python is an integral part of the gentoo system so it is very, very important that when python is updated the python-updater tool is run:

# python-updater

If you just upgraded PERL, be sure to run:

# perl-cleaner all

Which will take care of removing old header files, renaming files and rebuilding perl modules. It usually ends up rebuilding net-snmp on my systems and not much else in the way of ebuilds.

Lafilefixer fixes libtool archives and may not be needed on your system (does not come installed by default, emerge lafilefixer if you have not already) but it won't hurt to run it:

# lafilefixer --justfixit

Even if your first revdep-rebuild worked it is a good idea to run it again, in case any of the (possibly) newly installed packages have broken anything else. If some packages failed to build in the first world update you may now run it again.

It is now safe to clear out your old dependencies - should any exist. Most of the packages you will see listed are simply old versions of updated packages as a number of them occupy slots (such as GCC) for the sake of backwards compatibility. If you would like to retain a specific package, add it to your /var/lib/portage/world file, in full if you would like to target a specific version: =categ-gory/package-name-version.number

# emerge --depclean --deep --ask

Do not forget to update your configuration files. Be warned that simply overwriting all of the old configuration files may not be necessary and may render your system inaccessible:

# etc-update

That concludes my foolproof build order. If you followed it correctly there is a high probability your system is completely up-to-date and working, from a portage point of view. Of course, proper configuration is important and compile-stopping bugs are common and may require some mild footwork, but this usually amounts to copying and pasting the first error line into google or blocking/unblocking specific package versions with /etc/portage/package.mask, package.keywords and package.unmask. More information on package masking is available in the emerge man page.

Happy building!