Eskimo North

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Intermittant Service Issues

     Over the past month or so gradually building, we've had a number of
intermittant service issues caused by denial of service attacks.

     What follows is a somewhat technical discussion.  If you're not interested
in the details, the important thing to know is that I think I've got these
issues largely resolved.  If you are interested, read on.

     Starting friday and finishing tuesday morning, I complete re-worked our
firewall rules from a permitted unless explicitly denied stance to a denied
unless explicitly permitted stance in order to stop attacks using various typed
of invalid packets.

     Initially, we had some problems with the router overloading however, this
was largely due to logging and as I turned logging off for most of the known
stuff (still wanting to log stuff that be legitimate but that I neglected to
allow), this reduced the demands on the router and it appears to be stable now.

     The majority of the attacks were a modified Smurf attack and we were
actually being used as an intermediate "amplifier" to attack someone else.  The
way this works is to send an ICMP echo request or any other packet with the
echo bit set to a broadcast address on a network with the source address forged
as the target that you really want to attack.

     The modern RFC's call for hosts not to respond to ICMP on broadcast
addresses, unfortunately; Linux, at least in the 2.2.x kernels doens't behave
in this respect by default.  It has an option in /proc/sys for ignore echo
requests on broadcast, however, by default this was not set.

     However, even if it IS set, certain invalid packets sent to the broadcast
address will cause a Linux server to respond.  What's worse, if there are
virtual interfaces on a machine, every virtual interface will generate a

     Until last friday; I was never able to get logging to work on our routers.
This turned out to be a misconfiguration of syslog.conf rather than a problem
with the router and friday I got that resolved so we were actually able to log
and see what was crashing our servers.  Over the weekend, initially I put in
filters to block that specific type of packet but the attackers kept changing
their stratedgy which is what lead to the decision to change to a deny by
default stance in the firewall.

     This also severely limits visibility of our internal network
infrastructure to the outside world and this too is a good thing.  Now nmap
with a -P option only picks up ports that we intentionally make available to
the outside world to provide services, and the stealth mode doesn't even see
the hosts as alive.

     One thing that really has me puzzled is that one type of packet that is
being directed at broadcast addresses here, it has a protocol ID of 0, which
according to what I can find online is actually an IPv6 protocol, but we have
no IPv6 address space nor routing established for IPv6 so really as an IPv4
packet as near as I can tell that's a completely invalid protocol ID.  I've got
a call in to Integra to try to find out why they'd even route such a packet to
me but at this point it's a mystery as to how and why these make it here at

 Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
   Knowledgable human assistance, not telephone trees or script readers.
 See our web site: (206) 812-0051 or (800) 246-6874.