Boston Tunnel Fires and Packet Floods...

[Robert Dinse <nanook@eskimo.com>]

     Yesterday evening and part of today, authentication was broken for Qwest
lines and part of MegaPop.  This was the result of fried fiber in the tunnel in
Boston where the train fire was which isolated Qwests authentication server
from Star2's, a port aggregator we use, and cutting MegaPOP's authentication
servers off from a couple of their POPs. 

     Last night we were also hit with a SYN flood, a type of packet flood, the
IRC server was the target but it was very intense and caused about 90% packet
loss which pretty much isolated us from the net.  Sprint was very busy owing to
the Boston problems and a recent security problem in Cisco routers IOS and so
they couldn't get a NULL route in very quickly, took about 45 minutes which is
not usual for Sprint. 

     I left the NULL route for the IRC server in place today and then other
hosts were were attacked, including Eskimo, and because the shell server has to
be fairly exposed to the net for all of the service it provides to work, it
wasn't well protected and they were able to exhaust kernel memory repeatedly
and crash it several times.  They also targeted other hosts here and crippled
them enough that I eventually had to reboot them. 

     Because the IRC related servers is usually the draw for these kinds of
attacks, I've made some changes this evening that hopefully will somewhat limit
damage to the rest of the network.  One T1 now routes one class C incoming
which will be reserved for IRC related servers.  The other two route the
remaining network here.  This way when the IRC servers are flooded it won't
trash the rest of the network although it will not prevent attacks like the one
today where they targeted other machines. 

     I've got the client IRC server, the one accessible by the public, moved to
this seperately routed class C and will move chat shortly. 

     All three T1's are still used for outgoing traffic of all types so this
will not affect outbound capacity.