Network Trouble

[Robert Dinse <nanook@eskimo.com>]

     Last saturday night/sunday morning, Sprint did some work on something
called "router reflectors".  This work was not supposed to cause an outage
because of redundancy in the system but in fact did take us out of service
for about an hour from midnight to 1am. 

     After we came back up there was a fairly high packet loss, 8-10%, but
that is somewhat to be expected after an hour downtime because of mail and
news that is backed up that comes rushing through when the network
connection comes back up.

     But over the week it did not entirely go away; it was still around 4%
this weekend; and what's more many sites either could not reach us or were
experiencing a very high latency.  And we could not reach some sites.

     Working with Sprint last night and today I believe we now have this
resolved.  At least all of the sites that I had reports of being unable to
reach us can reach us and I can reach all the sites we previously could
not.

     At first I thought this problem was caused by unusually heavy web
traffic (web traffic WAS unusually heavy), but was able to prove that not
to the the primary cause when at 3am last night when traffic was light,
these difficulties persisted.

     I then had to move equipment around and play other games to prove to
Sprint that it was screwed up on their end; which I finally succeeded in
doing.  They then took both interfaces down for a few minutes and brought
them up again; this caused all the BGP-4 routes to be reset and after that
all the unreachable sites became reachable. 

     There is still light packet loss inside the Sprint network (about
3%), and they have isolated where it is occuring and are working on it.
There is also heavy packet loss to some other networks, primarily
alter-net, and they are referring those problems to them.  But the
majority of the problems have cleared up.

     If you are experiencing unusual lag telnetting in from to Eskimo from
some site; I need to know the IP address you are originating the
connection from.  The same holds if you are having difficulty reaching
other servers here.

     Likewise; if you are having problems reaching a remote site; I need
to know the site that you are trying to reach in specific terms; 
preferably an IP address; and at the very least a fully qualified domain
name such as www.figets.com, etc. 

     Sprint is holding this ticket open until monday and I want to make
sure that if there are any other connectivity problems we bring it to
their attention while this ticket is still open.

     I apologize to those people I was not able to get back to today; I
was on the phone with Sprint until 11am, slept until about 5pm, and then
was on the phone with them for another five hours before we finally got
some resolution.