Network

[Robert Dinse <nanook@eskimo.com>]

     Around 11:10pm we experienced severe packet loss reaching just about
anywhere.  All traffic inbound was coming in on one link (and one wasn't
sufficient to handle the traffic at that particular moment resulting in packet
loss). 

     Bouncing the port (which causes routing information to be repropogated
onto the net) cleared the problem of all traffic routing only to one link, but
even with the load balanced we still saw about a 10% packet loss. 

     Sprint told me that BGP-4 was eating up all the router CPU; a lot of
customers on that router reset their BGP-4 sessions at the same time they said. 
That explanation is bogus.  I pointed out that we had static routes and STILL
they lost our routing information, something was hosed with BGP-4 on that
router and something has been not right since they did the work on the route
reflectors a while back. 

     Things settled down and are working again but we are still seeing a small
residual packet loss (about 2%) and I'm convinced until they boot that thing it
isn't going to get better but I'm having trouble convincing them.  Hopefully it
will crash and burn and they will have to fix it; in the meantime I still have
a ticket outstanding and will escalate it to the presidents office if
necessary.