I apologize that the last two months have had some difficulties and interruptions but there was good reason for it.

     As you can see, web response latency spikes are GONE.  This means people viewing your web page hosted here will never see a delay.

     You can verify this for yourself by signing up for Google Webmaster tools, on your site hosted here, and following the Crawl Stats (under the Crawl pull-down).

     In addition to the improvement of web response time, all user data is now on RAID10 disk arrays.  RAID10 means both mirrored (for redundancy) and stripped (for speed).

     Drive failures will not result in lost customer data and the read speed is doubled. RAID10 keeps a fully redundant copy of your data, not just hash sums, so there is no rebuild time in the event of a failed disk.

     Mail is back up.  It took less time than I anticipated.

     I will be turning the client mail server down for approximately 45 minutes to move it to the host where the spool is located for better response time.

     Debian is back in service, but I don’t know what caused it to eat it’s network configuration.

     Something caused debian.eskimo.com to loose it’s primary IP address, effectively isolating itself from the net.  I’ve reconfigured the network interface and am in the process of rebooting.

     I still don’t know how people are stopping the name servers but I put a script in place on all the servers that checks them once a minute and restarts them if they stopped so at least there won’t be prolonged outages while I research this issue.

     Our public name servers were struck with some sort of denial of service attack today that stopped them all from functioning.  A restart brought them back online.  I am still researching this and trying to gain a better understanding of who is doing what and how to prevent a recurrence.

     If you find your website not working or if you are using our name servers and find that you can not resolve pages, please contact us right away.

     I have filed a bug report with Internet Systems Consortium.

     If anyone has any knowledge of this exploit and how to prevent it, please e-mail nanook@eskimo.com or call me at (206) 812-0051.

traceroute to 204.122.16.1 (204.122.16.1), 30 hops max, 60 byte packets
1 50-251-249-54-static.hfc.comcastbusiness.net (50.251.249.54) 0.441 ms 0.414 ms 0.406 ms
2 96.120.101.113 (96.120.101.113) 14.070 ms 14.064 ms 14.054 ms
3 te-0-2-0-2-sur03.seattle.wa.seattle.comcast.net (68.87.207.9) 15.024 ms 168.675 ms 267.358 ms

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Starts to get bad here…
4 be-20-ar01.seattle.wa.seattle.comcast.net (69.139.164.129) 1190.337 ms 1280.773 ms 1280.772 ms

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Really gets bad here.
5 be-33650-cr02.seattle.wa.ibone.comcast.net (68.86.93.165) 1784.715 ms 2266.942 ms 2349.390 ms
6 he-0-11-0-0-pe05.seattle.wa.ibone.comcast.net (68.86.88.150) 2346.864 ms 2814.403 ms 2814.620 ms
7 as11404-1-c.seattle.wa.ibone.comcast.net (23.30.206.34) 15.518 ms 14.991 ms 15.245 ms
8 agg2-sea-a-t8-3.bb.spectrumnet.us (174.127.140.174) 14.710 ms 14.713 ms agg2-sea-a-t8-2.bb.spectrumnet.us (174.127.140.170) 23.925 ms
9 isomedia-SEA-10000M.demarc.spectrumnet.us (216.243.31.250) 23.919 ms 23.909 ms 24.883 ms
10 sea-ring.isomedia.com (207.115.90.250) 29.859 ms 30.093 ms 44.073 ms
11 east-core.isomedia.com (66.114.129.7) 44.307 ms 140.992 ms 794.775 ms
12 bllvwaelrt0.eskimo.net (204.122.16.1) 871.223 ms 856.486 ms 856.458 ms

     The latency has moved farther up into Comcast’s network.  This is an issue that appears to be squarely rooted in Comcast, and not real surprising since it was down most of the night last night entirely, including telephone service.

     If anyone NOT on Comcast is experiencing latency, if you would e-mail a traceroute so I could look for common elements it would be appreciated.

     If you ARE on Comcast, please consider calling 1-800-391-3000 and generating a trouble ticket.

     Presently we are experiencing issues between our router and the co-lo facility router causing significant lag.  This might be related to the bad jack, it may be something else.

     We have tickets open on both issues.