Debian is back in service, but I don’t know what caused it to eat it’s network configuration.

     Something caused debian.eskimo.com to loose it’s primary IP address, effectively isolating itself from the net.  I’ve reconfigured the network interface and am in the process of rebooting.

     I still don’t know how people are stopping the name servers but I put a script in place on all the servers that checks them once a minute and restarts them if they stopped so at least there won’t be prolonged outages while I research this issue.

     Our public name servers were struck with some sort of denial of service attack today that stopped them all from functioning.  A restart brought them back online.  I am still researching this and trying to gain a better understanding of who is doing what and how to prevent a recurrence.

     If you find your website not working or if you are using our name servers and find that you can not resolve pages, please contact us right away.

     I have filed a bug report with Internet Systems Consortium.

     If anyone has any knowledge of this exploit and how to prevent it, please e-mail nanook@eskimo.com or call me at (206) 812-0051.

traceroute to 204.122.16.1 (204.122.16.1), 30 hops max, 60 byte packets
1 50-251-249-54-static.hfc.comcastbusiness.net (50.251.249.54) 0.441 ms 0.414 ms 0.406 ms
2 96.120.101.113 (96.120.101.113) 14.070 ms 14.064 ms 14.054 ms
3 te-0-2-0-2-sur03.seattle.wa.seattle.comcast.net (68.87.207.9) 15.024 ms 168.675 ms 267.358 ms

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Starts to get bad here…
4 be-20-ar01.seattle.wa.seattle.comcast.net (69.139.164.129) 1190.337 ms 1280.773 ms 1280.772 ms

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Really gets bad here.
5 be-33650-cr02.seattle.wa.ibone.comcast.net (68.86.93.165) 1784.715 ms 2266.942 ms 2349.390 ms
6 he-0-11-0-0-pe05.seattle.wa.ibone.comcast.net (68.86.88.150) 2346.864 ms 2814.403 ms 2814.620 ms
7 as11404-1-c.seattle.wa.ibone.comcast.net (23.30.206.34) 15.518 ms 14.991 ms 15.245 ms
8 agg2-sea-a-t8-3.bb.spectrumnet.us (174.127.140.174) 14.710 ms 14.713 ms agg2-sea-a-t8-2.bb.spectrumnet.us (174.127.140.170) 23.925 ms
9 isomedia-SEA-10000M.demarc.spectrumnet.us (216.243.31.250) 23.919 ms 23.909 ms 24.883 ms
10 sea-ring.isomedia.com (207.115.90.250) 29.859 ms 30.093 ms 44.073 ms
11 east-core.isomedia.com (66.114.129.7) 44.307 ms 140.992 ms 794.775 ms
12 bllvwaelrt0.eskimo.net (204.122.16.1) 871.223 ms 856.486 ms 856.458 ms

     The latency has moved farther up into Comcast’s network.  This is an issue that appears to be squarely rooted in Comcast, and not real surprising since it was down most of the night last night entirely, including telephone service.

     If anyone NOT on Comcast is experiencing latency, if you would e-mail a traceroute so I could look for common elements it would be appreciated.

     If you ARE on Comcast, please consider calling 1-800-391-3000 and generating a trouble ticket.

     Presently we are experiencing issues between our router and the co-lo facility router causing significant lag.  This might be related to the bad jack, it may be something else.

     We have tickets open on both issues.

     I managed to move the mail spool and a number of virtual machines to different hosts.  Fedora is caught in the middle right now and my quality Comcrap cable Internet keeps going down making work close to impossible so I’m going to leave Fedora down for the night and hopefully pick it up in the morning.

     Still have a number of virtual machines to move, both to align services with the disks that they are using and to clear one machine so I can do hardware upgrades and re-install an operating system (replacing Centos 6.7 with Ubuntu 15.04) so there will be some individual machines interrupted tomorrow to complete this.

     I replaced all the gigabit wiring with cat6, however, I discovered the Jack provided by Isomedia to connect back to their router is flaky.  I’ve put in a trouble ticket but this probably means there will be some interruption during the day to repair or replace it.

     I’ll be out of the office for a while as I am headed to Fry’s to get some cat-6e or better cable to replace the cat-5 cables that were not designed for 1G Ethernet usage.