Denial of Service Attack Mitigated

     We were hit with a denial of service attack that exhausted router resources.  This started just before 6AM this morning.  It has been mitigated by adding additional firewall rules around 11:15AM.  What made this difficult to troubleshoot was the traffic was being blocked by a firewall rule but on the outbound interface instead of the inbound one so the CPU was overloaded by it making it difficult to get any response from the router.  I will be restructuring firewall rules so invalid traffic is blocked on the inbound interface to prevent a future attack of this same type.

     Please do not rely on the Eskimo North Users group on Yahoo for information during an outage.  It often will not allow me to login or is not responsive so I can’t post.  There is also someone who posts to it maliciously so information you get there may not be accurate.  This morning inaccurate information was posted suggesting this was “just like 2009”, no it was in NO WAY like 2009, the phones worked, information was available there, and I also posted on Facebook where I could login.

     The most direct source of information is your phone, call us.  When something is down, I leave a voice message so at least you can know what is happening.  After that, check our Facebook page at https://www.facebook.com/EskimoNorth, which unlike the Yahoo group, is controlled by us and thus will have accurate information.

DoS Attack

       Our router is currently being subject to a denial of service attack that is sufficiently intense to block most traffic from passing through.  I am currently at the co-location facility and working with Isomedia to block it and restore service.

Ubuntu Upgrade Failed

      The upgrade of ubuntu.eskimo.com from 14.04 to 14.10 (an intermediate version before going to 15.04) failed.  Ubuntu is down while being restored from backup and then I will make another attempt.  If you need a Debian based shell server, please use debian.eskimo.com or mint.eskimo.com in the meantime.

Ubuntu Upgrade – Don’t Send E-mail from Ubuntu

      I am in the process of upgrading Ubuntu from Ubuntu 14.04 to Ubuntu 15.04.  The upgrade program removed the postfix user and group which will break postfix until they are re-installed.  For this reason, please do not send e-mail from Ubuntu for now.  I will post here again when the upgrade is completed and postfix fixed.

System Troubles

     We had problems with numerous servers this morning that I tracked down to a problem with NIS, a network information system used to map user names to user IDs and vice versa as well as distribute password information, home directories, etc, across the servers.  This caused issues across the board with web servers, ftp, e-mail, login to any service.

     The immediate issue causing the outage has been resolved and I am going to make some changes to improve redundancy with this sub-system to prevent a future recurrence.

Mail Reboot

   There will be a brief reboot of the client mail server, mail.eskimo.com, in order to install a kernel with samba support that is required by OpenChange, an open source program which provides support for MS-Exchange protocol.

X2Go MacOS

     The most recent X2Go client mostly works with MacOS.  There is a problem however, it only works if you use sh, ksh, bash, or some other Bourne shell derivative here.  If you use csh, tcsh, or other non Bourne shell derived shell it will fail to map the keyboard correctly.  This is because it uses bash (and apparently only the /bin/sh subset of commands) to execute xmodmap to do keyboard mapping.  It only does this on MacOS, and consequently this only is a problem if you are originating from a Mac OS/X based machine.

Shellx Reboot Saturday May 2nd

      I will need to reboot shellx to load a new kernel which has some capabilities that are required by some services on shellx.  I will be doing this later this evening (Saturday May 2nd).

O1 Access Numbers Resolved

     This is the current status of the ticket.  The outage was caused by a fiber cut that has been repaired. If you encounter any further problems please contact support@eskimo.com or call us at (206) 812-0051.  Thank you.

5/2/2015 11:10:00 AMGenoZ

Hello,

Our carrier reports the issue should now be resolved. We have placed multiple test calls through the reported access numbers and can no longer replicate any issues.

Please retest and let us know if you see any further issues.

Thank you.