Firewall Rules Restructured

     I have to totally restructured firewall rules in response to yesterdays Denial of Service attack.  Now all filtering is done on the incoming side of the interface cards.  This prevents the hostile packets from crossing the main bus and eating CPU.  Yesterdays attack consisted of a large number of small packets that exhausted CPU.  These changes will address exactly this type of attack.  The interface cards are intelligent and perform filtering actions without requiring the routers main processors to be involved.

     It also simplified the filtering by eliminating the necessity to allow local exceptions for local machine communications between subnets.

     In addition I have blocked access to all router interfaces and broadcast addresses from the outside which will prevent certain types of abuse.

Ubuntu Up – Upgrade in Progress

      Ubuntu is up but an upgrade from 14.10 to 15.04 is in progress.  Although it can be used in this state, it will be slow as the upgrade process is heavily taxing on I/O resources and it is probably best to minimize activity during the upgrade.

Ubuntu Out Of Service for Imaging

     I have successfully upgraded Ubuntu from 14.04 to 14.10. I am now going to take it offline to image it so that if the upgrade from 14.10 to 15.04 blows up, I don’t have to start all over at 14.04 again. This will take about 45 minutes.

Denial of Service Attack Mitigated

     We were hit with a denial of service attack that exhausted router resources.  This started just before 6AM this morning.  It has been mitigated by adding additional firewall rules around 11:15AM.  What made this difficult to troubleshoot was the traffic was being blocked by a firewall rule but on the outbound interface instead of the inbound one so the CPU was overloaded by it making it difficult to get any response from the router.  I will be restructuring firewall rules so invalid traffic is blocked on the inbound interface to prevent a future attack of this same type.

     Please do not rely on the Eskimo North Users group on Yahoo for information during an outage.  It often will not allow me to login or is not responsive so I can’t post.  There is also someone who posts to it maliciously so information you get there may not be accurate.  This morning inaccurate information was posted suggesting this was “just like 2009″, no it was in NO WAY like 2009, the phones worked, information was available there, and I also posted on Facebook where I could login.

     The most direct source of information is your phone, call us.  When something is down, I leave a voice message so at least you can know what is happening.  After that, check our Facebook page at https://www.facebook.com/EskimoNorth, which unlike the Yahoo group, is controlled by us and thus will have accurate information.

DoS Attack

       Our router is currently being subject to a denial of service attack that is sufficiently intense to block most traffic from passing through.  I am currently at the co-location facility and working with Isomedia to block it and restore service.

Ubuntu Upgrade Failed

      The upgrade of ubuntu.eskimo.com from 14.04 to 14.10 (an intermediate version before going to 15.04) failed.  Ubuntu is down while being restored from backup and then I will make another attempt.  If you need a Debian based shell server, please use debian.eskimo.com or mint.eskimo.com in the meantime.

Ubuntu Upgrade – Don’t Send E-mail from Ubuntu

      I am in the process of upgrading Ubuntu from Ubuntu 14.04 to Ubuntu 15.04.  The upgrade program removed the postfix user and group which will break postfix until they are re-installed.  For this reason, please do not send e-mail from Ubuntu for now.  I will post here again when the upgrade is completed and postfix fixed.

System Troubles

     We had problems with numerous servers this morning that I tracked down to a problem with NIS, a network information system used to map user names to user IDs and vice versa as well as distribute password information, home directories, etc, across the servers.  This caused issues across the board with web servers, ftp, e-mail, login to any service.

     The immediate issue causing the outage has been resolved and I am going to make some changes to improve redundancy with this sub-system to prevent a future recurrence.