Ghost Vulnerability – Reboots

There will be some brief interruptions of various services including the shell servers because of a serious vulnerability in glibc.

I will be applying updates to various systems as they become available and then rebooting so that old code no longer runs.

This will cause a brief interruption of all services.  The main file server and host machines take longer to reboot so I will do those after 10PM tonight.  There will be about twenty minutes between 10pm-midnight where virtually everything grinds to a halt while this is done.

Sick – No Voice or Frog Voice

I’m sick and have little to no voice, sometimes just a whisper, sometimes I can manage an intermittent frog voice with great difficulty.

If you need help and it’s something that can be handled via e-mail to support or fax, that’s much preferred at the moment.  If you need to send card information, fax is best, or login to webmail here and send to support from webmail, be sure to use https not http.

DSL Maintenance 1/26/2015

Date: 1/26/2015
Start time: 11:00 pm PST
End time: 1:00 am PST
Affected: ATM Terminations on Seattle Redback

Detail:

Maintenance is being performed in order to move the Redback to a new rack
within our space at Colo centers Estimated downtime for this is 1 hour while
the rack is moved and re-wired. Some affected customers will need to reboot
their equipment to restore services.

This will affect Western Washington DSL customers in CenturyLink territory.

Physical Host Down

One of the physical hosts wedged during a copy of a virtual machine.  The only shell servers available at present is shellx.eskimo.com and eskimo.com.

I may have to boot and run a file system check on the other as well, so everything may be down for about 20 minutes probably about 45-60 minutes from now.

Virtual Machine Migration

There may be some points where things are a bit sluggish today as I migrate some virtual machines from one box to another.  This involves copying images around 100GB.  With the old 100mb/s switch this would pretty much stop things.  I’m hoping not with the 1GB switch, still it’s going to task disk I/O and other resources on the machines pretty heavily.

The purpose for migrating these is for load balancing and to provide better redundancy when physical hosts are down by spreading functionality across multiple physical boxes.

Big Increase in BotNet Activity

Over the weekend, the amount IP addresses that brute force password guessing attacks originate from as detected by fail2ban, log scanning and automatic action script, has more than doubled from about 300 IPs per day to about 750 and that seems to be growing.

This pattern is usually indicative of some new Windows malware out in the wild successfully propagating to a huge number of machines that can then be used for things like password guessing and distributed denial of service attacks.

Relating to DDOS attacks, a large Botnet is attempting to use our DNS servers as DDOS amplifiers.  This won’t work because we have rate limiting configured on the external views for all of our servers, but it generated so much crap in the syslog’s that it ran some servers out of disk space.

As a result of this I’ve added code that bans IPs for an extended period if they exceed rate limit thresholds which both quieted down the logs and reduced CPU load on the name servers servers substantially.

The relevance to you is, if you are running Windows, make sure your anti-viral and anti-malware software and it’s databases are up to date and run scans frequently.

I recommend running Malware Bytes, as 9 times out of 10, when I have a customers computer that is infected, it’s the application that finds the infection.

Second thing, if your password is easily guessable, for example, a dictionary word, or a dictionary word with a number after it, it should be changed to something more complex.

An ideal password will contain no dictionary words, no proper names or anything related to your account such as your login, a combination of UPPER CASE, lower case, punctuation characters such as ~!@#$%^&*()_+=-`{}[]|\:;”‘<>,.?/ and numbers 0123456789.

Mail Server Maintenance 1/4/2015 00:01-00:25

I will be taking the client mail server down for maintenance just after midnight to image it for recovery in the event of file system corruption.  This should take less than about 25 minutes.

During this time, pop, imap, outbound smtp, and webmail will be unavailable.  Incoming mail will still be processed.  Mail can be viewed and/or deleted using shell mailers like pine, mail, mush, mh, but outgoing mail can not be sent during this time.  Some mailers that use the local mail smtp server will queue, but any that try to send via mail.eskimo.com will fail.

Switch Upgrade Completed

The work on the switch upgrade has been complete.  Now even copying an 8gb file only increases ping times to about 65ms which isn’t noticeable where before it would lock them machines up for sometimes seconds because the file transfers sucked up all the bandwidth.