Starting around 9pm, I will be taking various servers down for reboots (kernel updates) and imaging.  The main file server won’t be done until after midnight, and same for mail and web servers, but other shell servers will be done earlier starting with the least busy.

This work will take until about 2AM and might possibly be continued on Saturday evening if everything isn’t finished Friday evening / Saturday morning.  All the Redhat machines have kernel updates and a huge number of updates (over 200 each) to install, the Debian based systems mostly just need to be imaged.

Debian is broken.  I installed X2Go and FreeNX and then I needed to reboot but couldn’t because the shutdown program was not installed.  I installed that, rebooted, but somehow the virtual machine was corrupted beyond restoration.  I’m recovering from a backup but it will take some more work to get everything back to current so this machine may be down overnight.  Please use Ubuntu or Mint if you need a Debian-based machine in the meantime.

There is a new virus going around that clamav does not have a signature for yet and thus is not catching.  Our spam filtering software does recognize it as spam and is putting it in your spam folder.  If you find e-mail in your spam folder with an attachment, I strongly recommend against opening it as it most likely contains a virus.

The volume of this virus is so large that it has overloaded our incoming mail servers and caused mail to be backed up in queue.  I was able to make some adjustments that improved the mail servers ability to process mail and it is catching up, but as of 2AM it has worked through about half the queue, so at this rate it will probably be caught up by around 5AM.

Hopefully the clamav folks will kick out a database that includes a signature for this thing and put an end to it, but to prevent your machine from being infected, do not open any attachments you are not expecting and keep your anti-virus database up to date.

Debian is back up and functioning.  I’m still applying updates that were rolled back but the corruption is resolved and it is operational.

Our shell server debian.eskimo.com is down for repairs.  The ‘apt’ database has somehow gotten corrupted and I’m unable to figure out how to correct it so I’m reverting the machine to a image made prior to it’s corruption and then will re-apply updates.  This should require about 45 minutes.

After the power glitch this morning when everything rebooted, it left some of the MySQL tables corrupted.  I didn’t notice until I tried to go into the Forums just now and I received no complaints about broken blogs, etc.

I’ve run myisamchk -r on all the tables with the server down to repair any inconsistencies and brought the server back online.  Forums are working now.

If you notice any unusual behaviour pertaining to MySQL, please notify support@eskimo.com.

All of our machines simultaneously rebooted at 10:17AM this morning.  Isomedia claims there was no power hit at the facility but given that these machines have been otherwise rock steady I am sceptical.  I have opened a trouble ticket on the issue.

OpenSuse is down for a total rebuild.  An update today left it in pretty much an unusable state and I didn’t have a good backup since I was still installing after the build.

Because I have to deal with power issues at the co-lo facility, this probably will not happen tonight.

Due to updated info from Isomedia, we will be starting shutdown of our equipment at 11:30PM.  Work is expected to start at midnight and be completed by 12:30 and it will take about half an hour to bring all of our servers back up.  Below is the notice we received from Isomedia.

I have to say that while I’m not happy about the failure, I am happy that Isomedia both regularly tests their equipment and keeps their customers informed.  Our previous provider did neither.

This is a REMINDER of work happening tonight at our Eastgate Data Center.

Tonight between 12AM - 2AM 
Electricians complete the cutover of power service to the new transformer. 
THIS WILL CAUSE A PLANNED POWER DISRUPTION TO CUSTOMERS.  We expect the
disruption in power to be relatively short ~30 minutes.  We strongly
recommend all customer equipment be safely powered down prior to the start
of this work.  If you are unable to complete this remotely and are unable to
be onsite Data center technicians will be available to assist in powering
off or powering on equipment.  This step will also isolate the old UPS and
allow for the physical removal of the unit from the facility.  

If you need assistance during the Monday night event please reply to this
email with details on what you would like us to help with.  We apologize in
advance for any inconvenience caused by this event. Further updates will be
made available via ISOMEDIA's network status page at
http://www.isomedia.com/support-network-status.shtml. 


Below is the original notice sent last week

-------------------------


As mentioned previously, last Friday evening the Eastgate UPS encountered
error conditions resulting in the need to switch power into bypass. The
manufacturer (Eaton) has confirmed failures with the internal inverter and
rectifier of the UPS.  Eaton recommended we move forward in installing a new
UPS we had staged in an adjacent space for future expansion.  This work is
significant and as such is being completed in stages to ensure there is no
unexpected impact to customers.  Electricians started work Monday on the
installation of new conduit and electrical service to prepare for the new
UPS and transformer installation. The schedule at this time is as follows:

-Monday Afternoon - October 6th
A new upgraded electrical transformer will be set and connections made to a
new bypass panel.

-Monday Evening/Tuesday Morning 12AM-2AM - October 7th
Electricians complete the cutover of power service to the new transformer. 
THIS WILL CAUSE A PLANNED POWER DISRUPTION TO CUSTOMERS.  We expect the
disruption in power to be relatively short ~30 minutes.  We strongly
recommend all customer equipment be safely powered down prior to the start
of this work.  If you are unable to complete this remotely and are unable to
be onsite Data center technicians will be available to assist in powering
off or powering on equipment.  This step will also isolate the old UPS and
allow for the physical removal of the unit from the facility.  

-Tuesday Afternoon - October 7th
Electricians will connect the new UPS to the new bypass panel and power.

-Wednesday Morning - October 8th
Eaton will be onsite to complete the startup process and testing of the new
UPS. 

-Wednesday Afternoon - October 8th
Technicians will route power service to the new UPS.  This will restore
Eastgate's emergency power systems to fully working conditions.  

---

These time estimates are based on several external factors and technician
expectations.  Although not expected, there is a chance that these estimates
may change as work is completed.  If anything changes we will notify
customers with new expectations.

Other than the transformer cutover Tuesday morning at 12AM there is NO
expectation of disruptions in power to the data center or customer
equipment.  However, until the new UPS is in service the Eastgate Data
Center is not fully protected from a power disruption.  In the case of a
power event the emergency generator will start but there would be a brief
disruption as automatic systems switch power to the genset.

If you need assistance during the Monday night event please reply to this
email with details on what you would like us to help with.  We apologize in
advance for any inconvenience caused by this event. Further updates will be
made available via ISOMEDIA's network status page at
http://www.isomedia.com/support-network-status.shtml. 

Sincerely,
ISOMEDIA Inc.

Because all power will be removed from our equipment during this outage, DNS and authentication services will not be available.

If you establish a DSL or dial-up connection during this interval, that connection should remain up but none of our host services will be available, including DNS.

You can browse the web by using alternate public DNS servers.  There is a list of free public DNS servers at http://pcsupport.about.com/od/tipstricks/a/free-public-dns-servers.htm.