Outage Time Change 11:30PM Oct 6th – 1AM Oct 7th

Due to updated info from Isomedia, we will be starting shutdown of our equipment at 11:30PM.  Work is expected to start at midnight and be completed by 12:30 and it will take about half an hour to bring all of our servers back up.  Below is the notice we received from Isomedia.

I have to say that while I’m not happy about the failure, I am happy that Isomedia both regularly tests their equipment and keeps their customers informed.  Our previous provider did neither.

This is a REMINDER of work happening tonight at our Eastgate Data Center.

Tonight between 12AM - 2AM 
Electricians complete the cutover of power service to the new transformer. 
THIS WILL CAUSE A PLANNED POWER DISRUPTION TO CUSTOMERS.  We expect the
disruption in power to be relatively short ~30 minutes.  We strongly
recommend all customer equipment be safely powered down prior to the start
of this work.  If you are unable to complete this remotely and are unable to
be onsite Data center technicians will be available to assist in powering
off or powering on equipment.  This step will also isolate the old UPS and
allow for the physical removal of the unit from the facility.  

If you need assistance during the Monday night event please reply to this
email with details on what you would like us to help with.  We apologize in
advance for any inconvenience caused by this event. Further updates will be
made available via ISOMEDIA's network status page at
http://www.isomedia.com/support-network-status.shtml. 


Below is the original notice sent last week

-------------------------


As mentioned previously, last Friday evening the Eastgate UPS encountered
error conditions resulting in the need to switch power into bypass. The
manufacturer (Eaton) has confirmed failures with the internal inverter and
rectifier of the UPS.  Eaton recommended we move forward in installing a new
UPS we had staged in an adjacent space for future expansion.  This work is
significant and as such is being completed in stages to ensure there is no
unexpected impact to customers.  Electricians started work Monday on the
installation of new conduit and electrical service to prepare for the new
UPS and transformer installation. The schedule at this time is as follows:

-Monday Afternoon - October 6th
A new upgraded electrical transformer will be set and connections made to a
new bypass panel.

-Monday Evening/Tuesday Morning 12AM-2AM - October 7th
Electricians complete the cutover of power service to the new transformer. 
THIS WILL CAUSE A PLANNED POWER DISRUPTION TO CUSTOMERS.  We expect the
disruption in power to be relatively short ~30 minutes.  We strongly
recommend all customer equipment be safely powered down prior to the start
of this work.  If you are unable to complete this remotely and are unable to
be onsite Data center technicians will be available to assist in powering
off or powering on equipment.  This step will also isolate the old UPS and
allow for the physical removal of the unit from the facility.  

-Tuesday Afternoon - October 7th
Electricians will connect the new UPS to the new bypass panel and power.

-Wednesday Morning - October 8th
Eaton will be onsite to complete the startup process and testing of the new
UPS. 

-Wednesday Afternoon - October 8th
Technicians will route power service to the new UPS.  This will restore
Eastgate's emergency power systems to fully working conditions.  

---

These time estimates are based on several external factors and technician
expectations.  Although not expected, there is a chance that these estimates
may change as work is completed.  If anything changes we will notify
customers with new expectations.

Other than the transformer cutover Tuesday morning at 12AM there is NO
expectation of disruptions in power to the data center or customer
equipment.  However, until the new UPS is in service the Eastgate Data
Center is not fully protected from a power disruption.  In the case of a
power event the emergency generator will start but there would be a brief
disruption as automatic systems switch power to the genset.

If you need assistance during the Monday night event please reply to this
email with details on what you would like us to help with.  We apologize in
advance for any inconvenience caused by this event. Further updates will be
made available via ISOMEDIA's network status page at
http://www.isomedia.com/support-network-status.shtml. 

Sincerely,
ISOMEDIA Inc.

 

 

DNS and Authentication During Outage

Because all power will be removed from our equipment during this outage, DNS and authentication services will not be available.

If you establish a DSL or dial-up connection during this interval, that connection should remain up but none of our host services will be available, including DNS.

You can browse the web by using alternate public DNS servers.  There is a list of free public DNS servers at http://pcsupport.about.com/od/tipstricks/a/free-public-dns-servers.htm.

Maintenance Outage Oct 6 11:45PM – 2:15AM

The above time frame may not be exact but sometime around midnight there will be an outage of all of our servers as power needs to be turned off at the co-location facility to do electrical work to replace a UPS for the facility that has a fried inverter.

They have advised us that it would be best to have our equipment powered down before the proceed to protect it from potential damage so I will be there to do so and bring it back up afterwards.

Shellx – Sound Fixed – Other Info

I was able to get sound working again on Shellx, and fix a nasty problem with pirut, the add/remove software program, but in order to do so I had to remove a bunch of stuff and downgrade others to get versions in sync.  I’m still re-installing things but if there is something in particular that you need, please e-mail support@eskimo.com and I’ll bump it up on the priority stack.

Shellx – Planned Outage Friday 10/3 11pm – Saturday 10/4 2am

We will be taking shellx out of service late Friday evening and returning it to service early Saturday morning.  It will be up and down intermittently during this time frame in order to attempt to fix a problem with gstreamer audio so that flash videos and games and other applications that use gstreamer will properly produce sound.

Sound was broken by a kernel upgrade provided by the CentOS team that did not include all of the necessary drivers for gstreamer.  We will be rolling the kernel back to a working version or possibly compiling a kernel if we can’t get one supplied by the CentOS team to work.

Shellx

Shellx is restored to service.  X2Go was borked by an upgrade that was supposed to add support for gnome3-fallback and cinnamon sessions, neither of which are desktops available on this server.  Given that, I’ve downgraded to the previous working version.

Sound is still borked.  Fixing it is going to require downgrading kernel because drivers aren’t available for the current kernel.  Since this will most probably require multiple reboots, it’s going to wait until late Friday night.

Sound not reliant on gstreamer works but unfortunately that’s a minority of applications.

Shellx Borked

Something in shellx pertaining to X-sessions is borked in such a way that X2Go fails to launch a session.  I’ve totally reinstalled the X2Go software to no avail.

I am going to take shellx down to recover from a previously working state and then re-apply updates one at a time so I can determine which, if any, of them may have broke the server.

Shellx will be down for approximately one hour.

UPS Outage

The UPS at the co-location facility where we have our equipment located has failed.  They are working on repairing it.  In the meantime, if city power fails in Bellevue, a brief outage will result while equipment is switched to generator.

Maintenance 9/21/14 00:05 – 02:00

I will be performing maintenance tonight that will involve a reboot of the main file server to make a new Linux kernel active, and then taking down other machines, specifically shellx, scientific, mail, and ftp, for imaging.

I will not be taking down the other shell servers during this interval, so except for the initial file server reboot, the others will remain functional.  The file server reboot will take place between approximately 00:05 – 00:30.