Debian is back up and functioning. I’m still applying updates that were rolled back but the corruption is resolved and it is operational.
Our shell server debian.eskimo.com is down for repairs. The ‘apt’ database has somehow gotten corrupted and I’m unable to figure out how to correct it so I’m reverting the machine to a image made prior to it’s corruption and then will re-apply updates. This should require about 45 minutes.
After the power glitch this morning when everything rebooted, it left some of the MySQL tables corrupted. I didn’t notice until I tried to go into the Forums just now and I received no complaints about broken blogs, etc.
I’ve run myisamchk -r on all the tables with the server down to repair any inconsistencies and brought the server back online. Forums are working now.
If you notice any unusual behaviour pertaining to MySQL, please notify email@example.com.
All of our machines simultaneously rebooted at 10:17AM this morning. Isomedia claims there was no power hit at the facility but given that these machines have been otherwise rock steady I am sceptical. I have opened a trouble ticket on the issue.
OpenSuse is down for a total rebuild. An update today left it in pretty much an unusable state and I didn’t have a good backup since I was still installing after the build.
Because I have to deal with power issues at the co-lo facility, this probably will not happen tonight.
Due to updated info from Isomedia, we will be starting shutdown of our equipment at 11:30PM. Work is expected to start at midnight and be completed by 12:30 and it will take about half an hour to bring all of our servers back up. Below is the notice we received from Isomedia.
I have to say that while I’m not happy about the failure, I am happy that Isomedia both regularly tests their equipment and keeps their customers informed. Our previous provider did neither.
This is a REMINDER of work happening tonight at our Eastgate Data Center. Tonight between 12AM - 2AM Electricians complete the cutover of power service to the new transformer. THIS WILL CAUSE A PLANNED POWER DISRUPTION TO CUSTOMERS. We expect the disruption in power to be relatively short ~30 minutes. We strongly recommend all customer equipment be safely powered down prior to the start of this work. If you are unable to complete this remotely and are unable to be onsite Data center technicians will be available to assist in powering off or powering on equipment. This step will also isolate the old UPS and allow for the physical removal of the unit from the facility. If you need assistance during the Monday night event please reply to this email with details on what you would like us to help with. We apologize in advance for any inconvenience caused by this event. Further updates will be made available via ISOMEDIA's network status page at http://www.isomedia.com/support-network-status.shtml. Below is the original notice sent last week ------------------------- As mentioned previously, last Friday evening the Eastgate UPS encountered error conditions resulting in the need to switch power into bypass. The manufacturer (Eaton) has confirmed failures with the internal inverter and rectifier of the UPS. Eaton recommended we move forward in installing a new UPS we had staged in an adjacent space for future expansion. This work is significant and as such is being completed in stages to ensure there is no unexpected impact to customers. Electricians started work Monday on the installation of new conduit and electrical service to prepare for the new UPS and transformer installation. The schedule at this time is as follows: -Monday Afternoon - October 6th A new upgraded electrical transformer will be set and connections made to a new bypass panel. -Monday Evening/Tuesday Morning 12AM-2AM - October 7th Electricians complete the cutover of power service to the new transformer. THIS WILL CAUSE A PLANNED POWER DISRUPTION TO CUSTOMERS. We expect the disruption in power to be relatively short ~30 minutes. We strongly recommend all customer equipment be safely powered down prior to the start of this work. If you are unable to complete this remotely and are unable to be onsite Data center technicians will be available to assist in powering off or powering on equipment. This step will also isolate the old UPS and allow for the physical removal of the unit from the facility. -Tuesday Afternoon - October 7th Electricians will connect the new UPS to the new bypass panel and power. -Wednesday Morning - October 8th Eaton will be onsite to complete the startup process and testing of the new UPS. -Wednesday Afternoon - October 8th Technicians will route power service to the new UPS. This will restore Eastgate's emergency power systems to fully working conditions. --- These time estimates are based on several external factors and technician expectations. Although not expected, there is a chance that these estimates may change as work is completed. If anything changes we will notify customers with new expectations. Other than the transformer cutover Tuesday morning at 12AM there is NO expectation of disruptions in power to the data center or customer equipment. However, until the new UPS is in service the Eastgate Data Center is not fully protected from a power disruption. In the case of a power event the emergency generator will start but there would be a brief disruption as automatic systems switch power to the genset. If you need assistance during the Monday night event please reply to this email with details on what you would like us to help with. We apologize in advance for any inconvenience caused by this event. Further updates will be made available via ISOMEDIA's network status page at http://www.isomedia.com/support-network-status.shtml. Sincerely, ISOMEDIA Inc.
Because all power will be removed from our equipment during this outage, DNS and authentication services will not be available.
If you establish a DSL or dial-up connection during this interval, that connection should remain up but none of our host services will be available, including DNS.
You can browse the web by using alternate public DNS servers. There is a list of free public DNS servers at http://pcsupport.about.com/od/tipstricks/a/free-public-dns-servers.htm.
The above time frame may not be exact but sometime around midnight there will be an outage of all of our servers as power needs to be turned off at the co-location facility to do electrical work to replace a UPS for the facility that has a fried inverter.
They have advised us that it would be best to have our equipment powered down before the proceed to protect it from potential damage so I will be there to do so and bring it back up afterwards.
I was able to get sound working again on Shellx, and fix a nasty problem with pirut, the add/remove software program, but in order to do so I had to remove a bunch of stuff and downgrade others to get versions in sync. I’m still re-installing things but if there is something in particular that you need, please e-mail firstname.lastname@example.org and I’ll bump it up on the priority stack.
We will be taking shellx out of service late Friday evening and returning it to service early Saturday morning. It will be up and down intermittently during this time frame in order to attempt to fix a problem with gstreamer audio so that flash videos and games and other applications that use gstreamer will properly produce sound.
Sound was broken by a kernel upgrade provided by the CentOS team that did not include all of the necessary drivers for gstreamer. We will be rolling the kernel back to a working version or possibly compiling a kernel if we can’t get one supplied by the CentOS team to work.