Some of our dial access numbers in the Sacramento area are O1 numbers. I received this, not the first notice regarding this outage but thought I would post publicly in case others are having issues:
Dear O1 Customer,
Currently there is an outage affecting LATA 726 (Sacramento). Dialup users may experience busy signals, or trouble connecting while maintenance is being performed.
We apologize for this inconvenience and appreciate your patience in this matter.
As with all technical contacts, if you experience any issues that you believe may be related to the above message, or have any questions concerning this email, please contact our 24/7 Network Operations Center at (888-444-1111),OPT #2 or via email at email@example.com.
Since you are not a direct O1 customer you can not use these contacts to get help, but here are two Sacramento numbers on an alternate network you can use if the primary access numbers in Sacramento are down, 916-282-0155 – Sacramento (Main), and 916-609-0155 – Sacramento (North). These are both MegaPOP numbers and so should be unaffected by this outage. I believe this relates to an earlier fiber re-route being made necessary by flooding.
I am about a month and a half behind in accounting, getting notices out, expiring accounts, etc. If you know your account is near time for renewal feel free to contact me and I’ll be happy to look up the information for you. As account expirations come up, if you have not yet been notified your account will not be turned off until you’ve been notified of payment due and given an appropriate amount of time to respond.
This has come about through a number of things happening, failed equipment screwing up my sleep schedule, minor illnesses increasing sleep requirement, and a lot of software issues being discovered and need to devote more time to securing things.
Actually Eskimo did not crash, something just went wrong with sshd. I was able to login to the console fine.
Eskimo crashed, even though I’ve moved it to different hardware (I am still trying to repair the old), so I will head down to the co-lo in a bit to reboot.
Scientific Linux is slow since moving to new hardware. It’s not load or speed of the new hardware that is the issue, Centos6 is served off of the same hardware and does not have this problem.
Both are derived from the same Redhat 6.8 code, but I’m getting some weird errors on Scientific that I am not getting on Centos6. I’m going to try recreating the virtual machine in a bit and if that does not work perhaps reload it from scratch. I did have to do this with Centos6, (shellx became so corrupted it was unmaintainable). The RPM system that these Redhat derived operating systems use tends to be easily corrupted and difficult to fix beyond a certain point.
But this problem seems to relate to the virtual machine as it worked fine on the old host.
I’ve moved all the services off of failed hardware but because some machines still have file systems mounted off of there and will not let me umount, not even with the force option, I’m going to have to reboot a bunch of stuff to get that corrected so there will unfortunately have to be further interruptions although they should be relatively brief.
I am still in the process of shuffling virtual hosts around to get them all moved off of unstable hardware. Things will lag a bit while this is in progress as copying files of several hundred gigabytes across the net comes close to saturating the ethernet links.
Mail is now available on centos6.
Mail is now available via webmail, smtp/pop3/imap. I am still working on fixing mount points for shell servers. Incoming will be backed up and take a little while to catch up.
The physical machine is up and I have copied the mail spool to a new machine. Now I am moving the virtual machines off of old hardware and changing mount points to reflect the mail spools new location.
The physical host upon which the mail spool sits is down. It is also the NIS slave that many of the machines slave off of so they won’t authenticate.
I have to wait for a ride to get to where my car is and then from there make my way down to the co-location facility. Because it will be rush hour by the time I get to my car it is going to take a while to get to the co-location facility.
Once I get the mail server back up I am going to begin copying the spool from it to another physical machine and then move the virtual hosts off that box. The fact that it has crashed twice in a weeks time suggests there is some marginal hardware issue with the machine.
Since I had planned to upgrade it anyway I am going to move everything off the existing machine and then replace the motherboard, CPU, and memory.
I hope to have the old hardware back up and operational by around 9AM. It just depends upon how bad traffic is.