Fedora Up

     Fedora is back up.  There was nothing in the logs that gave any hint as to why it rebooted or why it started the emergency shell.

Starting Emergency Shell…
[ OK ] Started Emergency Shell.
[ OK ] Reached target Emergency Mode.

Fedora Down

     The Fedora 21 shell server is presently down.  Sometime during the night it rebooted into “emergency mode”.

     I am in the process of troubleshooting and working on returning it to service.

Again Mammoth Maintenance Not Done

     Again, Mammoth DSL maintenance that was supposed to have completed by 8AM mountain time (7AM Pacific) is still in progress.  It is taking longer for them to swing the circuits than anticipated.

     This effects most Seattle DSL customers on CenturyLink DSL circuits.  They are working to restore out of service circuits as fast as possible.


 

Date: 3/11/2015
Start time: 5:00 am MST
End time: 8:00 am MST
Affected: 74/OBGJ/090579//ACSO

Detail:

Extended: Maintenance is being extended in order to complete the OC3 Groom.
Most customers will see a short drop 30 seconds to 1 minute and restore to
service. As usual some customers may need to reboot to restore service.

Maintenance is being performed in order to groom the OC3 circuits to the new
OC12 circuit. All ATM customers on: 74/OBGJ/090579//ACSO will be affected.
Customers will go down and come up as their circuits are groomed to the new
service, some customers may need to reboot to restore services. Estimated
downtime is roughly 2 hours but maybe as short as 5 minutes for some
customers.

Mammoth DSL Maintenance Still In Progress

     Mammoth DSL maintenance that was supposed to have completed by 8AM mountain time (7AM Pacific) is still in progress.  It is taking longer for them to swing the circuits than anticipated.

     This effects most Seattle DSL customers on CenturyLink DSL circuits.  They are working to restore out of service circuits as fast as possible.

Reverted to 2.26 kernels

Had to revert to old kernels because there is a incompatibility between the nfs-utils, specifically idmapd, provided with CentOS 6 and the later 3.x kernels.

Unfortunately, I was unable to build a modern version of nfs-utils because of missing libraries I was unable to chase down this evening.

So I will continue to try to chase down the missing library and in the meantime will probably build a 2.26.39 (the last 2.26 kernel) with a lot of the garbage pulled out and pre-emptive scheduling. Will at least get some, but not all, of the performance gains that way.

It really takes the wind out of my sails to spend so much time on something, thought I had it working, and then have to backtrack.

Maintenance This evening 8pm-midnight

     I will be rebooting the host machines which will cause all Linux guests to freeze for about 15-20 minutes that are guests of any given host.

     The 3.19.1 kernel so hugely improved performance over the 2.26.x kernels on the guest machines that it makes sense to get them on the hosts as well.  Because of my wife’s work schedule, tonight is one of the few nights where I have the car and can go do this so tonight it is.

     The 3.19.1 kernel cut the memory usage for a given workload almost in half, made things go faster, and reduced the overall load on the machines I’ve put it on.  The memory usage reduction is most significant as it allows more RAM then to be used as cache so less disk access is required.  Whenever anything is slow, it almost always comes down to waiting on disk I/O.

Linux 3.1.19

     Linux 3.1.19 is the latest stable kernel but every distribution out there is at least three minor point releases behind, and in the case of CentOS 6, they’re still on a 2.6 kernel.

     I managed to compile a monolithic (all modules needed compiled in) 3.1.19 kernel under CentOS 6 and successfully got it to boot and run.  There are a few minor glitches such as nfs-utils needs to be updated, but the new kernel falls back to old behaviour so it still works even though it emits a few bitch messages.

     The newer kernels have some capabilities that allow Apache, Bind, Postfix, and some other programs to operate more efficiently.  Building a kernel from scratch also allows me to remove a lot of unneeded cruft that just wastes memory and CPU cycles.  It allows me to adjust parameters to optimize for our hardware environment and work loads.

     I’ve attempted this in the past but this is the first time I’ve been successful with 3.x kernels.

Spring Forward

     Remember everyone, tonight is the night we Spring Forward, technically tomorrow morning at 2AM.  Set your clocks before you go to bed tonight and get ready for not enough sleep but a bit more evening daylight.