There is presently a power outage at the co-location facility where our servers are housed and it is running on emergency generator power. There is enough fuel to last through the night during which time we hope that commercial AC power is restored. If not there may be problems. Just letting you all know there is the potential for service interruption due to power if fuel runs out before power is restored.
Around 7pm tonight I found one of my computers was unable to reach our website. After some investigation I found that DNS had failed, none of our servers responded to queries with information about our own servers although they still would resolve outside addresses okay.
I chased the problem down to apparmor on our master DNS server which is a stealth DNS server and not publicly accessible.
In the upgrade from Ubuntu 15.10 to Ubuntu 16.04LTS, Ubuntu renamed all of the DNS directories from /var/named/* to /var/bind/* but did not physically move the directories from their existing location.
This resulted in our master DNS server being unable to access it’s conf file, log file, or any of the zone records. This did not immediately cause problems but once the records in the slave servers expired, then they were unable to answer queries regarding those records.
I corrected the paths in apparmor and now it is able to access it’s conf file. The master records have been propagated to the slave servers which are now correctly answering queries.
Fedora 24 is now available at fedora.eskimo.com. I was unable to successfully do an online upgrade and so had to re-install fresh. That was fraught with some difficulties as well. The virt-manager in Ubuntu 16.04 selects a hardware profile for fedora that doesn’t work. I was able to manually change it and install it.
I have not yet re-installed all the applications we had or configured mail server on it, so sending mail from fedora probably will not work at this time.
I am impressed with the improvement from Fedora 23 which in my opinion should never have been released. It was just super ugly broken. The dnf command used to install software (does not function) now mostly does function, much better than it did in Fedora 23, and they have fixed groups in yum extender to where they mostly work again.
One thing yum extender does not do well is resolve dependencies, still issues there. There appears to be no tool that really works automatically for this so when there are dependency issues it is pretty much necessary to figure it out by hand. My opinion is they should get over the “not invented here” issue and take synaptic and modify it to work with rpms.
If you use x2go to connect, you may get a small 800×600 screen size. To fix this bring up a terminal and type xrandr -s HORZxVERT where HORZ is your screens horizontal resolution and VERT your screens vertical resolution in pixels. The cause of this problem is that x2go’s xserver has xrandr version 1.2 and with Fedora 24 and Ubuntu 16.04, at least version 1.3 is required. The command line version however is 1.5, so it will work fine.
Services are restored on mail and shell servers. There wasn’t really a problem with the shell server, rather it was an issue with the server that held the mail spool files.
For some reason, after an upgrade the network interfaces all went away. After a reboot it mysteriously revived. It would appear there are bugs in the systemd startup files in 16.04 yet.
While everything is currently backup, I am going to upgrade the operating system on the machine that holds users home directories so there will be another short interruption but once it is complete all operating system software will be current.
I am unable to login to either of the physical hosts hosting the mail spool and home directories. I will need to go to the co-location facility to reboot. This can be anywhere from a 30-minute to 2-hour drive depending upon traffic. I am headed that way to restore services as soon as I can.
We are having difficulties with the server that houses the mail spool files. It is in the middle of an upgrade and it has interrupted some services. The mail server is affected and thus web mail and customer smtp / imap are presently dysfunctional.
We are working on completing this software upgrade which will restore services. No mail will be lost, any incoming mail will be spooled and delivered when the server is back to being fully functional. The mail spool itself is on a RAID10 partition so even loss of hardware will not lose files and they are backed up weekly in addition to the RAID.
Because this is a physical host that is involved, I will need to drive down to the co-lo to reboot when this is finished and so will be unavailable for telephone support for several hours.
Please use an alternate shell server during this time frame. See Services->Shells->Servers for alternate servers.
I’ve upgraded the SquirrelMail Webmail to the most recent 1.5.2svn release as of 9/10/16. They never change the version number on the development version nor is there any decent documentation so no idea what has changed but it’s the very latest code now.