Web Server Broke

     Web server stopped serving pages this morning.  Not sure what went wrong but I was able to login to it okay and Apache still showed running.of life, will be stable.

     An update, it is not a problem with the web server but a problem with WordPress plugins.  For some reason the particular collection I use which got along fine until an update to Comet Cache now hangs.

     As a temporary fix I am going to remove Comet Cache from our main site.  This will make it slightly slower until an update to fix becomes available.

Power Outage

     There is presently a power outage at the co-location facility where our servers are housed and it is running on emergency generator power.  There is enough fuel to last through the night during which time we hope that commercial AC power is restored.  If not there may be problems.  Just letting you all know there is the potential for service interruption due to power if fuel runs out before power is restored.

DNS Failure Resolved

     Around 7pm tonight I found one of my computers was unable to reach our website.  After some investigation I found that DNS had failed, none of our servers responded to queries with information about our own servers although they still would resolve outside addresses okay.

     I chased the problem down to apparmor on our master DNS server which is a stealth DNS server and not publicly accessible.

     In the upgrade from Ubuntu 15.10 to Ubuntu 16.04LTS, Ubuntu renamed all of the DNS directories from /var/named/* to /var/bind/* but did not physically move the directories from their existing location.

     This resulted in our master DNS server being unable to access it’s conf file, log file, or any of the zone records.  This did not immediately cause problems but once the records in the slave servers expired, then they were unable to answer queries regarding those records.

     I corrected the paths in apparmor and now it is able to access it’s conf file.  The master records have been propagated to the slave servers which are now correctly answering queries.

Fedora Upgraded

Fedora 24     Fedora 24 is now available at fedora.eskimo.com.  I was unable to successfully do an online upgrade and so had to re-install fresh.  That was fraught with some difficulties as well.  The virt-manager in Ubuntu 16.04 selects a hardware profile for fedora that doesn’t work.  I was able to manually change it and install it.

     I have not yet re-installed all the applications we had or configured mail server on it, so sending mail from fedora probably will not work at this time.

     I am impressed with the improvement from Fedora 23 which in my opinion should never have been released.  It was just super ugly broken.  The dnf command used to install software (does not function) now mostly does function, much better than it did in Fedora 23, and they have fixed groups in yum extender to where they mostly work again.

     One thing yum extender does not do well is resolve dependencies, still issues there.  There appears to be no tool that really works automatically for this so when there are dependency issues it is pretty much necessary to figure it out by hand.  My opinion is they should get over the “not invented here” issue and take synaptic and modify it to work with rpms.

     If you use x2go to connect, you may get a small 800×600 screen size.  To fix this bring up a terminal and type xrandr -s HORZxVERT where HORZ is your screens horizontal resolution and VERT your screens vertical resolution in pixels.  The cause of this problem is that x2go’s xserver has xrandr version 1.2 and with Fedora 24 and Ubuntu 16.04, at least version 1.3 is required.  The command line version however is 1.5, so it will work fine.

Services Restored

     Services are restored on mail and shell servers.  There wasn’t really a problem with the shell server, rather it was an issue with the server that held the mail spool files.

     For some reason, after an upgrade the network interfaces all went away.  After a reboot it mysteriously revived.   It would appear there are bugs in the systemd startup files in 16.04 yet.

     While everything is currently backup, I am going to upgrade the operating system on the machine that holds users home directories so there will be another short interruption but once it is complete all operating system software will be current.

Servers

     I am unable to login to either of the physical hosts hosting the mail spool and home directories.  I will need to go to the co-location facility to reboot.  This can be anywhere from a 30-minute to 2-hour drive depending upon traffic.  I am headed that way to restore services as soon as I can.

Mail / Shellx

     We are having difficulties with the server that houses the mail spool files.  It is in the middle of an upgrade and it has interrupted some services.  The mail server is affected and thus web mail and customer smtp / imap are presently dysfunctional.

     We are working on completing this software upgrade which will restore services.  No mail will be lost, any incoming mail will be spooled and delivered when the server is back to being fully functional.  The mail spool itself is on a RAID10 partition so even loss of hardware will not lose files and they are backed up weekly in addition to the RAID.

     Because this is a physical host that is involved, I will need to drive down to the co-lo to reboot when this is finished and so will be unavailable for telephone support for several hours.

     Please use an alternate shell server during this time frame.  See Services->Shells->Servers for alternate servers.

SquirrelMail Webmail

     I’ve upgraded the SquirrelMail Webmail to the most recent 1.5.2svn release as of 9/10/16.  They never change the version number on the development version nor is there any decent documentation so no idea what has changed but it’s the very latest code now.

User Meeting

     About 20 people showed for the user meeting which was good.

     It was fun meeting a couple of my co-workers from PNB/USWest/Qwest days.  They had changed so much I did not recognize them at first but then it has been 21 years since I left.

     Regarding a few projects in the works, I am working on getting ownCloud 9.1 working.  I had setup a server with the previous long term version of Ubuntu in order to get the older ownCloud, osTicket, and one customers website working again since the upgrade of our server broke these things but then a new ownCloud came out that did seem to work with PHP7 so I decided instead to try to get it operational.

     PHP7 is desirable to PHP5.6 because it speeds up execution by about 30%, that makes php based sites like WordPress sites, and many of the applications we have run faster.

     I want Eskimo to succeed and I know for us to succeed we have to help you succeed.  Any business website is heavily dependent upon Google searches for new traffic.  Google has gotten very good at finding sites with relevant content.  When it has a bunch to offer, and it almost always does, several things are very important after relevancy, and that is mobile friendliness, SSL encryption, and speed.  Sites that have all of these things get bumped to the top of the search results and that’s where you want to me and we can help you get there.

     Another service I am working on is providing streaming audio (and in theory video) via Icecast so that people interested in creating an Internet radio station or offering some streaming service, will have a way to do so.

     This mostly is an old interest of mine, Radio, being re-kindled in the era of modern technologies, a desire to understand how to current way of doing thing works from a technical and operational perspective, and as always a desire to give the average person a venue for getting their information, ideas, views, art, etc, out there.