I believe I’ve got the website issues fixed.

     Ubuntu compiled MySQL 5.7 differently than MySQL 5.6.  Specifically, they included four things that were plugins into the base server.

     When the upgrade installed it, it asked me whether I wanted to keep the original configuration or accept a new configuration.  I opted to keep the original as I have a number of site specific things configured in.

     Also the plugins were kept in a table, mysql.plugins, and so in spite of having these functions compiled in, it also tried to load these plugins.  As you can imagine, this did not go over well.

     So I have deleted those plugins which are now hard coded into the server from the plugin table, and fixed the configuration issues.

     Another issue was that 5.7 now has support for encrypted connections but you have to specify a certificate and key to use them.  It is supposed to disable this functionality if none are found, but instead it tried and bitched about it.

     So I’ve now configured in the ability to support encrypted sessions.

     Mysql keeps hanging, it gets in a loop of errors with socket operation on a non-socket but the only socket I know of is the one to connect and it is still a socket.

     I am researching and trying to understand what is happening to correct it.  If you get cannot connect to the database when you go to our website or anything using MySQL here, that is why.

     This is unrelated to the upgrade.  It started happening several days before.  I had hoped the upgrade would resolve it.  The machine in question is quite generously equipped with memory (16GB), so not a memory issue.

     The upgrade appears mostly to have been a success.  However, on the web server it broke NFS version 4 so I’m currently running version 3.  I know this has to be something broken in my configuration because it is working properly on another 16.04.1 server here.

     I had hoped it would resolve the MySQL hanging and dying thing but it did not even though it replaced it with a newer version so I will continue to troubleshoot that.  I have some ideas on how to narrow it down but it will take some time as I have to wait for it to fail on it’s own as I do not yet know what induces a failure.

     The upgrade of the operating system on the web and ftp server has mostly been completed.  It us up and running on 16.04.1 LTS.

     However, it broke nfs version 4 in a way that cost all NFS mounted partitions to be not to allow any read or write which make it rather useless.  For now I’ve reverted back to NFSv3 until I can figure out what is wrong.

     The web server is in the process of being upgraded from Ubuntu 15.10, which is no longer receiving security updates, to Ubuntu 16.04.1 LTS, which will be supported for some time.

     I hope that it will be stable this time.  When 16.04 LTS came out I upgraded and found that it was not stable, but I’ve been running several other servers on 16.04.1 as well as a couple of workstations and thus far they have all been stable so I am optimistic that the stability issues have been fixed in the ‘.1’ release.

     If not, I have made an image of the machine prior to upgrading so we can revert if necessary.

     There may be an interval of several hours where MySQL and PHP are not working and our website will be down during this time.  This is because of the way that Ubuntu updates, first it downloads all the necessary package, then it unpacks them over the existing packages, THEN it configures the new packages.  Between the time of unpacking and configuring there may be some time where things do not function.

     If you have seen this article, Linux Bug leaves USA Today and other top sites vulnerable to serious TCP/IP hijacking attacks, just want you to know not our site, and if you have a site hosted here and provide an SSL certificate then not yours either.

     This attack only works if a site is not encrypted.  Protect your site from malware injection by encrypting it.  If you provide us with an SSL certificate your site will be encrypted, there is no charge for this by us, you probably will pay a fee to the certificate issuer although there are ways to get free certificates.

     We have also taken measures to protect you from another newly discovered attack that takes advantage of a flaw in Intel hardware having to do with the copy on write function for shared memory pages.

     I am going to take the web server down for about 1/2 hour tonight do back it up prior to starting an update to Ubuntu 16.04.1.  I’ve got 16.04.1 working on several other servers and so far now it has been stable.

     I am also having issues with mysql on this machine that I need to investigate further.  It occasionally is going off into lala land where it is still running but does not respond.

     Tekexplore has published an exploit that uses a hardware fault in Intel processors in combination with kvm, a daemon that finds memory pages with the same content on multiple virtual machines and merges them into one to save memory on the host machine marking that page copy on write so that in theory when it is written a copy is made and the original left alone so it doesn’t affect other virtual machines sharing that page.

     Apparently it is possible to alter a single bit in a page and not trip the copy on write and by altering just a single bit in ssh keys, it is possible to weaken ssh considerably allowing it to be compromised.

     I have turned off kvm on our machines.  It is of limited use in our environment because our servers are not overcommitted and thus the only thing freed up shared pages does is provide more memory for disk caching.  I monitored the logs for some time and found that this was infrequent and so we were getting little performance benefit.

     The mysql server on the web server died this morning with no apparent cause.  I restarted it, it ran for several hours, then died again.

     The second time it died, I ran myisamchk’s on all of the tables, there were quite a few with errors that it fixed.

     Hopefully that will take care of it, else I will reboot the server if it continues.

     Because neither owncloud, nor newly developed nextcloud, nor osTicket in spite of months of promises, will run under PHP 7, and I’ve also got a customers website consisting of 1910 PHP code that totally breaks, I am setting up a new server with two year old software specifically for these applications.

     The new server will be called “antique.eskimo.com”, it will not be something you can login to, it will just be a web server setup like the existing machine except that it will be running PHP 5.6 and Ubuntu 14.04 LTS so that it can run antique PHP code.  Being an LTS release it will still have security updates so will not represent a security threat.

     In addition to being able to run old code, it will also provide a back door to get to our ticket system or webmail when the main web server is inoperable as it will be on a totally different physical host machine.