I deeply apologize for the lengthy downtime of the website and MySQL database tonight.
Ubuntu kicked out an upgrade to mysql server that seriously broke it. With the update installed, as soon as someone tried to connect from anywhere but localhost, the server would get stuck in an infinite loop complaining of bad file descriptors and not a socket.
I tried to revert to the previous and only other version on the Ubuntu repository but it was also broke,
In addition, they hard coded in a limit of 65536 file descriptors which is not adequate for our sites traffic.
I tried to migrate to MariaDB but there was no clean path from MySQL 5.7 to MariaDB and nothing I tried would work.
I then attempted to install Community MySQL directly from the site but owing to the fact that I had a mysql user in the NIS database, it broke the install. However it did not log the cause or print an error that would give me a clue as to what the problem was so it took a lot of tracing and plodding around to find the cause and fix it.
We are now up and running with Community MySQL Server version 5.7 and it is back to functioning properly.