CenturyLink DSL

     Our customers using CenturyLink circuits, the backhaul provider, Mammoth Communications, experienced an equipment failure during maintenance that caused the  loss of both primary and secondary BGP-4 routers.  This basically resulted in a loss of routing.  All equipment is back in service and coming up now.  Service should be fully restored in about twenty minutes.

Client Mail Server “mail.eskimo.com” Up

     I was unable to determine the reason mail.eskimo.com was returning an error code when trying to send.  There was no resource exhaustion I could identify, plenty of RAM, Disk, load was low, memory not exhausted.  Restarting postfix didn’t fix it and there didn’t appear anything bad in the queue.

     I opted for the Windows solution at this point and rebooted the server.  After the reboot it accepted and sent mail again.

     A post mortem analysis of the logs showed that the smtp daemon was unable to open /dev/null.  Not sure why it needs that but apparently something did go wrong in the operating system kernel. This is the first time I’ve ever seen a kernel issue on that version of Linux.

Client Mail server mail.eskimo.com Sending

     We are experiencing difficulties with our client mail server, “mail.eskimo.com”, returning 451 service unavailable when sending.

     I do not yet know what is causing this.  I have been unable to identify any issue with the software configuration, mailq, or exhausted system resources.

     I am still troubleshooting.

Playing with caching

     I’m experimenting with caching out of memory by using /tmp as a tmpfs file system and putting the cache directory there.  This has the disadvantage of clearing the cache when the machine boots but it also serves pages faster and reduces disk I/O somewhat.

     The unfortunate side effect of this is that the picture at the top will no longer change.

MySQL Fixed

     Finally got our MySQL woes fixed.  The issue is that 5.7.14 which is distributed in the Ubuntu 16.04.1 LTS repositories, is broken, at least in our environment with our workload.

     I installed MySQL 5.7.15, one point issue newer from the MySQL Community Server repository and it has performed flawlessly thus far.

     You can now use MySQL tools on the shell server to connect to and manipulate your databases.  The server that has the same version of PHP as the main web server is ubuntu.eskimo.com, which like our web server, is running Ubuntu 16.04.1 LTS.

MySQL 5.7

     I’ve written a small script that pings mysql once a minute and if it does not respond immediately it restarts it.  This way at least we won’t have lengthy outages while I figure it out.

MySQL 5.7 Problems – MariaDB

     The version of mysql provided with Ubuntu 16.04 is version 5.7, an upgrade from the previous version 5.6.

     MySQL 5.7 has proven to be unstable and locks up periodically.  I have spent several days trying to troubleshoot this, fixed a number of issues, and optimized the configuration.  Still it locks up and dies from time to time.

     I tried to upgrade to MariaDB 10.  An in place upgrade failed, and dumping the old database and sourcing the resulting sql file in MariaDB also failed.  It completes but the grant tables aren’t recognized which basically makes it useless.  I have yet to find a fix for this.  I apologize for this instability.

     If anyone can find a clean path to migrate from MySQL 5.7 to MariaDB 10.0, please let me know (e-mail: nanook@eskimo.com) or call me at 206-812-0051.



I’ve got connections from the shell servers disabled temporarily until this issue can be resolved as it trips this bug instantly.  If anybody knows anyplace I can get help with this, a good forum for mysql issues etc, please do let me know (e-mail nanook@eskimo.com).