Mariadb, the database that is used to provide SQL service for most web apps requiring it (a handful use postgresql) hung when some internal process failed to remove a lock and everything else piled up behind it.  I could not restart it with either systemd or mysqladmin, and even a regular kill signal did not work, kill -9 eventually did but even the response to that was delayed by several minutes.  There was nothing in either the kernel logs or the mariadb logs that would provide a clue as to why this happened and it is a failure mode I have not seen before.

     Because mariadb was not shut-down properly, database corruption is a possibility.  Accordingly, I am running mariadb-repair –all-databases to check for and repair any inconsistencies. This may slow response slightly for the next hour or so.