A few weeks ago, I upgraded roundcube to make it compliant with php 8.x.  I did not notice at the time but this broke compose and settings.  At the heart of the issue were database structural changes that the upgrade script did not address.  To resolve this, I had to re-install with a clean database.  It is fixed now, but anything in the database, settings, signature, etc, are lost so you will need to re-enter these things.

If you get a letter like this:

    It is NOT from Eskimo, it is a phishing scam attempting to obtain your login credentials.  Do NOT respond, or at least do NOT provide them any identifiable information.  If you wish to tell them what orifice in which to insert it that is fine, except even doing that verifies to them that they have a valid spam address so better not to.  Do forward to spamtrap@eskimo.com.

Thank you.

     Kernel upgrades are complete.  Mostly went smoothly, some minor self-inflicted issues on a couple of machines but all good now.

     We will be upgrading the remaining systems including the physical hosts to the 6.0.0 kernel release.  We’ve got it presently installed on about a dozen servers and so far only one expedited RCU CPU Stall and that occurred during boot-up on the physical server with the most guest machines.  This can happen even with a good kernel because systemd’s parallelization results in large numbers of processes in run state simultaneously and this in turn can cause the RCU process not to get CPU in the 30ms allocated, but in the week we got only that one and no others where previous 6.0-rc5 and rc6 kernels generated 20-30 in the first five minutes and were basically unusable.

     No single service should be out for more than about ten minutes EXCEPT for yacy which rebuilds it’s indexes every time it restarts and this process takes about half an hour.

     The 6.0 kernel re-wrote the scheduler in a way that benefits multi-core systems.  Although the changes were largely aimed at AMD’s Ryzen CPU’s, we say substantial improvements on our Intel based servers as well, particularly in the area of scheduling latency.  It reduced the time it took our web server to load a WordPress page from about 280ms to 40ms, a seven fold improvement and the largest improvement I’ve seen from any Linux kernel upgrade since kernel .98.

     This affects all Eskimo North services including shared web hosting, private virtual servers, shell servers, e-mail, and our free federated services https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.  If you wish to support our free services, please consider signing up for a Linux shell or web hosting account at https://www.eskimo.com/services/free-trial/.

     Just found out a long time customer, Dave Bruels who had Interlake China Tours, passed yesterday.  I’ve spent many lunches with him and enjoyed his photos from China.  Great man and will miss him.  His wife had passed not long before and whatever the medical reason, I’m sure he passed of a broken heart.  She was a wonderful woman who fought cancer with a courage you rarely see in a human being, hiking and doing the things she loved right up to the end.

     Zorin will be down for a few days, well partially. It won’t be fully operational again for several days. Because Zorin 15 is based upon Ubuntu 16.04, there is NO support for bpfilters, and the currently Linux kernel has deprecated the old iptables filtering scheme so fail2ban and other firewall features do not work.

     Because even though Zorin is a complete rip-off of Ubuntu, if one performs a do-release-upgrade it only updates parts and you end up with a Mix of Ubuntu 16.04 and 20.04 that does not play well together.

     Thus the ONLY way to upgrade Zorin to Zorin 16, which is STILL based on a 2-1/2 year old Ubuntu, 20.04, is a fresh install which means re-installing all the applications and re-configuring everything which is several days work.  I paid for the “Pro” version of the previous release under the promise by Zorin developers that they were working on an in-place upgrade but a year and a half later it is still vaporware so I am only installing Zorin core this time around.

     The 6.0 kernel is working well, so far there has been only one expedited RCU CPU stall and that was on a virtual host during boot-up on a machine that has quite a few guests.  These can occur at this time just because ALL of the guests are busy with start-up and it simply exceeds available CPU cycles for a short time during boot-up.  If it continues to run this clean until this Friday then we’ll upgrade the physical servers to this kernel on that date at 11PM.

     At this point I have Linux kernel 6.0 installed on all but private virtual machines and the physical hosts.  So far it’s running MUCH better than rc5 or rc6,  but not perfect.  I’ve gotten ONE expedited RCU CPU stall in a days time on a dozen or so machines.  With 6.0rc5-6 I would see dozens in the same time frame.

     The scheduler has been substantially re-worked in the 6.0 kernel, most of these changes were aimed at AMD CPUs but it made for substantial performance improvements on Intel as well.  One stall in one day across a dozen machines means one process hung for up to two minutes in that time frame, not a HUGE concern but enough that I don’t want to put it on the physical hosts yet.  The scheduling changes cut the latency for the web server down by approximately 6 fold, that ain’t chicken feed!

     I’ve posted the most recent expedited RCU CPU stall with the bug report I had filed with bugzilla.kernel.org, for anyone interested it is bug #216501 and you can read the details here: https://bugzilla.kernel.org/show_bug.cgi?id=216501.

     I received a note from bugzilla.kernel.org that another user found 6.0.0rc7 no longer had the expedited RCU CPU stalls issue, but before I could grab and try it, the official release of 6.0 came out, so I’m going to build it and give it a try on a few select servers.  If it runs clean during the week I’ll propagate it to the other machines this coming weekend.

     Kernel upgrades are completed on the physical hosts and critical machines.  They are not done on vps1-vps7, but those machines are all single core and single core machines are not experiencing the CPU stall bugs in 5.16-6.0 kernels.

     Some of the other shell servers also are not updated yet but these are sufficiently non-busy that I can easily hit them when nobody is logged in, and for many I have to build kernels yet before I can update.  5.15.71 booted cleanly on all the machines that it is presently installed on.