I am having problems with the mail server being unstable with the new kernel, I am going to try 5.13.5 (came out since we installed 5.13.4) to see if it resolves.  This will take about 20 minutes to compile and install.

     Going to do another kernel upgrade this Friday starting at 11PM providing a crash doesn’t force one sooner.

     I didn’t want to move to 5.13 so early in the release cycle, actually didn’t want to move to it at all but was forced to by the Sequoia exploit.

     There are lots of bugs, most of them minor networking bugs affecting outlying cases but a few that are relevant and some memory leaks.  This early in the release cycle this is unfortunately the norm.

     This upgrade will not require any futzing with drivers since it is only a minor point release so everything should get done by 11:30PM or so.  Might be some NFS issues to clean up although after I last rebooted the mail server ALL of the NFS clients rebound properly so I think they are finally getting NFS stable.

     It has been a long grueling upgrade, made particularly difficult by one server having drivers not officially supported under 5.13.x kernels, but I got it working.

     All NFS mounts and NIS bindings have been verified.

     Another issue this time around, at some point some bug(s) have been introduced into systemd causing it to be unreliable at starting some services during boot.  But everything is up and running now.

     I’m behind in getting receipts and expiration notices out but should be able to get caught up on that this weekend providing no other major disasters.  Thanks for the folks at QualSys for not waiting for the kernel folks to really adequately test things before releasing an exploit for the Sequoia exploit (and yes I do mean this sarcastically).

    I have removed all tickless kernels prior to 5.13.4 because they are vulnerable to a recently announced root exploit that goes by “Sequoia”, CVE-2021-33909.

     5.13.4 tickless kernels are available for all Debian and now for all Redhat based Intel and AMD x86-64 systems (any 64 bit Intel or AMD CPU).  Previously I only made ‘.deb’ packages available but because I had some Redhat based systems for which the distributor has not made 5.13.4 available yet, I compiled for this platform as well.

     There are two kernels provided.  The ‘client’ kernel meant for home systems and work stations, optimized for low latency, it is 1000HZ clock tickless (tickless means clock interrupts cause context switches only when there is work scheduled, this saves a lot of wasted CPU cycles) and fully preemptive.  The ‘server’ kernel is meant for servers and is optimized for maximal throughput and has a 100HZ clock and is non-preemptive.  It is also, like the client kernel, tickless.

     On servers, the tickless feature is particularly useful where a large number of virtual machines are hosted as each machine adds to the host CPU load and all those clock interrupt reschedules add up to a lot of wasted CPU cycles.

     On laptops the tickless kernel is useful because saving CPU cycles extends battery life.

     To install these kernels, first download ALL three ‘.deb’ files for Debian based distributions or both ‘.rpm’ files for RPM distributions located in the client or server directory.  You can download via the web at https://www.eskimo.com/kernel, or via ftp at ftp://ftp.eskimo.com/pub/kernel using anonymous login.  If you are behind NAT, to use ftp after you login type passive to put the server in passive mode.

     Then  on Debian based systems, install with dpkg -i *.deb.  On Redhat based systems install with rpm -i *.rpm.  If on Redhat you get a complaint about headers conflicting with existing header package, remove the existing package with rpm –nodeps -e existing (whatever the existing package name is).  The –nodeps is important here otherwise removing the headers will remove some 300-odd dependent packages.

    If you run into any issues, please generate a ticket at: https://www.eskimo.com/support/osTicket/.  Thank you.

    All public facing servers except the private virtual servers have been upgraded to 5.13.4 which has the Sequoia exploit fixed.

     I will work on getting the virtual private servers done today, so your vps if you have one here, will be rebooted some time today.

     Tonight 11pm I will upgrade the file servers.  I hope this will be done within an hour but may take longer than usual owing to special drivers involved in the iomemory flash used for our MariaDB.

     I’m not sure what broke manjaro but it will not boot into a mode where I can login, not even console, so am restoring from backup.

     Because of the severity of the Sequoia security bug, kernel upgrades for shell servers and the web and mail servers, basically anything exposed to the public, will happen tonight instead of Friday as originally planned, and non-active servers will be upgraded sooner than 11pm.  The downtime for these will only be about 1 minute each.  The file servers that do not have public exposure will still be upgraded tomorrow.  VPS’s may or may not get  upgraded this evening.

     There are some worthwhile fixes to the kernel and as a result will be doing a kernel upgrade on July 20th, 11PM-Midnight.  Most services will experience an outage of about ten minutes during this interval.

     I am going to upgrade the router firmware shortly after 5pm tonight.  This will result in a brief interruption of Internet connectivity, usually lasting 1-2 minutes.  Usually this is brief enough that ssh sessions will not be knocked down but no guarantees.

     The reason that there have been no kernel upgrades for the last two weeks or this week is that 5.12.x has reached the point where it has been stable on our platform and I’ve been monitoring the progress and have not seen any changes that fix bugs that impact us, provide security improvements, or provide performance improvements.

     I do not anticipate going to 5.13 kernels because they don’t offer anything new that is of any benefit here.  5.14 when it reaches stable, we will try because they are replacing the E1000 Ethernet driver with a new copyless driver that will provide more efficient network I/O and hopefully also address the hardware offloading bug with this chip.  I do not expect this until sometime in August.