Tonight’s kernel upgrade is aborted because it will not build with the debugging options the developers wanted me to include so I’ve sent the compiler errors back to them and will resume when I have a fix.

     I am going to be upgrading the kernels on only the physical servers tonight in order to turn on some additional debugging options to help the developers chase down an error in the NFS code that is causing issues for us.  Apparently this bug only occurs when you have a mix of NFSv3 and NFSv4 clients as we do, (also an NFSv2 client).  So it’s an issue that is rarely triggered but our environment triggers it.  It is a use after freed error that for some reason KFENCE is not finding, they have asked me to turn on KASAN, a different somewhat higher overhead memory allocation troubleshooter, and this requires a rebuild of the kernel and rebooting of the physical servers.  Because this only affects the NFS servers, I will be installing this on Iglulik, Igloo, and Mail, but not the other servers at this time.  This will affect vps6, vps9, and all the shell servers and mail.  The interval will be between 11pm-11:30 with individual outages not lasting more than about 10 minutes with the exception of yacy.eskimo.com which takes about half an hour to 45 minutes to rebuild it’s database after a reboot.

     This will affect all Eskimo North services EXCEPT for vps1-vps6, vps7 and vps8.

     It will impact our Fediverse instances including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

     The kernel upgrades were successful and relatively trouble free.  We are now operating on Linux 6.1.3, which was released yesterday.

     Planning to upgrade to a 6.1.2 kernel Friday 1/6 at 11pm Pacific Time.  The present kernel, 6.0.15 has a nasty bug where it locks hard, no kernel dump, no auto reboot, no magic sys request key, only power cycling the affected machine restores service.  The inability to get a kernel dump makes this bug particularly difficult to troubleshoot.  Since this bug has persisted from 6.0.12, I’m going to try a 6.1 kernel and hope for better.

     This will result in outages between 11pm-11:30pm of all services lasting about 5-10 minutes each EXCEPT for yacy which takes close to 45 minutes to rebuild it’s database after every reboot.

     This will affect all of Eskimo North’s paid services such as mail, web hosting, virtual private servers, shell accounts, etc, as well as our free services including https://nextcloud.eskimo.com/, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and as I mentioned, https://yacy.eskimo.com/.

      The machine that hosts many of the virtual private servers, ice, is down.  I am headed down to the co-lo to find out why.

     My facsimile machine ran out of paper today and the spare paper I had all got wet from the tree limb through my roof, so it will be nonoperational until I can get more paper.  Please e-mail or call instead.

     We are going to attempt to change the default version of PHP to PHP 8.0.26 tonight.  The last time I attempted this there were too many applications that did not work but since that time several new releases have come out so we are going to try again.

     Just a reminder, if your own apps do not work well with PHP 8.0 you can override the PHP version: https://www.eskimo.com/support/override-php-version/

     Within half an hour of booting 6.0.15, the server which holds home directories and much shared disk space spontaneously booted.  While still not good, this is better than the behavior of 6.0.12 which locked up and did not reboot.

    Two customers reported mail failing to gmail, with the bounces indicating improper SPF records, however, I changed the SPF record along with the IP of the mail server at the same time AND pushed out the changes manually to all of our name servers.  Gmail was caching old data.  I have since tested and Google is again accepting our mail.

     I will be doing kernel upgrades tonight that were originally scheduled for last night.  I was able to recover my vehicle that my wife abandoned on the way home from work the night before last and most of the ice and snow has melted off the roads now.  I prefer doing them on Friday nights and especially don’t really want to be doing them Christmas evening but the existing kernel has a flaw that has so far resulted in the lock-up of four machines so really needs to be replaced as soon as possible.

     The photo gallery function of Friendica has been fixed.

     Talking to the provider of the infrastructure we use for dial-up, it is not yet set in stone, but we may be able to continue to offer this service into 2024, one of their larger customers is considering renewing their contract and if this happens they will continue into 2024, I will let you know as soon as I know.