Kernel Upgrade Date Correction – August 26th 11PM Friday

     It was brought to my attention that I erred in my announcement of the changed date of the kernel upgrade.  I posted it would be Friday August 24th, but the correct date is Friday August 26th.  A customer felt this was ambiguous since the 24th was on a Wednesday not Friday and asked me to clarify.

     For future reference, kernel upgrades and most major service interrupting maintenance takes place Fridays usually starting at 11PM.  The reason for this is to give the maximum amount of time before the business week to recover in the event of severe problems with the procedure in question.

Kernel Upgrade Postponed

     A kernel upgrade for all of’s services scheduled for August 17th at 11pm PDT has been postponed until August 24th because kernel 5.19.2 came out this Wednesday after I had already built and partially deployed 5.19.1.  5.19.1 contained only three very small fixes so did not require extensive testing but 5.19.2 contains far more significant changes and many so there is a much greater potential for instability to be introduced.  And one day is really too tight to deploy to all the machines so going to shoot for next Friday instead in order to give it time to soak and to deploy to all the machines.  5.19.0 has both performed extremely well and been stable so there is no real incentive to get this upgrade pushed out quickly.

Nextcloud Is Borked

     Nextcloud is down.  Something went horribly wrong during an update last night.  I tried to roll back to the previous version and it’s not working either.  It appears to be an issue with apps but I haven’t been able to isolate which yet and this is a painful and slow process.

Kernel Upgrades Friday August 19th 11PM Pacific Daylight Time

     I am planning a kernel upgrade this Friday, August 19th, at 11pm Pacific Daylight Time (GMT-0700).

     This will affect all Eskimo North services, shell servers, e-mail, web hosting, other hosting,,,,, and

     I do not expect the downtime for any one service to exceed ten minutes and the whole process should be completed by approximately 11:30PM.  I am expecting this to go reasonably smooth as 5.19.0 was very smooth and 5.19.1 only contained three very small fixes, one where a function was missing a return so if it did not return on some conditionals it could return with random results, and two that are bounds checks in the QEMU-KVM system that would not come into play unless something else went wrong.


     Fedora is now again accessible from the outside world.  The fail2ban folks have fixed fail2ban so that it works properly (if not a bit slowly) with python 3.11 used by Fedora.

Kernel Upgrades Completed

     The kernel upgrade to 5.19.0 went amazingly smooth this evening with only one service (dovecot) not starting on one machine (mail) and that because a dovecot upgrade had overwritten my systemd start-up file so that it tried to start before all of the necessary file systems were mounted.

     There was also one virtual private server that failed to start two services but that was because the customer has an SSL certificate configured in but the certificate file was not present.

Comcast Router Replaced

     Well I’ll have to give Comcast credit where credit is due.  They were out here five hours after the router failed with a working replacement.  So all good again, for now.  I think this is my 15th Comcast modem / router.

Comcrap Router Borked

     My Comcrap Cable router (with Cisco firmware) has had a melt down and is NAT’ing me to an address of  Since many of the servers I need to access to do ordinary maintenance are hard-wired to only allow connections from my IP address, I can not access many of them.  Comcrap is supposedly coming out at 8-9am, when I would normally be asleep, to replace this piece of crapola with a new piece of crapola.  Don’t know if it was hacked or a firmware update went bad, or something upstream on their network that pushes the static IP’s to the unit when south but for now I’m very restricted in what I can do to pretty much the same things that you can.  I have an even more colorful term for Comcrap but I am afraid delicate ears will be offended.

Fedora – No Outside Access

     I am taking outside access away from fedora until fail2ban can be made operational again.

     Fedora upgraded the python instance to 3.11 but fail2ban will not work with python greater than 3.10, so until this is fixed I am waking away access from the outside to prevent brute force password attacks.

     You can only access by logging in to one of the other shell servers and then from there ssh to fedora or from the web client.