Nextcloud Is Borked

     Nextcloud is down.  Something went horribly wrong during an update last night.  I tried to roll back to the previous version and it’s not working either.  It appears to be an issue with apps but I haven’t been able to isolate which yet and this is a painful and slow process.

Kernel Upgrades Friday August 19th 11PM Pacific Daylight Time

     I am planning a kernel upgrade this Friday, August 19th, at 11pm Pacific Daylight Time (GMT-0700).

     This will affect all Eskimo North services, shell servers, e-mail, web hosting, other hosting, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com, https://yacy.eskimo.com/, and https://www.eskimo.com/.

     I do not expect the downtime for any one service to exceed ten minutes and the whole process should be completed by approximately 11:30PM.  I am expecting this to go reasonably smooth as 5.19.0 was very smooth and 5.19.1 only contained three very small fixes, one where a function was missing a return so if it did not return on some conditionals it could return with random results, and two that are bounds checks in the QEMU-KVM system that would not come into play unless something else went wrong.

Kernel Upgrades Completed

     The kernel upgrade to 5.19.0 went amazingly smooth this evening with only one service (dovecot) not starting on one machine (mail) and that because a dovecot upgrade had overwritten my systemd start-up file so that it tried to start before all of the necessary file systems were mounted.

     There was also one virtual private server that failed to start two services but that was because the customer has an SSL certificate configured in but the certificate file was not present.

Comcrap Router Borked

     My Comcrap Cable router (with Cisco firmware) has had a melt down and is NAT’ing me to an address of 71.197.179.184.  Since many of the servers I need to access to do ordinary maintenance are hard-wired to only allow connections from my IP address, I can not access many of them.  Comcrap is supposedly coming out at 8-9am, when I would normally be asleep, to replace this piece of crapola with a new piece of crapola.  Don’t know if it was hacked or a firmware update went bad, or something upstream on their network that pushes the static IP’s to the unit when south but for now I’m very restricted in what I can do to pretty much the same things that you can.  I have an even more colorful term for Comcrap but I am afraid delicate ears will be offended.

Fedora – No Outside Access

     I am taking outside access away from fedora until fail2ban can be made operational again.

     Fedora upgraded the python instance to 3.11 but fail2ban will not work with python greater than 3.10, so until this is fixed I am waking away access from the outside to prevent brute force password attacks.

     You can only access by logging in to one of the other shell servers and then from there ssh to fedora or from the web client.

Kernel Upgrade Friday August 5th 11pm Pacific Daylight Time (GMT-0700)

     I am tentatively scheduling a kernel upgrade for all eskimo.com services this Friday pending successful testing of 5.19.0.  I’ve only installed it on two machines at present, the web server because it is really heavy on context switches and uses many computer language used in web programming, including C, C++, java, python, php, ruby, and context switching is often where Linux either barfs or has efficiency issues, and on my workstation where I also have several virtual machines and thus can test kvm-qemu which is heavily used to provide our services.

     This will affect all Eskimo North services including private virtual servers, web hosting, shell servers, and mail.  It will affect our public services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/, and our website https://www.eskimo.com/.

     So far I am highly optimistic.  It’s cut the load time for our PHP page down to 76ms cached, 82ms uncached.  Pingdom Tools won’t give you these numbers because there is too much latency in their tools, but Firefox web development tools will, and this is coming from a different network (Comcast).  This is at least an order of magnitude better than our fastest competitor and as much as two orders of magnitude better than our slower competitors so I am very happy with these results.

     We skipped 5.18 because it had stability issues with kvm-qemu but so far these have not manifested in 5.19.  I am particularly impressed with the results because 5.19 added many new device drivers and debug tools as well as other tools.  Usually when you increase the complexity of a kernel to this degree it doesn’t result in good things for performance, but not the case with 5.19.

     These are tickless kernels, I always build them this way because why waste CPU cycles rescheduling tasks when there is nothing in the run queue?  If you care to try them and perhaps assist with testing or just want to maximize the performance of your hardware, I make them available at https://www.eskimo.com/kernel/.  Please take the time to read the README file as it describes the differences in the various kernels available.  The .config’s are also present should you care to build yourself.  These are all compiled with the most current release of gcc, gcc 12.1.