I am taking outside access away from fedora until fail2ban can be made operational again.

     Fedora upgraded the python instance to 3.11 but fail2ban will not work with python greater than 3.10, so until this is fixed I am waking away access from the outside to prevent brute force password attacks.

     You can only access by logging in to one of the other shell servers and then from there ssh to fedora or from the web client.

     I am tentatively scheduling a kernel upgrade for all eskimo.com services this Friday pending successful testing of 5.19.0.  I’ve only installed it on two machines at present, the web server because it is really heavy on context switches and uses many computer language used in web programming, including C, C++, java, python, php, ruby, and context switching is often where Linux either barfs or has efficiency issues, and on my workstation where I also have several virtual machines and thus can test kvm-qemu which is heavily used to provide our services.

     This will affect all Eskimo North services including private virtual servers, web hosting, shell servers, and mail.  It will affect our public services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/, and our website https://www.eskimo.com/.

     So far I am highly optimistic.  It’s cut the load time for our PHP page down to 76ms cached, 82ms uncached.  Pingdom Tools won’t give you these numbers because there is too much latency in their tools, but Firefox web development tools will, and this is coming from a different network (Comcast).  This is at least an order of magnitude better than our fastest competitor and as much as two orders of magnitude better than our slower competitors so I am very happy with these results.

     We skipped 5.18 because it had stability issues with kvm-qemu but so far these have not manifested in 5.19.  I am particularly impressed with the results because 5.19 added many new device drivers and debug tools as well as other tools.  Usually when you increase the complexity of a kernel to this degree it doesn’t result in good things for performance, but not the case with 5.19.

     These are tickless kernels, I always build them this way because why waste CPU cycles rescheduling tasks when there is nothing in the run queue?  If you care to try them and perhaps assist with testing or just want to maximize the performance of your hardware, I make them available at https://www.eskimo.com/kernel/.  Please take the time to read the README file as it describes the differences in the various kernels available.  The .config’s are also present should you care to build yourself.  These are all compiled with the most current release of gcc, gcc 12.1.

    Router maintenance at eskimo.com has been completed.  The automatic upgrade failed but manually uploading the new firmware succeeded.

     This affects all eskimo.com services including our shell servers, virtual private servers, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://www.eskimo.com.

     This evening there will be several times when I/O will freeze for router reboots.  I am trying to upgrade software but having some difficulty so need to follow a procedure given to me by Ubiquiti that will require at least two reboots.  This machine isn’t the fastest when it comes to reboots so outages may be as long as five minutes in duration.

     I am looking for some suggestions for open source mail list software capable of running under Linux to replace the SmartList mail lists we are currently hosting.

     SmartList has several fundamental problems.  First, neither it nor procmail which it relies on has not been under active development for at least a decade.  This means it may be deprecated at any time which would be bad.

     Second, SmartList does not re-write the sender address and in situations where the sender’s domain signs their e-mail with SPF, DKIM, or DMARC records, and mail list recipients domains check these things, the e-mail is rejected, dropped, or marked as spam as a result of the server the mail is coming from (ours) not matching the domain of the sender which is still in the header.

     Third, many people prefer to administer their lists from a web interface, SmartList provides no method for Web Admin.

     Fourth, I’ve had a number of requests for mail list software that has a web archive of lists.  SmartList lacks this capability.

     So with these things in mind, I attempted to implement Mailman, and I was able to install it but Mailman relies on Python 2.7 which is deprecated and will go away soon, and I was unable to get it to properly recognize the fully qualified domain name and it kept trying to create lists with just the hostname.  I was unable to figure out why this is since I do not have just the hostname alone configured anywhere in the software.

     So then the next thing I looked at was Mailman3, mailman3 is based upon Python 3.x so current software is supported, BUT it’s split into pieces and does not have a native web interface, this is provided by some separate third party software such as Kitty, Kitty is hardwired to NgNix, and I have a very highly customized and optimized Apache2 that beats NgNix for security, speed, and capability that I prefer to stick with.  Second problem with Mailman3 is the documentation is sparse to non-existent and has huge gaps in it’s installation instructions.

     So I’m looking for something else, recommendations would be much appreciated.

     We are going to do a software upgrade tonight.  This requires a router reboot which takes about five minutes during which we will temporarily disconnect from the Internet.  This affects ALL eskimo.com services.

     Kernel upgrades were completed with only minor systemd issues like not starting timedate service on some machines.  Nothing severe or even consequential beyond requiring restart of those services.

     This will affect ALL of Eskimo North’s services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://www.eskimo.com/.

     We will be upgrading to 5.17.15 which is the last iteration of the 5.17 kernel, we will not be upgrading to 5.18 series yet as it has not been entirely stable on my workstation yet.  Keep getting stuck processes with 5.18 and that is no good.

     The outages should mainly be between 11pm-11:30pm though some servers may be slightly longer if systemd gives me a hard time.

     If you get an e-mail bounced back with a message something to the effect of no inverse DNS for 204.122.16.222, this was not our doing but something that went wrong at ARIN, the entity responsible for assignment of IP space and associated records.  Re-send your e-mail and it should go through successfully now unless the receiving site is caching bad data.

     I had a customer query me as to why the mail bounced, got this message, did an nslookup and indeed there was no inverse DNS for that, or any of our other servers.

     I checked each of our name servers and they were all responding correctly.  For some reason ARIN was failing to delegate inverse DNS requests for our IP space to our DNS servers.  I went in to the section to update the records, made no changes what so ever, but after saving the non-changes it worked again.

     I am going to reboot a bunch of machines tonight sometime midnight or after to change some configuration options.  It turns out that there is a bug in virt-manager where if you tell it to copy the host CPU configuration, it doesn’t, instead it takes an four core, eight thread, CPU, and turns it into eight 1 thread CPU’s in the guest OS.  This has less than optimal scheduling results in the guest kernel.

     I can, however, manually set the architecture correctly, and this is what I’ll be doing, but it does require a guest shut-down and reboot to be effective.