One of our customers account was compromised, in spite of all the facilities in place to prevent this, and used to send spam triggering the addition of our mail server IP to abuseix.  Yahoo and AOL have also blocked it for excessive spam.  I have requested delisting at abuseix and with AOL and Yahoo there isn’t much to do but wait.

     Kernel test is completed, 5.16 did not perform adequately.

     We are trialing a new kernel on our web server.  We were running 5.13.19 however it is past end-of-life upstream.  We attempted 5.15.x but the performance of this kernel was garbage.  However, I’ve tested 5.16.0 on my workstation and it performed much better than 5.15 and at least on par if not slightly better than 5.13, so I’m giving it a try on our web server (the most heavily loaded machine).  If it holds up we will be upgrading all the machines next Friday the 21st between 11PM and Midnight.

     If you notice any difference in the performance please let us know.

     Because 2.4.52 version of Apache seems to have a bug that causes it to occasionally stall, and because earlier versions have a root exploit that makes returning to them unfeasible, I wrote an automatic recovery script to recover the server in the event of a failure.

     What it does is automatically test the web server for a proper response every five minutes.  If it fails to respond, it attempts to restart apache, waits a minute to give it time to spool up, then retests.

     If the second test is good it logs the failure so I can keep track of how often it is failing.  If the second test fails, it turns my speaker volume up to 100%, and I have 100-watts per channel so that’s 200 watts worth of noise and then sends the Star Trek Next Generation Red Alert sound effect to the speakers.  I should hear that pretty much no matter what if I’m here, and also sends an e-mail so if I’m out I will be notified via my tablet or laptop.

     Hopefully this will prevent long outages like that which occurred last night when Apache decided to silently hang.

    I am going to kill our web server for about five minutes a couple of time to test some automatic failure recovery scripts I’ve written to try to prevent a repeat of last night’s outage.

     Our fax machine has bitten the dust.

     I’ve got a new one on order but it won’t be here until around the 28th.

     In the meantime, to get payment info, or other material you don’t feel comfortable sending plain-text, to me securely please either send e-mail from your eskimo.com e-mail address or use the ticket system to send the info.

The mail server took longer than I had expected.  And then after the backup was done it would not start properly.  So took some mucking around to figure out why and fix.  There were some things that used files on /misc which is an NFS mounted partition that were not set to wait for the mount.  So mail wasn’t back up to about 12:30AM.

     I am going to be taking the web server, mail server, and some of the shell servers down tonight at 11pm for between 20-60 minutes (mail server closer to 20, web server closer to 60) to image them (a form of backup) as I’ve made substantial changes to fix various issues to all of them and want to capture those changes so I don’t have to re-apply them in the event restoration from a backup is required.

     This will affect all customer websites for about 20 minutes, it will also affect all eskimo.com websites including friendica, hubzilla, and nextcloud, and mail will be unavailable for about an hour, but incoming mail will not be lost, it will be queued and delivered after the server comes back up.

This affected Eskimo North and customer sites as well as friendica.eskimo.com, hubzilla.eskimo.com, and nextcloud.eskimo.com.

We had a web server issue causing intermittent failure of the web server.  I thought this was load related as the server is quite heavily loaded, but it did it today during a time of very light load.  I was able to chase the problem down to a cronjob being run on a wordpress site automatically configured to redirect to secure mode when the certificate had expired.  The cronjobs were never exiting and ran once a minute and simply piled up until memory was exhausted causing failure of other services.  Normally I get automatic notifications when certificates are about to expire but did not in this instance. I’ve ordered a new certificate and commented out the cron job until it arrives and can be installed.

     Mail server changes are completed.  Mail that fails SPF, DKIM, or DMARC checks will no longer be rejected outright, instead they will be placed in your spam folder by default.  In the process of implementing this, I discovered many perl modules needed by spamassassin where not installed and so portions of spamassassin were not functioning.  Spam filtering should therefore be more effective now.

     You can now whitelist these as you could other e-mails scored as spam using the spam control facilities described here:

     https://www.eskimo.com/support/mail/spam-control-facilities/

     Similarly you can write procmail rules that handle such failures as you desire.  E-mails will now contain a header line like this:

     Authentication-Results: mx2.eskimo.com; dmarc=pass (p=none dis=none)
header.from=gmail.com

     That dmarc= may be “pass”, “fail”, or “none” and you can write rules to key off of these if you so desire.

     By default, any e-mail that arrives in your “spam” box should not be trusted, forged mails will go here.  So if you get e-mail from your bank saying you need to update your authentication or some such, do NOT click on the link if said mail is in your spam box.

     As a general safety note, I recommend NEVER clicking on these types of links, instead go directly to the site in question and make any necessary changes there.