Kernel Upgrades Completed

     Kernel upgrades are finally completed.  It took abnormally wrong because something went wrong with the kernel install on mail and it ended up with a grub configuration pointing to two ancient kernels that no longer exist and thus it was unable to boot.

     I was able to fix the grub configuration with boot repair but then had to manually mount all the shell servers NFS partitions from it as it was dead when they booted.

Kernel and Hardware Upgrades

     I am planning yet another kernel upgrade tonight starting at 11pm as there is an issue with the kernel taking memory from the wrong end of a linked list resulting in poor guest performance.  Since we use VM’s to implement most of our services, guest performance is important.  If this goes as smoothly as the last upgrade it should be completed well before midnight.

     With respect to MariaDB database hardware, I’ve been notified by Amazon that it will be as much as a week later than originally promised, so the upgrade may not happen until the weekend of March 12th.

PCIe SSD with Atomic Writes Located

     I found one of MariaDB’s recommended drive model on Amazon for a very reasonable price and ordered.  I don’t know if my motherboard will be able to boot from it because I don’t know if the BIOS knows about it, but it should be compatible from a hardware standpoint and Linux talks directly bypassing the BIOS once booted and all this drive is going to be used for is MariaDB storage so it should be fine.

     So thank you to all of you who sent me suggestions and web references.  I expect to have it in hand next Friday and baring no snow will install it that weekend, else will wait for a weekend without snow.

Need PCIe SSD with Atomic Writes for MariaDB

     I recently turned up a Hubzilla in addition to the Friendica Fediverse social media site here.  Sunday it seemed to be okay but come Monday the server was seeing loads of around 150 or so at which point a number of web apps turned themselves off.  Also it was exhausted on connections which made many peoples WordPress sites break.

     After much fine tuning I’ve gotten MySQL to handle a touch over 400 transactions per second, but occasionally hubzilla spikes higher than that and the load will temporarily rise.

     After doing some research on MariaDB and tuning it as best I could, now load only goes up to about 8 as opposed to 150, but virtually all waiting processes are waiting on DIsk I/O at that point so it appears that peaks are saturating the drives I/O capability.  Already this is on a 4-disk RAID 10 array with everything pretty finely tuned.

     Researching I am finding most people are not getting much better than 400 TPS on rotary drives and perhaps 500 TPS with M.2 SSD drives, but with a PCIE SSD drive that supports atomic writes, up to 1500 TPS has been obtained.

     So I am searching for same but not having much luck.  I can find lots of M.2s but can not find PCIE drives.  When I search I only find PCIE SSD adapters, not what I need.

     Does anybody know where I can source these and who makes them, perhaps some make and model numbers?

Maintenance

I am going to be taking our webserver down at midnight to backup and then to re-partition the disk so that I can create a separate partition for mariadb tables so I can better optimize it for performance.  This is pacific standard time.  It may be down for several hours.

Wave Cable E-mail

     Wave cable uses a company called Synacor to provide their e-mail.  Synacor is allowing spammers to use their servers.

     We have had this problem ourselves in the past with hacked accounts being used to spam.  We have solved it by installing fail2ban to limit brute force password attacks and postfwd to force a password change if/when a clients account is abused.  It detects abuse by looking for a customer coming from IP’s in three or more countries in 24 hours.  This pattern only happens with botnets which spammers are using to send their spam.

     If anyone knows how to contact their network or mail administrator I’d be glad to point them at these resources.

     When spammers use unclean e-mail lists that contain addresses that don’t exist on our server, it triggers a program called fail2ban to ban them after several unsuccessful attempts because it thinks someone is probing for user addresses to spam.

     I have unbanned their servers twice but as long as they continue to allow their resources to be used for spamming, mail delivery from wavecable.com is going to be problematic.

     If you are a customer of wavecable, it may help to complain to them that your e-mail is not being delivered because of other customers on synacor.com’s servers abuse them.

Hubzilla

New Federated Social Media Hubzilla

     We’ve added a new federated social media platform called “Hubzilla“, https://hubzilla.eskimo.com/.  It is similar to Friendica https://friendica.eskimo.com/ in that it is a federated social media platform but it has many more capabilities including channels that are similar to groups or blogs or user pages in Facebook but federated across the entire fediverse grid, although not all fediverse servers like friendica support all the features of hubzilla.  It is considerably more complex however it is well documented online.  Discovery of both users and resources such as channels is much easier and faster than it is with friendica and in my view the layout is somewhat nicer.

Kernel Upgrades Completed

     Kernel upgrades are completed.  All NFS and NIS relationships verified operational.  This was the fastest (15 minutes to boot, 9 minutes to check) kernel upgrade yet, all NFS and NIS relationships established themselves successfully first time around.  Only problem I ran into is postfix failed to start on mx2 but started fine manually.

Phishing Scams

Customer Asks:

— Robert

is this from you?  style seems wrong…  and there was no heads up (unless I
missed it)

——– Forwarded Message ——–
Subject: New Message Received
Date: Fri, 12 Feb 2021 12:18:41 +0100
From: Eskimo.com Notification <margaretbaros@gvec.net>

——— End Relevant Part of Forwarded Message ———

I’m sending this to everyone because these phishing scams are a daily occurrence these days and I’m wasting WAY too much time responding to them.

Please read this and note, the portion inside the <>, the portion in RED is the REAL e-mail address.  This portion of the address is verified by SPF, DKIM, and DMARC records and so is difficult to forge (it is not impossible but I’ve seen it done successfully in like maybe two times in the last ten years).  The portion outside the <> in BLUE whatever the sending client puts in the name field of their mail client and is easily set to whatever they want it to be.

If you do not see <nanook@eskimo.com> or <support@eskimo.com> INSIDE the <> marks then the mail is NOT from us.  Please forward these to spamtrap@eskimo.com.

If when you attempt to forward them to spamtrap@eskimo.com, they are rejected, this means that someone who has previously forwarded them has already generated a virus signature so this particular instance is already addressed.