I was not where I thought I was and did an rm -rf removing the friendica directory in the process.  So it’s being restored from compressed backups.  Because the partition that is backed up is 13TB partition, this is going to take a little while.

     In an attempt to further the freedom of speech, I have added a new Misskey social media site to Eskimo North.  It can be reached through our Web-Apps menu on the main website https://www.eskimo.com/ or directly at https://misskey.eskimo.com/.

     It is a minimal setup at present so not super pretty but I’ll fill it out as time permits.

     This joins our other social media sites, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://mastodon.eskimo.com/, https://nextcloud.eskimo.com/ and our federated search engine https://yacy.eskimo.com/.

     At the request of one of our customers, “anki” has been installed on all of our shell servers.  The version installed is the qt6 version from the ankiweb website with the exception of Mint.

     On the Mint server, the distribution version is installed because system security would not permit some of the actions required by the install script.

     For servers available see: https://www.eskimo.com/services/shells/servers/

     Presently quacamole and vnc access is broken on manjaro and rocky8.

     SSH, rdp, and x2go works on all of the servers.

     For those without an account, free trials: https://www.eskimo.com/services/free-trial/

    A new shell server based upon the Alma Linux distribution is now available. The server name is “alma.eskimo.com”.  It is available via ssh, vnc, rdp, and guacamole.

     To access via guacamole (web) go to https://www.eskimo.com/

     Then hover over web apps and select Terminal or Desktop.  At the login prompt type “public” in both the login and password fields.  Then select Alma Terminal or Alma Desktop, as you prefer, from the menu.  Then type your login and password when prompted.

     I believe it is reasonably complete, but if there is software not present that you would like, please send email to support@eskimo.com.

If you do not have an account with us, sign up for a free trial: https://www.eskimo.com/services/free-trial/

If you are having issues connecting to a shell server using port 22, we have found at least one cable company is blocking connections to port 22. On most of our shell servers, we also listen to port 443. If ssh times out, try “ssh -p 443 server.eskimo.com”.  Substitute “server” with the desired host, i.e., ubuntu, debian, rocky8, etc.

     Our Inuvik server which hosts manjaro, friendica, hubzilla, mastodon, and yacy, proved to be unstable under 6.11.4 as it was on 6.11.3, so I am headed over to the co-lo facility to reboot back onto 6.11.2 which is stable.  Estimated return to service time 01:15 Pacific Daylight Time.

     Inuvik now has a brand new Seasonic 1200 watt supply, it also has the good thermal paste now which lowered CPU max temps by about 10C, and I wire tied the fans to the heat sink because the fan clips kept slipping off.

     It took me longer than expected because I forgot two of the drives required a power adapter because they had a feature for remote power control that utilized one of the power pins and without said adapter they would not power up.  In addition, I accidentally hit the bios erase button instead of the power button and had to reset all the settings, but it is done and operational.

     All of the services it is supporting, roundcube, yacy, friendica, hubzilla, and mastodon, are now all operational.

     This outage, I learned three resources were particularly important to have available on more than one machine:

• DNS – Without this mail will returned no such address.
• SSL Certificates – Without this no encrypted services, mail, web, databases, can be started.
• Mail spool – Without no mail services.
• Home directories – Without no mail folders other than INBOX, no customer websites, and no shell services.

     All of these are single points of failure that bring important services down entirely.  Here are some general priorities I try to achieve.

• No loss of customer data, e-mail or home directory contents.  I have been very successful at this objective.
• No loss of incoming mail.  Not 100% on this mostly because of network outages making DNS unavailable.
• The uninterrupted ability to send and receive mail, more severe issues in this area.
• Uninterrupted availability of your websites.
• The availability of some shell servers.  I do not strive to have all of them up all of the time because there is enough overlap that services available on any one server are also available on others. Because of the direct access to the OS on these machines, security is a more difficult challenge than on other services so I do require more time to address issues on these.
• Ancillary services such as Nextcloud, Friendica, Hubzilla, Mastodon, and Yacy.

     Here are plans I have to address these issues at this point.  They are not fully developed which is part of the reason I am sharing this now, to get your input during the process.

My workstation at home is an 8-core i7-9700k with 128GB of RAM and about 20TB of disk.  It is on 24×7 but occasionally I boot windows to play games, so not available 100% of the time but obviously I am not going to be playing games during an outage.  Nothing I do really requires 128GB on this machine, I just happen to have an opportunity to acquire 128GB of RAM for about $36 so did.  But now since I’m also running Linux on it, and I have five static IPs with Comcast, my idea to address not ever disappearing from the net or loosing incoming mail is to setup a virtual machine on my box here and on it install bind, postfix, and set it up as a name server that is outside of the co-lo facility so if our network connection goes down we will still have a working name server and then postfix will be setup as a store and forward server, that is it will be a lower priority MX server that if the first two are unreachable mail will come to it, store, and then when the primary servers come back online it will forward to them.  This would address the second issue, not ever losing incoming mail.

     The third issue is more difficult to address because the mail spool is a single point of failure.  I could use rsync to maintain a near time duplicate, but the issue is if we switch to that during an outage of the primary server and then rsync then stop the primary incomings and let mail go to the store and forward server while we rsync any changed mail spools back to the original spool directory, any mail that came in between the last rsync from the spool to the secondary spool would be lost.  I have to do some experimentation to determine how often rsync can reasonably be run and how minimal that time span can be made.

     I can do something similar with home directories, this is less problematic than mail spool because the mail spool contains all INBOX mails for a given user in one file, but most home directory files are not subject to as rapid change and only those people who use procmail to sort into folders will risk any loss in this case, and we can rsync any files with a more current update when primary storage goes back online.  If we can duplicate home directories then duplicating the web server is pretty trivial, in fact when we get the big machine stable we will have two web servers operational under normal circumstances.

So while not totally thought out I’m letting you know how I plan to address these issues but open to input.  Particularly if there is some risk of losing mail between the time of last rsync and the primary system going down, is that risk worth having the ability to have access to mail during an outage of the primary server?

     Now in the more immediate future, the motherboard arrived for the ice, I don’t know for 100% sure if it is the motherboard or power supply, I replaced the supply with one I had on hand but still had the same problem but I’m not 100% sure that supply isn’t also dodgy as it is from the same vendor and I do not remember it’s history.  At any rate, I’m going to try to replace the motherboard tonight and if the machine works, I will return it to the co-lo facility Friday evening and take down Inuvik which has friendica, hubzilla, mastodon, yacy, and roundcube on it and take it back home to replace the power supply, and probably return it Saturday depending upon time frame.  Power on that machine is kind of a nightmare but I should be able to replace it in one night.

     Lastly, I am preparing kernel 6.11.1 for installation, 6.11 fixes a couple of issues.  6.10.x had an issue with some of our CPUs when it came to changing clock speeds in response to loads.  It detects an error when writing the MSR register, this is a register in the CPU that controls, among other things, the clock multiplier.  It actually succeeds and so it does change clock speeds appropriately but it doesn’t know it succeeds and so generates kernel splats.  This is fixed in 6.11.x.  I will apply this when I am at the co-lo so there will be a brief (around 2-3 minute) interruption in every service.