One disk has failed in the /dev/md0 on Iglulik.  The RAID is currently running in degraded mode, one disk has no mirror at present.  As long as that drive holds up this is not service affecting.  A new drive has been ordered and should be here Wednesday.

     If the mirrored drive should fail before a replacement arrives, unlikely but not impossible, it contains three partitions, /owncloud which has been decomissioned, /opt/iso which simply holds operating system install images and is not needed unless a new virtual machine is created, and /var/lib/libvirt/images which holds important virtual domain images, these are all backed up elsewhere, so if we lost the entire partition they can be readily restored.

     The drive in question has not completely failed but did develop two bad sectors that for some reason did not automatically map out.  I am attempting to run badblocks now to identify any bad blocks and then will attempt to manually lock them out.  If that succeeds I will restore the drive to the RAID until the new drive arrives.  If not we will operate in degraded mode until that time.  Degraded mode is somewhat slower because in RAID10 arrays Linux reads different strips from both drives along with their stripped drives to effectively get 4x disk speed, in degraded mode it is only 2x.

     I got part of but not all of what I had hoped to accomplish done tonight.

     Our client mail server is now chroot’d as are the incoming servers.

     Our client mail server now signs mail with a DKIM signature and checks mail that comes to it with a signature though from a client that would be a rarity.

     I also installed clam-AV on our client mail server so there is no longer any e-mail ingress routes for viruses that are known.  Always there are some out there in the wild for which signatures haven’t yet been developed, but at least this will stop propagation of those that have.

     I installed spamassassin on the incoming server but it will only affect local delivery and not relayed e-mail at present.  I hope to fix that in the future.

     I did not get DMARC installed yet.  That is still a work in progress.

     Mail server maintenance is completed.

     October 8th, midnight – 12:30AM, I will be taking this machine down to image it.  This should take approximately 1/2 hour.  During this time you will be unable to send mail or view mail via IMAP/POP/Webmail, you can still view mail via shell mailers.

     Lists are operational with the new mail server now.  I was somehow missing procmail_wrapper from the smartlist.bin binaries.

     Our client mail server, “mail.eskimo.com”, has been replaced with a newer server based upon Ubuntu 19.10.

     I am having difficulty getting smartlist to work right but not only with this server but the two existing incoming servers.  I believe it is a permissions problem, smartlist on CentOS is owned by “smartlst” but under Ubuntu it is owned by “list”, and not only are the names different but so are the user ID’s and group ID’s.

     I’ve attempted to change everything to the new permission scheme but I believe I’ve missed some piece somewhere.  I’m still trying to chase it down.

     We have a new shell server available for your use.  Centos8.yellow-snow.net.  This is not available via the web yet but is available via “ssh centos8.yellow-snow.net” or X2go using “centos8.yellow-snow.net” as the host name.

     Please also note the installed software is extremely limited right now, the only desktops available at this time are KDE and Mate, gnome-flashback has not yet been ported, nor has xfce, lxde, or lxqt.

     I have not installed and configured the mail environment yet so please do not use it for e-mail yet, I will announce further when that is ready.  Unfortunately much software is not ported so makes this a difficult task at best.

     The new mail server is up.  I am not sure if list mail is working.  I will do some further testing after I have a few hours of sleep.

     I am going to replace the existing Centos6 based system powering mail.eskimo.com with a brand new Ubuntu 19.10 based system.

     This brings with it immediately only some minor benefits, new stronger key exchange methods for key exchange for example.

     In the longer term it will also bring better spam and virus filtering.

     Because I’ve had this new server on a different hostname and IP address getting it ready, I will need to take the old machine down and then change the hostname and IP address of the temporary name to mail.eskimo.com.

     In addition because of changes in the ownership convention between CentOS6 and Ubuntu, I will need to change the owner on all the list files.  This may take some time so mail lists may be out of service for an hour or more.

     During this interval you will not be able to send mail and you will only be able to read mail via non IMAP based shell mail programs.

     All affected systems have been rebooted.