The failed drive has been replaced.  The RAID array is copying to the new drive, it will take about ten hours to complete.

     I will be taking most services down around midnight for approximately two hours to replace a failed hard drive on the server that provides NFS service for everyone’s home directories.

     Maintenance completed.  Mail is available.

     I am taking the client mail server down for about 1/2 hour to image it after getting smartlists working again.

     I will be taking various shell servers and the client mail server down for about 1/2 hour each tonight, either to move to a different host machine to do some load balancing, or for backups in the case of the mail server.

     When I installed the new mail server, I changed the ownership of smartlists from “smartlist” to “list” because the newer Linux distros use a different permissions scheme.

     However, some files, like the “dist” file were inadvertently changed to and this broke the ability for people to administer their lists.

     I am working on correcting this but it involves restoring from backups so your lists will be in a state they were in last Saturday or possibly the week before when all is said and done.  I apologize for this, I made the mistake of doing chown -R list.list /usr/smartlist when I should have done something like find /usr/smartlist -user smartlist -exec chown list.list {} \;

     One disk has failed in the /dev/md0 on Iglulik.  The RAID is currently running in degraded mode, one disk has no mirror at present.  As long as that drive holds up this is not service affecting.  A new drive has been ordered and should be here Wednesday.

     If the mirrored drive should fail before a replacement arrives, unlikely but not impossible, it contains three partitions, /owncloud which has been decomissioned, /opt/iso which simply holds operating system install images and is not needed unless a new virtual machine is created, and /var/lib/libvirt/images which holds important virtual domain images, these are all backed up elsewhere, so if we lost the entire partition they can be readily restored.

     The drive in question has not completely failed but did develop two bad sectors that for some reason did not automatically map out.  I am attempting to run badblocks now to identify any bad blocks and then will attempt to manually lock them out.  If that succeeds I will restore the drive to the RAID until the new drive arrives.  If not we will operate in degraded mode until that time.  Degraded mode is somewhat slower because in RAID10 arrays Linux reads different strips from both drives along with their stripped drives to effectively get 4x disk speed, in degraded mode it is only 2x.

     I got part of but not all of what I had hoped to accomplish done tonight.

     Our client mail server is now chroot’d as are the incoming servers.

     Our client mail server now signs mail with a DKIM signature and checks mail that comes to it with a signature though from a client that would be a rarity.

     I also installed clam-AV on our client mail server so there is no longer any e-mail ingress routes for viruses that are known.  Always there are some out there in the wild for which signatures haven’t yet been developed, but at least this will stop propagation of those that have.

     I installed spamassassin on the incoming server but it will only affect local delivery and not relayed e-mail at present.  I hope to fix that in the future.

     I did not get DMARC installed yet.  That is still a work in progress.

     Mail server maintenance is completed.

     October 8th, midnight – 12:30AM, I will be taking this machine down to image it.  This should take approximately 1/2 hour.  During this time you will be unable to send mail or view mail via IMAP/POP/Webmail, you can still view mail via shell mailers.