Fedora is currently down for upgrade from Fedora 20 to Fedora 21. Since the machine has been getting very little use and nobody was on it, I didn’t think it would hurt to do it during normal hours.
I’m attempting a network update, but if that fails I will just re-install from scratch. This won’t affect users home directories or mail, those are on a separate file server.
Shellx is under a botnet based brute force password guessing attack. The attackers seem to be under the mistaken belief that it is a D-link router as they are primarily trying passwords for a non-existent user D-link.
To limit the resources consumed, I’ve set the fail2ban time to an hour instead of the usual ten minutes so if you get your password wrong three times you will either need to wait an hour or use a different shell server.
Need to reboot the main file server, and physical machines that host various virtual machines. This will be done just after midnight and will take approximately 25 minutes each, starting with the main file server.
Then I have to take various machines down for imaging, this will take approximately 25 minutes per machine as well, although that varies somewhat with each machine, shellx is the longest, some of the others are quicker.
This work is necessary to install the most recent software and make sure we have current backups in the event anything breaks.
I hope everyone had a pleasant Thanksgivings. I apologize for not getting something out prior. I got to spend the afternoon / evening with my oldest son Carl, second oldest son Raymond, and my wife Tina over at Carl’s house where we cooked up a turkey.
Tonight I will need to be taking servers down for maintenance, reboots to load new kernels, some down time to image machines with significant updates. Anything involving machines with customers on them such as occupied shell servers or servers which aren’t duplicated, will be done after midnight early Saturday morning, other machines which don’t have customers currently logged in or which are duplicated and thus can be taken down without impacting service, will be done earlier in the evening.
As you may or may not have noticed, I’ve been working on the website. If you have any input with respect to issues or things you’d like to see on it, please let me know.
I will be a bit late into the office today as I have to make a trip to the co-location facility to resurrect a broken Radius server. Radius infrastructure is duplicated so this should not affect authentication except to delay it slightly if it is the first server tried.
Starting around 9pm, I will be taking various servers down for reboots (kernel updates) and imaging. The main file server won’t be done until after midnight, and same for mail and web servers, but other shell servers will be done earlier starting with the least busy.
This work will take until about 2AM and might possibly be continued on Saturday evening if everything isn’t finished Friday evening / Saturday morning. All the Redhat machines have kernel updates and a huge number of updates (over 200 each) to install, the Debian based systems mostly just need to be imaged.
Debian is broken. I installed X2Go and FreeNX and then I needed to reboot but couldn’t because the shutdown program was not installed. I installed that, rebooted, but somehow the virtual machine was corrupted beyond restoration. I’m recovering from a backup but it will take some more work to get everything back to current so this machine may be down overnight. Please use Ubuntu or Mint if you need a Debian-based machine in the meantime.
There is a new virus going around that clamav does not have a signature for yet and thus is not catching. Our spam filtering software does recognize it as spam and is putting it in your spam folder. If you find e-mail in your spam folder with an attachment, I strongly recommend against opening it as it most likely contains a virus.
The volume of this virus is so large that it has overloaded our incoming mail servers and caused mail to be backed up in queue. I was able to make some adjustments that improved the mail servers ability to process mail and it is catching up, but as of 2AM it has worked through about half the queue, so at this rate it will probably be caught up by around 5AM.
Hopefully the clamav folks will kick out a database that includes a signature for this thing and put an end to it, but to prevent your machine from being infected, do not open any attachments you are not expecting and keep your anti-virus database up to date.
Debian is back up and functioning. I’m still applying updates that were rolled back but the corruption is resolved and it is operational.