RDP / VNC / NX CentOS6

     In addition to x2go graphical connections, Centos6 now supports RDP (which will automatically size to your screen), NX (which also will automatically size to your screen if you tell it to use the full display), and VNC which starts at 1024×768 but if you go into System->Preferences->Display you can the set it to your native resolution and it will resize the screen accordingly.

Mail Stability

     Today I logged in to the physical machine that the mail virtual host runs on and it died.

     I was able to determine that the cause was an out of memory condition on the machine even though it normally has about 18gb free RAM available.

     I believe it died when I logged in because it was probably a buggy version of x2goserver that caused it.  The x2goserver had been acting weird on that machine, giving me a tiny display instead of a normal full screen display.

     I use a nightly builds version because the stable release is incompatible with the rest of the graphical environment on that machine.  That machine requires xrandr version 1.3 or greater, the stable x2goserver uses version 1.2, but the version in nightly builds uses 1.3.

     The nightly builds may have serious bugs since it is the development version and in this case it appeared it had a bug that ate up all memory.  I have since de-installed and re-installed the x2goserver and all associated libraries and applications.  Now it works properly and it doesn’t crash mail.

     A stable release with this version of xrandr is expected around the second week of February but in the meantime I will not re-install unless there is some other problem.

     I also wrote a small script that runs once a minute and checks to make sure mail is alive and if not boots it so that if this does occur again it will automatically come back up.  It logs it when this occurs with a time stamp so I will be able to examine logs on the machine to see what else was going on at the time it went down.

Mail Is Up ; Future Maintenance

     Mail is backup.  I have a bunch of updates to apply then will need to take down for about twenty minutes to image the machine.

      Also, SSL certificates are currently expired, will fix shortly.  In the meantime just tell your mail client to accept the security exception.

Mail

     The physical host is back up.  This means you can access and receive mail from shell servers.

     I am working on restoring the virtual machine mail from an earlier backup before instabilities crept in.

Mail Server

     The mail server crashed again this afternoon.  In an attempt to fix it rather than just reboot again I decided to restore from a backup made prior to the time it started acting up.

     During the restoration the physical host hung so I will need to drive down to the co-location facility to restore the physical server then finish bringing mail backup.

 

NX and Remmina Support

     Shellx had support for NX and Remmina.  This was lost when I replaced the server with CentOS 6 owing to incompatibilities with the newer LibNX libraries used by x2go.

     I had thought there was little hope of this being fixed as freenx is no longer being actively developed as x2go has replaced it’s functionality, is available on a broader range of platforms, and had much additional functionality such as sound.

     But… I ran across a version to point releases later of freenx on one of the third party repositories and I installed and tested it and it worked, so this functionality has returned to centos6 which replaces shellx.

Mail Client Server Stability

     We have been  plagued with a recent rash of hardware and software problems as of late.

     A current problem I am seeing is the older Redhat 6 servers are mysteriously powering themselves down.

     It may be related to kernel upgrades I attempted recently.  The newer kernels required a newer version of acpid.  The acpid daemon is a program that listens to things like power and reset buttons and then acts accordingly.

     I had to back out the kernels because the mandatory locks which mail really requires to operate properly were broken under 4.8 at least.  I neglected to back out the updates to acpid at the same time.

     In addition, on mail, sogo, a package we have in place to provide some interoperability with ms-exchange clients, overwrote some of the system python modules and broke some of the admin tools.

     I’ve fixed these things and I’ve turned on some additional logging so that if it happens again it will hopefully provide additional information for troubleshooting.

Web Server

     I deeply apologize for the lengthy downtime of the website and MySQL database tonight.

     Ubuntu kicked out an upgrade to mysql server that seriously broke it.  With the update installed, as soon as someone tried to connect from anywhere but localhost, the server would get stuck in an infinite loop complaining of bad file descriptors and not a socket.

     I tried to revert to the previous and only other version on the Ubuntu repository but it was also broke,

     In addition, they hard coded in a limit of 65536 file descriptors which is not adequate for our sites traffic.

     I tried to migrate to MariaDB but there was no clean path from MySQL 5.7 to MariaDB and nothing I tried would work.

     I then attempted to install Community MySQL directly from the site but owing to the fact that I had a mysql user in the NIS database, it broke the install.  However it did not log the cause or print an error that would give me a clue as to what the problem was so it took a lot of tracing and plodding around to find the cause and fix it.

     We are now up and running with Community MySQL Server version 5.7 and it is back to functioning properly.

MySQL Database

     MySQL continues to run for a while and then exhausts file descriptors even though it’s set to more than a million.  We are running a version that previously was stable.  I’ve fixed the stuff in the start up script that it removed.

     I am still trying to identify the cause and correct it.