After much experimentation, I was able to determine that the NFS version 4 behaviour changed between Linux kernel 3.3.8 and 3.4 so I’ve been able to get a pre-emptive 3.3.8 kernel working on the web server. It does cut latency somewhat.
Moreover, I’ve come to understand the compatibility issues. The problem is that newer NFSv4 went to using nfsidmap which uses upcalls within the kernel rather than an daemon to handle mapping.
What I haven’t determined yet, is exactly at what kernel version this change was made and if it is at all possible to build a kernel with kernel nfsd support that will work with both. That would be the easiest, else I’ll have to change all the machines over at once which would be challenging.
Further research suggests that this isn’t going to fix it. I’m going to update Dovecot anyway just to get it current but will probably do this later this evening instead of at 5pm.
It appears that this problem is because of the POODLE exploit that came out which RedHat “solved” by disabling SSLv3.
The only fix at this end would be compiling OpenSSL from source, and then recompiling a whole bunch of stuff not to use the system version because RedHat isn’t going to fix it properly, or build a new server based on a non-broken operating system, and that is problematic because Red Hat’s EL6, upon which CentOS 6 is based, has a broken implementation of NFS version 4, which is really needed for mail to work properly owing to the lack of mandatory locking on earlier versions of NFS.
In the long term I am going to work towards moving our infrastructure away from Red Hat and towards Ubuntu. Although the Ubuntu people occasionally screw things up, they almost always fix them quickly. Red Hat is becoming impossible to maintain and have properly interact with other operating systems, kind of like Microsoft twenty years ago.
Since there is no good short term fix on this end, those affected will either need to upgrade their software to something capable of TLS or use a mailer that doesn’t override their encryption selections, such as Thunderbird.
There are some versions of Outlook, and I don’t know which yet, that forces SSL encryption even if a customer selections non-encrypted mail and it tries to use a version not supported by our servers, causing mail retrieval to fail.
People who are affected have systems that do not support TLS (mostly very old) and only support a version of SSL that our server won’t accept. If you are retrieving mail okay at present then your machine isn’t affected.
In order to try to resolve this issue for others, I’m going to be upgrading Dovecot from version 2.0.8 to 2.2.16 around 5pm today. I’ve got the software built and ready to go but don’t want to install it in the middle of a business day in case something goes wrong and have to back it out. We’ve been using the version that was part of the CentOS distribution and it is out of date.
If there is an interruption, it should be brief, (I have saved the old configuration and can re-install the CentOS supplied version in seconds if need be) but I wanted to let you know in advance just in case.
Often what starts as a little problem has a way of growing. What I found out today, it’s not only newer kernels I build but also the kernels that exist in any non Redhat 6 based Linux, so idmapd is not working properly on Centos7, Scientif7, Fedora, Debian, Mint, or Ubuntu. Oddly, an actual operation on a file seems to work correctly, if I su to a user and touch to create a file, then go back and look at it on the server, it is created with the proper UID even though it isn’t displayed correctly with ls on the client. This is why the problem has essentially gone unnoticed for so long.
As near as I can tell there is some fundamentally different way that idmapd works under RedHat 6, then any other Linux distribution or even later versions of Redhat. I’ve found copious documentation of this bug but no fix that actually works except to revert to NFSv3. The problem with NFSv3, it does not have mandatory locking that works.
With a 3.5.9 kernel, idmap fails to resolve some UIDs but does not generate the error messages the 3.9.x kernel did. This may be a configuration issue however as I did see an option relating to idmap. It would be good to get it to work because it does significantly reduce the load time of web objects. Faster is good because Google gives a higher ranking to faster servers.
I am attempting a kernel build of a 3.5.9 kernel. I’ve read that it will work with CentOS 6.x and still provide some features that hopefully will provide better performance although initially it’s going to be built with pretty much the same options as the old kernel so that I can differentiate problems caused by the kernel version from problems caused by operator malfunction.
This may cause the web server to operate slightly slower during the build although I’ve checked with Firebug and it doesn’t seem to be affecting it greatly so far.
While KDE doesn’t work for me (and another customer) on Mint, it does work for a different customer using X2Go, which suggests it may be a settings issue. However, I deleted everything in .kde and it still doesn’t work correctly for me. I still think something is wrong with the install but haven’t been able to figure it out yet.
We have three Debian based shell servers:
- debian.eskimo.com – Debian Wheezy
- mint.eskimo.com – Mint 17 Qiana
- ubuntu.eskimo.com – Ubuntu 14.04 Trusty Tahr
On the Debian server, KDE works with both freenx and X2Go. On the Ubuntu server, KDE works with X2Go but not with freenx. With freenx, after login you’ll get an initial black screen then it exits. On the Mint server, KDE will not work with either freenx or X2Go. On freenx it just exits, with X2Go, you get a wallpaper but no panels or widgets. Right mouse click brings a pull down menu but most of the objects including add panel and widgets do not work. On all of these systems mate gnome and xfce work (although gnome is gnome-fallback and painfully slow, mate is much better).
I have not been able to find anything in system logs or searching with DuckDuckGo or Google that gives me a clue was to why these problems with KDE. Whatever broke KDE happened some time ago as I re-installed Mint from a backup about a month ago and still the same problems.
Even with Debian, Synaptik complains about an old version of Xinput but I can’t find anything that provides this with synaptic package manager.
I invite any suggestions that may help me determine why this is failing and resolve it. It’s odd because all three machines are derived from the same code base.
I am in the process of restoring Mint from backups. It will be back to Mint 17, I can not get 17.1 to work easily. You’d think they’d test this stuff before they kicked out a distribution, and normally I would before attempting an install but since Mint 17 had issues with KDE anyway I attempted without first testing.
For now I’m going to put it back to Mint 17, build a new virtual machine to work on 17.1 on, and when I figure out how to get it working I’ll upgrade the existing machine.
Sorry for the inconvenience everyone. This was really bad judgement on my part, owing largely to lack of sleep (I got up at 4:50AM and attempted this around midnight, too many hours up).