Got scientific updated and running on 3.3.8 pre-emptive, although it’s bloated at present like the stock kernel. The kernels I had built for CentOS were missing things needed by scientific, well at least it thinks it needs them, start up scripts try to load modules we really have no need for in our environment. So running a 3.3.8 kernel that is pretty much configured like the stock 2.2.26 kernel for now which is inefficient but better than the original. I’ll lean it out as time allows but for now bigger fish to fry.
Category Archives: Eskimo
Shellx Linux 3.3.8 Pre-emptive
Shellx is now upgraded to a 3.3.8 Pre-emptive kernel. The only EL6 based server not updated is scientific. For reasons I don’t understand, it requires a different configuration, some modules are required by the SL that aren’t required by CentOS. Not really sure why since it involves hardware I don’t have.
Mail, Web, FTP, Mx1, Mx2 Updated
The web and ftp server, client e-mail server, mx1 and mx2 incoming and list expansion servers, have all been updated to Linux kernel 3.3.8 pre-emptive which has been optimized for use in a KVM guest environment. This results in lower latency for these machines and faster responses.
Mail Server Is Restored
The mail server is restored to service. The virtual machine disk image has been restored from backup. This will not affect your mail as the spool directory is independent of the mail servers and backed up separately so it is not affected by a server restore.
Mail Server Down
When I attempted to restart the mail server and it corrupted the virtual machine image on disk and will not boot. I am in the process of restoring from backups. Estimated downtime is approximately 30 minutes.
Linux-3.3.8 / Linux-3.4
I am tired now.
After much experimentation, I was able to determine that the NFS version 4 behaviour changed between Linux kernel 3.3.8 and 3.4 so I’ve been able to get a pre-emptive 3.3.8 kernel working on the web server. It does cut latency somewhat.
3.x Pre-emptive Kernel
I succeeded in compile a 3.0 pre-emptive kernel that actually functions properly with NFSv4 and rpc.idmapd.
Moreover, I’ve come to understand the compatibility issues. The problem is that newer NFSv4 went to using nfsidmap which uses upcalls within the kernel rather than an daemon to handle mapping.
What I haven’t determined yet, is exactly at what kernel version this change was made and if it is at all possible to build a kernel with kernel nfsd support that will work with both. That would be the easiest, else I’ll have to change all the machines over at once which would be challenging.
Imapd / Pop3
Further research suggests that this isn’t going to fix it. I’m going to update Dovecot anyway just to get it current but will probably do this later this evening instead of at 5pm.
It appears that this problem is because of the POODLE exploit that came out which RedHat “solved” by disabling SSLv3.
The only fix at this end would be compiling OpenSSL from source, and then recompiling a whole bunch of stuff not to use the system version because RedHat isn’t going to fix it properly, or build a new server based on a non-broken operating system, and that is problematic because Red Hat’s EL6, upon which CentOS 6 is based, has a broken implementation of NFS version 4, which is really needed for mail to work properly owing to the lack of mandatory locking on earlier versions of NFS.
In the long term I am going to work towards moving our infrastructure away from Red Hat and towards Ubuntu. Although the Ubuntu people occasionally screw things up, they almost always fix them quickly. Red Hat is becoming impossible to maintain and have properly interact with other operating systems, kind of like Microsoft twenty years ago.
Since there is no good short term fix on this end, those affected will either need to upgrade their software to something capable of TLS or use a mailer that doesn’t override their encryption selections, such as Thunderbird.
Imapd and Pop3 E-mail
There are some versions of Outlook, and I don’t know which yet, that forces SSL encryption even if a customer selections non-encrypted mail and it tries to use a version not supported by our servers, causing mail retrieval to fail.
People who are affected have systems that do not support TLS (mostly very old) and only support a version of SSL that our server won’t accept. If you are retrieving mail okay at present then your machine isn’t affected.
In order to try to resolve this issue for others, I’m going to be upgrading Dovecot from version 2.0.8 to 2.2.16 around 5pm today. I’ve got the software built and ready to go but don’t want to install it in the middle of a business day in case something goes wrong and have to back it out. We’ve been using the version that was part of the CentOS distribution and it is out of date.
If there is an interruption, it should be brief, (I have saved the old configuration and can re-install the CentOS supplied version in seconds if need be) but I wanted to let you know in advance just in case.
Idmapd
Often what starts as a little problem has a way of growing. What I found out today, it’s not only newer kernels I build but also the kernels that exist in any non Redhat 6 based Linux, so idmapd is not working properly on Centos7, Scientif7, Fedora, Debian, Mint, or Ubuntu. Oddly, an actual operation on a file seems to work correctly, if I su to a user and touch to create a file, then go back and look at it on the server, it is created with the proper UID even though it isn’t displayed correctly with ls on the client. This is why the problem has essentially gone unnoticed for so long.
As near as I can tell there is some fundamentally different way that idmapd works under RedHat 6, then any other Linux distribution or even later versions of Redhat. I’ve found copious documentation of this bug but no fix that actually works except to revert to NFSv3. The problem with NFSv3, it does not have mandatory locking that works.
