Web Server / MXLinux Shell Server

     Mxlinux is now upgraded to Mxlinux-21.

     The following describes what I am working on to address slow web response.  If you like tech details read on, else just know I’m working on a solution.

     I am in the process of acquiring components for a new web server.  The existing server is no longer able to handle the peak loads.  This came about rather suddenly and is a combination of increased traffic and a larger mix of applications verses flat-files being served.  Applications require CPU to execute something to create a page or possibly an interactive situation such as connecting to a shell server via the web.  In particular the popularity of friendica and nextcloud has increased, and nextcloud has become ever more bloated with each new version.

     There is a big plus to this, it has generated new customer interest and with that the money is available for new hardware.  But we were already running some fairly high performance hardware so now really reaching for stratospheric performance.  It’s easy to get a lot of threads at a low CPU speed or a few threads at a high speed but both at once are a challenge.

     The existing web server is on an i7-6850k based system which is a 6-core 12-thread CPU, and it is maxed out at 128GB of RAM, that’s as much as that platform can support.  Only the database is on SSD which means any fork/exec that isn’t cached involves disk access.  When the machine gets busy everything gets blown out of cache because there isn’t enough RAM and that’s when performance really suffers.  The machine is shared with Mint, Debian, and Ubuntu shell servers, those and the web server are all virtual machines on the same physical box.

     To address this I am building a new server that will be based on an i9-10940x which is a 14 core 28 thread CPU, most importantly it is capable of addressing 256GB of RAM.  The shell servers will remain on the existing machine and the web server will be on bare metal on the new server so it will have 4x the RAM available as the existing machine.  Additionally, not only the database but also system programs will be on SSD in a RAID-1 configuration so that fork/exec operations will be much faster.  The i9-10940x is rated at 3.6 Ghz but can be clocked at 5Ghz IF the heat can be adequately removed, so finding an adequate cooling solution is the main challenge.  I’m going to start with a Noctua 15 dual fan and replace the fans with some noisy but very high CFM.  The co-lo already sounds like you’re standing behind a 747 during take-off so noisy fans not a huge concern.  If that does not suffice may have to go liquid but I really prefer to avoid that since maintenance of liquid cooled systems tends to be a major headache.  So we’ll have more than twice the CPU cycles, 4x the memory, and much faster disk infrastructure.

     One thing making this take longer is I have to get rid of two old Sparc machines in order to make the power budget this will require available.  At the co-lo facility I have one 20 AMP circuit and converting the old antique free radius to the new version is some learning curve that is straining my 63 year old brain but making slow progress.

News for Shell Users

     You can login to any of our shell servers without a password by setting up ssh keys.  To do this, from a terminal on your machine type:

     ssh-copyid username@host.eskimo.com

     Where username is your login, and host is the shell server you want to connect to, for example, if I wanted to set them up for ubuntu I would type:

     ssh-copyid nanook@ubuntu.eskimo.com

     It will then prompt for your password.  After you type it you can then ssh username@ubuntu.eskimo.com or whichever host(s) you set up, and you should never need to type a password again.

     This prevents two potential issues, one it will prevent someone who installs a keyboard logger on your machine from obtaining authentication information for your accounts here.  Two, it will prevent you from getting locked out from too many mistyped passwords.

     Second bit of news, I will be attempting an upgrade of mxlinux from 19 to 21 this evening.  The dpkg program in mxlinux 19 is so old it will no longer install a modern kernel.  The machine will need to be rebooted during this procedure and may be unavailable for some time if things go wrong.  Since mxlinux does not include an updater program, this is, like debian, a very manual process fraught with potential errors.