I will be going to the co-location facility to attempt to resurrect the old Eskimo.Com SunOS shell server.
In addition I will be working on some firewall rules to prevent idiots brute force password guessing from locking me out of the router.
If you call during this time, please leave a message and include the latest time that it will be okay for me to call you back. If it’s before that time when I return I will call.
The eskimo.com shell server (old SunOS 4.1.4 machine) is down and will remain that way until tomorrow afternoon. It is stuck in a mode where the kernel is still running and it can be pinged, but no user mode programs get CPU time. This is an old SunOS bug.
We only have one car and my wife has it right now. In the meantime please use one of the other shell servers: shellx.eskimo.com, debian.eskimo.com, ubuntu.eskimo.com, opensuse.eskimo.com, centos7.eskimo.com, scientific.eskimo.com, scientific7.eskimo.com, fedora.eskimo.com, or mint.eskimo.com.
Even with all the changes I made, there was STILL an occasional slow load of images, and the delay was always 5 seconds when this happened.
I finally was able to chase this down. Rate limiting on the name servers was kicking in and 5 seconds is the default resolver time before it gives up and moves on to the next name server.
I have adjusted the rate limiting up so that they will rarely be invoked, added a rotate argument to the resolv.conf configuration so that it spreads the load across the servers, and I’ve added timeout:1 to make it give up after one second instead of five if a server doesn’t respond to minimize the page load delay in the event of a server failure.
This seems to have completely eliminated the delayed loading.
I have made some configuration changes so that when you go to eskimo.com or www.eskimo.com, you will connect SSL even if you do not specify https in the URL. This is to prevent user authentication information in applications like WebMail, the Forums, or PHPMyAdmin from being sniffed en route and also to secure your data.
I will be taking the web server down approximately five minutes after midnight tonight in order to image it so that the changes I made to improve responsiveness are not lost in the event a restoration from backup becomes necessary.
There will be a brief interruption between now and 6PM while our router at the co-location facility is rebooted. This is necessary because idiots trying to brute force password guess have caused it to disallow remote logins so I can not monitor traffic or make configuration changes.
I found and fixed the web server issue causing images to load slow for one client I knew of. The issue was that the server was starving for threads, the Apache defaults are not adequate for a larger busier site and we have sufficient machine resources to support a much larger configuration. Now this customers images load in milliseconds instead of seconds. The seconds were the time Apache was taking to ramp up processes. Now the minimum is sufficient to support traffic bursts and it can ramp up to very higher if necessary.
The log rotation issues have been resolved. Fail2ban is no longer grinding on huge logs and making the server slow. In addition, I’ve implemented file system caching so that local copies of frequently used files are kept on local disk negating the need to fetch them via NFS. I’ve also recompiled the server using maximum optimization and made other optimizations.
The result of these things is that it’s cut the time it takes to load our main page (which has some PHP and database lookups) from around 800ms to about 250ms, a notable improvement.
However, I have one customer who has a site that loads a small number of images and those images are still taking from 5 seconds to 15 seconds to load. I have a games page here that loads several hundred tiles that are actually larger than the images on his site and they all load in under 100ms.
We recently upgraded to 2.4.12, the latest version of Apache. I am wondering if anyone else is seeing any issues on their site. If so, please let me know, seeing what is common might help me diagnose and resolve this problem.
Our web server is slower than normal presently because of a failure of logrotate, a program which moves and compresses old logs. This has a allowed a large amount of log data to build up.
That in turn has caused fail2ban to consume a lot of resources, particularly disk I/O, and also flush the disk cache because it has to go through a couple of months worth of logging instead of the current days logs.
I am working on correcting this and cleaning out the old logs, it may take a day or two because of the volume of logs involved.
If you would like to have more control over your WordPress posts, for example change fonts, or change the size of fonts, or the font color, or perhaps you need subscripts for things like H2O, or super scripts for things like X2 + Y2 = Z2, or add special characters like a ♥ for valentines day, or maybe I ♣ my dog, you can do all this and more with a plugin called WP Edit that replaces the TinyMCE editor that comes with WordPress.