This morning I finally figured out the source of the most recent instability (since we changed out the bad NIC card).
We kept having this incidence where I’d go to the co-lo, thought I had
everything working but in minutes or hours or sometimes a few days it would
just stop talking to the Internet.
One of those occurred this morning, I went down, looked at settings, nothing appeared to have changed, but it wasn’t routing. Rebooted the server,
started routing again. Went back home, couldn’t ping anything.
And I’m really half a sleep and my workstation is busted on account of the
fact that the night before I tried to upgrade the OS, it failed, and so I tried
to restore from backup but afterwards I could not boot. And at this point I
have had approximately two hours of sleep in the past 48 hours.
So I drove back down to the co-lo center again, and keep in mind it’s
22 miles each way and pretty close to rush hour so not at all a pleasant
drive. This time I rebooted again but still no route, several times, still
the same.
So at this point I got the NOC involved, neither of us could ping the
other end of the wire and that should not have been difficult since it’s just
a single Ethernet to Ethernet cable. But we couldn’t, so I got the idea of
unplugging the LAN interface and after doing that the WAN interface immediately
came up, so this pointed to something wrong on my end, didn’t know what but
had to be something on my end.
So I took a look at the routing table and it quickly became apparent what
was wrong, there were not one but TWO default routes. This is not legal under
Linux so how did it come about?
Well on the WAN interface, I had the correct gateway address, but on the
LAN side I was pointed at my own machine instead of his router. Why I did
his is because if I had a rounder in between my machines and LAN and his router
my router’s IP would be the gateway IP we point all the local machines to.
So corrected this to the correct gateway IP, everything came up from a
routing perspective and has remained that ever since.
But once I got home, I got another telephone call, not able to receive
e-mail from g-mail. At first I had the problem of not having a workstation
to use, totally forgot about my laptop, probably sleep deprivation.
But I did remember I had an antique Dell loaded with Linux, fired it up
and SSH’d into the mail servers, found both were operational but both had a
bunch of jobs stopped but no logged errors indicating why. I rebooted them
and they came up and ran fine except that the load went up to around 200 for about half an hour then settled down to a normal below 1 load.
At this point I’m speculating that without a network queue runs got stuck
until they exhausted memory then things died.
At this point I returned to my workstation. I had restored a corrupted
root partition from backups but after doing so it would not boot. I finally
chased this down to the fact that I did a reformat prior to reloading from
backups and this changed the partition UUID so that it no longer matched the
fstab file. Fixed that up and now it’s running properly again.
I expect to receive the new router sometime between this Friday and next
Monday, I don’t know how long it will take for me to learn how to use it will
enough to put it into service but it is very thoroughly documented, has eight
manuals, the administrative manual is 3061 pages. Also some free online courses and if you want to get into it deeply you can spend as much as $6000 on
non-free training. It supports damned near every communication protocol known
to man.