I have found that what kernel resource ran out on the client mail server was the number of open files.  I’ve bumped that up by 5x, hopefully that will be sufficient.

Dec 3 08:57:22 mail kernel: [371342.776244] VFS: file-max limit 194527 reached
Dec 3 09:43:33 mail kernel: [374113.872327] VFS: file-max limit 194527 reached
Dec 3 09:43:33 mail kernel: [374113.905264] VFS: file-max limit 194527 reached
Dec 3 09:45:17 mail kernel: [374217.999876] VFS: file-max limit 194527 reached
Dec 3 09:45:23 mail kernel: [374223.425007] VFS: file-max limit 194527 reached
Dec 3 09:45:24 mail kernel: [374224.590837] VFS: file-max limit 194527 reached
Dec 3 10:21:11 mail kernel: [376372.038404] VFS: file-max limit 194527 reached
Dec 3 10:21:24 mail kernel: [376385.096118] VFS: file-max limit 194527 reached

     Our client mail server got flaky between around 11AM and 1PM today.

     I am pretty sure the cause is brute force password attacks exhausting some kernel resource but I have not been able to identify the resource being exhausted.

     The reason I believe this is the cause is that in the last two days the number of IP addresses we lock out for these sorts of attacks has increased from a typical number of several hundred to over 15,000.  This is probably the result of a new Windows virus that is allowing the creation of huge botnets.  This is something we see periodically.

     I rebooted the server which restored it to normal functionality and will continue to try to determine what is being exhausted and correct it.