Eskimo North


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Web Server Trouble




     The web server started acting up again, this time with no sign of DoS
attack.

     Something caused the system to increase CPU usage, increase memory usage,
and a lot of I/O happened between it and Ultra7, the main NFS server.

     Still, I couldn't find any unusual requests, ps didn't show any processes
eating an abornal amount of CPU or memory.

     ypbind and ypserv did start acting up again.

     I thought maybe something was goofy with portmapper since ypbind needs to
use it to find ypserv, NFS needs it, etc.  So I decided to reboot the machine
which is really the only way to recover from a portmap problem.

     It started shutting down but didn't kill everything and got stuck.  I
manually tried to kill httpd's that were still running and got "Memory Error",
which really isn't a memory error per se' but rather is something corrupt with
the memory allocation tables in the kernel.

     So then I was going to force boot it since a graceful shutdown seemed out
of the picture.  I attempted to sync the file systems first so there was a
minimal possibility of file system damage, but when I typed sync, that command
hung.

     I then stupidly tried it from another terminal, the ONLY other terminal I
had access to at that point and it too hung.  So now I'm at a point where it's
half-way shut down, I can't communicate with it.

     So I thought I was going to have to drive to Bellevue but after locking
things up I made one last check and it was in the process of rebooting.
However, when it came up it was without a file system check so I had to reboot
one more time and then when it came up there were some MySQL database issues
which I repaired.

     However, I still do not know what got things messed up to begin with.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
 Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
   Knowledgable human assistance, not telephone trees or script readers.
 See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Sat, 21 Jun 2008, David Larsen wrote:

> Date: Sat, 21 Jun 2008 18:10:18 -0700
> From: David Larsen <dslarsen@eskimo.com>
> To: Nanook <nanook@eskimo.com>
> Subject: Re: Web Server Trouble
> 
> I'm not able to get to eskimo.com myself now at all.
> 
> 
> 6:10 pm.
> 
> 
> David
> 
> 
> 
> On Jun 21, 2008, at 3:32 PM, Nanook wrote:
> 
> >
> >     We had issues with the web server starting around 3pm today but  
> > to be
> > honest I wasn't able to isolate the cause before the trouble  
> > resolved itself.
> >
> >     There was a DoS attack, a SYN flood on port 80, but it was low  
> > level and
> > the system is capable of sustaining a fairly high level SYN flood  
> > without
> > adverse effects and under ordinary circumstances this wouldn't have  
> > caused any
> > issues.
> >
> >     There was an NIS communications problem where YPBIND lost it's  
> > ability to
> > communicate with YPSERV, but again under ordinary circumstances, it  
> > would have
> > just connected to YPSERV on another server without any noticable  
> > interruption.
> > It eventually did do this but for some reason it took longer than  
> > normal to
> > reconfigure.
> >
> >     The actual request rate did not change during this period from the
> > background level, it stayed at around 30 hits/second and this  
> > machine has
> > handled 90 hits/second during traffic spikes in the past and more  
> > than 60
> > hits/second sustained without any problems so it's not a traffic  
> > issue.
> >
> >     The bottom line is I was unable to identify the cause and it  
> > returned to
> > normal operation on it's own without a reboot.
> >
> > -_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_- 
> > _-_-_-_-_-
> > Eskimo North Linux Friendly Internet Access, Shell Accounts, and  
> > Hosting.
> >   Knowledgable human assistance, not telephone trees or script  
> > readers.
> > See our web site: http://www.eskimo.com/ (206) 812-0051 or (800)  
> > 246-6874.
> >
> >
> 
>