httpd go boom

For the past few weeks I've been dealing with a puzzling problem where this server becomes completely unresponsive. The first few times I caught I had to hit the power button to bring the system down. The third time I managed to catch it in the act with top(1) and discovered the httpd processes going absolutely bonkers. I also happened to be tail -f-ing the webserver access log at the time too. Oddly enough, there were httpd processes going nuts spawning and terminating processes over and over (load average spiked to over 100), but there were very few pages being served up. Took me a while to stop everything and kill off the httpd daemon. A check of the error logs didn't reveal anything enlightening unfortunately.

An upgrade to Apache 2.0.53 didn't seem to make things any better. It actually seems to have made things a little worse, since the problem seems to be cropping up more frequently.

At the moment, my prime suspect is mod_security, although I haven't read or seen anything about mod_security causing this kind of problem. But it started happening a few weeks ago on a previously stable server with no other changes other than adding mod_security to the mix. I haven't ruled out bad configuration on account of my own stupidity though. Also haven't ruled out that it could be because of using the 1.9dev1 version of mod_security either. I suppose I ought to test that out first.