spike in process number on resources.ovirt.org

Description

The number of processes on resources.ovirt.org jumped from 300 to 500 today. Need to check why that happened and if it's normal - reconfigure nagios accordingly

Activity

Show:
Evgheni Dereveanchin
August 15, 2016, 9:56 AM

Here's the graph from nagios:

The majority of processes on the system seem to be apache threads:

ps aux | grep httpd | wc -l
258

need to check the logs to see if this was some kind of usage spike that caused the number of threads to grow.

Evgheni Dereveanchin
August 15, 2016, 12:00 PM

The number of processes went down so this probably was a spike in requests.
We've got HTTPD working in prefork mode with MaxClients 256 so the number of apache processes seems accurate to the scenario when all workers were started. Now it dropped down to 88 processes. I'll check the logs for suspicious requests as this has signs of DoS as all workers were busy at some point.

Eyal Edri
August 28, 2016, 11:33 AM

any action item for this ticket?

Evgheni Dereveanchin
August 29, 2016, 8:20 AM

Root cause identified (not a false positive) and after a few weeks of monitoring no new spikes were detected (just two on 15.08 and 16.08). As this did not impact anything and was only detected by nagios, there's no action to be taken here. Closing this. If it starts happening often and affecting us, we can investigate tools like mod_security or other tools to combat this.

Done

Assignee

Evgheni Dereveanchin

Reporter

Evgheni Dereveanchin

Blocked By

None

Priority

Low