The number of processes on resources.ovirt.org jumped from 300 to 500 today. Need to check why that happened and if it's normal - reconfigure nagios accordingly
Here's the graph from nagios:
The majority of processes on the system seem to be apache threads:
ps aux | grep httpd | wc -l
need to check the logs to see if this was some kind of usage spike that caused the number of threads to grow.
The number of processes went down so this probably was a spike in requests.
We've got HTTPD working in prefork mode with MaxClients 256 so the number of apache processes seems accurate to the scenario when all workers were started. Now it dropped down to 88 processes. I'll check the logs for suspicious requests as this has signs of DoS as all workers were busy at some point.
any action item for this ticket?
Root cause identified (not a false positive) and after a few weeks of monitoring no new spikes were detected (just two on 15.08 and 16.08). As this did not impact anything and was only detected by nagios, there's no action to be taken here. Closing this. If it starts happening often and affecting us, we can investigate tools like mod_security or other tools to combat this.