Root cause identified (not a false positive) and after a few weeks of monitoring no new spikes were detected (just two on 15.08 and 16.08). As this did not impact anything and was only detected by nagios, there's no action to be taken here. Closing this. If it starts happening often and affecting us, we can investigate tools like mod_security or other tools to combat this.

any action item for this ticket?

The number of processes went down so this probably was a spike in requests.
We've got HTTPD working in prefork mode with MaxClients 256 so the number of apache processes seems accurate to the scenario when all workers were started. Now it dropped down to 88 processes. I'll check the logs for suspicious requests as this has signs of DoS as all workers were busy at some point.

Here's the graph from nagios:

Open

The majority of processes on the system seem to be apache threads:

ps aux | grep httpd | wc -l
258

need to check the logs to see if this was some kind of usage spike that caused the number of threads to grow.