Jenkins running stable and no OOMs are present in the log after increasing the limit. Closing the ticket.

For now I’ve created /etc/security/limits.d/22-jenkins-nproc.conf with the following content:

jenkins soft nproc 16384

Restarting Jenkins will apply the new limit while I can work on a patch to puppet since we’ll need to add an extra module to define this

The jenkins user can currently run 4096 processes:

ulimit -u
4096

Looking at the java process it currently goes up to 1500 threads during normal operation:

cat /proc/490/status | grep Threads
Threads: 1720

As for memory usage, it seems to be under control:

One of the noteworthy error messages in jenkins.log is one like this:

SEVERE: GerritMissedEventsPlaybackEnabledChecker thread failed with error
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at hudson.model.AsyncPeriodicWork.doRun(AsyncPeriodicWork.java:119)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

This may mean that we have to bump ulimit or check memory allocation on the Jenkins. I recall @Barak Korren hitting similar issues on containerized Jenkins deployments.

As soon as the UI died, new container workers could no longer spawn as they need to grab http://jenkins.ovirt.org//jnlpJars/remoting.jar in order to connect back to Jenkins.