All loader-container nodes are offline, 99 jobs in the queue, jobs stuck for 2 hours

This is a duplicate of OVIRT-2765, basically after the last upgrade Jenkins started using more OS threads and ran out of the limit set in CentOS which is 4096 for regular users. I've bumped this to 16k but was waiting for a weekend to apply changes. Unfortunately, the issue happened sooner and I had to restart Jenkins during the working day of July 31. There has not been a single java.lang.OutOfMemoryError ever since.

This was caused by a bug in Jenkins that caused the thread that listens to
JNLP to die.
Evgheni, can you give some info on how it was fixed?

On Wed, Jul 31, 2019 at 6:36 PM Nir Soffer <nsoffer@redhat.com> wrote:

> We depend on jenkins to run our tests and build artifacts in a timely
> manner.
>
> Here is example job, stuck for 2 hours:
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9212/
>
> This looks like the issue:
>
> [image: collapse]
> <https://jenkins.ovirt.org/toggleCollapse?paneId=executors>Build Executor
> Status <https://jenkins.ovirt.org/computer/>
> master <https://jenkins.ovirt.org/computer/(master)/>
> 1 Idle
> 2 Idle
> 3 Idle
> 4 Idle
> lfedora1.lf-dev.marist.edu
> <https://jenkins.ovirt.org/computer/lfedora1.lf-dev.marist.edu/>
> 1 Idle
> loader-container-28ljp
> <https://jenkins.ovirt.org/computer/loader-container-28ljp/> (offline)
> (suspended)
> loader-container-35tlc
> <https://jenkins.ovirt.org/computer/loader-container-35tlc/> (offline)
> (suspended)
> loader-container-3j0ps
> <https://jenkins.ovirt.org/computer/loader-container-3j0ps/> (offline)
> (suspended)
> loader-container-4391q
> <https://jenkins.ovirt.org/computer/loader-container-4391q/> (offline)
> (suspended)
> loader-container-58v3d
> <https://jenkins.ovirt.org/computer/loader-container-58v3d/> (offline)
> (suspended)
> loader-container-66r5r
> <https://jenkins.ovirt.org/computer/loader-container-66r5r/> (offline)
> (suspended)
> loader-container-6k57j
> <https://jenkins.ovirt.org/computer/loader-container-6k57j/> (offline)
> (suspended)
> loader-container-7813f
> <https://jenkins.ovirt.org/computer/loader-container-7813f/> (offline)
> (suspended)
> loader-container-8dkj1
> <https://jenkins.ovirt.org/computer/loader-container-8dkj1/> (offline)
> (suspended)
> loader-container-8hb98
> <https://jenkins.ovirt.org/computer/loader-container-8hb98/> (offline)
> (suspended)
> loader-container-8v609
> <https://jenkins.ovirt.org/computer/loader-container-8v609/> (offline)
> (suspended)
> loader-container-99f36
> <https://jenkins.ovirt.org/computer/loader-container-99f36/> (offline)
> (suspended)
> loader-container-c16rq
> <https://jenkins.ovirt.org/computer/loader-container-c16rq/> (offline)
> (suspended)
> loader-container-c1vkj
> <https://jenkins.ovirt.org/computer/loader-container-c1vkj/> (offline)
> (suspended)
> loader-container-c4g8c
> <https://jenkins.ovirt.org/computer/loader-container-c4g8c/> (offline)
> (suspended)
> loader-container-cqrx5
> <https://jenkins.ovirt.org/computer/loader-container-cqrx5/> (offline)
> (suspended)
> loader-container-d1r8d
> <https://jenkins.ovirt.org/computer/loader-container-d1r8d/> (offline)
> (suspended)
> loader-container-dpp70
> <https://jenkins.ovirt.org/computer/loader-container-dpp70/> (offline)
> (suspended)
> loader-container-gz52q
> <https://jenkins.ovirt.org/computer/loader-container-gz52q/> (offline)
> (suspended)
> loader-container-h9kcf
> <https://jenkins.ovirt.org/computer/loader-container-h9kcf/> (offline)
> (suspended)
> loader-container-hjq0g
> <https://jenkins.ovirt.org/computer/loader-container-hjq0g/> (offline)
> (suspended)
> loader-container-ht75p
> <https://jenkins.ovirt.org/computer/loader-container-ht75p/> (offline)
> (suspended)
> loader-container-htstk
> <https://jenkins.ovirt.org/computer/loader-container-htstk/> (offline)
> (suspended)
> loader-container-j47s3
> <https://jenkins.ovirt.org/computer/loader-container-j47s3/> (offline)
> (suspended)
> loader-container-k2p10
> <https://jenkins.ovirt.org/computer/loader-container-k2p10/> (offline)
> (suspended)
> loader-container-kc066
> <https://jenkins.ovirt.org/computer/loader-container-kc066/> (offline)
> (suspended)
> loader-container-l1vd9
> <https://jenkins.ovirt.org/computer/loader-container-l1vd9/> (offline)
> (suspended)
> loader-container-l7lqn
> <https://jenkins.ovirt.org/computer/loader-container-l7lqn/> (offline)
> (suspended)
> loader-container-lqqkj
> <https://jenkins.ovirt.org/computer/loader-container-lqqkj/> (offline)
> (suspended)
> loader-container-m8fkq
> <https://jenkins.ovirt.org/computer/loader-container-m8fkq/> (offline)
> (suspended)
> loader-container-n2zml
> <https://jenkins.ovirt.org/computer/loader-container-n2zml/> (offline)
> (suspended)
> loader-container-p7rxw
> <https://jenkins.ovirt.org/computer/loader-container-p7rxw/> (offline)
> (suspended)
> loader-container-pqg56
> <https://jenkins.ovirt.org/computer/loader-container-pqg56/> (offline)
> (suspended)
> loader-container-qtsdr
> <https://jenkins.ovirt.org/computer/loader-container-qtsdr/> (offline)
> (suspended)
> loader-container-qxwnp
> <https://jenkins.ovirt.org/computer/loader-container-qxwnp/> (offline)
> (suspended)
> loader-container-rdlww
> <https://jenkins.ovirt.org/computer/loader-container-rdlww/> (offline)
> (suspended)
> loader-container-rp1rc
> <https://jenkins.ovirt.org/computer/loader-container-rp1rc/> (offline)
> (suspended)
> loader-container-s6ndk
> <https://jenkins.ovirt.org/computer/loader-container-s6ndk/> (offline)
> (suspended)
> loader-container-sv0pt
> <https://jenkins.ovirt.org/computer/loader-container-sv0pt/> (offline)
> (suspended)
> loader-container-t6qd9
> <https://jenkins.ovirt.org/computer/loader-container-t6qd9/> (offline)
> (suspended)
> loader-container-vb5dj
> <https://jenkins.ovirt.org/computer/loader-container-vb5dj/> (offline)
> (suspended)
> loader-container-vhdck
> <https://jenkins.ovirt.org/computer/loader-container-vhdck/> (offline)
> (suspended)
> loader-container-vmww3
> <https://jenkins.ovirt.org/computer/loader-container-vmww3/> (offline)
> (suspended)
> loader-container-vv202
> <https://jenkins.ovirt.org/computer/loader-container-vv202/> (offline)
> (suspended)
> loader-container-w7n89
> <https://jenkins.ovirt.org/computer/loader-container-w7n89/> (offline)
> (suspended)
> loader-container-x5gwl
> <https://jenkins.ovirt.org/computer/loader-container-x5gwl/> (offline)
> (suspended)
> loader-container-xmthb
> <https://jenkins.ovirt.org/computer/loader-container-xmthb/> (offline)
> (suspended)
> loader-container-xxcl1
> <https://jenkins.ovirt.org/computer/loader-container-xxcl1/> (offline)
> (suspended)
> loader-container-zp4w9
> <https://jenkins.ovirt.org/computer/loader-container-zp4w9/> (offline)
> (suspended)
> _______________________________________________
> Devel mailing list – devel@ovirt.org
> To unsubscribe send an email to devel-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7YBUQRA6HSAG4ZSOCJG72ZR6NDLOB6AA/
>

–

Eyal edri

He / Him / His

MANAGER

CONTINUOUS PRODUCTIZATION

SYSTEM ENGINEERING

Red Hat <https://www.redhat.com/>
<https://red.ht/sig>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)

can you add info on what was the bug and how you fixed it?