All loader-container nodes are offline, 99 jobs in the queue, jobs stuck for 2 hours

Description

We depend on jenkins to run our tests and build artifacts in a timely
manner.

Here is example job, stuck for 2 hours:
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9212/

This looks like the issue:

[image: collapse]
<https://jenkins.ovirt.org/toggleCollapse?paneId=executors>Build Executor
Status <https://jenkins.ovirt.org/computer/>
master <https://jenkins.ovirt.org/computer/(master)/>
1 Idle
2 Idle
3 Idle
4 Idle
lfedora1.lf-dev.marist.edu
<https://jenkins.ovirt.org/computer/lfedora1.lf-dev.marist.edu/>
1 Idle
loader-container-28ljp
<https://jenkins.ovirt.org/computer/loader-container-28ljp/> (offline)
(suspended)
loader-container-35tlc
<https://jenkins.ovirt.org/computer/loader-container-35tlc/> (offline)
(suspended)
loader-container-3j0ps
<https://jenkins.ovirt.org/computer/loader-container-3j0ps/> (offline)
(suspended)
loader-container-4391q
<https://jenkins.ovirt.org/computer/loader-container-4391q/> (offline)
(suspended)
loader-container-58v3d
<https://jenkins.ovirt.org/computer/loader-container-58v3d/> (offline)
(suspended)
loader-container-66r5r
<https://jenkins.ovirt.org/computer/loader-container-66r5r/> (offline)
(suspended)
loader-container-6k57j
<https://jenkins.ovirt.org/computer/loader-container-6k57j/> (offline)
(suspended)
loader-container-7813f
<https://jenkins.ovirt.org/computer/loader-container-7813f/> (offline)
(suspended)
loader-container-8dkj1
<https://jenkins.ovirt.org/computer/loader-container-8dkj1/> (offline)
(suspended)
loader-container-8hb98
<https://jenkins.ovirt.org/computer/loader-container-8hb98/> (offline)
(suspended)
loader-container-8v609
<https://jenkins.ovirt.org/computer/loader-container-8v609/> (offline)
(suspended)
loader-container-99f36
<https://jenkins.ovirt.org/computer/loader-container-99f36/> (offline)
(suspended)
loader-container-c16rq
<https://jenkins.ovirt.org/computer/loader-container-c16rq/> (offline)
(suspended)
loader-container-c1vkj
<https://jenkins.ovirt.org/computer/loader-container-c1vkj/> (offline)
(suspended)
loader-container-c4g8c
<https://jenkins.ovirt.org/computer/loader-container-c4g8c/> (offline)
(suspended)
loader-container-cqrx5
<https://jenkins.ovirt.org/computer/loader-container-cqrx5/> (offline)
(suspended)
loader-container-d1r8d
<https://jenkins.ovirt.org/computer/loader-container-d1r8d/> (offline)
(suspended)
loader-container-dpp70
<https://jenkins.ovirt.org/computer/loader-container-dpp70/> (offline)
(suspended)
loader-container-gz52q
<https://jenkins.ovirt.org/computer/loader-container-gz52q/> (offline)
(suspended)
loader-container-h9kcf
<https://jenkins.ovirt.org/computer/loader-container-h9kcf/> (offline)
(suspended)
loader-container-hjq0g
<https://jenkins.ovirt.org/computer/loader-container-hjq0g/> (offline)
(suspended)
loader-container-ht75p
<https://jenkins.ovirt.org/computer/loader-container-ht75p/> (offline)
(suspended)
loader-container-htstk
<https://jenkins.ovirt.org/computer/loader-container-htstk/> (offline)
(suspended)
loader-container-j47s3
<https://jenkins.ovirt.org/computer/loader-container-j47s3/> (offline)
(suspended)
loader-container-k2p10
<https://jenkins.ovirt.org/computer/loader-container-k2p10/> (offline)
(suspended)
loader-container-kc066
<https://jenkins.ovirt.org/computer/loader-container-kc066/> (offline)
(suspended)
loader-container-l1vd9
<https://jenkins.ovirt.org/computer/loader-container-l1vd9/> (offline)
(suspended)
loader-container-l7lqn
<https://jenkins.ovirt.org/computer/loader-container-l7lqn/> (offline)
(suspended)
loader-container-lqqkj
<https://jenkins.ovirt.org/computer/loader-container-lqqkj/> (offline)
(suspended)
loader-container-m8fkq
<https://jenkins.ovirt.org/computer/loader-container-m8fkq/> (offline)
(suspended)
loader-container-n2zml
<https://jenkins.ovirt.org/computer/loader-container-n2zml/> (offline)
(suspended)
loader-container-p7rxw
<https://jenkins.ovirt.org/computer/loader-container-p7rxw/> (offline)
(suspended)
loader-container-pqg56
<https://jenkins.ovirt.org/computer/loader-container-pqg56/> (offline)
(suspended)
loader-container-qtsdr
<https://jenkins.ovirt.org/computer/loader-container-qtsdr/> (offline)
(suspended)
loader-container-qxwnp
<https://jenkins.ovirt.org/computer/loader-container-qxwnp/> (offline)
(suspended)
loader-container-rdlww
<https://jenkins.ovirt.org/computer/loader-container-rdlww/> (offline)
(suspended)
loader-container-rp1rc
<https://jenkins.ovirt.org/computer/loader-container-rp1rc/> (offline)
(suspended)
loader-container-s6ndk
<https://jenkins.ovirt.org/computer/loader-container-s6ndk/> (offline)
(suspended)
loader-container-sv0pt
<https://jenkins.ovirt.org/computer/loader-container-sv0pt/> (offline)
(suspended)
loader-container-t6qd9
<https://jenkins.ovirt.org/computer/loader-container-t6qd9/> (offline)
(suspended)
loader-container-vb5dj
<https://jenkins.ovirt.org/computer/loader-container-vb5dj/> (offline)
(suspended)
loader-container-vhdck
<https://jenkins.ovirt.org/computer/loader-container-vhdck/> (offline)
(suspended)
loader-container-vmww3
<https://jenkins.ovirt.org/computer/loader-container-vmww3/> (offline)
(suspended)
loader-container-vv202
<https://jenkins.ovirt.org/computer/loader-container-vv202/> (offline)
(suspended)
loader-container-w7n89
<https://jenkins.ovirt.org/computer/loader-container-w7n89/> (offline)
(suspended)
loader-container-x5gwl
<https://jenkins.ovirt.org/computer/loader-container-x5gwl/> (offline)
(suspended)
loader-container-xmthb
<https://jenkins.ovirt.org/computer/loader-container-xmthb/> (offline)
(suspended)
loader-container-xxcl1
<https://jenkins.ovirt.org/computer/loader-container-xxcl1/> (offline)
(suspended)
loader-container-zp4w9
<https://jenkins.ovirt.org/computer/loader-container-zp4w9/> (offline)
(suspended)

Activity

Show:
Evgheni Dereveanchin
August 7, 2019, 12:42 PM

This is a duplicate of OVIRT-2765, basically after the last upgrade Jenkins started using more OS threads and ran out of the limit set in CentOS which is 4096 for regular users. I've bumped this to 16k but was waiting for a weekend to apply changes. Unfortunately, the issue happened sooner and I had to restart Jenkins during the working day of July 31. There has not been a single java.lang.OutOfMemoryError ever since.

Eyal Edri
August 1, 2019, 9:37 AM

This was caused by a bug in Jenkins that caused the thread that listens to
JNLP to die.
Evgheni, can you give some info on how it was fixed?

On Wed, Jul 31, 2019 at 6:36 PM Nir Soffer <nsoffer@redhat.com> wrote:

> We depend on jenkins to run our tests and build artifacts in a timely
> manner.
>
> Here is example job, stuck for 2 hours:
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9212/
>
> This looks like the issue:
>
> [image: collapse]
> <https://jenkins.ovirt.org/toggleCollapse?paneId=executors>Build Executor
> Status <https://jenkins.ovirt.org/computer/>
> master <https://jenkins.ovirt.org/computer/(master)/>
> 1 Idle
> 2 Idle
> 3 Idle
> 4 Idle
> lfedora1.lf-dev.marist.edu
> <https://jenkins.ovirt.org/computer/lfedora1.lf-dev.marist.edu/>
> 1 Idle
> loader-container-28ljp
> <https://jenkins.ovirt.org/computer/loader-container-28ljp/> (offline)
> (suspended)
> loader-container-35tlc
> <https://jenkins.ovirt.org/computer/loader-container-35tlc/> (offline)
> (suspended)
> loader-container-3j0ps
> <https://jenkins.ovirt.org/computer/loader-container-3j0ps/> (offline)
> (suspended)
> loader-container-4391q
> <https://jenkins.ovirt.org/computer/loader-container-4391q/> (offline)
> (suspended)
> loader-container-58v3d
> <https://jenkins.ovirt.org/computer/loader-container-58v3d/> (offline)
> (suspended)
> loader-container-66r5r
> <https://jenkins.ovirt.org/computer/loader-container-66r5r/> (offline)
> (suspended)
> loader-container-6k57j
> <https://jenkins.ovirt.org/computer/loader-container-6k57j/> (offline)
> (suspended)
> loader-container-7813f
> <https://jenkins.ovirt.org/computer/loader-container-7813f/> (offline)
> (suspended)
> loader-container-8dkj1
> <https://jenkins.ovirt.org/computer/loader-container-8dkj1/> (offline)
> (suspended)
> loader-container-8hb98
> <https://jenkins.ovirt.org/computer/loader-container-8hb98/> (offline)
> (suspended)
> loader-container-8v609
> <https://jenkins.ovirt.org/computer/loader-container-8v609/> (offline)
> (suspended)
> loader-container-99f36
> <https://jenkins.ovirt.org/computer/loader-container-99f36/> (offline)
> (suspended)
> loader-container-c16rq
> <https://jenkins.ovirt.org/computer/loader-container-c16rq/> (offline)
> (suspended)
> loader-container-c1vkj
> <https://jenkins.ovirt.org/computer/loader-container-c1vkj/> (offline)
> (suspended)
> loader-container-c4g8c
> <https://jenkins.ovirt.org/computer/loader-container-c4g8c/> (offline)
> (suspended)
> loader-container-cqrx5
> <https://jenkins.ovirt.org/computer/loader-container-cqrx5/> (offline)
> (suspended)
> loader-container-d1r8d
> <https://jenkins.ovirt.org/computer/loader-container-d1r8d/> (offline)
> (suspended)
> loader-container-dpp70
> <https://jenkins.ovirt.org/computer/loader-container-dpp70/> (offline)
> (suspended)
> loader-container-gz52q
> <https://jenkins.ovirt.org/computer/loader-container-gz52q/> (offline)
> (suspended)
> loader-container-h9kcf
> <https://jenkins.ovirt.org/computer/loader-container-h9kcf/> (offline)
> (suspended)
> loader-container-hjq0g
> <https://jenkins.ovirt.org/computer/loader-container-hjq0g/> (offline)
> (suspended)
> loader-container-ht75p
> <https://jenkins.ovirt.org/computer/loader-container-ht75p/> (offline)
> (suspended)
> loader-container-htstk
> <https://jenkins.ovirt.org/computer/loader-container-htstk/> (offline)
> (suspended)
> loader-container-j47s3
> <https://jenkins.ovirt.org/computer/loader-container-j47s3/> (offline)
> (suspended)
> loader-container-k2p10
> <https://jenkins.ovirt.org/computer/loader-container-k2p10/> (offline)
> (suspended)
> loader-container-kc066
> <https://jenkins.ovirt.org/computer/loader-container-kc066/> (offline)
> (suspended)
> loader-container-l1vd9
> <https://jenkins.ovirt.org/computer/loader-container-l1vd9/> (offline)
> (suspended)
> loader-container-l7lqn
> <https://jenkins.ovirt.org/computer/loader-container-l7lqn/> (offline)
> (suspended)
> loader-container-lqqkj
> <https://jenkins.ovirt.org/computer/loader-container-lqqkj/> (offline)
> (suspended)
> loader-container-m8fkq
> <https://jenkins.ovirt.org/computer/loader-container-m8fkq/> (offline)
> (suspended)
> loader-container-n2zml
> <https://jenkins.ovirt.org/computer/loader-container-n2zml/> (offline)
> (suspended)
> loader-container-p7rxw
> <https://jenkins.ovirt.org/computer/loader-container-p7rxw/> (offline)
> (suspended)
> loader-container-pqg56
> <https://jenkins.ovirt.org/computer/loader-container-pqg56/> (offline)
> (suspended)
> loader-container-qtsdr
> <https://jenkins.ovirt.org/computer/loader-container-qtsdr/> (offline)
> (suspended)
> loader-container-qxwnp
> <https://jenkins.ovirt.org/computer/loader-container-qxwnp/> (offline)
> (suspended)
> loader-container-rdlww
> <https://jenkins.ovirt.org/computer/loader-container-rdlww/> (offline)
> (suspended)
> loader-container-rp1rc
> <https://jenkins.ovirt.org/computer/loader-container-rp1rc/> (offline)
> (suspended)
> loader-container-s6ndk
> <https://jenkins.ovirt.org/computer/loader-container-s6ndk/> (offline)
> (suspended)
> loader-container-sv0pt
> <https://jenkins.ovirt.org/computer/loader-container-sv0pt/> (offline)
> (suspended)
> loader-container-t6qd9
> <https://jenkins.ovirt.org/computer/loader-container-t6qd9/> (offline)
> (suspended)
> loader-container-vb5dj
> <https://jenkins.ovirt.org/computer/loader-container-vb5dj/> (offline)
> (suspended)
> loader-container-vhdck
> <https://jenkins.ovirt.org/computer/loader-container-vhdck/> (offline)
> (suspended)
> loader-container-vmww3
> <https://jenkins.ovirt.org/computer/loader-container-vmww3/> (offline)
> (suspended)
> loader-container-vv202
> <https://jenkins.ovirt.org/computer/loader-container-vv202/> (offline)
> (suspended)
> loader-container-w7n89
> <https://jenkins.ovirt.org/computer/loader-container-w7n89/> (offline)
> (suspended)
> loader-container-x5gwl
> <https://jenkins.ovirt.org/computer/loader-container-x5gwl/> (offline)
> (suspended)
> loader-container-xmthb
> <https://jenkins.ovirt.org/computer/loader-container-xmthb/> (offline)
> (suspended)
> loader-container-xxcl1
> <https://jenkins.ovirt.org/computer/loader-container-xxcl1/> (offline)
> (suspended)
> loader-container-zp4w9
> <https://jenkins.ovirt.org/computer/loader-container-zp4w9/> (offline)
> (suspended)
> _______________________________________________
> Devel mailing list – devel@ovirt.org
> To unsubscribe send an email to devel-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7YBUQRA6HSAG4ZSOCJG72ZR6NDLOB6AA/
>

Eyal edri

He / Him / His

MANAGER

CONTINUOUS PRODUCTIZATION

SYSTEM ENGINEERING

Red Hat <https://www.redhat.com/>
<https://red.ht/sig>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)

Eyal Edri
August 1, 2019, 9:02 AM

can you add info on what was the bug and how you fixed it?

Duplicate

Assignee

Evgheni Dereveanchin

Reporter

Nir Soffer

Blocked By

None

Priority

Medium