[URGENT] Jenkins queue up to 600 jobs with lots of unused slaves

Description

Hi, this morning there are 639 jobs in queue and just 1 running build.
This is critical with the upcoming 4.1 rc release coming.
Please fix urgently!


Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

Activity

Show:

Sandro Bonazzola January 13, 2017 at 12:12 PM

we're good to close this ticket.

Former user January 13, 2017 at 9:06 AM

I can confirm that our slaves are in good use at the moment. We have a total of 150 online slaves, 16 of which are PPC ones and the current slave utilization is at 87% which I would call very good:
http://graphite.phx.ovirt.org/dashboard/db/jenkins-monitoring?panelId=16&fullscreen
Unfortunately some reporting in Graphana is broken, that's why it's showing "1 running build" on the appropriate graph - please ignore that.

The build queue went down from 1500 to 1200 in 1 hour, so I would estimate the whole queue will be processed within 4-5 hours.

Looking at teh oVirt admin UI, hypervisors have around 87% CPU load as well, so adding more slaves will just slow down the others: we're running near hardware capacity at this point.
could you please confirm if there is anything I can do regarding this or we're good to wait for all the builds to complete?

Barak Korren January 13, 2017 at 7:53 AM

looks like my estimates were off by a factor. we have over 112 patches running in CI ATM: https://gerrit.ovirt.org/#/q/topic:mockito

We also have more slaves, not sure about the exact number but looks like about 100.
how many x86_64 slaves do we have ATM?

Barak Korren January 13, 2017 at 7:40 AM

Load seem to be due to ~20 engine patches sent at once, each running multiple upgrade jobs... probably not neccessary so maybe we should just abort the upgrade jobs if important tasks are starved atm.

Barak Korren January 13, 2017 at 7:37 AM

This is a false alarm, we have more then 50 slaves working.

The unused ones are the PPC slaves.

Fixed

Details

Assignee

Reporter

Priority

Created January 13, 2017 at 7:26 AM
Updated January 30, 2017 at 9:25 AM
Resolved January 13, 2017 at 7:37 AM