Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

deploy-to-experimental job can 'miss' builds when Jenkins is loaded

Description

When Jenkins is highly loaded, the global deploy-to-experimental job can fail to find the build jobs that were triggered with it.

This could be because those jobs were not yet started by the time the deploy searched for them.

Example of such deploy-to-experimental run:
http://jenkins.ovirt.org/view/experimental%20jobs/job/deploy-to-ovirt_experimental/781/

We need to ensure that the deploy-to-experimental job can find queued builds as well.

Activity

Show:

Barak Korren May 23, 2017 at 6:58 AM

Patch merged

Barak Korren May 10, 2017 at 8:53 AM

Created and verified a patch

Blocking on code review

Barak Korren May 7, 2017 at 9:12 AM

We need to fix this. This may not be very important for the 'deploy' job because typically when Jenkins is loaded, it is quite likely that another patch for the same project will be submitted soon enough so that 'missd' changes will make it to 'experimental'. But the same sibliong job search code will also be used by the change queue scheduling jobs, and there it would be critical to not miss any patches.

Barak Korren May 7, 2017 at 8:33 AM

I have managed to reproduce the issue on the staging instance. I did it by creating multiple jobs that trigger on the dame Gerrit event and then limiting one of them to run on a slave label that does not exist.

It turns out that queued jobs are handle in Jenkins with different data structures and objects then builds that are already running or have completed. Here is a reference to some relevant Jenkins APIs:
http://javadoc.jenkins.io/hudson/model/Queue.html

Fixed

Details

Assignee

Reporter

Blocked By

Components

Priority

Parent

Created May 4, 2017 at 2:13 PM
Updated June 1, 2017 at 11:31 AM
Resolved May 23, 2017 at 6:58 AM