gerrit does not print check_merged output

Description

Hello,

Recently vdsm gerrit job check-merged doesn't send report after its done -
check all recent jobs - they were done but the gerrit url doesn't show the
output nor send email about failures.

Thanks in advance
Yaniv Bronhaim.


Yaniv Bronhaim.

Activity

Show:

Eyal Edri August 27, 2017 at 3:04 PM

Please try also to ask on the users list: https://groups.google.com/forum/#!forum/jenkinsci-users

Former user August 27, 2017 at 9:32 AM

I was able to reproduce this issue in our staging environment.
What happens is that if you configure more than 1 job for a certain gerrit event and the first triggered will fail - jenkins will fail to exand the gerrit-trigger-plugin parameters and will throw a 'ConcurrentModificationException'

Here is what I've found in the logs:

Aug 23, 2017 3:14:17 PM com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.ParameterExpander expandParameters SEVERE: Failed to expand env vars into gerrit cmd. Gerrit won't be notified!! java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:885) at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.ParameterExpander.expandParameters(ParameterExpander.java:266) at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.ParameterExpander.getBuildStartedCommand(ParameterExpander.java:132) at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.GerritNotifier.buildStarted(GerritNotifier.java:84) at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.job.ssh.BuildStartedCommandJob.run(BuildStartedCommandJob.java:80) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

My guess is that it has to do with the fact that Jenkins runs different 'Run' (A particular execution of Job) classes in different threads, while the gerrit-trigger-plugin parameters are shared among those threads so maybe the lock mechanism between the threads is not working so well.
Anyway, I've opened a ticket for the guys from Jenkins. I'll follow up on it and see if we get any information.
JENKINS-46483

Barak Korren August 22, 2017 at 1:59 PM

This is blocked on https://ovirt-jira.atlassian.net/browse/OVIRT-1532#icft=OVIRT-1532 where we failed to make CI post-merge jobs run so far. This may or may not be and instance of the issue.

This may also be related to https://ovirt-jira.atlassian.net/browse/OVIRT-1156#icft=OVIRT-1156 and the somewhat strange voting behaviour we have.

danken August 22, 2017 at 1:16 PM

Another example: https://gerrit.ovirt.org/#/c/80213/ broke master vdsm, but was not marked with check-merge -1.

Eyal Edri August 22, 2017 at 1:07 PM

What can we do to try and find the root cause for this?

Fixed

Details

Assignee

Reporter

Priority

Created July 3, 2017 at 11:35 AM
Updated October 1, 2017 at 10:57 AM
Resolved September 3, 2017 at 3:01 PM

Flag notifications