Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Improve CI logging and reporting to Gerrit

Description

On Fri, Aug 12, 2016 at 8:08 PM, Nir Soffer <nsoffer@redhat.com> wrote:

> Hi all,
>
> Lately vdsm builds are failing in the install packages stage with
> this unhelpful error:
>
> 13:59:42 INFO: installing package(s): autoconf automake gdb ...
> 13:59:53 ERROR: Command failed. See logs for output.
>
> I downloaded the logs.tgz file from
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> x86_64/1154/artifact/exported-artifacts/logs.tgz
>
> And indeed in /./vdsm/logs/mocker-fedora-24-x86_64.fc24.install_packages/
> root.log
> I found found this error:
>
> DEBUG util.py:421: Error: Package:
> python-ioprocess-0.17.0-1.201608111609.gitbd272f2.fc24.noarch
> (ovirt-snapshot)
> DEBUG util.py:421: Requires: ioprocess =
> 0.17.0-1.201608111609.gitbd272f2.fc24
> DEBUG util.py:421: Available:
> ioprocess-0.15.1-1.fc24.x86_64 (fedora)
> DEBUG util.py:421: ioprocess = 0.15.1-1.fc24
> DEBUG util.py:421: Available:
> ioprocess-0.16.1-1.fc24.x86_64 (updates)
> DEBUG util.py:421: ioprocess = 0.16.1-1.fc24
> DEBUG util.py:421: Available:
> ioprocess-0.17.0-1.201607121058.gitbd272f2.fc24.x86_64
> (ovirt-snapshot)
> DEBUG util.py:421: ioprocess =
> 0.17.0-1.201607121058.gitbd272f2.fc24
> DEBUG util.py:421: Available:
> ioprocess-0.17.0-1.201608111129.gitbd272f2.fc24.x86_64
> (ovirt-snapshot)
> DEBUG util.py:421: ioprocess =
> 0.17.0-1.201608111129.gitbd272f2.fc24
>
> So we have 2 issues here:
>
> 1. We need all errors in the console
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> x86_64/1154/console

Not sure its possible or easy to do, these errors usually come from mock
and are not relevant to the code running on the console.
I've cc'd infra-support and updated topic to track it and check our options.
Going foward, we might want to install a log collector like logstash and
then searching errors there should be much easier.

>
>
> 2. Someone need to fix the ovirt-snapshot repository - it should have
> the missing ioprocess
> package.
> Maybe the project xml is not correct?
>
> Nir
>

–
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)

Activity

Show:

Barak Korren June 10, 2018 at 3:24 PM

This ticket is overly broad and generic, closing it as we now have more specific tickets for specific aspects of this.

Barak Korren July 12, 2017 at 1:39 PM

Gerrit trigger feedback configuration is not flexible enough to let us have each job customize the output it sends to Gerrit.

Instead we will need to send the results to Gerrit ourselves with something like the GerritServer object (See Gerrit 74874). But when sending results, the Gerrit trigger also calculates the votes for the flags by looking at all the jobs that it invoked. To do this ourselves we will need to be able to track all the checking scripts that are running in parallel for a given patchset, this probably means we have to implement https://ovirt-jira.atlassian.net/browse/OVIRT-1522#icft=OVIRT-1522 first.

Barak Korren February 27, 2017 at 4:05 PM

One more thing we can do if we implement extended reporting to Gerrit as I suggested above, is let the CI system issue warnings, like for example if the job used external repos instead of mirrors.

Barak Korren February 26, 2017 at 9:55 AM
Edited

We can probably improve the user experience by putting a better message in Gerrit, something like:

You can see standard-CI script output log here(1)

For individual test results please click here(2), or use the test results analyser(3)

Additional built artifacts can be found here(4)

For full, unfiltered, job output click here(5)

To re-trigger the job click here(6) or type @ci check

Where:

  • (1) points to the check-patch.sh, {check-merged.sh}}, etc. script output.

  • If any JUNIT XML files were found (If not the line shouldnt be shown):

    • (2) points to the test results page

    • (3) point to the test results analyser

  • If any artifacts that are not mock logs or JUNIT XML files wee found (4) points to them. If <= 3 artifacts found we may even include direct links (This will be useful for things like differences.html in the Jenkins repo.

  • (5) Will point to the good-old job console output screen

  • If the job failed, (6) will point to the re-trigger button

We will need to check if its possible to embed links in Gerrit comments. If no we will unfortunately have to use ugly full URLs.

Eyal Edri January 12, 2017 at 3:46 PM

some improvements were added recently to mock runner, but there is more we can do to simplify the output of jobs to ease debugging.
keeping this as place holder for any improvement we'll add or consider

Won't Fix

Details

Assignee

Reporter

Components

Priority

Parent

Created August 14, 2016 at 6:16 AM
Updated September 2, 2018 at 3:51 PM
Resolved June 10, 2018 at 3:24 PM

Flag notifications