[standard-ci] build-artifacts should reuse the artifacts built on check-merged if any

Description

Right now most projects just rebuild twice the artifacts, what for some means a big waste of resources, if we could just reuse what was built previously there would be no rebuilding.

Something to take into account is that the distributions that check-merged and build-artifacts run for might not match, so you can be running check-merged only on fc23 but building artifacts for fc23, fc22 an el7 (as an example), maybe we should not allow different distros on each stage?|

Activity

Show:

Eyal Edri November 29, 2016 at 11:54 AM

we moved to build minimal profile,
if we'll want any more optimizations, we can either open a new ticket or just send a patch to engine

Eyal Edri November 6, 2016 at 3:28 PM

Any patches sent on this?
Also, wanted to mention #OVIRT-751 is handling similar purpose with enabling better maven caching, so it might suffice as a starting point in optimizing ovirt-engine runtime for builds.

Vojtech Szocs September 19, 2016 at 3:09 PM

> So the question is, can we reduce this to 1-2 permutations? if this is just going to nightly / hourly repo then we don't need all these permutations and we'll enjoy a much faster verification on the experimental flow which collects artifacts from build-artifacts jobs.

+1

Full GWT build (all browsers & languages) is not needed for oVirt nightly (snapshot) builds. It's only needed for oVirt official (release) builds.

> We are now adding the 'build official manual' option to standard CI, so we can keep the 27 permutations there for official builds.

Sounds good to me.

Eyal Edri August 14, 2016 at 1:07 PM

OK,
looking at the code it seems check-merged is running a draft build:

  1. create the rpms
    rpmbuild \
    -D "_rpmdir $PWD/output" \
    -D "_topmdir $PWD/rpmbuild" \
    -D "release_suffix ${SUFFIX}" \
    -D "ovirt_build_ut $BUILD_UT" \
    -D "ovirt_build_extra_flags $EXTRA_BUILD_FLAGS" \
    -D "ovirt_build_draft 1" \
    --rebuild output/*.src.rpm

while build artifacts is running full blown matrix:

  1. create the rpms
    rpmbuild \
    -D "_rpmdir $PWD/output" \
    -D "_topmdir $PWD/rpmbuild" \
    -D "release_suffix ${SUFFIX}" \
    -D "ovirt_build_ut $BUILD_UT" \
    -D "ovirt_build_extra_flags $EXTRA_BUILD_FLAGS" \
    --rebuild output/*.src.rpm

and currently running 27 permutations.

So the question is, can we reduce this to 1-2 permutations? if this is just going to nightly / hourly repo then we don't need all these permutations and we'll enjoy a much faster verification on the experimental flow which collects artifacts from build-artifacts jobs.

We are now adding the 'build official manual' option to standard CI, so we can keep the 27 permutations there for official builds.

I believe this will reduce dramatically the CI flow runtime and will give us much faster feedback on failures.

Also, it will simplify the move to running experimental flow per patch as the full run-time will be significantly lower.

Eyal Edri July 28, 2016 at 7:14 AM

do you know if today check-merged.sh in ovirt-engine is building the same RPMs as build-artifacts?

Potentially they can be different since its a different bash script, but if they are the same we can save a lot of time running duplicate rpm builds...

Now that we enabled GWT compilation per patch, maybe we can drop the rpm build on check-merged.sh and just do it in build-artifacts?

we'll find an error in either jobs anyway.. what do you think?

Fixed

Details

Assignee

Reporter

Priority

Created February 15, 2016 at 12:35 PM
Updated November 29, 2016 at 12:18 PM
Resolved November 29, 2016 at 11:54 AM