its basic and upgrade suites that run.

the basic suite covers multiple projects and should continue running.
I think that we should not add any specific components suites for 3 reasons:
1. Basic suites tests basic tests on all components, testing your specific area would not give the coverage we would like as sometimes the code change would effect multiple components and we are likely to miss serious regressions.
2. if we start adding more and more suites to CQ we would end up testing like QE and we are not providing those sort of tests
3. if we add the network suite we will have to add storage, virt, sla atc and we do not have resources to run every suite on every change.

I think that the basic and upgrade suite provide the coverage we need for CQ and other suites are running in check-patch.
If you believe that the basic suite is not stable or lacking, please contribute to stabilizing and adding to it.

Nir, in change-queue context, "gating" has a clear meaning: if you do not pass the suite, your package does not get to "tested". There is no meaning to "gate only vdsm-network".

Currently, only basic-suite-* is gating. this means that someone can post an engine patch which breaks the network suite inadvertently. That's what I would like to prevent.

You are right that this can be costly. We need to measure what is the time and resources to enable this gate, vs. our gain. Currently we do not know the price.

The network suite is currently shorter in tests and more stable the then the basic suite.

On Mon, Nov 5, 2018 at 8:59 AM danken (oVirt JIRA) <

I'm ok with this only if it will gate only vdsm-network package, and it
will run in parallel, so
it does not slow down OST run.

For basic suite, we have too many random failures, so I don't think it is
useful, and we should
make it optional.

Nir

We still need to allocate bare metal to run it on each CQ run, and I'm not
sure we have available resources for it, pretty sure we don't have since
KubeVirt took half of the hosts.

On Mon, Mar 4, 2019 at 2:09 PM danken (oVirt JIRA) <

–

Eyal edri

MANAGER

RHV/CNV DevOps

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)

Is there news here? Would you try adding it?

A benefit for sequential execution, is that you don't need to waste resources on the network suite if the basic suite has finished successfully.

To save time I prefer the network suite to run in parallel to the basic suite.

Its not a matter of preference - there is no other option.

WDYT? will this prolong our CQ iterations?

Also how are we with capacity? Do we have enough? would this be a good candidate to run in a container?

cc:

https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/buildTimeTrend shows that it runs for 44-48 minutes.

https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/buildTimeTrend runs slightly quicker (41-45 minutes)

To save time I prefer the network suite to run in parallel to the basic suite.

how long does it take the suit to run?