The network suite is executing nightly for almost a year. It has a caring team that tends to it, and it does not have false positives.
Currently, it currently fails for a week on the 4.2 branch, but it is due to a production code bug.
I would like to see this suite escalated to the importance of the basic suite, making it a gating condition for marking a collection of packages as "tested".
, what should be done to make this happen?
On Mon, Nov 5, 2018 at 8:59 AM danken (oVirt JIRA) <
I'm ok with this only if it will gate only vdsm-network package, and it
will run in parallel, so
it does not slow down OST run.
For basic suite, we have too many random failures, so I don't think it is
useful, and we should
make it optional.
Nir, in change-queue context, "gating" has a clear meaning: if you do not pass the suite, your package does not get to "tested". There is no meaning to "gate only vdsm-network".
Currently, only basic-suite-* is gating. this means that someone can post an engine patch which breaks the network suite inadvertently. That's what I would like to prevent.
You are right that this can be costly. We need to measure what is the time and resources to enable this gate, vs. our gain. Currently we do not know the price.
The network suite is currently shorter in tests and more stable the then the basic suite.
its basic and upgrade suites that run.
the basic suite covers multiple projects and should continue running.
I think that we should not add any specific components suites for 3 reasons:
1. Basic suites tests basic tests on all components, testing your specific area would not give the coverage we would like as sometimes the code change would effect multiple components and we are likely to miss serious regressions.
2. if we start adding more and more suites to CQ we would end up testing like QE and we are not providing those sort of tests
3. if we add the network suite we will have to add storage, virt, sla atc and we do not have resources to run every suite on every change.
I think that the basic and upgrade suite provide the coverage we need for CQ and other suites are running in check-patch.
If you believe that the basic suite is not stable or lacking, please contribute to stabilizing and adding to it.
we can consider this once we enable zuul/patch gating.
will we be able to enable network suite for gating network patches?
Not for specific patches - but we can easily enable it for ALL patches.
If we only want it for specific patches - we can consider allowing some customization of the Zull configuration at project level to allow running specific suits for specific patches, but this will require some code changes in several places. We can plan this once we're in production with the current set of suits.