It may be just a mismatch on terminology indeed. What I mean is that tagging is IMHO a good enough system to achieve that. For an equivalent of -candidate tag I don't really need anyone else to review that, it's good enough for nightlies and non-release usage. Sure, you need to record what you have at the exact time and you need to be able to reproduce it, so set of hashes, NVR or whatever make sense, but only as a tracking record.
And for oVirt compose, as long as project is generally accepted as "part of oVirt" I think it should be sole responsibility of the sub-project maintainer to include/tag whatever he/she sees fit.
I do not think gerrit is a good fit, I think Fedora tooling has all this sorted out in a better way.
But it's not a big deal, the "gerrit" way just looks a bit cumbersome, but it might be the most easy thing to do right now.

not sure I understand "3". Yes, when you do a compose next day you have different "latest" packages. That's typically what you want.

No its not. You don't want the outcome of the compose process to be dependant on the time in which you run it, just on the sources of the packages that were included in it (As represented by a set of git hashes). Otherwise you cannot do any effective system testing or CI because the compose the CI system would make and test on day X may end up being different from the compose you release on day Y.

I do not see a point of "review and merge it" though. Why?

A few reasons:

1. We want a git hash that represents the point in which a new package version was introduced to oVirt from the reasons stated above

2. We want the action of introducing a change to oVirt to be explicit, so it is clear who is responsible for it

3. Doing it through a review system would provide a natural place where to provide feedback about a certain change causing issues

4. Doing it through a system like Gerrit would allow for easy rolling back of changes.

5. If some cases peer feedback may be useful - for example the integration team might prefer not to have changes introduced near new release.

We might me meaning different things when we say "review and merge". I just mean that it need to end up in Gerrit (or possibly Github), and someone needs to go and explicitly click "submit". I do not mean that one has to wait for peers to come by and vote it. You will be the owner of the "link" repo, so you get to set the rules about what a review means there.

BTW it might be even possible to fully automate step 3 when we reach the point where the CI system can take unmerged patches (or PRs), run system tests on them (essentially simulate the merge) and merge if successful. This is what we refer to as "Patch Gating" in other places. I`m afraid this is still ways off.

not sure I understand "3". Yes, when you do a compose next day you have different "latest" packages. That's typically what you want.
That new feature you mention seems ok, I do not see a point of "review and merge it" though. Why?

Ok. Here is how we get this with the existing tooling, and with very little work:

1. We create a new repo that will serve as the "link" between oVirt and mom

2. We place an 'automation/build-artifacts.sh' file in the new repo that will 'wget' or 'curl' a specific build of mom and place it in 'exported-artifacts' (You can even use 'repoman' to do it and get the same file format that you've seen in 'releng-tools')

3. When a new mom is released, a patch to the repo with the url of the new build needs to be submitted.

4. The "link" repo can be branched like oVirt versions to indicate which version of mom goes into which oVirt release (You can choose any branch names you'd like as the mapping to oVirt versions happens in the project yaml file, you can also have the same branch go into multiple oVirt versions).

WRT automating step 3 - we want builds to be reproducible. e.g. given a specific set of git hashes you`re supposed to get an equivalent release no matter when you build it. If we just take the latest mom every time we compose, we will just end up with a different composition depending on when we do it. This is why we need the "link" repo - to point to the exact mom build that goes into a certain oVirt version.

We are working on a new feature that may allow semi-automating step 3 in the future - It will allow the upstream repo to be queried periodically, and when a new release is found, a patch to the "link" repo will be created. The maintainer would still need to review and merge it.

WRT ovirt-web-ui I don't see how this kind of process makes any sense for such a core component. Quite frankly I'm having a hard time understanding why prefer this to "I submit a patch and the CI system does the rest".

+1 for ovirt-web-ui, we're going to use the same flow

Exactly what I would like to achieve. With one improvement in step 3. I usually release in the same copr repo so in the ideal world I would just provide the repo url once and every time you do a compose (or nightly) the tooling would check for updates.

The difference is that you depend on gerrit and on specific branch name to know which build-artifact script to execute. My requests does not require that, I will provide the git url and tarball url (or a dir with those) instead.

It does not have to be in Gerrit, nor is there a need for specific branches ATM. The discussion about branches was trying to come up with a scheme to auto-detect things that are currently specified in the project YAML in the Jenkins repo. Since we did not implement that yet, you can use any branch names you'd like and map them to oVirt releases in the YAML file.

The one thing we do strongly depend on, is that every change starts with a commit somewhere so that we can track it, and use the hash as a reliable unique identifier for the change. I think this is important. Without it you cannot track changes.

Something very similar to what I want is the releng tools repository Sandro uses. An example: https://gerrit.ovirt.org/gitweb?p=releng-tools.git;a=blob;f=releases/ovirt-4.1.1.conf;h=f20c23f5f7d88225e2e729ebab552e5d69544c41;hb=0988bf1bd9d5f91f35733031571f1e8407e28647

The only issue we have with this is that it is not properly documented (inheritance rules, ...), does not support source tarballs and CI is probably not using it at all.

That is a repoman configuration file. All it does is collect files together. This is not a reliable source of information, not is it located in the right place, because AFAIK deciding which package builds go to which oVirt releases is the responsibility of the individual package maintainers, not the integration team.

But still, I'm not sure I understand where you are trying to go. Let me try to describe the process you are tying to get to as I understand it:

1. You merge a new commit to mom

2. You go and launch a build in copr manually.

3. When the build is done you go to some file in some repo, specify the url to the build in it and submit a new commit.

4. Once the new commit is merged the CY system downloads the new build and publishes it in the oVirt repos.

The difference is that you depend on gerrit and on specific branch name to know which build-artifact script to execute. My requests does not require that, I will provide the git url and tarball url (or a dir with those) instead.

Something very similar to what I want is the releng tools repository Sandro uses. An example: https://gerrit.ovirt.org/gitweb?p=releng-tools.git;a=blob;f=releases/ovirt-4.1.1.conf;h=f20c23f5f7d88225e2e729ebab552e5d69544c41;hb=0988bf1bd9d5f91f35733031571f1e8407e28647

The only issue we have with this is that it is not properly documented (inheritance rules, ...), does not support source tarballs and CI is probably not using it at all.

I'm not sure I understand the process, and how it differs from what we have now.

A "build-artifacts" job can build in any way the developer chooses, including invoking remote build services. As long as it eventually leaves RPMs in exported-artifacts, they will be taken, system-tested and published. Every 'build-artifacts' job is also a package repository so you even get publishing "for free" without passing OST.

If you want to separate the build/release process from source that is also easy to do, you just create a separate repo, place build scripts in it, and make them clone the "source" repo. This is essentially what the UX team did for various 3rd party dependencies.

We will probably not further automate the use of specific build systems, nor is it probable that we will deploy any new build system.

WRT GitHub - using it does not imply using a different process, we've been running a very oVirt-like process on GitHub for Lago for quite a long while now. And we've recently started doing this for other projects as well.

Do not make this more complex than it has to be. A simple text or ini file (in ovirt specific repository, not in the project itself!) where the maintainer can say which build (a link to jenkins, copr, koji, github, ...) should go to which release is enough. The same for source tarball. Just let us provide a link (git hash url, gitweb export or github for example).

All your tooling like publishers or CI or anything else can then consume this configuration without forcing us to use a fixed tag or branch structure. The maintainer should be in control of the releases or platform, not the automation.

oVirt is a distribution of pieces and some pieces have slightly different development model. Even some core oVirt project use GitHub as the primary source now.

This has been discussed couple of times already, for example here http://lists.ovirt.org/pipermail/devel/2017-February/029439.html and here http://lists.ovirt.org/pipermail/devel/2017-January/029161.html