Solution implemented and verified:
http://jenkins.ovirt.org/job/ovirt-host-deploy_4.1_check-patch-fc24-x86_64/51

Closing this now.

On Thu, Apr 27, 2017 at 10:06 AM, Barak Korren <bkorren@redhat.com> wrote:
> On 27 April 2017 at 10:02, Yedidyah Bar David <didi@redhat.com> wrote:
>>>
>>> Anything else that's missing?
>>
>> I still think we should consider on-demand caching instead of mirroring.
>>
>> On my local (VMs on my laptop and minidells at home) I use [1], with a
>> variation on the squid conf there. Works more-or-less ok.
>>
>> [1] https://github.com/yevmel/squid-rpm-cache/blob/master/store_id_program/store_id_program.py
>>
>
> Been there, done that, switched to mirroring.
> Please read the following to understand why we need mirrors:
> http://ovirt-infra-docs.readthedocs.io/en/latest/CI/Transactional_mirrors/index.html

Wasn't aware of this document, thanks for writing and sharing it.

It doesn't seem like it was a simple job, and it seems to still require
some significant on-going maintenance. Did you consider doing something
like this (snapshots, special treatment for repo metadata, etc) with
on-demand proxy caching? Is there some inherent reason why it won't
work, or you simply concluded it will require more work and not be
worth it?

I do not know squid very well, no idea if it's powerful enough to
implement something like this on it (or whatever other proxy cache
tool you want). So going this way might require implementing one
from scratch (probably using existing libraries to do much of the
low-level work). It would still be interesting to try, and imo much
more useful - if done well, can be used more-or-less unchanged,
with the default configuration, for many different uses/projects.

Also, it's not clear to me how (if at all) you handle mirroring
where the source changed in the middle of syncing from it.

>
> –
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

–
Didi

On 27 April 2017 at 10:02, Yedidyah Bar David <didi@redhat.com> wrote:
>>
>> Anything else that's missing?
>
> I still think we should consider on-demand caching instead of mirroring.
>
> On my local (VMs on my laptop and minidells at home) I use [1], with a
> variation on the squid conf there. Works more-or-less ok.
>
> [1] https://github.com/yevmel/squid-rpm-cache/blob/master/store_id_program/store_id_program.py
>

Been there, done that, switched to mirroring.
Please read the following to understand why we need mirrors:
http://ovirt-infra-docs.readthedocs.io/en/latest/CI/Transactional_mirrors/index.html

–
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

On Thu, Apr 27, 2017 at 9:41 AM, Barak Korren <bkorren@redhat.com> wrote:
>
>
> On 27 April 2017 at 09:28, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
>>
>>
>>
>> Opening a ticket, multiple rpm files are missing on the mirror.
>> Also:
>>
>> eclipse-filesystem-1.0-7.fc24.x86_64: failed to retrieve
>> Packages/e/eclipse-filesystem-1.0-7.fc24.x86_64.rpm from fedora-base-fc24
>> error was [Errno 14] HTTP Error 404 - Not Found
>
>
> Yeah, so I shouldnt have tried to remove eclipse I guess...
> I guess its a -v for this:
> https://gerrit.ovirt.org/#/c/75913/

Seems like that.

>
> Anything else that's missing?

I still think we should consider on-demand caching instead of mirroring.

On my local (VMs on my laptop and minidells at home) I use [1], with a
variation on the squid conf there. Works more-or-less ok.

[1] https://github.com/yevmel/squid-rpm-cache/blob/master/store_id_program/store_id_program.py

>
> –
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

–
Didi

Updated patch to restore eclipse on the mirrors, waiting for them to resync now.