centos-base-el8 repo mirror inconsistent

Description

A repo mirror for Centos8 Base was added today and it seems to be inconsistent. Disabled it for now and removed from the mirror index and loggin ticket to investigate the root cause and fix the repo.

Activity

Show:
Evgheni Dereveanchin
April 2, 2020, 4:08 PM

First I've seen reports of the repo missing multiple files:
https://jenkins.ovirt.org/job/oVirt_ovirt-cockpit-sso_standard-check-pr/3/console

Most of these files are now in place.
Later some repos were failing to fetch individual RPMs:
https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6729/console

This is confirmed on the mirror itself:
grep primary repodata/repomd.xml
<data type="primary">
<location xml:base="http://mirrors.phx.ovirt.org/repos/yum/centos-base-el8/base" href="repodata/4a6818ea5c413f2164bd434ed7dc44ebb363755dcd7cc8e9549bc2ecda4e907e-primary.xml.gz"/>
...

zgrep python3-audit repodata/4a6818ea5c413f2164bd434ed7dc44ebb363755dcd7cc8e9549bc2ecda4e907e-primary.xml.gz
<name>python3-audit</name>
<description>The python3-audit package contains the bindings so that libaudit
<location xml:base="http://mirrors.phx.ovirt.org/repos/yum/centos-base-el8/base" href="Packages/python3-audit-3.0-0.10.20180831git0047a6c.el8.i686.rpm"/>
<rpm:entry name="python3-audit" flags="EQ" epoch="0" ver="3.0" rel="0.10.20180831git0047a6c.el8"/>
<rpm:entry name="python3-audit(x86-32)" flags="EQ" epoch="0" ver="3.0" rel="0.10.20180831git0047a6c.el8"/>
<name>python3-audit</name>
<description>The python3-audit package contains the bindings so that libaudit
<location xml:base="http://mirrors.phx.ovirt.org/repos/yum/centos-base-el8/base" href="Packages/python3-audit-3.0-0.10.20180831git0047a6c.el8.x86_64.rpm"/>
<rpm:entry name="python3-audit" flags="EQ" epoch="0" ver="3.0" rel="0.10.20180831git0047a6c.el8"/>
<rpm:entry name="python3-audit(x86-64)" flags="EQ" epoch="0" ver="3.0" rel="0.10.20180831git0047a6c.el8"/>
<name>python3-audit</name>
<description>The python3-audit package contains the bindings so that libaudit
<location xml:base="http://mirrors.phx.ovirt.org/repos/yum/centos-base-el8/base" href="Packages/python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64.rpm"/>
<rpm:entry name="python3-audit" flags="EQ" epoch="0" ver="3.0" rel="0.13.20190507gitf58ec40.el8"/>
<rpm:entry name="python3-audit(x86-64)" flags="EQ" epoch="0" ver="3.0" rel="0.13.20190507gitf58ec40.el8"/>

...

ls -l Packages/ | grep python3-audit
rw-rw-r-. 1 jenkins jenkins 87216 Jul 2 2019 python3-audit-3.0-0.10.20180831git0047a6c.el8.i686.rpm
rw-rw-r-. 1 jenkins jenkins 87220 Jan 3 20:19 python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64.rpm

So there's 3 files in metadata and just 2 on disk with one dated 2019 (i.e. from the last mirror introduction attempt).

There is no snapshot from 2019 however:
$ ls -l
total 4
drwxr-xr-x. 3 jenkins root 22 Apr 2 13:05 2020-04-02-13-05
drwxr-xr-x. 3 jenkins root 22 Apr 2 14:56 2020-04-02-14-56
drwxr-xr-x. 4 jenkins root 154 Apr 2 14:55 base
rw-rw-r-. 1 jenkins jenkins 17 Apr 2 14:56 latest.txt

May be a combination of the fact that the mirror existed before and the bug of the cleanup code that we solved in the morning resulting in the deletion. could you please take a look at what could have deleted python3-audit-3.0-0.10.20180831git0047a6c.el8.x86_64.rpm?

Can we get the list of deleted facts exported to build artifacts for these kinds of troubleshooting?

Evgheni Dereveanchin
April 2, 2020, 4:10 PM

for now I’ll back up the sync cache and mirror contents, then re-create it from scratch. Other mirrors that have existed before may be affected so we need to find the root cause here

 

Evgheni Dereveanchin
April 2, 2020, 10:17 PM

Re-creating mirrors seems to have helped. Re-did EPEL and Extras too. Leaving this open to try and find the root cause. Existence of stale data from previous versions of the mirror seems like the prime suspect but still strange to have missing data listed in metadata even after re-running the sync.

Evgheni Dereveanchin
April 7, 2020, 2:20 PM

did you have a chance to look through the mirror snapshots I backed up to see what could cause file removal? If it’s impossible to define without we can probably close this one.

Evgheni Dereveanchin
April 14, 2020, 2:57 PM

Likely cause found and fixed by moving the cleanup from the middle of repo generation process with https://gerrit.ovirt.org/108375/

Fixed

Assignee

Evgheni Dereveanchin

Reporter

Evgheni Dereveanchin

Blocked By

None

Priority

High
Configure