centos-updates-el7 mirror sync fails

Description

CentOS 7.8 is out yet our mirror is failing to sync the Updates repo which can cause problems and inconsistencies.

Seems to be failing on sync'ing ppp-devel-2.4.5-34.el7_7.i686.rpm:
https://jenkins.ovirt.org/view/System%20and%20Monitoring/job/system-sync_mirrors-centos-updates-el7-x86_64/3745/console#L272

Activity

Show:
Evgheni Dereveanchin
April 30, 2020, 9:07 AM

Looking closer at this, this RPM is from 7.7 and we already have it in our mirror:

[root@mirrors Packages]# pwd
/var/www/html/repos/yum/centos-updates-el7/base/Packages
[root@mirrors Packages]# ls -l | grep ppp
rw-rw-r-. 1 jenkins jenkins 172696 Feb 5 20:15 NetworkManager-ppp-1.18.0-5.el7_7.2.x86_64.rpm
rw-rw-r-. 1 jenkins jenkins 366284 Feb 27 19:33 ppp-2.4.5-34.el7_7.x86_64.rpm
rw-rw-r-. 1 jenkins jenkins 51960 Feb 27 19:33 ppp-devel-2.4.5-34.el7_7.i686.rpm
rw-rw-r-. 1 jenkins jenkins 51908 Feb 27 19:33 ppp-devel-2.4.5-34.el7_7.x86_64.rpm

However, the checksum is different as compared to what's upstream:

$ wget http://ewr.edge.kernel.org/centos/7.8.2003/updates/x86_64/Packages/ppp-devel-2.4.5-34.el7_7.i686.rpm
$ md5sum ppp-devel-2.4.5-34.el7_7.i686.rpm
9eec903a90e6938d818d6c2f4ce487f4 ppp-devel-2.4.5-34.el7_7.i686.rpm

[root@mirrors Packages]# md5sum ppp-devel-2.4.5-34.el7_7.i686.rpm
c20a529a948316af805557c746acd7a4 ppp-devel-2.4.5-34.el7_7.i686.rpm

Was the package rebuilt upstream without changing its version? Will archive the file from the mirror and attempt another resync.

Barak Korren
April 30, 2020, 9:09 AM

try deleting the particular file that is causing issues from the mirror - this is a typical issue with the yum downloading libraries (used by both `yum` and `reposync`) - if a file gets corrupted or truncated during download, it does not know how to re-download it without manual intervention.

In the Lago code we have some logic that is capturs the particular error message that this kind of failure raises and automatically deletes the relevant files before rerunning reposync. I think we also have a ticket somewhere to add this kind of login to the mirrors sync code...

Evgheni Dereveanchin
April 30, 2020, 9:39 AM

Indeed, getting the file out of the way fixed the sync. Comparing the two files they had the same size but a diff around the start of the file - probably the file got re-signed or something like that.

cmp ppp-devel-2.4.5-34.el7_7.i686.rpm_old ppp-devel-2.4.5-34.el7_7.i686.rpm_new

ppp-devel-2.4.5-34.el7_7.i686.rpm_old ppp-devel-2.4.5-34.el7_7.i686.rpm_new differ: byte 232, line 1

Querying both RPMs with rpm -qpi reveals that indeed they're the same RPM but signed on different times:

Size : 126469
License : BSD and LGPLv2+ and GPLv2+ and Public Domain
Signature : RSA/SHA256, Thu 27 Feb 2020 07:33:18 PM UTC, Key ID 24c6a8a7f4a80eb5
Source RPM : ppp-2.4.5-34.el7_7.src.rpm
Build Date : Thu 27 Feb 2020 04:48:18 PM UTC


Size : 126469
License : BSD and LGPLv2+ and GPLv2+ and Public Domain
Signature : RSA/SHA256, Thu 16 Apr 2020 01:57:32 PM UTC, Key ID 24c6a8a7f4a80eb5
Source RPM : ppp-2.4.5-34.el7_7.src.rpm
Build Date : Thu 27 Feb 2020 04:48:18 PM UTC


Fixed

Assignee

Evgheni Dereveanchin

Reporter

Evgheni Dereveanchin

Blocked By

None

Priority

Medium
Configure