CI failures due to oVirt proxy getting forbidden access to CentOS mirrors

Description

Hi,
we are experiencing build failures due to

08:49:10 # /usr/bin/yum --installroot
/var/lib/mock/epel-8-x86_64-6c17b26f3795335ce1ac45b0dcdbc9d5-bootstrap-8304/root/
--releasever 8 install dnf dnf-plugins-core distribution-gpg-keys
--setopt=tsflags=nocontexts

<https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86_64/708/console#L859>08:49:10
Failed to set locale, defaulting to C

<https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86_64/708/console#L860>08:49:10
http://mirror.centos.org/centos-8/8/AppStream/x86_64/os/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden

<https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86_64/708/console#L861>08:49:10
Trying other mirror.

...

08:49:10 failure: repodata/repomd.xml from centos-appstream-el8:
[Errno 256] No more mirrors to try.

<https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86_64/708/console#L901>08:49:10
http://mirror.centos.org/centos-8/8/AppStream/x86_64/os/repodata/repomd.xml:
[Errno 14] HTTP Error 403 - Forbidden

(see https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86_64/708/console)

I checked our squid logs on proxy01.phx.ovirt.org and I see:

1623395369.584 1 38.145.50.116 TCP_DENIED/403 4148 GET
http://mirror.centos.org/centos-8/8/AppStream/x86_64/os/repodata/repomd.xml

  • HIER_NONE/- text/html

Apparently there's no option in oVirt CI system to turn off the proxy
since mock_runner is always called with --try-proxy option.

Anyone that can look into this?

Sandro Bonazzola

MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

Red Hat EMEA <https://www.redhat.com/>

sbonazzo@redhat.com
<https://www.redhat.com/>

*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*

Activity

Show:

Former user June 14, 2021 at 9:31 AM

Seems that Puppet has reverted my changes, re-opened the ticket.

Fix sent for review, puppet stopped on the proxy for now to unblock CI

https://gerrit.ovirt.org/c/infra-puppet/+/115240

Former user June 11, 2021 at 9:09 AM
Edited

I see that connections are passing successfully now:

1623402416.131    755 66.187.230.121 TCP_MISS/200 3023 GET http://mirrors.ocf.berkeley.edu/centos/7/os/x86_64/ - HIER_DIRECT/169.229.226.30 text/html

Former user June 11, 2021 at 8:57 AM

I’ve added the 38.145.50.0/24 to squid’s ACL - could you please confirm if appliance builds are back to normal?

Former user June 11, 2021 at 8:53 AM

I think I know what’s wrong, some systems were moved to a new subnet in preparation of the PSI migration and it is likely not authorized on the proxy. Will fix and report here.

Fixed

Details

Assignee

Reporter

Components

Priority

Created June 11, 2021 at 7:42 AM
Updated June 14, 2021 at 9:31 AM
Resolved June 11, 2021 at 1:24 PM