Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

dnf install in jobs setup scripts fails due to corrupted rpmdb

Description

10:24:59 + sudo dnf -y install python3-PyYAML PyYAML 10:24:59 error: rpmdb: BDB0113 Thread/process 27225/140313046546176 failed: BDB1507 Thread died in Berkeley DB library 10:24:59 error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery 10:24:59 error: cannot open Packages index using db5 - (-30973) 10:24:59 error: cannot open Packages database in /var/lib/rpm 10:24:59 Error: Error: rpmdb open failed 10:24:59 + failed=true

http://jenkins.ovirt.org/job/lago_master_github_build-artifacts-el7-x86_64/91/console

relates to

Activity

Show:

Eyal Edri July 24, 2017 at 7:09 AM

so the question is how much effort do we estimate needed to implement #OVIRT-1468, if it might take time, we should consider a temp workaround if possible, via rebuilddb during setup stage?

Barak Korren July 24, 2017 at 6:59 AM

I suspect that the RPMDB gets broken b/c of broken data from repos, as we've seen these RPMDB issues start after the Fedora repos broke down.

Eyal Edri July 24, 2017 at 6:51 AM

can you elaborate how implementing mirrors for slaves will help us avoid rpmdb issues on the slaves?

Barak Korren July 24, 2017 at 6:46 AM

This might get fixed by OVIRT-1468, so lets implement it first and then see what happens.

Barak Korren June 26, 2017 at 11:55 AM

, FYI - The following build failed because of this:
http://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_master_build-artifacts-el7-x86_64/153/

I re-triggered it after fixing node and it was successful:
http://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_master_build-artifacts-el7-x86_64/154/

I then re-triggered the deploy-to-experimental job for that merge event to make sure the package makes it into the 'experimental' repo:
http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental/2817/

Barak Korren June 26, 2017 at 11:34 AM

Saw (and fixed) this twice today:

  1. vm0156.workers-phx.ovirt.org (FC25)

  2. vm0149.workers-phx.ovirt.org (FC24)

vm0157.workers-phx.ovirt.org is FC25.

Maybe this has something to do with the repo failure last week.

, , Please note any occurrences of this you come across. We're trying to gauge how big of an issue is this, and if we need to add safeguards to the code.

Nadav Goldin June 26, 2017 at 11:12 AM

Fixed manually on vm0157 just now by executing: rpm --rebuilddb

Fixed

Details

Assignee

Reporter

Components

Priority

Parent

Created June 26, 2017 at 11:08 AM
Updated December 28, 2017 at 9:30 AM
Resolved December 26, 2017 at 12:34 PM

Flag notifications