Setup a cleanup job for the mirrors

Description

Motivation

Out transactional mirrors do not currently have a cleanup job. This means packages are kept forever, or at least until specific mirrors are removed.

This means we may run out of space.

Need to first verify this is actually needed in practice once and are implemented.

Acceptance criteria

  • Snapshots that are older than 14 days are removed.

  • Snapshots that are newer than 13 days are kept.

  • Packages that are referenced only by old (14 days and older) snapshots are removed.

  • Packages that are referenced by new (13 days and newer) snapshots are kept.

  • At least 1 snapshot and its referenced packages are kept, even if it's older than 14 days.

Activity

Show:
Daniel Belenky
October 3, 2019, 3:18 PM

Continued from our discussion offline, I've added acceptance criteria to the ticket. As for the implementation details, we probably will want to create two sets, one with old packages and one with new packages, then remove the new packages from the old packages set. It will give us a set of packages that we can safely remove.
While building the "new packages set", make sure to have at least 1 snapshot (even if it's older than 14 days).

Let me know if you have any questions.

Barak Korren
October 3, 2019, 3:37 PM

Looking at the acceptance criteria - it may need to change a little, there is no reason to keep any package that are not referenced by any snapshots, so we can just delete all packages that are not referenced by new snapshots, this may simplify implementation, for example, an implementation strategy could be:

  1. Delete all old snapshot metadata (older then 14 days)

  2. Delete all packages that are no longer referenced by any snapshot

I suggest this would be implemented as part of the mirror_mgr.sh script.

Evgheni Dereveanchin
December 16, 2019, 1:45 PM

As an update, we better ensure we don’t drop snapshots used by CI slaves to avoid issues like OVIRT-2689

Shlomi Zidmi
March 3, 2020, 10:28 AM

the patch that adds the cleanup functionality is waiting to be reviewed for few weeks now:

https://gerrit.ovirt.org/#/c/106118/

 

Shlomi Zidmi
April 16, 2020, 7:57 AM

Cleanup merged and now active for all repos

Assignee

Shlomi Zidmi

Reporter

Barak Korren

Blocked By

None

Components

Priority

High
Configure