Verify Gerrit backup works

Description

We have regular gerrit backups in place. Need to check that data included in these backups is enough to restore productivity. This is very important for DR purposes and also useful to test planned upgrades on real data.

Activity

Show:
Evgheni Dereveanchin
March 17, 2017, 4:14 PM

Checked the options for backup and the best is to use duplicity as it is more effective thanks to incremental capabilities. We already use duplicity for Jenkins.

The current backup consists of two scripts and two cron jobs which run 20 minutes apart: one backs up files in /home and the other one dumps the DB. This is not the best solution as data may become inconsistent within these twenty minutes.

Duplicity supports hooks but can only back up one source path at a time. Possible solutions:
1) use / as the source path, write an exclude file to not back up unneeded files, create a pre-backup hook to dump the DB
2) create a wrapper script that will dump the DB, then run three duplicity backups: /home/gerrit, /usr/local/src/, /path/to/pg_dump

Evgheni Dereveanchin
March 17, 2017, 5:14 PM

Ran a manual full duplicity backup of the directory with repos, stats below:

--------------[ Backup Statistics ]--------------
StartTime 1489768830.90 (Fri Mar 17 12:40:30 2017)
EndTime 1489770659.39 (Fri Mar 17 13:10:59 2017)
ElapsedTime 1828.49 (30 minutes 28.49 seconds)
SourceFiles 304567
SourceFileSize 2236025323 (2.08 GB)
NewFiles 304567
NewFileSize 2236025323 (2.08 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 304567
RawDeltaSize 1746894732 (1.63 GB)
TotalDestinationSizeChange 1412449586 (1.32 GB)
Errors 0
-------------------------------------------------

Will try to restore this data plus existing backups to confirm if it is enough to recover the service.

Evgheni Dereveanchin
March 21, 2017, 7:43 PM

Ran another backup of git repos to verify the diff size:

--------------[ Backup Statistics ]--------------
StartTime 1490123948.77 (Tue Mar 21 15:19:08 2017)
EndTime 1490124759.61 (Tue Mar 21 15:32:39 2017)
ElapsedTime 810.84 (13 minutes 30.84 seconds)
SourceFiles 144570
SourceFileSize 2039243069 (1.90 GB)
NewFiles 85090
NewFileSize 1250104062 (1.16 GB)
DeletedFiles 173522
ChangedFiles 807
ChangedFileSize 156824701 (150 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 259419
RawDeltaSize 833024808 (794 MB)
TotalDestinationSizeChange 784171944 (748 MB)
Errors 0
-------------------------------------------------

It's a 750 MB delta after several days which is around 30% of the total backup. We can do full backups bi-weekly or monthly and have the backup chain of around 7GB per month for the git part.

Eyal Edri
May 1, 2017, 8:43 AM

can you update on status?
Do we have automated backups running already?

Evgheni Dereveanchin
September 13, 2019, 9:07 AM

I can confirm we have full backups running after migrating Gerrit to EL7 so closing this request

Done

Assignee

Evgheni Dereveanchin

Reporter

Evgheni Dereveanchin

Blocked By

None

Components

Priority

Highest
Configure