upgrade Gerrit to 2.16

Description

Gerrit 2.14 is no longer receiving updates and we’ve got some plugin bugs reported.

The minimum currently supported is 2.16 and we can jump to it through 2.15 using the standard upgrade process. The only culprit is the removal of drafts which are going to be transformed into Private changes (WIP is the default but I think Provate is a better fit for existing drafts)

Activity

Show:
Evgheni Dereveanchin
August 18, 2020, 4:07 PM

Very similar use case, worth trying the fix-legacy-user-id option with the new plugin:

Evgheni Dereveanchin
September 14, 2020, 1:37 PM
Edited

Using the option together with the latest version of gerrit-oauth-provider seems to work just fine for me, I can log in using both Google and GitHub.

https://github.com/davido/gerrit-oauth-provider/releases/download/v2.16.1/gerrit-oauth-provider.jar

 

This means that the ticket is unblocked and we can bump prod to 2.16 after testing the migration one last time on a disconnected system to ensure the NotesDB migration works without issues.

Evgheni Dereveanchin
November 13, 2020, 7:19 PM
Edited

tested the upgrade on the production dump. New versions released since last upgrade so here are the new steps:

 

cd gerrit_old_versions/
wget https://gerrit-releases.storage.googleapis.com/gerrit-2.15.19.war
wget https://gerrit-releases.storage.googleapis.com/gerrit-2.16.23.war
systemctl stop gerrit

#yum install java-1.8.0-openjdk-headless

java -jar /home/gerrit2/gerrit_old_versions/gerrit-2.15.19.war init -d /home/gerrit2/review_site/
java -jar /home/gerrit2/review_site/bin/gerrit.war reindex -d /home/gerrit2/review_site/

java -jar /home/gerrit2/gerrit_old_versions/gerrit-2.16.23.war init -d /home/gerrit2/review_site/
java -jar /home/gerrit2/review_site/bin/gerrit.war reindex -d /home/gerrit2/review_site/
cd /home/gerrit2/review_site/plugins/

mv reviewers.jar ../../gerrit_old_plugins/reviewers_214.jar
mv avatars-gravatar.jar ../../gerrit_old_plugins/avatars-gravatar_214.jar
#mv events-log.jar ../../gerrit_old_plugins/events-log_214.jar
#mv webhooks.jar ../../gerrit_old_plugins/webhooks_214.jar

mv gerrit-oauth-provider.jar ../../gerrit_old_plugins/gerrit-oauth-provider_214.jar

wget https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.16/job/plugin-reviewers-bazel-stable-2.16/lastSuccessfulBuild/artifact/bazel-bin/plugins/reviewers/reviewers.jar

wget https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.16/job/plugin-avatars-gravatar-bazel-stable-2.16/lastSuccessfulBuild/artifact/bazel-bin/plugins/avatars-gravatar/avatars-gravatar.jar
#wget https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.16/job/plugin-events-log-bazel-stable-2.16/lastSuccessfulBuild/artifact/bazel-bin/plugins/events-log/events-log.jar
#wget https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.16/job/plugin-webhooks-bazel-stable-2.16/lastSuccessfulBuild/artifact/bazel-bin/plugins/webhooks/webhooks.jar

wget https://github.com/davido/gerrit-oauth-provider/releases/download/v2.16.1/gerrit-oauth-provider.jar

add fix-legacy-user-id = true option to /home/gerrit2/review_site/etc/gerrit.config

then start the service


systemctl start gerrit

the test system ran out of memory during re-index, so will re-run the upgrade and post a new comment.

 

A thing to note: I moved drafts to WIP changes, there’s also an option to convert them to private ones instead but IMHO the default is fine.

Evgheni Dereveanchin
November 13, 2020, 9:16 PM

The upgrade seems to progress fine, there’s only a couple of errors during the 2.15 reindex:

Reindexing changes: projects: 27% (36/129), 87% (95327/108849) (-)[2020-11-13 16:01:40,792] [Index-Batch-2] ERROR com.google.gerrit.server.project.SubmitRuleEvaluator : Error looking up change 43881
com.google.gwtorm.server.OrmException: No patch set found
Reindexing changes: 5% (5591/108849) (\)[2020-11-13 15:48:33,415] [Index-Batch-2] ERROR com.google.gerrit.server.project.SubmitRuleEvaluator : Error looking up change 51604
com.google.gwtorm.server.OrmException: No patch set found
Reindexing changes: 9% (10379/108849) (/)[2020-11-13 15:49:20,523] [Index-Batch-2] ERROR com.google.gerrit.server.project.SubmitRuleEvaluator : Error looking up change 50408
com.google.gwtorm.server.OrmException: No patch set found
Reindexing changes: projects: 10% (14/129), 69% (75252/108849) (\)[2020-11-13 15:58:48,678] [Index-Batch-1] ERROR com.google.gerrit.server.project.SubmitRuleEvaluator : Error looking up change 73909
com.google.gwtorm.server.OrmException: No patch set found
Reindexing changes: projects: 10% (14/129), 69% (75743/108849) (-)[2020-11-13 15:58:51,714] [Index-Batch-2] ERROR com.google.gerrit.server.project.SubmitRuleEvaluator : Error looking up change 67563
com.google.gwtorm.server.OrmException: No patch set found
Reindexing changes: projects: 23% (30/129), 85% (92825/108849) (/)[2020-11-13 16:01:20,335] [Index-Batch-2] ERROR com.google.gerrit.server.project.SubmitRuleEvaluator : Error looking up change 40579
com.google.gwtorm.server.OrmException: No patch set found

Given the fact it’s just 6 patches out of 100k it’s probably fine and we can proceed. I also saw no errors during the 2.16 reindex so what ever caused these inconsistencies has been cleaned up.

 

After upgrading plugins I started the Gerrit process and it ran just fine. I cut off network to ensure we don’t interfere with GitHub replication and could not test authentication but we can for sure proceed with the upgrade and take a snapshot of PROD beforehand to roll back if something unexpected happens during real-world operation.

Evgheni Dereveanchin
November 25, 2020, 8:58 AM

Upgrade applied. Had to also add the auth fix to GitHub, otherwise it wend according to plan.

Fixed

Assignee

Evgheni Dereveanchin

Reporter

Evgheni Dereveanchin

Blocked By

None

Priority

Medium
Configure