Prod update complete to version 2.14.3

During the upgrade ECDSA keys were generated:
Generating SSH host key ... ed25519...unknown key type ed25519
ecdsa 256... ecdsa 384... ecdsa 521... done

Now however, parsing errors are shown throughout the logs even if keys look correct at first glance:

[2017-08-30 18:23:49,771] [sshd-SshServer[60b4c754]-nio2-thread-1] WARN org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/review_site/etc/ssh_host_ecdsa_key: Invalid DER: object is not an OID: SEQUENCE
[2017-08-30 18:23:49,772] [sshd-SshServer[60b4c754]-nio2-thread-1] WARN org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/review_site/etc/ssh_host_ecdsa_384_key: Invalid DER: object is not an OID: SEQUENCE
[2017-08-30 18:23:49,772] [sshd-SshServer[60b4c754]-nio2-thread-1] WARN org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/review_site/etc/ssh_host_ecdsa_521_key: Invalid DER: object is not an OID: SEQUENCE

So far I was only able to find a mailing list post without an answer. Cloning over SSH works fine regardless of these warnings.

Staging instance upgraded in a similar way as before during :

First - update the system and install JAVA8

yum update
yum install java-1.8.0-openjdk

This should change the alternative symlink, check that it points to java-1.8.0:

java -showversion
ls -l /etc/alternatives/java

Now all is set for the upgrade:

wget https://gerrit-releases.storage.googleapis.com/gerrit-2.14.2.war
java -jar gerrit-2.14.2.war init -d /home/gerrit2/review_site/

Select all defaults, except for BitBucket auth at the very end of the upgrade wizard.
After completion - upgrade the reviewers plugin manually and ensure the directory is owned by gerrit2:

wget https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.14/job/plugin-reviewers-bazel-stable-2.14/lastSuccessfulBuild/artifact/bazel-genfiles/plugins/reviewers/reviewers.jar
mv reviewers.jar /home/gerrit2/review_site/plugins
chown -R gerrit2 /home/gerrit2/review_site

Now reindex the database - this step may take up to one hour:

java -jar /home/gerrit2/review_site/bin/gerrit.war reindex -d /home/gerrit2/review_site/

All done, the service can be started now:

/etc/init.d/gerrit start

If there are any issues with service start or UI operation, check /home/gerrit2/review_site/logs/error_log

Upgrade to 2.13 completed successfully so we can test upgrading the staging instance to 2.14.

The important preparation step is to use Java 8 as 2.14 no longer works with Java 7. Otherwise there should be no major changes so we can stay on EL6 for now. Will report after I've conducted the tests.

We'll upgrade to 2.13 first, then the 2.14 release can be tested (which may require an OS migration as we're currently on el6)

This is probably a duplicate, we already plan to upgrade Gerrit to 2.13
soon, 2.14 will follow if all goes well.
Evgheni is on PTO today, so he can reply with dates on the planned upgrade
to 2.13 early next week.

We also want to make sure the Gerrit backups are verified and working well
before upgrading.

On Fri, Jun 16, 2017 at 8:08 AM, sbonazzo (oVirt JIRA) <

–

Eyal edri

ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)