Unavialable projects in Gerrit

Description

Hi,

We've encountered an issue where some projects (we've noticed only ovirt-engine project, but other projects might have been affected too) were unavailable from the gerrit ui.
Clone with ssh was unavailable, but clone with git worked.
After restarting the gerrit service, the issue resolved.
We assume that the gerrit-exproll job ran, and maybe because that the ovirt-engine repo is much larger and others, something went wrong.
system_gerrit-exproll jenkins job log

Attached is the gerrit error_log with the expetions

Attachments

1

Activity

Show:

Former user June 29, 2017 at 1:35 AM

Gerrrit updated to 2.13.8 which should have a newer JGit version. This should permanently fix the issue described in this ticket.

Former user April 6, 2017 at 8:58 AM

I still see "Missing unknown" errors so the issue is probably still there:

[2017-04-06 04:48:28,619] [SSH gerrit query limit:500 (status:open OR status:closed) --all-approvals --comments --format=JSON --start=0 (owl)] ERROR com.google.gerrit.server.events.EventFactory : Cannot load patch set data for 74970,1
org.eclipse.jgit.errors.MissingObjectException: Missing unknown d0056f655822400064be849c35188aa933d7ef91
[2017-04-06 04:48:28,619] [SSH gerrit query limit:500 (status:open OR status:closed) --all-approvals --comments --format=JSON --start=0 (owl)] ERROR com.google.gerrit.server.events.EventFactory : Cannot load patch set data for 74970,2
org.eclipse.jgit.errors.MissingObjectException: Missing unknown c86648097c9963a4e109968f8b4d04babab399dc
[2017-04-06 04:48:28,620] [SSH gerrit query limit:500 (status:open OR status:closed) --all-approvals --comments --format=JSON --start=0 (owl)] ERROR com.google.gerrit.server.events.EventFactory : Cannot load patch set data for 74970,3
org.eclipse.jgit.errors.MissingObjectException: Missing unknown b4417aebc048faeaf854e8135dbcc3c13bfe982f

This however may be related to changeset inconsistencies as reported in since the change mentioned above is unavailable with error 500:
https://gerrit.ovirt.org/#/c/74970/

Former user April 4, 2017 at 3:13 PM

Closing as Gerrit core was updated to newer version. Please reopen if this happens again.

Former user March 7, 2017 at 1:35 PM

From Jenkins logs this error first appeared around this timestamp:

[2017-03-04 20:55:33,667] [SSH gerrit query --format=JSON --current-patch-set status:open age:70d project:infra-puppet (jenkins_ro)] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /usr/local/src/git
/infra-puppet.git/objects/pack/pack-82359f9d358849634f6e6ec243beb1c87e19236a.pack was deleted, removing it from pack list
[2017-03-04 20:55:33,667] [SSH gerrit query --format=JSON --current-patch-set status:open age:70d project:infra-puppet (jenkins_ro)] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /usr/local/src/git
/infra-puppet.git/objects/pack/pack-619c979282a59e91be3a9d83c3f4903464085504.pack is corrupt, removing it from pack list
[2017-03-04 20:55:48,743] [SSH gerrit query --format=JSON --current-patch-set status:open age:160d project:mom (jenkins_ro)] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /usr/local/src/git/mom.git
/objects/pack/pack-85e61c79aca4545c05541eec3bd177cf3897ce82.pack was deleted, removing it from pack list
[2017-03-04 20:55:48,744] [SSH gerrit query --format=JSON --current-patch-set status:open age:160d project:mom (jenkins_ro)] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /usr/local/src/git/mom.git
/objects/pack/pack-1af261f9779700f5179f10eb964ed3e7f0d281bb.pack is corrupt, removing it from pack list
[2017-03-04 20:55:51,771] [SSH gerrit query --format=JSON --current-patch-set status:open age:160d project:otopi (jenkins_ro)] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /usr/local/src/git/otopi
.git/objects/pack/pack-cf4fb1b4310362de6228d58f0d639548b262bc2e.pack was deleted, removing it from pack list
[2017-03-04 20:55:51,771] [SSH gerrit query --format=JSON --current-patch-set status:open age:160d project:otopi (jenkins_ro)] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /usr/local/src/git/otopi
.git/objects/pack/pack-c30d9264f991eaf2f27d9ecb864dab215cb7be6b.pack is corrupt, removing it from pack list
[2017-03-04 20:56:07,667] [SSH gerrit query --format=JSON --current-patch-set status:open age:160d project:ovirt-engine (jenkins_ro)] WARN com.google.gerrit.server.patch.PatchListLoader : Error computing PatchListKey[BASE..
7e43d939527c9b09c1610aeb64bf2d546d45fc0d IGNORE_NONE]
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 8faae80a63593daebea22a9564a51464c287463c
...

Looking at possible causes (given the fact that a service restart fixed it) it looks like an incompatibility between older Git and JGit due to naming collisions:
https://groups.google.com/d/msg/repo-discuss/Qdmbl-YZ4NU/q3Hewer4BvwJ

I am not familiar with garbage collection processes in git, this was reported fixed (name collisions removed) in git 1.9 while RHEL6 ships with version 1.7 (even RHEL7 ships 1.8)
To battle this we have a cron job that restarts gerrit daily but apparrently it doesn't always help.
any ideas how we could improve or what else may be causing this?

On a side note, there's a lot of stuff in /usr/local/src/git while our backup scripts don't seem to cover it. If it is important (looks so from its contents) we need to address this.

Cannot Reproduce

Details

Assignee

Reporter

Priority

Created March 5, 2017 at 10:41 AM
Updated June 29, 2017 at 1:35 AM
Resolved April 4, 2017 at 3:13 PM