Very slow push/pull to/from gerrit

Description

Recently pushing changes to gerrit is very slow. It seems that the issue
is pulling changes during rebase in "git review".

Sometimes "git review" is very slow, but "git review -R" is very quick. This
show that the issue was getting changes from gerrit during when git-review
try to rebase the current branch on master.

I noticed that origin/master and gerrit/master do not show the same commit:

$ git log origin/master | head -1
commit 6e6e93cb1d237f9b6fdd533d5afdc1b9ba1c197a

$ git log gerrit/master | head -1
commit 353e7b1e322aa02d4767b6617ed094be0643b094

$ git remote -v
gerrit http://gerrit.ovirt.org/p/vdsm.git (fetch)
gerrit ssh://nsoffer@gerrit.ovirt.org:29418/vdsm.git (push)
origin http://gerrit.ovirt.org/p/vdsm.git (fetch)
origin http://gerrit.ovirt.org/p/vdsm.git (push)

I've been using this configuration for several years, and the slowness
started only recently, so I guess the issue is not on my side.

Same issues also on ovirt-engine, using similar configuration:

$ git remote -v
gerrit http://gerrit.ovirt.org/ovirt-engine (fetch)
gerrit ssh://nsoffer@gerrit.ovirt.org:29418/ovirt-engine.git (push)
origin git://gerrit.ovirt.org/ovirt-engine (fetch)
origin git://gerrit.ovirt.org/ovirt-engine (push)

Anyone experienced this?

I guess that restarting the gerrit server will "fix" this issue.

Nir

Activity

Show:
Evgheni Dereveanchin
September 30, 2020, 2:31 PM

Hi Nir, sorry for the delay. I’ve looked at this issue a few days ago and compared the speed using the commands you provided getting 70 Mbps speeds so this may be related you your ISP’s connectivity to AWS where gerrit is hosted. I must also note that the git:// protocol is deprecated and we advertise using https:// however speeds in both cases should be roughly the same.

Are you still experiencing slowness when cloning? Maybe we can try to fetch via the TLV office to compare speeds.

Nir Soffer
October 12, 2020, 5:35 PM

Yes, I still experience very slow pulls, specially in ovirt-engine.

I have now a stuck pull, no progress after several minutes.

Nir Soffer
October 12, 2020, 8:07 PM

Comparing github and gerrit for same project:

$ time git clone
Cloning into 'ovirt-engine'...
remote: Enumerating objects: 3015, done.
remote: Counting objects: 100% (3015/3015), done.
remote: Compressing objects: 100% (1784/1784), done.
remote: Total 1189325 (delta 1175), reused 2422 (delta 582), pack-reused 1186310
Receiving objects: 100% (1189325/1189325), 303.35 MiB | 751.00 KiB/s, done.
Resolving deltas: 100% (551618/551618), done.

real 7m6.307s
user 0m33.054s
sys 0m5.362s

 

$ time git clone http://gerrit.ovirt.org/ovirt-engine
Cloning into 'ovirt-engine'...
remote: Counting objects: 1176123, done
remote: Finding sources: 100% (1176123/1176123)
remote: Total 1176123 (delta 606354), reused 1132012 (delta 606354)
Receiving objects: 100% (1176123/1176123), 305.42 MiB | 134.00 KiB/s, done.
Resolving deltas: 100% (606354/606354), done.

real 40m4.511s
user 0m33.484s
sys 0m9.772s

Nir Soffer
October 12, 2020, 8:13 PM

Note the difference in the number of objects:

github: remote: Counting objects: 100% (3015/3015), done.

gerrit: remote: Counting objects: 1176123, done

Also, github use compression:

remote: Compressing objects: 100% (1784/1784), done.

No such log in gerrit clone.

Maybe we need to do some gc/packing in gerrit repos?

Evgheni Dereveanchin
October 13, 2020, 10:35 AM

Hi Nir, thanks for providing the test commands. I ran it off the office WiFi and here are my results:

 

$ time git clone https://gerrit.ovirt.org/ovirt-engine
Cloning into 'ovirt-engine'...
remote: Counting objects: 1176196, done
remote: Finding sources: 100% (1176196/1176196)
remote: Total 1176196 (delta 606359), reused 1132054 (delta 606359)
Receiving objects: 100% (1176196/1176196), 305.52 MiB | 5.12 MiB/s, done.
Resolving deltas: 100% (606359/606359), done.

real 1m31.826s
user 0m31.851s
sys 0m4.255s


$ time git clone https://github.com/oVirt/ovirt-engine.git
Cloning into 'ovirt-engine'...
remote: Enumerating objects: 3088, done.
remote: Counting objects: 100% (3088/3088), done.
remote: Compressing objects: 100% (1835/1835), done.
remote: Total 1189398 (delta 1205), reused 2465 (delta 582), pack-reused 1186310
Receiving objects: 100% (1189398/1189398), 303.36 MiB | 9.71 MiB/s, done.
Resolving deltas: 100% (551648/551648), done.

real 0m45.266s
user 0m32.290s
sys 0m3.128s

Do you experience this slowness at all times or just periodically? I seem to get fast download speeds most of the times I check and the same applies to the CI setup in PHX which is also remote compared to AWS where Gerrit is hosted.

 

As for compression, in this particular case the amount of compressed data is the same as for uncompressed so I don’t think it plays a role (i.e. it does not save bandwidth but adds extra CPU load)

We really need more data from your side here as it does not look like an issue on Gerrit that would reproduce easily.

Could you please run this test to possibly spot connectivity issues to AWS?

https://cloudharmony.com/speedtest-for-aws

Assignee

Shlomi Zidmi

Reporter

Nir Soffer

Blocked By

None

Priority

Medium
Configure