Managed to make the packages cached as well, and may have also shortened the uncached run times in the process...
https://gerrit.ovirt.org/#/c/67801/1
Compare:
http://jenkins.ovirt.org/job/jenkins_master_check-patch-el7-x86_64/165/ (uncached, 1m 39s)
With:
http://jenkins.ovirt.org/job/jenkins_master_check-patch-el7-x86_64/167/ (cached, 23s)

The impact for projects with more packages (like vdsm) will hopefully be bigger.

Ok. Just cleaning up less aggressively seems effective.
Compare:
http://jenkins.ovirt.org/job/jenkins_master_check-patch-el7-x86_64/159/console (90 seconds total)
with:
http://jenkins.ovirt.org/job/jenkins_master_check-patch-el7-x86_64/161/console (26 seconds total)

This patch makes the change:
https://gerrit.ovirt.org/#/c/67795/

We can probably get further speedup by also caching the package installation step (As opposed to just the mock setup step)

This can be seen in the mock logs (Checked both for fc24 and for el7):

This means that we don't need to explicitly enable the options I mentioned in the ticket description.

Checking locally without zapping caches, the first time I run muck_chroot.sh:

The 2nd time:

The 3rd time:

So not zapping the cache will probably provide significant improvement.

Right now my plan is:

1. make mock_runner.sh touch the mock cache directory

2. make mock_cleanup.sh delete only directories that were not touched in the last N days.

The packages install is done by mock itself during the setup of the chroot - so could probably be sped up.

All the rest:

• The pip stuff

• The autogen w/o locale

• other package installs
  All come from inside check_patch.sh (from vdsm repo) so do not belong on this ticket.

Having said that, yum in the mock env is running through oVirt's proxy - so not really going to the internet (pre-post install script still takt time though which is why the mock setup is long).

We could probably find one way or another to make pip go through the proxy too to make it a little faster (Maybe export the "http_proxy" env var in to the mock env .to make scripts able to leverage it).

Looking deeper into our scripts, this can be found in mock_cleanup.sh:

We probably need need remove that before we can gain any speed ups. But we must heed David's warning and make sure we zap the caches frequently enough so we don't fill up the slaves.

I think we see the slowness also on experimental flows, same actions which takes a few seconds w/o mock takes 2 min when running with mock runner.
So we should probably give priority to checking the caching options.

1. I'm not sure if it's here or requires a different ticket, but this is SLOW (taken from http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/consoleFull ):

00:01:28.338 INFO: installing package(s): autoconf automake gdb git libguestfs-tools-c libselinux-python3 libvirt-python3 m2crypto make mom openvswitch policycoreutils-python PyYAML python-blivet python-coverage python2-decorator python-devel python-inotify python-ioprocess python-mock python-netaddr python-pthreading python-setuptools python-six python-requests python3-decorator python3-netaddr python3-nose python3-six python3-yaml rpm-build sanlock-python sudo yum yum-utils
00:03:48.999 INFO: None

2:20 to install packages, is VERY slow.

2. Then there are the pip stuff - which probably are also going somewhere off to the 'net:
00:03:52.839 + easy_install pip
...
00:03:53.401 + pip install -U tox==2.1.1
...
00:03:55.843 Downloading pluggy-0.3.1-py2.py3-none-any.whl
00:03:55.868 Installing collected packages: virtualenv, py, pluggy, tox
00:03:56.223 Successfully installed pluggy-0.3.1 py-1.4.31 tox-2.1.1 virtualenv-15.1.0

So only 4 seconds, but probably not needed at all.

3. Why not set locale?
00:03:56.670 + ./autogen.sh --system --enable-hooks --enable-vhostmd
00:03:56.678 perl: warning: Setting locale failed.
00:03:56.678 perl: warning: Please check that your locale settings:
00:03:56.679 LANGUAGE = (unset),
00:03:56.679 LC_ALL = (unset),
00:03:56.679 LANG = "en_US.UTF-8"

4. The 5 seconds here are probably due to slow storage!
00:04:05.096 client/Makefile.am:23: installing 'build-aux/py-compile'
00:04:10.239 Running ./configure with --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=/usr/lib64 --enable-hooks --enable-vhostmd

5. This is also going out to the Internet?
00:04:14.989 + debuginfo-install -y python
...
00:04:30.879 glibc-debuginfo-common.x86_64 0:2.23.1-11.fc24

6. Again some more deps:
00:05:23.621 tox -e tests
00:05:23.912 tests create: /home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/.tox/tests
00:05:29.991 tests installdeps: nose==1.3.7

7. it's not clear if the nose tests are running in parallel. Is that supported by VDSM tests?
8. Not sure what is going on here:
00:09:34.017 + coverage html -d /home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/exported-artifacts/htmlcov
00:09:49.213 + popd
00:09:49.213 ~
00:09:49.213 + shopt -s extglob
00:09:49.213 + git diff-tree --no-commit-id --name-only -r HEAD
00:09:49.213 + egrep --quiet 'vdsm.spec.in|Makefile.am'
00:19:07.994 Took 915 seconds