Stats from logs for the last 5 days:

POST requests are what h5ai uses to request tree portions, so people are using this.

Given that amount of repositories we have, and the directory structure within - the directory tree is important. Maybe we can find a non-php alternative to h5ai? I'm all for removing php if we can.

Tested h5ai 0.29 and it requires PHP 5.5 or newer while the default in CentOS is 5.4. We could invest time into adding newer PHP from software collections but I'm not sure we actually gain anything from having h5ai as that's the only reason PHP is on this web server. We can tune Apache index listings by adding headers/footers and changing fonts/icons to make the style more consistent with our other websites instead without. The only thing apache is not able to do is draw the directory tree on the left but do we actually need it?

, , , what do you think?

Reviewed existing configs of h5ai and there are just a couple of differences from defaults, mainly disabled previews:

"preview-aud": {

• "enabled": true,
  + "enabled": false,

"preview-vid": {

• "enabled": true,
  + "enabled": false,

• "txt": [".text", ".txt"],
  + "txt": [".text", ".txt", "*.README"],

Checked the system itself as it is slow when multiple createrepo/repoman processes are running:

top - 13:40:53 up 13 days, 4:23, 1 user, load average: 2.27, 1.41, 1.37
Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie
%Cpu(s): 33.8 us, 4.1 sy, 0.0 ni, 7.4 id, 54.3 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 1809444 total, 75884 free, 623748 used, 1109812 buff/cache
KiB Swap: 2097148 total, 1886412 free, 210736 used. 867424 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13899 deploy-+ 20 0 342164 21372 7160 S 36.8 1.2 0:01.28 worker.py
13909 deploy-+ 20 0 342044 21380 7160 D 34.8 1.2 0:01.25 worker.py
13897 deploy-+ 20 0 341848 20956 7160 D 10.9 1.2 0:00.56 worker.py
13901 deploy-+ 20 0 345064 24268 7160 S 9.6 1.3 0:00.44 worker.py
7968 root 20 0 0 0 0 S 7.9 0.0 0:01.46 kworker/2:0
13905 deploy-+ 20 0 342324 21660 7160 R 7.9 1.2 0:00.43 worker.py
534 ovirtag+ 20 0 441888 6228 2640 S 6.6 0.3 94:59.65 python
13895 deploy-+ 20 0 344536 23668 7160 R 6.3 1.3 0:00.36 worker.py
13896 deploy-+ 20 0 342072 21408 7160 D 3.3 1.2 0:00.27 worker.py
41 root 20 0 0 0 0 S 2.6 0.0 63:04.40 kswapd0
13894 deploy-+ 20 0 341848 20968 7160 D 2.3 1.2 0:00.37 worker.py
13888 deploy-+ 20 0 342408 21616 7156 S 2.0 1.2 0:00.26 genpkgmetadata.
13891 deploy-+ 20 0 342408 21604 7152 S 1.7 1.2 0:00.22 genpkgmetadata.
13877 deploy-+ 20 0 342316 21952 7552 D 1.3 1.2 0:00.20 genpkgmetadata.

memory and CPU are enough, 54% IOWait is the bottleneck and we can't really get around this: the VM is on shared storage and we can't enable caching as this may lead to data corruption. What we can do is bump RAM usage so that there is more IO cache in the kernel itself or make repo generation happen on a different machine if possible. I will update h5ai to the newest release in any case while I'm at it since we are mostly using built in features only.

could you please confirm if h5ai is managed by puppet or installed manually?
The only reference I found in puppet manifests is apache-related and not covering the actual h5ai install. Is this correct?

We are using version 0.27 while the newest release is 0.29 from half a year ago.

On another note I see this script allows generating directory tarballs on the fly to download multiple files at once. We have a couple of huge directories where this feature may consume a lot of resources. Need to check if this is needed or can be disabled.