Functionality is fixed, deployed and tested on gerrit.ovirt.org. Announcement is also sent to devel list.

The patch [1] restores the functionality by fixing the parameters passed to rerun() function and by disabling the "return" preventing any comments left from being handled by the hooks handling.

[1] https://gerrit.ovirt.org/#/c/91292/

All patches are verified on staging gerrit. I will wait a bit so people can review the patches and then will merge and deploy to the production gerrit.

The first problem was that all exceptions for the hooks code are essentially lost. This might or it might be possible to get the delivered to some gerrit log file, but I was not able to find them. The patch [1] redirect items sent to STDOUT and STDERR to the hooks log file so we can see them.

With this patch and some code modifications we see the problem:

2018-05-16 11:22:16,763::26675::ERROR::Traceback (most recent call last):
2018-05-16 11:22:16,763::26675::ERROR:: File "/home/gerrit2/review_site/hooks/comment-added", line 581, in <module>
2018-05-16 11:22:16,764::26675::ERROR::main()
2018-05-16 11:22:16,764::26675::ERROR:: File "/home/gerrit2/review_site/hooks/comment-added", line 568, in main
2018-05-16 11:22:16,765::26675::ERROR::hooks = fun(tag_val, hooks, known_args)
2018-05-16 11:22:16,765::26675::ERROR::TypeError
2018-05-16 11:22:16,765::26675::ERROR:::
2018-05-16 11:22:16,766::26675::ERROR::rerun() takes exactly 2 arguments (3 given)

the current rerun() function is not compatible with the way it is called and takes less arguments than passed. Also the current code just "returns" and does nothing when comment is added, so I also disabled that "return".

Now proceeding to check if it is easy to modify rerun() function to function properly as I see the logic inside it is implemented, so we might end up with few modifications only.

[1] https://gerrit.ovirt.org/#/c/91284/

Alas in https://gerrit.ovirt.org/#/c/81187/ the reason for hooks being rerun is the updated of the commit message, but not "Rerun-hooks" comment. The code for the dispatcher is the same as was written by David 2 years ago and did not changed. Not sure yet how it was working before.

It was tricky to get understanding on how to debug it on staging gerrit. But I now have everything needed for that, so processing with actual debug. The current code patch for all comments is just "return" so at that sense it is disabled.

Trying to pinpoint the change which caused that, here is approximate timing infromation: