After stopping the engine, updating postgres setting and restarting it I hit the symptoms described in this bug causing the upgrade to fail as DWH lost connection and couldn't shut down properly:

[ ERROR ] dwhd is currently running. Its hostname is engine-phx.ovirt.org. Please stop it before running Setup.
[ ERROR ] Failed to execute stage 'Transaction setup': dwhd is currently running

2017-04-22 09:17:43 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/bin/systemctl', 'status', 'ovirt-engine-dwhd.service') stdout:
● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse
Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Sat 2017-04-22 09:17:42 EDT; 1s ago
Process: 2098 ExecStart=/usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.py --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, status=0/SUCCESS)
Main PID: 2098 (code=exited, status=0/SUCCESS)

Apr 22 08:36:50 engine-phx systemd[1]: Starting oVirt Engine Data Warehouse...
Apr 22 08:36:51 engine-phx systemd[1]: Started oVirt Engine Data Warehouse.
Apr 22 09:17:38 engine-phx systemd[1]: Stopping oVirt Engine Data Warehouse...
Apr 22 09:17:42 engine-phx systemd[1]: Stopped oVirt Engine Data Warehouse.

2017-04-22 09:17:43 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/bin/systemctl', 'status', 'ovirt-engine-dwhd.service') stderr:

2017-04-22 09:17:43 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:200 Database: 'None', Statement: '
select * from GetDwhHistoryTimekeepingByVarName(
%(name)s
)
', args: {'name': 'DwhCurrentlyRunning'}
2017-04-22 09:17:43 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:205 Creating own connection
2017-04-22 09:17:43 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:250 Result: [{'var_value': '1', 'var_datetime': None, 'var_name': 'DwhCurrentlyRunning'}]
2017-04-22 09:17:43 ERROR otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.core.single_etl single_etl._transactionBegin:137 dwhd is currently running.
Its hostname is engine-phx.ovirt.org.
Please stop it before running Setup.
2017-04-22 09:17:43 DEBUG otopi.context context._executeMethod:142 method exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod
method['method']()
File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-dwh/core/single_etl.py", line 140, in _transactionBegin
raise RuntimeError(_('dwhd is currently running'))
RuntimeError: dwhd is currently running

Rebooted the system to let all services start in normal order. After this the upgrade worked properly and is now complete.

All hosts updated. For engine update to proceed epel had to be disabled otherwise it failed with error:

[ INFO ] Checking for product updates...
[ ERROR ] Yum [u'collectd-write_http-5.7.0-2.el7.x86_64 requires collectd(x86-64) = 5.7.0-2.el7', u'collectd-disk-5.7.0-2.el7.x86_64 requires collectd(x86-64) = 5.7.0-2.el7']

After that, the upgrade failed with error:

Found the following problems in PostgreSQL configuration for the Engine database:
autovacuum_vacuum_scale_factor required to be at most 0.01
autovacuum_analyze_scale_factor required to be at most 0.075
autovacuum_max_workers required to be at least 6
maintenance_work_mem required to be at least 65536
Please set:
autovacuum_vacuum_scale_factor = 0.01
autovacuum_analyze_scale_factor = 0.075
autovacuum_max_workers = 6
maintenance_work_mem = 65536
in postgresql.conf on '127.0.0.1'. Its location is usually /var/lib/pgsql/data , or somewhere under /etc/postgresql* .

I updated one of the hosts to VDSM 4.19.10 on Friday and it is working fine. Will continue with other hosts and then proceed with the Engine.
Need to be careful with the Production cluster as live migrations are broken on VMs with QoS in 4.1.0.