update OpenShift to OKD 3.10

Another upgrade blocker is https://github.com/openshift/openshift-ansible/issues/9750 that causes the process to fail

It's been quite a while and our staging instance is still destroyed after a failed upgrade to 3.10 with various issues still blocking upgrades. At the same time we're moving most of CI to OpenShift so we need to plan upgrades. To save time we need to decide on an approach. Some options I see:
1) stay on version 3.9 until the next version of OKD comes out that uses operators. That will require a reinstall anyway.
2) deploy a fresh 3.11 instance now and move CI workloads there. Then move production pods as they're not that dependent on new features but uptime is critical.

what do you think? How much do we need 3.10 or 3.11 features for CI? Also, can Jenkins connect to several OpenShift instances at once? That would allow us to move between environments very seamlessly.

I think it'll probably be best to install a fresh install of OKD 3.11 (instead of the staging if we still have it) or take one host and reinstall it as master and slowly move nodes to the new instance.
That way it will be easier to make it production grade with latest monitoring, etc..,
It will also make it easier to upgrade to 4.0 in a few months.

Yes, Jenkins can connect to multiple Openshifts at once.

I think that we should use the latest Openshift (it has CPU affinity, device plugins, and more good stuff).

I agree with , we should create a new cluster, migrate one host, test it and then migrate the other hosts.

Closing this as we'll work on v4 in instead