Today's outage was a clear reminder that our current storage configuration does not serve us well. We hardly know how to debug it, it seems to not be resistant to the very issues it was supposed to protect against and introduce potential failure scenarios of its own.

I suggest we implement a new storage layout that meets the following criteria:

1. Ultimate simplicity at the lower level of the stack. More specifically:

   1. The storage severs should be simple NFS or iSCSI servers. No DRBD and no exotic file-systems.

   2. Only simple storage will be presented to oVirt for use as storage domains

2. Separation of resources between critical services - The 'Jenkins" master for e.g. should not share resources with the "resources" server or anything else.The separation should hold true down to the physical spindle level.

3. Duplication of services and use of local storage where possible - this is a longer term effort - but we have some low hanging fruits here like artifactory, where simple DNS/LB-based fail-over between two identical hosts would probably suffice.

4. Complexity only where needed and up the stack. For example we can just have the storage for Jenkins be mirrored at the VM level with fail-over to a backup VM.