Simplify oVirt storage configuration

Description

Today's outage was a clear reminder that our current storage configuration does not serve us well. We hardly know how to debug it, it seems to not be resistant to the very issues it was supposed to protect against and introduce potential failure scenarios of its own.

I suggest we implement a new storage layout that meets the following criteria:

  1. Ultimate simplicity at the lower level of the stack. More specifically:

    1. The storage severs should be simple NFS or iSCSI servers. No DRBD and no exotic file-systems.

    2. Only simple storage will be presented to oVirt for use as storage domains

  2. Separation of resources between critical services - The 'Jenkins" master for e.g. should not share resources with the "resources" server or anything else.The separation should hold true down to the physical spindle level.

  3. Duplication of services and use of local storage where possible - this is a longer term effort - but we have some low hanging fruits here like artifactory, where simple DNS/LB-based fail-over between two identical hosts would probably suffice.

  4. Complexity only where needed and up the stack. For example we can just have the storage for Jenkins be mirrored at the VM level with fail-over to a backup VM.

Activity

Show:
Eyal Edri
April 1, 2018, 11:51 AM

can you give an update on this? what is the plan for reprovisoning the next storage server?

Eyal Edri
April 30, 2018, 7:34 AM

let's target it to post 4.2.3, after we upgrade HE to latest 4.2.3 and it works well

Evgheni Dereveanchin
June 1, 2018, 11:52 AM

4.2.3 is out and the production HE environment is up-to-date.

New ticket logged to track NFS migration: OVIRT‌-2075

Evgheni Dereveanchin
August 8, 2018, 9:10 AM
Edited

NFS migration complete, storage01 shut down and can be rebuilt.

Here's the partitioning from storage02 - I will likely repeat it unless we need some other specifics:

Physical size

RAID

Logical size

name

use

2x900G

0.9T

RAID1

centos

OS plus NFS shares

4x900G

1.8T

RAID10

jenkins

Jenkins

6x900G

3.6T

RAID50

prod-1

prod systems tier 1 like resources

4x900G

2.7T

RAID5

prod-2

prod systems tier 2

Need to have VLANs configured on it as well as patch BIOS and enable PXE.

Evgheni Dereveanchin
August 23, 2018, 3:05 PM

Both storage servers rebuilt to offer block storage via iSCSI plus NFS for some use cases. Closing the tracker ticket.

Done

Assignee

Evgheni Dereveanchin

Reporter

Barak Korren

Blocked By

pending upgrade of HE to 4.2.3 and verifying no open bugs on HE

Components

Priority

Highest
Configure