he-basic-iscsi-master suite fails with 'disk_space requirements are not satisfied'

Description

https://pastebin.com/HfpwWfgi

we have 3 disks: Main NFS device (101G), Main iSCSI device (191G) and Hosted engine storage (80G)

Gal, should we increase the 80- GB to 101 as well?
Adding patch - if needed please merge: https://gerrit.ovirt.org/93493

Activity

Show:

Dafna Ron August 16, 2018 at 10:06 AM

The suite is now passing so moving to resolved.
please re-open if needed.

Dafna Ron August 14, 2018 at 2:40 PM

I am seeing it on other hosts now for check-patch:
http://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/1649/consoleFull
failed with server
'13:18:05 [check-patch.he-basic_suite_master.el7.x86_64] Running on ovirt-srv25.phx.ovirt.org in /home/jenkins/workspace/ovirt-system-tests_standard-check-patch'

and also, the entire suite is failing and its running on other servers
http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/426/consoleFull
'10:37:01 Building remotely on ovirt-srv20.phx.ovirt.org (el7 phx integ-tests older-libvirt physical) in workspace /home/jenkins/workspace/ovirt-system-tests_he-basic-iscsi-suite-master'

are we sure its the hosts and not the suites?
if we can't solve it soon we should remove it from check-patch.

Former user August 9, 2018 at 8:22 AM

The error specifies that /dev/shm didn't have enough space:

15:37:21 [check-patch.he-basic-iscsi_suite_master.el7.x86_64] ERROR:_main_:disk_space requirements are not satisfied.
15:37:21 [check-patch.he-basic-iscsi_suite_master.el7.x86_64] /dev/shm/ost/deployment-he-basic-iscsi-suite-master, Required space: 85899345920, Free space: 67242180608

The job ran on ovirt-srv06 that has since been rebuilt so I can't say more to what filled the shm device. The bare metal has 128GB of RAM so probably something didn't get cleaned up. I can't confirm how cleanups happen for /dev/shm but they may need improvement. We could also add a line to the logs that prints the size of the partition just to confirm it was the right size initially.

Dafna Ron August 9, 2018 at 8:01 AM

can you please check if there was an issue with the host?

Gal Ben Haim August 6, 2018 at 7:57 AM

It checks for free disk space the Jenkins slave, not Lago VMs.

Fixed

Details

Assignee

Reporter

Priority

Created August 6, 2018 at 7:29 AM
Updated September 2, 2018 at 3:50 PM
Resolved August 16, 2018 at 10:06 AM