No space left on device

Description

Hi,
looks like on some slave there's not enough space for handling the jobs:
https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x86_64/160/console

Can you please check if we need larger disks or if it's a case of cleanup
failing?
In both cases, please take action on fixing the issue.

SANDRO BONAZZOLA

MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

Red Hat EMEA <https://www.redhat.com/>

sbonazzo@redhat.com
<https://red.ht/sig>

Activity

Show:
Eyal Edri
December 30, 2018, 9:13 AM

fixed issue with current slave, will track additional actions in another open ticket.

Eyal Edri
December 30, 2018, 9:13 AM

I see the last 2/3 runs worked on vm0013, and it has 21GB available space.
The last run fails on a different issue, which should be checked, but doesn't look infra related.

We have another ticket on adding monitoring to VMs, or improving slave cleanup, so we can track it there,
though I don't think this happens that common.

Eyal Edri
December 27, 2018, 3:02 PM

If it keeps happening, we need to see if we can clear more files using slave_cleanups.sh

Eyal Edri
December 27, 2018, 2:55 PM

I looked at the job and it seems it always run on vm0013.
Looking at the VM, I saw it has enough free space 13 GB, so not sure why it is failing.
It was up 200 days and running Centos 7.4, I upgraded it and rebooted, now the job runs on a new VM.
Let's see how it goes, not sure what is the root cause for it, but if its not reproducing on multiple jobs, I suggest to fix/upgrade bad slaves if it solves the issue.

Galit Rosenthal
December 26, 2018, 6:27 AM

Removed data from /var/mock - release some space on the vm.

Fixed
Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Galit Rosenthal

Reporter

Sandro Bonazzola