No space left on device
looks like on some slave there's not enough space for handling the jobs:
Can you please check if we need larger disks or if it's a case of cleanup
In both cases, please take action on fixing the issue.
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
fixed issue with current slave, will track additional actions in another open ticket.
I see the last 2/3 runs worked on vm0013, and it has 21GB available space.
The last run fails on a different issue, which should be checked, but doesn't look infra related.
We have another ticket on adding monitoring to VMs, or improving slave cleanup, so we can track it there,
though I don't think this happens that common.
If it keeps happening, we need to see if we can clear more files using slave_cleanups.sh
I looked at the job and it seems it always run on vm0013.
Looking at the VM, I saw it has enough free space 13 GB, so not sure why it is failing.
It was up 200 days and running Centos 7.4, I upgraded it and rebooted, now the job runs on a new VM.
Let's see how it goes, not sure what is the root cause for it, but if its not reproducing on multiple jobs, I suggest to fix/upgrade bad slaves if it solves the issue.
Removed data from /var/mock - release some space on the vm.