03.2020 - Jenkins data partition running out of disk space
Description
Jenkins is running out of disk space and this caused issues with CI last week.
Points to take:
1) extend disk space as needed
2) verify what is taking up space - likely some job is not deleting old artifacts
3) confirm why we weren't notify sooner since we do monitor jenkins disk space via nagios
Activity
looking at monitoring graphs, something changed in the second part of February and disk space usage increased significantly:
30+GB jobs:
126G | ovirt-4.3_change-queue-tester |
150G | ovirt-appliance_4.3_build-artifacts-el7-x86_64 |
94G | ovirt-appliance_master_build-artifacts-el8-x86_64 |
134G | ovirt-appliance_standard-check-patch |
32G | ovirt-engine-nodejs-modules_standard-on-merge |
151G | ovirt-master_change-queue-tester |
67G | ovirt-node-ng-image_4.3_build-artifacts-el7-x86_64 |
43G | ovirt-node-ng-image_master_build-artifacts-el8-x86_64 |
185G | ovirt-system-tests_gate |
115G | ovirt-system-tests_manual |
90G | ovirt-system-tests_standard-check-patch |
63G | system-sync_mirrors-fedora-base-fc29-x86_64 |
please take a look at the fc29 job - it should not contain that much data.
Let's also go through appliance jobs and remove old "keep forever" builds if any. Same must be done for OST manual/gate and we should review retention policies there since most of the artifacts are probably not needed there.
After cleaning up some "keep forever" builds the situation is back to normal. Closing the ticket.
Assignee
Reporter
Blocked By
Priority
