03.2020 - Jenkins data partition running out of disk space

Description

Jenkins is running out of disk space and this caused issues with CI last week.

Points to take:
1) extend disk space as needed
2) verify what is taking up space - likely some job is not deleting old artifacts
3) confirm why we weren't notify sooner since we do monitor jenkins disk space via nagios

Activity

Show:
Evgheni Dereveanchin
March 16, 2020, 11:11 AM

looking at monitoring graphs, something changed in the second part of February and disk space usage increased significantly:

 

Evgheni Dereveanchin
March 16, 2020, 4:48 PM

30+GB jobs:

126G

ovirt-4.3_change-queue-tester

150G

ovirt-appliance_4.3_build-artifacts-el7-x86_64

94G

ovirt-appliance_master_build-artifacts-el8-x86_64

134G

ovirt-appliance_standard-check-patch

32G

ovirt-engine-nodejs-modules_standard-on-merge

151G

ovirt-master_change-queue-tester

67G

ovirt-node-ng-image_4.3_build-artifacts-el7-x86_64

43G

ovirt-node-ng-image_master_build-artifacts-el8-x86_64

185G

ovirt-system-tests_gate

115G

ovirt-system-tests_manual

90G

ovirt-system-tests_standard-check-patch

63G

system-sync_mirrors-fedora-base-fc29-x86_64

please take a look at the fc29 job - it should not contain that much data.

Let's also go through appliance jobs and remove old "keep forever" builds if any. Same must be done for OST manual/gate and we should review retention policies there since most of the artifacts are probably not needed there.

Evgheni Dereveanchin
April 3, 2020, 9:45 AM

After cleaning up some "keep forever" builds the situation is back to normal. Closing the ticket.

Fixed

Assignee

Evgheni Dereveanchin

Reporter

Evgheni Dereveanchin

Blocked By

None

Priority

Medium
Configure