Jenkins STD-CI are running in dirty mock chroots
Description
Activity
Eyal Edri December 26, 2017 at 12:23 PM
please reopen if its still relevant
Eyal Edri November 28, 2017 at 2:24 PM
anything else needed here? looks like the current flow is by design as barak mentioned.
Barak Korren November 5, 2017 at 9:33 AM
I'm pretty sure in the past it was completely removed from the slave.
I'm pretty sure it was not, since code handling this has not been significantly changed since David first wrote it.
The workspace including the source checkout does get completely remove when a different job starts running on the node. But as long as its the same job, the workspace stays as-is.
Is there any good reason for not cleaning the slave after job completion?
Yes. Re-cloning the whole git repo every time we run something will significantly slow things down.
Also we do not clean the slave up after the execution, we clean it before the next one. This has the benefit of allowing us to debug failed runs and also prevent the next run from failing if the post-run code was not reached for some reason. But a side-affect of this is that the more cleanup steps we have the longer the latency our users experience between submitting patchs and having their check-patch code run, so we try to keep it to a minimum.
Sandro Bonazzola November 3, 2017 at 10:35 AM
I think that the source directory should be removed after the job execution, not git cleaned.
I'm pretty sure in the past it was completely removed from the slave.
Is there any good reason for not cleaning the slave after job completion?
Eyal Edri October 31, 2017 at 2:40 PM
Is there something to be done here if its done outside of the mock env? should the scripts change in build-artifacts?
Barak Korren October 16, 2017 at 12:03 PM
Not sure what is $PWD for the git command in the log above. If you are creating files in the source directory, it is not part of the chroot, it is bind mounted.
We do have git clean running on the clone directories, but it has its own rules.
Sandro Bonazzola October 16, 2017 at 11:15 AM
2017-10-16 13:09 GMT+02:00 eyal edri (oVirt JIRA) <
Yes, I'm sure it hasn't been created during the same instance of the job.
As a workaround I changed the code for deleting the directory if found
there but yes, having a dirty chroot is a risk.
–
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
<http://www.teraplan.it/redhat-osd-2017/>
Eyal Edri October 16, 2017 at 11:09 AM
You mean that because it found an existing dir, the mock wasn't cleaned properly?
We do have cache for RPMs, but I agree other files shouldn't exists from previous runs, are we positive this wasn't created during the same job?
Details
Assignee
infrainfraReporter
Sandro BonazzolaSandro BonazzolaPriority
Medium
Details
Details
Assignee
infraReporter
Sandro BonazzolaPriority
Medium
Example:
http://jenkins.ovirt.org/job/ovirt-wgt_master_check-patch-fc26-x86_64/11/console
09:44:06 + ./automation/build-artifacts.sh*09:44:06* + [[ -d
exported-artifacts ]]09:44:06 + mkdir -p
exported-artifacts*09:44:06* + [[ -d tmp.repos/SOURCES ]]09:44:06 +
mkdir -p tmp.repos/SOURCES*09:44:06* + git clone
git://anongit.freedesktop.org/spice/spice-nsis*09:44:06* fatal:
destination path 'spice-nsis' already exists and is not an empty
directory.
jobs should run in clean chroots.
–
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
<http://www.teraplan.it/redhat-osd-2017/>