Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Java updates without restarting Jenkins agent are probably causing jobs to fail

Description

A recently invoked system teasts job failed with the following error:

Looking at el7-vm26.phx.ovirt.org failed where this happened proved that indeed the file '/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64/jre/lib/resources.jar' does not exist on the node. The file '/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/resources.jar' does exist however.

Further examination in the logs revealed that the system's java version was recently updated:

I suppose the issue is does to the system Jave and libraries being updated while the Jenkins agent was still running and trying to use the older libraries.

I suggest we endure that the Jenkins agent is shut down every time package updates are done on the node and restarted afterwards. This could be done by setting the node to "offline" in Jenkins.

Activity

Show:

Former user February 28, 2017 at 4:24 PM

patch merged, closing

Former user January 24, 2017 at 1:24 PM

patch submitted for review

Barak Korren January 24, 2017 at 12:02 PM

This is probably a relic from the time we didn't have enough manpower to do the updates properly...
Can probably change to "ensure => installed" now.

Former user January 24, 2017 at 9:08 AM

Here's the manifest that caused this:
https://gerrit.ovirt.org/gitweb?p=infra-puppet.git;a=blob;f=site/ovirt_jenkins_slave/manifests/base.pp;h=4ec794b36b1a619fa230612bba6feb5d0eb98443;hb=refs/heads/production#l99

do you know any background of why this is set to "ensure => latest" ? As this seems to cause troubles I propose changing this to "ensure => installed" and apply updates manually while slaves are offline to avoid this. Same goes for Fedora a few lines above.

Former user January 23, 2017 at 2:51 PM

All el7 slaves rebooted to ensure the agent starts with the right java version.

Fixed

Details

Assignee

Reporter

Priority

Parent

Created January 22, 2017 at 3:34 PM
Updated April 2, 2017 at 12:51 PM
Resolved February 28, 2017 at 4:24 PM