All builds on vm0145.workers-phx.ovirt.org (fc28 phx nested) fail with: Error fetching remote repo 'origin'

Description

Looks like all builds on Fedora 28 fail with this error now:

-
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/3457/pipeline
vm0145.workers-phx.ovirt.org (fc28 phx nested)

-
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/3455/pipeline/101
vm0145.workers-phx.ovirt.org (fc28 phx nested)

-
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/3461/pipeline
vm0145.workers-phx.ovirt.org (fc28 phx nested)

https://jenkins.ovirt.org/job/vdsm_standard-check-patch/buildTimeTrend

For some reason we always get the same slave for the py27 build.


No credentials specified

Fetching changes from the remote Git repository

Cleaning workspace

ERROR: Error fetching remote repo 'origin'

hudson.plugins.git.GitException: Failed to fetch from
https://gerrit.ovirt.org/vdsm

at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:904)

at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1144)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1175)

at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)

at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)

at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)

at
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: hudson.plugins.git.GitException: Command "git clean -fdx"
returned status code 1:

stdout:

stderr: warning: failed to remove
.tox/lib-py27/lib/python2.7/site-packages/pytest_cov/_pycache_/plugin.cpython-27-PYTEST.pyc:
Permission denied

warning: failed to remove
.tox/lib-py27/lib/python2.7/site-packages/pytest_cov/_pycache_/compat.cpython-27-PYTEST.pyc:
Permission denied

warning: failed to remove
.tox/lib-py27/lib/python2.7/site-packages/pytest_cov/_pycache_/engine.cpython-27-PYTEST.pyc:
Permission denied

warning: failed to remove
.tox/lib-py27/lib/python2.7/site-packages/pytest_cov/_pycache_/embed.cpython-27-PYTEST.pyc:
Permission denied

warning: failed to remove
tests/_pycache_/pywatch_test.cpython-27-PYTEST.pyc: Permission denied

warning: failed to remove
tests/_pycache_/prlimit_test.cpython-27-PYTEST.pyc: Permission denied

warning: failed to remove tests/_pycache_/ssl_test.cpython-27-PYTEST.pyc:
Permission denied

warning: failed to remove
tests/_pycache_/hooking_test.cpython-27-PYTEST.pyc: Permission denied

warning: failed to remove
tests/common/_pycache_/systemd_test.cpython-27-PYTEST.pyc: Permission
denied

warning: failed to remove
tests/common/_pycache_/systemctl_test.cpython-27-PYTEST.pyc: Permission
denied

warning: failed to remove
tests/common/_pycache_/commands_test.cpython-27-PYTEST.pyc: Permission
denied

at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)

at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2248)

at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2244)

at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1777)

at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1789)

at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:901)

at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:912)

at hudson.plugins.git.GitAPI.clean(GitAPI.java:311)

at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)

at
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)

at
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)

at hudson.remoting.UserRequest.perform(UserRequest.java:212)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:369)

at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)

... 4 more

Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
vm0145.workers-phx.ovirt.org

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)

at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)

at hudson.remoting.Channel.call(Channel.java:955)

at
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)

at com.sun.proxy.$Proxy96.clean(Unknown Source)

at
org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:463)

at
hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:30)

at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:900)

at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1144)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1175)

at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)

at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)

at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)

at
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

... 4 more

> git rev-parse --is-inside-work-tree # timeout=10

> git config remote.origin.url https://gerrit.ovirt.org/vdsm # timeout=10

> git rev-parse --verify HEAD # timeout=10

Resetting working tree

> git reset --hard # timeout=10

> git clean -fdx # timeout=10

Error fetching remote repo 'origin'

Activity

Show:

Former user March 6, 2019 at 8:22 AM

removed the workspace, bringing the slave back online.

Eyal Edri March 5, 2019 at 1:36 PM

you can also just remove the workspace or chown to jenkins user.

Anton Marchukov March 5, 2019 at 1:09 PM

I guess we can just reprovision the slave when convenient. The reason is found and the slave is disabled. Not urgent.

Anton Marchukov March 5, 2019 at 12:37 PM

Disabling slaves in jenkins does not prevent finding the solution, but prevents further failures in case it is specific to single slave. Assigning to current infra owner.

Barak Korren March 4, 2019 at 9:30 AM
Edited

This is not a real clone failure - the issue is this:

Jenkins is failing to run git clean -fdx because root-owned files were left in the workspace - this ususally happens when someone manually aborts mock_runner.sh.

I think you're the infra owner - please clean up the workspaces on the node and restore it to working condition.

please make an effort to avoid offlining slaves without following up with a solution

Anton Marchukov March 4, 2019 at 8:30 AM

I have disabled this node for now till this is being investigated.

Fixed

Details

Assignee

Reporter

Priority

Created March 3, 2019 at 3:19 PM
Updated August 29, 2019 at 2:12 PM
Resolved March 6, 2019 at 8:23 AM