Why is Jenkins suddenly unable to delete a workspace

jenkins bitbucket hudson
MarkA picture MarkA · Jun 10, 2018 · Viewed 36k times · Source

We have a Jenkins server which has successfully built our code over 200 times - until a couple of days ago.

We are now getting an error to indicate that Jenkins was unable to delete the workspace (full message to follow with identifying elements redacted.)

I have checked through the recent code changes, and can see nothing which may have contributed to this issue, and nothing on that server has changed for weeks.

The stack trace suggests that "the context class hudson.FilePath is missing", but the config has not been changed from the config which worked over 200 times.

Can anybody suggest steps which could be taken to fix this ? 

Started by user <REDACTED>
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://<REDACTED>@bitbucket.org/<REDACTED>/<REDACTED>.git
 > git config remote.origin.url             
https://<REDACTED>@bitbucket.org/<REDACTED>/<REDACTED>.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials <REDACTED>@bitbucket
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen branch in repository origin/temp
Seen 2 remote branches
Obtained code/Jenkinsfile from <REDACTED>
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/<REDACTED>
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://<REDACTED>@bitbucket.org/<REDACTED>/<REDACTED>.git
ERROR: Failed to clean the workspace
java.io.IOException: Unable to delete '/var/lib/jenkins/workspace/<REDACTED>. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
    at hudson.Util.deleteContentsRecursive(Util.java:252)
    at     org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:555)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/workspace/<REDACTED>/code/<REDACTED>-query/target/classes/application.properties: Operation not permitted
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238)
at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260)
at java.nio.file.Files.setPosixFilePermissions(Files.java:2045)
at hudson.Util.makeWritable(Util.java:332)
at hudson.Util.tryOnceDeleteFile(Util.java:292)
at hudson.Util.tryOnceDeleteRecursive(Util.java:383)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.deleteContentsRecursive(Util.java:247)
... 14 more
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Failed to delete workspace
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:558)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to delete '/var/lib/jenkins/workspace/<REDACTED>'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at hudson.Util.deleteContentsRecursive(Util.java:252)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:555)
... 13 more
Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/workspace/<REDACTED>/code/<REDACTED>-query/target/classes/application.properties: Operation not permitted
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:238)
at sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:260)
at java.nio.file.Files.setPosixFilePermissions(Files.java:2045)
at hudson.Util.makeWritable(Util.java:332)
at hudson.Util.tryOnceDeleteFile(Util.java:292)
at hudson.Util.tryOnceDeleteRecursive(Util.java:383)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.tryOnceDeleteRecursive(Util.java:382)
at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:402)
at hudson.Util.deleteContentsRecursive(Util.java:247)
... 14 more
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] step
Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: Error cloning remote repo 'origin'
Finished: FAILURE

Answer

Sudhanshu Patel picture Sudhanshu Patel · Jan 11, 2020

I finally found the solution that explains everything.

Cause: I was using docker within Jenkins and mounted Jenkins work directory in Docker (-v pwd:/code:rw). During runtime my program generate few files that also goes in to Jenkins work directory as it is mounted. but the user is docker root not Jenkins user, because of that Jenkins user is unable to delete those files and cause this error.

Solution: you can also share Jenkins user with docker, So all file created is same as file created by Jenkins itself.

docker run --user "$(id -u):$(id -g)" -i --rm -v /etc/passwd:/etc/passwd:ro  -v `pwd`:/code:rw docker_image:tag

So Adding This basically solve the issue by Using Jenkins user inside docker.

--user "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro

Related questions

How to connect Bitbucket to Jenkins properly

Since about 1 week now, Bitbucket doesn't (?) send a request to my Jenkins server. I've set it all up like this: Endpoint http://username:apitoken@jenkinshost/ username = username in Jenkins apitoken = apitoken connected to the username in Jenkins jenkinshost = my host …

Read More
How do I get Jenkins to build on push to a BitBucket git repository?

I have a git repository hosted on BitBucket, and have set up SSH authentication between the repository and my Jenkins server. I can build on Jenkins manually, but cannot get the Jenkins service on BitBucket to trigger builds. Jenkins configuration: …

Read More
Why is Jenkins failing when fetching from git, while the command line isn't?

All of my Jenkins builds are failing at the git fetch line. It's failing at git fetch --tags --progress [email protected]:ethenwilson/whentoact.git Started by user anonymous Building in workspace /Users/ethen/.jenkins/workspace/Build NikNik > git …

Read More