Hudson fails with error "No space left on device" though there is enough space on the system
Hudson is repeatedly failing after building few projects with following stack trace with error "No space left on device", though there is enough space on disk. There are no limits of quotas on any folder. Below is output of different system commands.
Here is key system information:
Hudson ver. 1.361
executable-war /opt/hudson/hudson.war
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.6.0_18-b18
os.name Linux-Ubuntu 10.04
os.version 2.6.32-19-generic
There is 50% free space according to df
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 147550696 67382688 72672840 49% /
none 1535580 304 1535276 1% /dev
none 1539732 504 1539228 1% /dev/shm
none 1539732 96 1539636 1% /var/run
none 1539732 0 1539732 0% /var/lock
none 1539732 0 1539732 0% /lib/init/rw
none 147550696 67382688 72672840 49% /var/lib/ureadahead/debugfs
next i checked open file handles by lsof, thats also within limit
lsof | wc -l
694
then i checked file handles by this command
cat /proc/sys/fs/file-nr
3392 0 306935
Detailed Error
[INFO] No tests to run.
[HUDSON] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] [antrun:run {execution: default}]
[INFO] Executing tasks
[unzip] Expanding: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources/CRMServices.Driver-soapui-project.zip into /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/pom.xml to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/pom.xml
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/eif.deployment.CRMServices-1.2-bt.jar
[HUDSON] Re-archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to serialize hudson.model.Actionable#actions for class hudson.maven.MavenModuleSetBuild
No space left on device
[INFO] ------------------------------------------------------------------------
[INFO] Trace
java.langchannel stopped
ERROR: Failed to parse POMs
java.io.IOException: Remote call on Channel to Maven [/opt/bea/jdk160_05/bin/java, -cp, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.363.jar:/opt/hudson/tools/Maven_2.2.1/boot/classworlds-1.1.jar, hudson.maven.agent.Main, /opt/hudson/tools/Maven_2.2.1, /opt/hudson/war/WEB-INF/lib/remoting-1.363.jar, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.363.jar, 55951, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven2.1-interceptor-1.2.jar] failed
at hudson.remoting.Channel.call(Channel.java:564)
at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:156)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:483)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1253)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:127)
Caused by: java.lang.Error: Unable to load resource hudson/maven/Messages.properties
at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:198)
at java.lang.ClassLoader.getResource(ClassLoader.java:977)
at java.lang.Class.getResource(Class.java:2074)
at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:83)
at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:139)
at hudson.maven.Messages.MavenBuilder_AsyncFailed(Messages.java:233)
at hudson.maven.MavenBuilder.call(MavenBuilder.java:184)
at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:696)
at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:640)
at hudson.remoting.UserRequest.perform(UserRequest.java:114)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at hudson.remoting.RemoteClassLoader.makeResource(RemoteClassLoader.java:267)
at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:194)
... 19 more
FATAL: : No space left on device
hudson.util.IOException2: : No space left on device
at hudson.XmlFile.write(XmlFile.java:168)
at hudson.model.Run.save(Run.java:1383)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
at hudson.model.Run.run(Run.java:1276)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:127)
Caused by: com.thoughtworks.xstream.io.StreamException: : No space left on device
at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:73)
at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:99)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:840)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:829)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:804)
at hudson.XmlFile.write(XmlFile.java:165)
... 7 more
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(FileOutputStream.java)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at java.io.BufferedWriter.flush(BufferedWriter.java:236)
at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:91)
at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:49)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
at hudson.XmlFile.write(XmlFile.java:165)
at hudson.model.Run.save(Run.java:1384)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
Answer
The "no space left on device" error isn't necessary caused by running out of storage capacity, as it suggests, it can also be cause by running out of i-nodes on the filesystem. In other words, a given filesystem can only contain so many files. Running df will suggest everything's fine.
See this article for more info.
You either need to delete some files that you don't need any more, or put Hudson on a different filesystem.
It's not uncommon in situations where yo know you'll have a lot of small files to build a filesystem with an explicitly larger inode table.