SocketException Broken pipe in Solr/tomcat

java tomcat socketexception solr4
Muthu kader picture Muthu kader · Sep 19, 2013 · Viewed 10.1k times · Source

I'm using Solr 4.3 with tomcat7. When the number of concurrent writes and reads increase it is crash throwing this SocketException error. I do concurrent commits and searches through multiple clients. Any kind of help is appreciated. Here is the tomcat log(catalina.out) and solr.log

 tomcat log(catalina.out):

    59887611 [http-bio-8080-exec-988] ERROR org.apache.solr.servlet.SolrDispatchFilter  – null:ClientAbortException:  java.net.SocketException: Broken pipe
     at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
     at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
     at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:431)
     at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:419)
     at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
     at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
     at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
     at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
     at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
     at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
     at org.apache.solr.util.FastWriter.write(FastWriter.java:126)
     at java.io.Writer.write(Writer.java:157)
     at org.apache.solr.response.XMLWriter.startTag(XMLWriter.java:144)
     at org.apache.solr.response.XMLWriter.writePrim(XMLWriter.java:347)
     at org.apache.solr.response.XMLWriter.writeLong(XMLWriter.java:305)
     at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:139)
     at org.apache.solr.schema.TrieField.write(TrieField.java:221)
     at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:130)
     at org.apache.solr.response.XMLWriter.writeSolrDocument(XMLWriter.java:199)
     at org.apache.solr.response.TextResponseWriter.writeDocuments(TextResponseWriter.java:275)
     at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:172)
     at org.apache.solr.response.XMLWriter.writeResponse(XMLWriter.java:111)
     at org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:39)
     at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:644)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:724)
    Caused by: java.net.SocketException: Broken pipe
     at java.net.SocketOutputStream.socketWrite0(Native Method)
     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
     at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
     at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
     at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
     at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
     at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
     at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119)
     at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
     at org.apache.coyote.Response.doWrite(Response.java:517)
     at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:401)
     ... 39 more


         solr.log:



    2013-09-19 05:32:13.978; org.apache.solr.common.SolrException; null:ClientAbortException:  java.net.SocketException: Broken pipe
     at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
     at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
     at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:431)
     at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:419)
     at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
     at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
     at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
     at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
     at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
     at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
     at org.apache.solr.util.FastWriter.write(FastWriter.java:126)
     at java.io.Writer.write(Writer.java:157)
     at org.apache.solr.response.XMLWriter.startTag(XMLWriter.java:144)
     at org.apache.solr.response.XMLWriter.writePrim(XMLWriter.java:347)
     at org.apache.solr.response.XMLWriter.writeLong(XMLWriter.java:305)
     at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:139)
     at org.apache.solr.schema.TrieField.write(TrieField.java:221)
     at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:130)
     at org.apache.solr.response.XMLWriter.writeSolrDocument(XMLWriter.java:199)
     at org.apache.solr.response.TextResponseWriter.writeDocuments(TextResponseWriter.java:275)
     at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:172)
     at org.apache.solr.response.XMLWriter.writeResponse(XMLWriter.java:111)
     at org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:39)
     at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:644)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:724)
    Caused by: java.net.SocketException: Broken pipe
     at java.net.SocketOutputStream.socketWrite0(Native Me...

Answer

lcestari picture lcestari · Sep 19, 2013

This is more low level error. I found this question&answers in stackoverflow and the summary of the solutions is to verify if you are really closing all the sockets (it would be indirectly due the framework) and check if it doesn't have other exceptions swallowed (like it needs more memory).

Related questions

ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error

I am getting the following error frequently while retrieving file object from database column. How can I resolve this problem? May 8, 2009 3:18:14 PM org.apache.catalina.core.StandardHostValve status WARNING: Exception Processing ErrorPage[errorCode=404, location=/error.jsp] ClientAbortException: java.net.SocketException: …

Read More
java.net.SocketTimeoutException: Read timed out

I have an application with client server architecture. The client use Java Web Start with Java Swing / AWT and the sert uses HTTP server / Servlet with Tomcat. The communication is made from the serialization of objects, create a ObjectOutput serializes …

Read More
How to stop printing exception stack trace on Console?

I wrote a servlet to handle the exceptions occurring in my web app and mapped them in web.xml <error-page> <exception-type>java.lang.Exception</exception-type> <location>/exceptionHandler</location> </error-page&…

Read More