Why am i seeing lots of sockets in CLOSE_WAIT status when webservice stops working?

java tomcat tcp jetty load-balancing
Paul Taylor picture Paul Taylor · Mar 5, 2015 · Viewed 33.7k times · Source

My java webservice running on Jetty falls over after a period of a few hours and investigation indicate many sockets in CLOSE_WAIT status. Whilst it is working ok there seems to be no sockets in CLOSE_WAIT status but when it goes wrong there are loads.

I found this definition

CLOSE-WAIT: The local end-point has received a connection termination request and acknowledged it e.g. a passive close has been performed and the local end-point needs to perform an active close to leave this state.

With netstat on my server I see a list of tcp sockets in CLOSE_WAIT status, the local address is my server and the foreign address my load balancer machine. So I assume this means the client (load balancer) has just terminated the connection at its end in some improper way, and my server has not properly closed the connection at its end.

But how do I do that, my Java code doesn't deal with low level sockets ?

Or is the load-balancer terminating connection because of an earlier problem caused by something my server is doing wrong in the code.

Answer

Eirenliel picture Eirenliel · Mar 11, 2015

Sounds like a bug in Jetty or JVM, maybe this workaround will work for you: http://www.tux.hk/index.php?entry=entry090521-111844

Add the following lines to /etc/sysctl.conf

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 1800

And then execute

sysctl -p

or do a reboot

Related questions

How to deploy a war file in Tomcat 7

I have copied the sample.war file into the webapps directory of Tomcat, and I can access localhost:8080. Now how will Tomcat deploy it, I mean do I need to open it in browser? How can I access the application?

Read More
Several ports (8005, 8080, 8009) required by Tomcat Server at localhost are already in use

I'm getting the following error when I try to run a simple JSP program on Tomcat in Eclipse. Several ports (8005, 8080, 8009) required by Tomcat v6.0 Server at localhost are already in use. The server may already be running in another process, …

Read More
Tomcat 404 error: The origin server did not find a current representation for the target resource or is not willing to disclose that one exists

I was following the tutorial on o7planning and got stuck at step 6: http://o7planning.org/en/10169/java-servlet-tutorial It's just a simple project that show HelloWorld but for some reason I keep getting 404 error. Detail: However the Tomcat welcome …

Read More