RabbitMQ error timeout
I've set up RabbitMQ in order to parse some 20.000 requests from an external API but it keeps timing out after a few minutes. It does get to correctly parse about 2000 out of the total 20.000 requests.
The log file says:
=INFO REPORT==== 16-Feb-2016::17:02:50 ===
accepting AMQP connection <0.1648.0> (127.0.0.1:33091 -> 127.0.0.1:5672)
=ERROR REPORT==== 16-Feb-2016::17:03:21 ===
closing AMQP connection <0.1648.0> (127.0.0.1:33091 -> 127.0.0.1:5672):
{writer,send_failed,{error,timeout}}
I've already increased the heartbeat value but I cannot figure out why it's timing out. Configuration is: Ubuntu 14.04, NGINX 1.8.1, RabbitMQ 3.6.0
I'd appreciate your time and input !
Answer
I've just solved a similar problem in python. In my case, it was solved by reducing the prefetch count on the consumer, so that it had fewer messages queued up in its receive buffer.
My theory is that the receive buffer on the consumer gets full, and then RMQ tries to write some other message to the consumer's socket and can't due to the consumer's socket being full. RMQ blocks on this socket, and eventually timeouts and just closes the connection on the consumer. Having a smaller prefetch queue means the socket receive buffer doesn't get filled, and RMQ is able to write whatever bookkeeping messages it was trying to do and so doesn't timeout on its writes nor close the connection.
This is just a theory though, but it seems to hold in my testing.
In Python, setting the prefetch count can be done like so:
subChannel.basicQos(10);
(Thanks to @shawn-guo for reminding me to add this code snippet)