Jmeter vs LoadRunner in terms of vusers
I have found contradicting information, one saying JMeter can produce much more load then LR can, the other saying the opposite. From what I know (if we do not consider licencing), each LoadGenerator is only limited by hardware. But so is JMeter. The documentation did not help me much. Does anyone have an experience with both of these so he can compare? I am speaking about 2 000- 4000 users. Thanks
Answer
LoadRunner is known to run well with very high volume tests, as is, out of the box.
JMeter can typically hit issues with high throughput, high threaded tests in the following scenarios:
- Using one machine, with lots of listeners, running in GUI mode - this eats memory.
- Using distributed mode in default configuration with versions < 2.9 where there is not a problem to run the test on the Load Generator but there was a bottleneck sending results to the Master machine. This issue is reported to have been solved in 2.9 and throughput is claimed to be higher in 2.10.
The thing is, it's not that hard to solve JMeter's problems. It's simply a matter of best practice.
- Run from the command line and don't use lots of listeners. Lean and Mean mode.
- In distributed execution, use batch mode to reduce the volume of samples being written to one file in versions < 2.9 or use default configurations of >= 2.9.
- Make sure you distribute the test over sufficient hardware. This is the same for LoadRunner by the way.
You should read those 2 documents for other best-practices:
- http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
- http://jmeter.apache.org/usermanual/best-practices.html
LoadRunner also has issues at high load - the Analysis and data collation phases can take hours (literally) and you can't get around this. If you have too much data to analyze you can also run into memory issues. Jmeter is not as comprehensive at results analysis but it is much quicker.
If you really need high volume tests then I wrote a script that effectively gives you infinite scalability with JMeter - I've tested it up to 20000 users making 8000 hits a second running over 50 servers. It's 'infinite' because it works by running lots of isolated tests that do not talk to each other until the end of the test, that way there is no bottleneck with compiling results. But there's always another bottleneck somewhere...