I am searching for a tool that tests SIP calls. A platform that makes a call from SIP device A to SIP device B and reports results...
Any idea? A simulation platform would be ideal.
thnx, cateof
Answer
There are many solutions. Some more broken than others. Here's a quick summary of what I've found while looking for a base for a proper automated testing solution.
It's ok if you want only single dialog at a time. What doesn't work here is complex solutions where you need to synchronise 2 call legs, do registration, call and presence in the same scenario. If you go this way, you'll end up with running multiple sipp scenarios for each conversation element separately. Sipp also doesn't scale at all for media transfers. Even though it's multithreaded, something stops it from running concurrently - if you look at htop for example, you'll see that sipp never crosses the 100% line. Around 50 media calls it starts to cut audio and take all CPU of the machine.
It can sometimes lose track of what's happening, some packets which don't even belong to the call really, can fail the test. It's got some silly bugs like case-sensitive comparing of the headers.
Ruby-based solution where you have to write your own scenarios in Ruby. It's got its own SIP stack and lots of tests. While it's generally good and handles a lot of complex scenarios nicely, its design is terrible. Bugs are hard to track and after a week I had >10 patches that I needed just to make it do basic stuff. Later I learned that some of the scenarios are just written in a different way, but SIPr developers were not really responsive and it took a lot of time to find it out. Synchronising actions of many agents if a hard problem, since they'd rather use an event-based, but still single-threaded version... it just makes you concentrate too much on "what order can this happen in and do I handle it correctly", rather than writing the actual test.
Commercial solution. Never tested it properly since the basic functionality is missing from the evaluation version and it's hard to spend that much money on something you're not sure works...
Java-based solution reusing Jain-SIP stack. It can do almost any scenario and is fairly good. It tries to make everything non-blocking / action based leading to the same problems SIPr has, but in this case it's trivial to make it parallel / threaded. It has its own share of bugs, so not everything works well in the vanilla package, but most of the stuff is patchable. The developers seem to be busy with other projects, so it's not updated for a long time. If you need transfers, presence, dialog-info, custom messages, RTP handling, etc. - you'll have to write your own modifications to support them. It is not good for performance testing.
If you're a Java-hater like me, it can be used in a simple way from Jython, JRuby or any other JVM language.
In the end, I chose SIPunit as the least broken/evil/unusable solution. It is by no means perfect, but... it works in most cases. If I was doing the project once again with all this knowledge, I'd probably reuse SIPp configurations and try to write my own, sane solution that uses proper threading - but that's at least a ½ year project for one person, to make it good enough for production.