Have you used Quickcheck in a real project
Quickcheck and its variants (even there is one in Java), seems to be interesting. However, apart from academic interest, is it really useful in a real application testing (Eg. a GUI application or Client/Server or even take StackOverflow itself)? Any experiences you had with similar test generators is appreciated.
Answer
Yes, well. Actually no, but I've studied under the man who originally developed QuickCheck and he's a really interesting guy.
Back in 2004, we were forced to use QuickCheck to test our Haskell programs and it was combination of good and bad. Mostly bad because Haskell was a bit daunting itself but never the less wonderful when you got it working.
John has since then perfected that which he wrote years back and actually helped Ericssion test their complex telecom hardware, and he found bugs in 20 millions or so lines of code reducing that to a mere three steps through his approach. He's a great speaker so it's always a joy listening to him present what he does so well, but all in all, what he did with QuickCheck was new to me. So I asked him, what his interest was in bringing this to the market. He was open to the idea, but at the time his business (based around QuickCheck) was relatively new and so there were other areas he would focus on. This is now 2007. My point is, you could learn from QuickCheck even if you wont end up using it.
But what is QuickCheck? It's a a combinatorial testing framework and an interesting way to test programs. The people over at Microsoft Research has built Pex which is sort of similar. Pex generates tests automatically by examining your IL. However, John would write a generator for possible input and test properties of a function. A property is something which can easily be tested and it's a lot more formal. e.g. reversing a list? Well, reversing a list, is the same thing as splitting a list in two halves, reversing them each individually and then concatenating the two reversed halves in reverse order.
1,2,3,4 // original
1,2 3,4 // split into A and B
2,1 4,3 // reverse A and B
4,3,2,1 // concat B and A
This is a great property to test with QuickCheck called the specification and the result is quite astonishing.
Pex is nice, but not as cool as QuickCheck, Pex simplifies things, QuickCheck does to but it takes a lot of effort to write a good specification.
The power of QuickCheck is that when it runs into a failure it will reduce the input which caused your test to fail, to the smallest possible form. Leaving you with a detailed description of what progression of state caused your test to fail. In comparison to other testing frameworks which will just try to break your code in a brute force manner.
This is made possible due to how you write your testing specification. QuickCheck relies on pseudo-randomness to invent input and it's because of this, its capable of backtracking and find really small input which does not pass your test.
It's a lot more work to write QuickCheck properties but the end result is better testing. As John himself said, 70% of bugs are caught by unit testing, but it's that other 30% which causes your program to crash. QuickCheck is testing those last 30%.