Using JUnit as an acceptance test framework
OK, so I work for a company who has openly adopted agile practices for development in recent years. Our unit tests and code quality are improving. One area we still are working on is to find what works best for us in the automated acceptance test arena. We want to take our well formed user stories and use these to drive the code in a test driven manner. This will also give us acceptance level tests for each user story which we can then automate.
To date, we've tried Fit, Fitnesse and Selenium. Each have their advantages, but we've also had real issues with them as well. With Fit and Fitnesse, we can't help but feel they overcomplicate things and we've had many technical issues using them. The business haven't fully bought in these tools and aren't particularly keen on maintaining the scripts all the time (and aren't big fans of the table style). Selenium is really good, but slow and relies on real time data and resources.
One approach we are now considering is the use of the JUnit framework to provide similiar functionality. Rather than testing just a small unit of work using JUnit, why not use it to write a test (using the JUnit framework) to cover an acceptance level swath of the application? I.e. take a new story ("As a user I would like to see basic details of my policy...") and write a test in JUnit which starts executing application code at the point of entry for the policy details link but covers all code and logic down to the stubbed data access layer and back to the point of forwarding to the next page in the application, asserting on what data the user should see on that page.
This seems to me to have the following advantages:
- Simplicity (no additional frameworks required)
- Zero effort to integrate with our Continuous Integration build server (since it already handles our JUnit tests)
- Full skillset already present in the team (its just a JUnit test after all)
And the downsides being:
- Less customer involvement (though they are heavily involved in writing the user stories in the first place from which the acceptance tests will be written)
- Perhaps more difficult to understand (or make understood) the user story and acceptance criteria in a JUnit class verses a freetext specification ala Fit or Fitnesse
So, my question is really, have you ever tried this method? Ever considered it? What are your thoughts? What do you like and dislike about this approach? Finally, please only mention alternative frameworks if you can say why you like or dislike them more than this approach.
Answer
My experience using JUnit for an Acceptance Testing framework.
Our company did a very similar thing where we started with FitNesse and then went to JUnit. We did this mainly because FitNesse was sitting on top of Java already, so switching to JUnit was easy. FitNesse seemed to not fit (no pun intended) our needs.
Here are my pros and cons of using JUnit:
Pros:
- JUnit has a lot of flexibility (because it's Java), so it's relatively easy to create an Internal Domain Specific Language[Fowler]. The DSL can be modified so that it's easy for domain experts (not software engineers) to read/write.
- Since it's JUnit you can use Eclipse as an IDE, which gives you auto-complete, syntax checking, debugging, etc.
- Specific for our application, we have a CORBA interface to the UI. Java can easily use this CORBA interface as well.
Cons:
- People hear JUnit and think unit test. This gets confusing when people start referring to the acceptance tests as "JUnit tests".
- To provide a simplified testing environment you have to spend some time extending the framework and developing your DSL.
So in summary, JUnit works just fine for Acceptance Tests if you are willing to spend the time to extend it into you own domain specific testing environment. Otherwise I think I would look elsewhere for something more out of the box.