Best practice for debug Asserts during Unit testing
Does heavy use of unit tests discourage the use of debug asserts? It seems like a debug assert firing in the code under test implies the unit test shouldn't exist or the debug assert shouldn't exist. "There can be only one" seems like a reasonable principle. Is this the common practice? Or do you disable your debug asserts when unit testing, so they can be around for integration testing?
Edit: I updated 'Assert' to debug assert to distinguish an assert in the code under test from the lines in the unit test that check state after the test has run.
Also here is an example that I believe shows the dilema: A unit test passes invalid inputs for a protected function that asserts it's inputs are valid. Should the unit test not exist? It's not a public function. Perhaps checking the inputs would kill perf? Or should the assert not exist? The function is protected not private so it should be checking it's inputs for safety.
Answer
This is a perfectly valid question.
First of all, many people are suggesting that you are using assertions wrongly. I think many debugging experts would disagree. Although it is good practice to check invariants with assertions, assertions shouldn't be limited to state invariants. In fact, many expert debuggers will tell you to assert any conditions that may cause an exception in addition to checking invariants.
For example, consider the following code:
if (param1 == null)
throw new ArgumentNullException("param1");
That's fine. But when the exception is thrown, the stack gets unwound until something handles the exception (probably some top level default handler). If execution pauses at that point (you may have a modal exception dialog in a Windows app), you have the chance to attach a debugger, but you have probably lost a lot of the information that could have helped you to fix the issue, because most of the stack has been unwound.
Now consider the following:
if (param1 == null)
{
Debug.Fail("param1 == null");
throw new ArgumentNullException("param1");
}
Now if the problem occurs, the modal assert dialog pops up. Execution is paused instantaneously. You are free to attach your chosen debugger and investigate exactly what's on the stack and all the state of the system at the exact point of failure. In a release build, you still get an exception.
Now how do we handle your unit tests?
Consider a unit test that tests the code above that includes the assertion. You want to check that the exception is thrown when param1 is null. You expect that particular assertion to fail, but any other assertion failures would indicate that something is wrong. You want to allow particular assertion failures for particular tests.
The way you solve this will depend upon what languages etc. you're using. However, I have some suggestions if you are using .NET (I haven't actually tried this, but I will in the future and update the post):
- Check Trace.Listeners. Find any instance of DefaultTraceListener and set AssertUiEnabled to false. This stops the modal dialog from popping up. You could also clear the listeners collection, but you'll get no tracing whatsoever.
- Write your own TraceListener which records assertions. How you record assertions is up to you. Recording the failure message may not be good enough, so you may want to walk the stack to find the method the assertion came from and record that too.
- Once a test ends, check that the only assertion failures that occured were the ones you were expecting. If any others occurred, fail the test.
For an example of a TraceListener that contains the code to do a stack walk like that, I'd search for SUPERASSERT.NET's SuperAssertListener and check its code. (It's also worth integrating SUPERASSERT.NET if you're really serious about debugging using assertions).
Most unit test frameworks support test setup/teardown methods. You may want to add code to reset the trace listener and to assert that there aren't any unexpected assertion failures in those areas to mimimize duplication and prevent mistakes.
UPDATE:
Here is an example TraceListener that can be used to unit test assertions. You should add an instance to the Trace.Listeners collection. You'll probably also want to provide some easy way that your tests can get hold of the listener.
NOTE: This owes an awful lot to John Robbins' SUPERASSERT.NET.
/// <summary>
/// TraceListener used for trapping assertion failures during unit tests.
/// </summary>
public class DebugAssertUnitTestTraceListener : DefaultTraceListener
{
/// <summary>
/// Defines an assertion by the method it failed in and the messages it
/// provided.
/// </summary>
public class Assertion
{
/// <summary>
/// Gets the message provided by the assertion.
/// </summary>
public String Message { get; private set; }
/// <summary>
/// Gets the detailed message provided by the assertion.
/// </summary>
public String DetailedMessage { get; private set; }
/// <summary>
/// Gets the name of the method the assertion failed in.
/// </summary>
public String MethodName { get; private set; }
/// <summary>
/// Creates a new Assertion definition.
/// </summary>
/// <param name="message"></param>
/// <param name="detailedMessage"></param>
/// <param name="methodName"></param>
public Assertion(String message, String detailedMessage, String methodName)
{
if (methodName == null)
{
throw new ArgumentNullException("methodName");
}
Message = message;
DetailedMessage = detailedMessage;
MethodName = methodName;
}
/// <summary>
/// Gets a string representation of this instance.
/// </summary>
/// <returns></returns>
public override string ToString()
{
return String.Format("Message: {0}{1}Detail: {2}{1}Method: {3}{1}",
Message ?? "<No Message>",
Environment.NewLine,
DetailedMessage ?? "<No Detail>",
MethodName);
}
/// <summary>
/// Tests this object and another object for equality.
/// </summary>
/// <param name="obj"></param>
/// <returns></returns>
public override bool Equals(object obj)
{
var other = obj as Assertion;
if (other == null)
{
return false;
}
return
this.Message == other.Message &&
this.DetailedMessage == other.DetailedMessage &&
this.MethodName == other.MethodName;
}
/// <summary>
/// Gets a hash code for this instance.
/// Calculated as recommended at http://msdn.microsoft.com/en-us/library/system.object.gethashcode.aspx
/// </summary>
/// <returns></returns>
public override int GetHashCode()
{
return
MethodName.GetHashCode() ^
(DetailedMessage == null ? 0 : DetailedMessage.GetHashCode()) ^
(Message == null ? 0 : Message.GetHashCode());
}
}
/// <summary>
/// Records the assertions that failed.
/// </summary>
private readonly List<Assertion> assertionFailures;
/// <summary>
/// Gets the assertions that failed since the last call to Clear().
/// </summary>
public ReadOnlyCollection<Assertion> AssertionFailures { get { return new ReadOnlyCollection<Assertion>(assertionFailures); } }
/// <summary>
/// Gets the assertions that are allowed to fail.
/// </summary>
public List<Assertion> AllowedFailures { get; private set; }
/// <summary>
/// Creates a new instance of this trace listener with the default name
/// DebugAssertUnitTestTraceListener.
/// </summary>
public DebugAssertUnitTestTraceListener() : this("DebugAssertUnitTestListener") { }
/// <summary>
/// Creates a new instance of this trace listener with the specified name.
/// </summary>
/// <param name="name"></param>
public DebugAssertUnitTestTraceListener(String name) : base()
{
AssertUiEnabled = false;
Name = name;
AllowedFailures = new List<Assertion>();
assertionFailures = new List<Assertion>();
}
/// <summary>
/// Records assertion failures.
/// </summary>
/// <param name="message"></param>
/// <param name="detailMessage"></param>
public override void Fail(string message, string detailMessage)
{
var failure = new Assertion(message, detailMessage, GetAssertionMethodName());
if (!AllowedFailures.Contains(failure))
{
assertionFailures.Add(failure);
}
}
/// <summary>
/// Records assertion failures.
/// </summary>
/// <param name="message"></param>
public override void Fail(string message)
{
Fail(message, null);
}
/// <summary>
/// Gets rid of any assertions that have been recorded.
/// </summary>
public void ClearAssertions()
{
assertionFailures.Clear();
}
/// <summary>
/// Gets the full name of the method that causes the assertion failure.
///
/// Credit goes to John Robbins of Wintellect for the code in this method,
/// which was taken from his excellent SuperAssertTraceListener.
/// </summary>
/// <returns></returns>
private String GetAssertionMethodName()
{
StackTrace stk = new StackTrace();
int i = 0;
for (; i < stk.FrameCount; i++)
{
StackFrame frame = stk.GetFrame(i);
MethodBase method = frame.GetMethod();
if (null != method)
{
if(method.ReflectedType.ToString().Equals("System.Diagnostics.Debug"))
{
if (method.Name.Equals("Assert") || method.Name.Equals("Fail"))
{
i++;
break;
}
}
}
}
// Now walk the stack but only get the real parts.
stk = new StackTrace(i, true);
// Get the fully qualified name of the method that made the assertion.
StackFrame hitFrame = stk.GetFrame(0);
StringBuilder sbKey = new StringBuilder();
sbKey.AppendFormat("{0}.{1}",
hitFrame.GetMethod().ReflectedType.FullName,
hitFrame.GetMethod().Name);
return sbKey.ToString();
}
}
You can add Assertions to the AllowedFailures collection at the start of each test for the assertions you expect.
At the end of every test (hopefully your unit testing framework supports a test teardown method) do:
if (DebugAssertListener.AssertionFailures.Count > 0)
{
// TODO: Create a message for the failure.
DebugAssertListener.ClearAssertions();
DebugAssertListener.AllowedFailures.Clear();
// TODO: Fail the test using the message created above.
}