Assert.Inconclusive and IgnoreAttribute

Question

What is the right way to use Assert.Inconclusive and IgnoreAttribute in MS Unit test framework?

We are using Assert.Inconclusive mainly for tests which are:

• Not implemented yet
• Somehow broken or incomplete = requires futher attention
• When test body is for any reason commented out

We are doing this because:

• Inconclusive test can have message
• We want to see such tests in test results on TFS

Our problem is that Inconclusive tests are marked as error in both TFS and Resharper. If we use IgnoreAttribute instead we will see these tests in Resharper but MS Test runner and TFS will ignore them at all. Using IgnoreAttribute in TFS and MS Test runner is same like commenting whole test which is useless.

Answer 1

i also see a dilemma in the current implementation.

• Inconclusive assertions are included in the TRX report, but mstest.exe (and also vstest.console.exe) will return 1 (meaning error) after execution.
• TestMethods with the Ignore attribute will not be reported as an error, but they are completely hidden from the TRX report.

my personal understanding is as follows:

use the [Ignore] attribute to (temporarily) disable / skip the method:

[TestMethod]
[Ignore] // <== disabled through "Ignore" attribute
public void Test001()
{
   //execute some stuff ...
   Assert.IsTrue(...);

   //execute some stuff ...
   Assert.AreEqual(...);
}

do not misuse the Inconclusive assertion for this purpose:

[TestMethod]
public void Test002()
{
    Assert.Inconclusive(); // <== misuse of "Inconclusive" to skip this test

    //execute some stuff ...
}

instead, Inconclusive should be used conditionally: only if we can't tell whether the component to be tested does work as expected or not.
for example in case an external resource we depend on is not available at time of test execution:

[TestMethod]
public void Test003()
{
    //check if the server is running,
    //otherwise can can't test our local client component!
    if (!WebServiceAvailable())
    {
        Assert.Inconclusive(); // <== skip remaining code because the resource is not available
    }

    //execute some stuff ...
    Assert.AreEqual(...);

    //execute some stuff ...
    Assert.AreEqual(...);
}

_ _

conclusion:
to disable / skip a test the logical way is to use the [Ignore] attribute.
i clearly see the current behaviour of mstest.exe not reporting on any ignored test as a bug that should be fixed.

feel free to up-vote the following bug reports:

• https://connect.microsoft.com/VisualStudio/feedback/details/779743/
• http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2088845