Comparing .NET Testing Frameworks.

xunit/xunit : Community-focused unit testing tool

nunit/nunit : NUnit Framework

microsoft/testfx : MSTest framework and adapter

thomhurst/TUnit : A modern, fast and flexible .NET testing framework

Target audience: A developer switching frameworks :)

Test Suites

Typically test classes and/or methods are decorated with Attributes.

xUnit IClassFixture

Setup in xUnit doesn’t work with Attributes!
xUnit injects the same XUnitTestsFixture to the constructor before running each [Fact].

It’s also possible to do setup/teardown for all tests within classes decorated with a CollectionAttribute with ICollectionFixture<T>, see shared-context for more info.

public class XUnitTestsFixture : IDisposable
{
    public XUnitTestsFixture() { /* BeforeAllTests */ }
    public void Dispose() { /* AfterAllTests */ }
}

public class XUnitTestsWithSetUp : IClassFixture<XUnitTestsFixture>
{
    private readonly XUnitTestsFixture _fixture;

    public XUnitTestsWithSetUp(XUnitTestsFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public void Test1() { }
}

Assembly setup

xUnit

As per the IClassFixture<T> example above, the XUnitTestsFixture will be injected in all test classes in the assembly.

[assembly: AssemblyFixture(typeof(XUnitTestsFixture))]

NUnit

[SetUpFixture]
public class Config
{
    [OneTimeSetUp]
    public void SetUp() { }

    [OneTimeTearDown]
    public void TearDown() { }
}

MSTest

[TestClass]
public class MyTestClass
{
    [AssemblyInitialize]
    public static void AssemblyInitialize(TestContext testContext)
    {
        // Or return Task!
    }

    [AssemblyCleanup]
    public static void AssemblyCleanup()
    {
        // Or return Task!
    }
}

TUnit

public class TUnitSetup
{
  [Before(Assembly)]
  public static async Task AssemblySetup() { }

  [After(Assembly)]
  public static async Task AssemblyTeardown()
  {
    // Runs after the last test in GetType().Assembly
  }

  [AfterEvery(Assembly)]
  public static async Task AllAssembliesTeardown(AssemblyHookContext context)
  {
    // Runs after the last test in all assemblies
    // For all Before/After there exists a Before/AfterEvery
    // Aside from Test/Class/Assembly there also is TestSession/TestDiscovery
  }
}

Assertions

bool? actual = true;

// NUnit
Assert.That(actual, Is.EqualTo(true));

// xUnit
Assert.Equal(true, actual);

// MSTest
Assert.AreEqual(true, actual);

// TUnit
await Assert.That(actual).IsEqualTo(true);

Basic

Strings

All but NUnit use an overload to ignore case for Start/End with.

Exceptions

Action sut = () => throw new Exception("cause");
Func<Task> asyncSut = () => throw new Exception("cause");

// NUnit
var ex = Assert.Throws<Exception>(() => sut(), "failure message");
Assert.That(ex.Message, Is.EqualTo("cause"));

var ex2 = Assert.ThrowsAsync<Exception>(() => asyncSut());
Assert.That(ex2.Message, Is.EqualTo("cause"));


// xUnit
var ex = Assert.Throws<Exception>(sut);
Assert.Equal("cause", ex.Message);

var ex2 = await Assert.ThrowsAsync<Exception>(() => asyncSut());
Assert.Equal("cause", ex2.Message);


// MSTest
// DEPRECATED: ExpectedExceptionAttribute(typeof(Exception))
// DEPRECATED: ThrowsException<T>()
var ex = Assert.ThrowsExactly<Exception>(sut);
Assert.AreEqual("cause", ex.Message);

var ex2 = await Assert.ThrowsExactlyAsync<Exception>(asyncSut);
Assert.AreEqual("cause", ex2.Message);


// TUnit
// Also: .WithMessageMatching and .WithInnerException
await Assert.That(sut).ThrowsExactly<Exception>().WithMessage("cause");

await Assert.That(asyncSut)
  .ThrowsExactly<ArgumentException>()
  .WithMessageMatching(StringMatcher.AsRegex("^cause$"));

Exact vs Derived

The above assertions expect the exceptions to be of exactly the type provided. If you do not want a failure when the actual type is a derived type:

• xUnit: Assert.ThrowsAny(Async)<T>
• NUnit: Assert.Catch(Async)<T>
• MSTest: Assert.Throws(Async)<T>
• TUnit: var ex = Assert.Throws(Async)<T>(sut)

Collections

xUnit Assert.Collection

Specific assertions per element in the collection.

Assert.Collection(
  collection,
  el1 => Assert.Null(el1),
  el2 => Assert.False(el2),
  ...
);

NUnit Dictionary Assertions

var dict = new Dictionary<int, int>
{
  { 1, 4 },
  { 2, 5 }
};

Assert.That(dict, Does.ContainKey(1).WithValue(4));

Assert.That(dict, Contains.Value(4));
Assert.That(dict, Does.ContainValue(5));
Assert.That(dict, Does.Not.ContainValue(3));

Numbers

// xUnit
// Only InRange
Assert.InRange(actual, 0, 100);

// NUnit
Assert.That(actual, Is.InRange(0, 100));
Assert.That(actual, Is.AtMost(0));
// Also: AtLeast, Zero, LessThanOrEqualTo, Negative/Positive
// Greater/LessThan uses IComparable

// MSTest: N/A

// TUnit
await Assert.That(actual).IsBetween(0, 100);
await Assert.That(actual).IsLessThan(0);
// Also: IsGreaterThan, IsNotLessThanOrEqualTo
// Is(Not)Zero, IsNegative/Positive

Soft Assertions

A test stops running as soon as an assertion fails. But sometimes it is handy that all assertions still run so that you have more context to pinpoint the exact problem. Both xUnit and NUnit have a mecanism to do this.

// xUnit
Assert.Multiple(
  () => Assert.NotEqual(12, 24),
  () => Assert.NotNull("Hello world")
);

// NUnit
using (Assert.EnterMultipleScope())
{
  Assert.That(12, Is.Not.EqualTo(24));
  Assert.That("Hello world", Is.Not.Null);
}

// TUnit
// Both NUnit & TUnit can also use block scoped using
using var _ = Assert.Multiple();
await Assert.That(12).IsNotEqualTo(24);
await Assert.That("Hello world").IsNotNull();

Before NUnit 4.2 the syntax was: Assert.Multiple(() => { /* assertions */ })

NUnit and TUnit Only

Dates

// NUnit
Assert.That(later, Is.EqualTo(now).Within(TimeSpan.FromHours(3.0)));
Assert.That(later, Is.EqualTo(now).Within(3).Hours);

// TUnit
await Assert.That(later).IsAfter(now);
await Assert.That(later).IsBetween(now, now.AddHours(3));

Directories

Also: Is.SamePath(OrUnder)

// NUnit only
Assert.That("/folder1/./junk/../folder2", Is.SubPathOf("/folder1/"));
Assert.That(new DirectoryInfo("c:\\temp"), Does.Exist);
Assert.That(new DirectoryInfo("c:\\temp"), Is.Not.Empty);

And/Or

// NUnit
Assert.That(3, Is.LessThan(5).Or.GreaterThan(10));

// TUnit
await Assert.That(3).IsLessThan(5).Or.IsGreaterThan(10);

TUnit: For more complicated AND/OR logic, check AssertionGroup.For