.NET Testing Frameworks

Comparing NUnit, xUnit and MSTest

.NET Testing Frameworks

posted in dotnet on  •   • 

Comparing .NET Testing Frameworks.

xunit/xunit : Community-focused unit testing tool

nunit/nunit : NUnit Framework

microsoft/testfx : MSTest framework and adapter

Target audience: A developer switching frameworks :)

Test Suites

Typically test classes and/or methods are decorated with Attributes.

  NUnit xUnit MSTest
Namespace NUnit.Framework Xunit Microsoft.VisualStudio
.TestTools.UnitTesting
Class Attribute TestFixture (optional)   TestClass
Method Attribute Test Fact TestMethod
Ignoring Ignore Fact(Skip=””) Ignore
Setup & Teardown      
Before all tests OneTimeSetUp IClassFixture<T> ClassInitialize
Before each test SetUp Constructor TestInitialize
After each test TearDown IDisposable.Dispose TestCleanup
After all tests OneTimeTearDown IClassFixture<T> ClassCleanup
Meta data      
Description Description   Description
Categories Category   TestCategory
Custom properties Property Trait TestProperty

xUnit IClassFixture

Setup in xUnit doesn’t work with Attributes!
xUnit injects the same XUnitTestsFixture to the constructor before running each [Fact].

public class XUnitTestsFixture : IDisposable
{
    public XUnitTestsFixture() { /* BeforeAllTests */ }
    public void Dispose() { /* AfterAllTests */ }
}

public class XUnitTestsWithSetUp : IClassFixture<XUnitTestsFixture>
{
    private readonly XUnitTestsFixture _fixture;

    public XUnitTestsWithSetUp(XUnitTestsFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public void Test1() { }
}

Assembly setup

xUnit

As per the IClassFixture<T> example above, the XUnitTestsFixture will be injected in all test classes in the assembly.

[assembly: AssemblyFixture(typeof(XUnitTestsFixture))]

NUnit

[SetUpFixture]
public class Config
{
    [OneTimeSetUp]
    public void SetUp() { }

    [OneTimeTearDown]
    public void TearDown() { }
}

MSTest

[TestClass]
public class MyTestClass
{
    [AssemblyInitialize]
    public static void AssemblyInitialize(TestContext testContext)
    {
        // Or return Task!
    }

    [AssemblyCleanup]
    public static void AssemblyCleanup()
    {
        // Or return Task!
    }
}

Assertions

bool? actual = true;

// NUnit
Assert.That(actual, Is.EqualTo(true));

// xUnit
Assert.Equal(true, actual);

// MSTest
Assert.AreEqual(true, actual);

Basic

NUnit xUnit MSTest Notes
Is.EqualTo Equal AreEqual Using IEquatable
Is.Not.EqualTo NotEqual AreNotEqual  
Is.Null Null IsNull  
Is.True True IsTrue  
Is.SameAs Same AreSame Same referenced object

Strings

NUnit xUnit MSTest Notes
Is.Empty Empty    
Does.Contain Contains StringAssert.Contains  
Does.Not.Contain DoesNotContain   Only StringAssert.DoesNotMatch for MSTest
Does.StartWith StartsWith StringAssert.StartsWith  
Does.EndWith().IgnoreCase EndsWith StringAssert.EndsWith xUnit & MSTest use an overload to ignore case
Does.Match Matches StringAssert.Matches Regex

Exceptions

Action sut = () => throw new Exception("cause");
Func<Task> asyncSut = () => Task.CompletedTask;

// NUnit
var ex = Assert.Throws<Exception>(() => sut(), "failure message");
Assert.That(ex.Message, Is.EqualTo("cause"));

Assert.DoesNotThrowAsync(() => asyncSut());


// xUnit
var ex = Assert.Throws<Exception>(sut);
Assert.Equal("cause", ex.Message);

Assert.ThrowsAsync<Exception>(() => asyncSut());


// MSTest
// via Attribute
[TestMethod]
[ExpectedException(typeof(Exception))]
public void Exceptions() {}

// Via code
Assert.ThrowsExceptionAsync<Exception>(asyncSut);

Collections

NUnit xUnit MSTest Notes
Is.Empty Empty    
Is.EqualTo Equal CollectionAssert.AreEqual Same order
Is.EquivalentTo Equivalent CollectionAssert.AreEquivalent Allow different order
Has.Some.EqualTo() Contains CollectionAssert.Contains  
Contains.Item Contains CollectionAssert.Contains  
Is.Ordered.Descending      
Is.All.GreaterThan(1) All(col, x => Assert.True(x > 1))    
Has.None.Null All CollectionAssert.AllItemsAreNotNull  
Has.Exactly(3).Items      
Is.SubsetOf   CollectionAssert.IsSubsetOf  
Is.Unique   CollectionAssert.AllItemsAreUnique  

xUnit Assert.Collection

Specific assertions per element in the collection.

Assert.Collection(
  collection,
  el1 => Assert.Null(el1),
  el2 => Assert.False(el2),
  ...
);

NUnit Dictionary Assertions

var dict = new Dictionary<int, int>
{
  { 1, 4 },
  { 2, 5 }
};

Assert.That(dict, Does.ContainKey(1).WithValue(4));

Assert.That(dict, Contains.Value(4));
Assert.That(dict, Does.ContainValue(5));
Assert.That(dict, Does.Not.ContainValue(3));

Numbers

// xUnit
// Only InRange
Assert.InRange(actual, 0, 100);

// NUnit
Assert.That(actual, Is.InRange(0, 100));
Assert.That(actual, Is.AtMost(0));
// Also: AtLeast, Zero, LessThanOrEqualTo, Negative/Positive
// Greater/LessThan uses IComparable

// MSTest: N/A

NUnit Only

Dates

Assert.That(later, Is.EqualTo(now).Within(TimeSpan.FromHours(3.0)));
Assert.That(later, Is.EqualTo(now).Within(3).Hours);

Directories

Also: Is.SamePath(OrUnder)

Assert.That("/folder1/./junk/../folder2", Is.SubPathOf("/folder1/"));
Assert.That(new DirectoryInfo("c:\\temp"), Does.Exist);
Assert.That(new DirectoryInfo("c:\\temp"), Is.Not.Empty);

And/Or

Assert.That(3, Is.LessThan(5).Or.GreaterThan(10));

Stuff that came into being during the making of this post
Other interesting reads
Tags: testing cheat-sheet