Microsoft, Pleeease Expose Your MSTest API

First world problems, eh?

Regardless…

Almost exactly two years ago, I raised a suggestion on the Visual Studio UserVoice site regarding what I perceive a big problem with the MSTest suite.

It would be very useful if MSTest could be made available through a programmatic API, and have programmatic access to the results.

Whilst it’s true that we can lunch MSTest.exe from a separate process, and then query the corresponding results file, in reality this is clunky, inelegant and annoying.

Other unit test frameworks (NUnit, xUnit) are already publicly available, so why can’t we have this for MSTest?

Before you read on, go and vote for this, please. And use all 3 votes if you could. Thanks.

The problem is encapsulated quite well here:

Is there an API for running Visual Studio Unit Tests programmatically?

Running VSTS tests without mstest.exe

So why would one want this? Well, one justification posted on the original request offered goes some way of explaining it:

In combination with Roslyn this would now bring even more value as it would ease writing unit testing related tools quite a bit.

In general, though this is just a special case of the broader goal of Extensibility.

With an available API we can consider some of the following:

  • Hook up test run stats to advanced reporting facilities.
  • Hook up coverage stats for the same.
  • Hook up to intellisense tolls to allow identification of unit tests that cover the changed code.
  • Upon checkin, determine which tests exercise the affected code, and RUN THOSE ONLY! Drastically speeding up gated checkins, and CI builds.
  • Extend with a mutation testing capability.
  • [Post your suggestion below]

And there’s other problems.

What if you wanted the code to run on other implementations of the .NET Framework, say Mono?

Tough.

Here’s a nice snippet from Travis CI (an online open-source CI builder) explaining how to run unit tests from the build.

If you’re using other test frameworks the process is similar. Please note that the MSTest framework is not supported, as it only works on Windows/Visual Studio.

Really? At a time where Microsoft are embracing open source, and opening up their code bases, for example .NET, C#, MVC, and the like, we still encounter this ludicrous corner-case tying us to Windows and Visual Studio.

Then, out of the blue, I received a response!

Thank you the feedback. You will be aware that it is possible to write a Visual Studio Test Adapter. See here: http://blogs.msdn.com/b/visualstudioalm/archive/2012/07/31/writing-a-visual-studio-2012-unit-test-adapter.aspxfor reference.
Does this meet your needs, or do you have something else in mind? Please let me know.
Pratap Lakshman (@pvlakshm on twitter)
Visual Studio

Fantastic. So I replied (tl;dr;  thanks, but that’s not quite what I wanted):

Many thanks for your response.

Looking for the post, It seems to instruct how to write a test adaptor for Visual Studio. If I have misunderstood, then my apologies. I’ve outlined my thoughts on what I’d like to see below:

I am wanting to assume there is no visual studio, just the .NET framework and my program.

I am interested in the ability to stipulate via an MSTest API, which tests to run in the context of some executing program (let’s say a console app).

Let say, for the sake of argument, something like:

var testRunner = new MsTestRunner(); 
testRunner.Run(testClass => testClass.TestMethod()); //Assuming, that the parameter to Run() is some sort of Expression Lambda

You’ll note from these articles that it’s possible with NUnit:
https://gist.github.com/fijiaaron/1387474
http://stackoverflow.com/questions/2825918/how-to-run-nunit-tests-from-c-sharp-code

Moreover, I think I’d like to see MSTest to be independent of Visual Studio, and executable, for example on Mono. I’ve read articles that say it isn’t, but might be mistaken.

An extra added bonus for this would be if code coverage statistics could be made available via an API also, in such a way that it doesn’t matter which test framework we use. Particularly useful would be the stat that says: given a line of code, which unit tests ran?

I got a further response saying they’d get back to me.

I’d like to close though with a general comment about MSTest. Although I’m ranting about its extensibility, it looks like MSTest is in need of a major overhaul. Check out some of the following, especially the first by Roy Osherove, author of The Art of Unit Testing.

Why MSTest is the IE6 of unit test frameworks

Why Not MsTest

Moving to NUnit from MsTest – experience report

So, rant over, with one final reminder: GO VOTE FOR THIS SUGGESTION on the Visual Studio UserVoice site. You get 3 votes. Use all 3!

Thanks.

5 thoughts on “Microsoft, Pleeease Expose Your MSTest API

  1. Pingback: Dew Drop – October 19, 2015 (#2114) | Morning Dew

  2. In response to the above questions:
    1) Microsoft may have changed focus, but MSTest is still a prominent option for a lot of developers. This means that many people will still be picking up and using MSTest. It doesn’t matter that there better options out there. Whilst it’s still supported, people will still use it.
    2) You can’t ignore legacy If you want to build a generic tool that extends test frameworks, you are going to need to include MSTest. Currently this is difficult to extend for the above reasons.

  3. What’s the point? I mean, why keep using MSTests when there are better test frameworks out there? Is there anything that MSTest does better than e.g. NUnit or xUnit?

  4. I’m not sure the request makes sense at this point. With VS2015, aren’t they switching focus to xUnit anyway?

Leave a Reply