First world problems, eh?
It would be very useful if MSTest could be made available through a programmatic API, and have programmatic access to the results.
Whilst it’s true that we can lunch MSTest.exe from a separate process, and then query the corresponding results file, in reality this is clunky, inelegant and annoying.
Other unit test frameworks (NUnit, xUnit) are already publicly available, so why can’t we have this for MSTest?
Before you read on, go and vote for this, please. And use all 3 votes if you could. Thanks.
The problem is encapsulated quite well here:
So why would one want this? Well, one justification posted on the original request offered goes some way of explaining it:
In combination with Roslyn this would now bring even more value as it would ease writing unit testing related tools quite a bit.
In general, though this is just a special case of the broader goal of Extensibility.
With an available API we can consider some of the following:
- Hook up test run stats to advanced reporting facilities.
- Hook up coverage stats for the same.
- Hook up to intellisense tolls to allow identification of unit tests that cover the changed code.
- Upon checkin, determine which tests exercise the affected code, and RUN THOSE ONLY! Drastically speeding up gated checkins, and CI builds.
- Extend with a mutation testing capability.
- [Post your suggestion below]
And there’s other problems.
What if you wanted the code to run on other implementations of the .NET Framework, say Mono?
Here’s a nice snippet from Travis CI (an online open-source CI builder) explaining how to run unit tests from the build.
If you’re using other test frameworks the process is similar. Please note that the MSTest framework is not supported, as it only works on Windows/Visual Studio.
Really? At a time where Microsoft are embracing open source, and opening up their code bases, for example .NET, C#, MVC, and the like, we still encounter this ludicrous corner-case tying us to Windows and Visual Studio.
Then, out of the blue, I received a response!
Thank you the feedback. You will be aware that it is possible to write a Visual Studio Test Adapter. See here: http://blogs.msdn.com/b/visualstudioalm/archive/2012/07/31/writing-a-visual-studio-2012-unit-test-adapter.aspxfor reference.
Does this meet your needs, or do you have something else in mind? Please let me know.
Pratap Lakshman (@pvlakshm on twitter)
Fantastic. So I replied (tl;dr; thanks, but that’s not quite what I wanted):
Many thanks for your response.
Looking for the post, It seems to instruct how to write a test adaptor for Visual Studio. If I have misunderstood, then my apologies. I’ve outlined my thoughts on what I’d like to see below:
I am wanting to assume there is no visual studio, just the .NET framework and my program.
I am interested in the ability to stipulate via an MSTest API, which tests to run in the context of some executing program (let’s say a console app).
Let say, for the sake of argument, something like:var testRunner = new MsTestRunner(); testRunner.Run(testClass => testClass.TestMethod()); //Assuming, that the parameter to Run() is some sort of Expression Lambda
You’ll note from these articles that it’s possible with NUnit:
Moreover, I think I’d like to see MSTest to be independent of Visual Studio, and executable, for example on Mono. I’ve read articles that say it isn’t, but might be mistaken.
An extra added bonus for this would be if code coverage statistics could be made available via an API also, in such a way that it doesn’t matter which test framework we use. Particularly useful would be the stat that says: given a line of code, which unit tests ran?
I got a further response saying they’d get back to me.
I’d like to close though with a general comment about MSTest. Although I’m ranting about its extensibility, it looks like MSTest is in need of a major overhaul. Check out some of the following, especially the first by Roy Osherove, author of The Art of Unit Testing.
So, rant over, with one final reminder: GO VOTE FOR THIS SUGGESTION on the Visual Studio UserVoice site. You get 3 votes. Use all 3!