Lately, I exchanged some arguments with Derick Bailey about some details of
the red-green-refactor cycle of
the Test-driven development process. In short,
the issue revolved around
the fact that it’s not enough to have a test red or green, but it’s also important to have it red or green for
the right reasons. While for me, it’s sufficient to initially have a NotImplementedException in place, Derick argues that this is not totally correct (see these two posts: Red/Green/Refactor, For
The Right Reasons and Red For
The Right Reason: Fail By Assertion, Not By Anything Else). And he’s right.
But on
the other hand, I had no idea how his insights could have any practical consequence for my own individual interpretation of
the red-green-refactor cycle (which is not really red-green-refactor, at least not in its pure sense, see
the rest of this article). This made me think deeply for some days now. In
the end I found out that
the ‘right reason’ changes in my understanding depending on what development phase I’m in.
To make this clear (at least I hope it becomes clear…) I started to describe my way of working in some detail, and then something strange happened:
The scope of
the article slightly shifted from focusing ‘only’ on
the ‘right reason’ issue to something more general, which you might describe as something like 'Doing real-world TDD in .NET , with massive use of third-party add-ins’.
This is because I feel that there is a more general statement about Test-driven development to make: It’s high time to speak about
the ‘How’ of TDD, not always only
the ‘Why’. Much has been said about this, and me myself also contributed to that (see here: TDD is not about testing, it's about how we develop software). But always justifying what you do is very unsatisfying in
the long run, it is inherently defensive, and it costs time and effort that could be used for better and more important things. And frankly: I’m somewhat sick and tired of repeating time and again that
the test-driven way of software development is highly preferable for many reasons - I don’t want to spent my time exclusively on stating
the obvious…
So, again, let’s say it clearly: TDD is programming, and programming is TDD. Other ways of programming (code-first, sometimes called cowboy-coding) are exceptional and need justification. – I know that there are many people out there who will disagree with this radical statement, and I also know that it’s not a description of
the real world but more of a mission statement or something. But nevertheless I’m absolutely sure that in some years this statement will be nothing but a platitude.
Side note: Some parts of this post read as if I were paid by Jetbrains (the manufacturer of
the ReSharper add-in – R#), but I swear I’m not. Rather I think that Visual Studio is just not production-complete without it, and I wouldn’t even consider to do professional work without having this add-in installed...
The three parts of a software component
Before I go into some details, I first should describe my understanding of what belongs to a software component (assembly, type, or method) during
the production process (i.e.
the coding phase). Roughly, I come up with
the three parts shown below:
First, we need to have some initial sort of requirement. This can be a multi-page formal document, a vague idea in some programmer’s
brain of what might be needed, or anything in between. In either way, there has to be some sort of requirement, be it explicit or not. – At
the C# micro-level,
the best way that I found to formulate that is to define interfaces for just about everything, even for internal classes, and to provide them with exhaustive xml comments.
The next step then is to re-formulate these requirements in an executable form. This is specific to
the respective programming language. - For C#/.NET,
the Gallio framework (which includes MbUnit) in conjunction with
the ReSharper add-in for Visual Studio is my toolset of choice.
The third part then finally is
the production code itself. It’s development is entirely driven by
the requirements and their executable formulation. This is
the delivery,
the two other parts are ‘only’ there to make its production possible, to give it a decent quality and reliability, and to significantly reduce related costs down
the maintenance timeline.
So while
the first two parts are not really relevant for
the customer, they are very important for
the developer.
The customer (or in Scrum terms:
the Product Owner) is not interested at all in how
the product is developed, he is only interested in
the fact that it is developed as cost-effective as possible, and that it meets his functional and non-functional requirements.
The rest is solely a matter of
the developer’s craftsmanship, and this is what I want to talk about during
the remainder of this article…
An example
To demonstrate my way of doing real-world TDD, I decided to show
the development of a (very) simple Calculator component.
The example is deliberately trivial and silly, as examples always are. I am totally aware of
the fact that real life is never that simple, but I only want to show some development principles here…
The requirement
As already said above, I start with writing down some words on
the initial requirement, and I normally use interfaces for that, even for internal classes -
the typical question “intf or not” doesn’t even come to mind. I need them for my usual workflow and using them automatically produces high componentized and testable code anyway. To think about their usage in every single situation would slow down
the production process unnecessarily.
So this is what I begin with:
namespace Calculator
{
/// <summary>
/// Defines a very simple calculator component for demo purposes.
/// </summary>
public interface ICalculator
{
/// <summary>
/// Gets
the result of
the last successful operation.
/// </summary>
/// <value>The last result.</value>
/// <remarks>
/// Will be <see langword="null" /> before
the first successful operation.
/// </remarks>
double? LastResult { get; }
} // interface ICalculator
} // namespace Calculator
So, I’m not beginning with a test, but with a sort of code declaration - and still I insist on being 100% test-driven. There are three important things here:
Starting this way gives me a method signature, which allows to use IntelliSense and AutoCompletion and thus eliminates
the danger of typos - one of
the most regular, annoying, time-consuming, and therefore expensive sources of error in
the development process.
In my understanding,
the interface definition as a whole is more of a readable requirement document and technical documentation than anything else. So this is at least as much about documentation than about coding.
The documentation must completely describe
the behavior of
the documented element.
I normally use an IoC container or some sort of self-written provider-like model in my architecture. In either case, I need my components defined via service interfaces anyway. - I will use
the LinFu IoC framework here, for no other reason as that is is very simple to use.
The ‘Red’ (pt. 1)
First I create a folder for
the project’s third-party libraries and put
the LinFu.Core dll there. Then I set up a test project (via a Gallio project template), and add references to
the Calculator project and
the LinFu dll. Finally I’m ready to write
the first test, which will look like
the following:
namespace Calculator.Test
{
[TestFixture]
public class CalculatorTest
{
private readonly ServiceContainer container = new ServiceContainer();
[Test]
public void CalculatorLastResultIsInitiallyNull()
{
ICalculator calculator = container.GetService<ICalculator>();
Assert.IsNull(calculator.LastResult);
}
} // class CalculatorTest
} // namespace Calculator.Test
This is basically
the executable formulation of what
the interface definition states (part of).
Side note:
There’s one principle of TDD that is just plain wrong in my eyes: I’m talking about
the Red is 'does not compile' thing. How could a compiler error ever be interpreted as a valid test outcome? I never understood that, it just makes no sense to me. (Or, in Derick’s terms: this reason is as wrong as a reason ever could be…) A compiler error tells me: Your code is incorrect, but nothing more.
Instead,
the ‘Red’ part of
the red-green-refactor cycle has a clearly defined meaning to me: It means that
the test works as intended and fails only if its assumptions are not met for some reason.
Back to our Calculator. When I execute
the above test with R#,
the Gallio plugin will give me this output:
So this tells me that
the test is red for
the wrong reason: There’s no implementation that
the IoC-container could load, of course. So let’s fix that. With R#, this is very easy:
First, create an ICalculator - derived type:
Next, implement
the interface members:
And finally, move
the new class to its own file:
So far my ‘work’ was six mouse clicks long,
the only thing that’s left to do manually here, is to add
the Ioc-specific wiring-declaration and also to make
the respective class non-public, which I regularly do to force my components to communicate exclusively via interfaces: This is what my Calculator class looks like as of now:
using System;
using LinFu.IoC.Configuration;
namespace Calculator
{
[Implements(typeof(ICalculator))]
internal class Calculator : ICalculator
{
public double? LastResult
{
get
{
throw new NotImplementedException();
}
}
}
}
Back to
the test fixture, we have to put our IoC container to work:
[TestFixture]
public class CalculatorTest
{
#region Fields
private readonly ServiceContainer container = new ServiceContainer();
#endregion // Fields
#region Setup/TearDown
[FixtureSetUp]
public void FixtureSetUp()
{
container.LoadFrom(AppDomain.CurrentDomain.BaseDirectory, "Calculator.dll");
}
...
Because I have a R# live template defined for
the setup/teardown method skeleton as well,
the only manual coding here again is
the IoC-specific stuff: two lines, not more…
The ‘Red’ (pt. 2)
Now,
the execution of
the above test gives
the following result:
This time,
the test outcome tells me that
the method under test is called. And this is
the point, where Derick and I seem to have somewhat different views on
the subject: Of course,
the test still is worthless regarding
the red/green outcome (or: it’s still red for
the wrong reasons, in that it gives a false negative). But as far as I am concerned, I’m not really interested in
the test outcome at this point of
the red-green-refactor cycle. Rather, I only want to assert that my test actually calls
the right method. If that’s
the case, I will happily go on to
the ‘Green’ part…
The ‘Green’
Making
the test green is quite trivial. Just make LastResult an automatic property:
[Implements(typeof(ICalculator))]
internal class Calculator : ICalculator
{
public double? LastResult { get; private set; }
}
One more round…
Now on to something slightly more demanding (cough…). Let’s state that our Calculator exposes an Add() method:
...
/// <summary>
/// Adds
the specified operands.
/// </summary>
/// <param name="operand1">The operand1.</param>
/// <param name="operand2">The operand2.</param>
/// <returns>The result of
the additon.</returns>
/// <exception cref="ArgumentException">
/// Argument <paramref name="operand1"/> is < 0.<br/>
/// -- or --<br/>
/// Argument <paramref name="operand2"/> is < 0.
/// </exception>
double Add(double operand1, double operand2);
} // interface ICalculator
A remark: I sometimes hear
the complaint that xml comment stuff like
the above is hard to read. That’s certainly true, but irrelevant to me, because I read xml code comments with
the CR_Documentor tool window. And using that, it looks like this:
Apart from that, I’m heavily using xml code comments (see e.g. here for a detailed guide) because there is
the possibility of automating help generation with nightly CI builds (using MS Sandcastle and
the Sandcastle Help File Builder), and then publishing
the results to some intranet location. This way, a team always has first class, up-to-date technical documentation at hand about
the current codebase.
(And, also very important for speeding up things and avoiding typos: You have IntelliSense/AutoCompletion and R# support, and
the comments are subject to compiler checking…).
Back to our Calculator again: Two more R# – clicks implement
the Add() skeleton:
...
public double Add(double operand1, double operand2)
{
throw new NotImplementedException();
}
} // class Calculator
As we have stated in
the interface definition (which actually serves as our requirement document!),
the operands are not allowed to be negative. So let’s start implementing that. Here’s
the test:
[Test]
[Row(-0.5, 2)]
public void AddThrowsOnNegativeOperands(double operand1, double operand2)
{
ICalculator calculator = container.GetService<ICalculator>();
Assert.Throws<ArgumentException>(() => calculator.Add(operand1, operand2));
}
As you can see, I’m using a data-driven unit test method here, mainly for these two reasons:
Because I know that I will have to do
the same test for
the second operand in a few seconds, I save myself from implementing another test method for this purpose. Rather, I only will have to add another Row attribute to
the existing one.
From
the test report below, you can see that
the argument values are explicitly printed out. This can be a valuable documentation feature even when everything is green: One can quickly review what values were tested exactly -
the complete Gallio HTML-report (as it will be produced by
the Continuous Integration runs) shows these values in a quite clear format (see below for an example).
Back to our Calculator development again, this is what
the test result tells us at
the moment:
So we’re red again, because there is not yet an implementation…
Next we go on and implement
the necessary parameter verification to become green again, and then we do
the same thing for
the second operand. To make a long story short, here’s
the test and
the method implementation at
the end of
the second cycle:
// in CalculatorTest:
[Test]
[Row(-0.5, 2)]
[Row(295, -123)]
public void AddThrowsOnNegativeOperands(double operand1, double operand2)
{
ICalculator calculator = container.GetService<ICalculator>();
Assert.Throws<ArgumentException>(() => calculator.Add(operand1, operand2));
}
// in Calculator:
public double Add(double operand1, double operand2)
{
if (operand1 < 0.0)
{
throw new ArgumentException("Value must not be negative.", "operand1");
}
if (operand2 < 0.0)
{
throw new ArgumentException("Value must not be negative.", "operand2");
}
throw new NotImplementedException();
}
So far, we have sheltered our method from unwanted input, and now we can safely operate on
the parameters without further caring about their validity (this is my interpretation of
the Fail Fast principle, which is regarded here in more detail).
Now we can think about
the method’s successful outcomes. First let’s write another test for that:
[Test]
[Row(1, 1, 2)]
public void TestAdd(double operand1, double operand2, double expectedResult)
{
ICalculator calculator = container.GetService<ICalculator>();
double result = calculator.Add(operand1, operand2);
Assert.AreEqual(expectedResult, result);
}
Again, I’m regularly using row based test methods for these kinds of unit tests.
The above shown pattern proved to be extremely helpful for my development work, I call it
the Defined-Input/Expected-Output test idiom: You define your input arguments together with
the expected method result. There are two major benefits from that way of testing:
In
the course of refining a method, it’s very likely to come up with additional test cases. In our case, we might add tests for some edge cases like ‘one of
the operands is zero’ or ‘the sum of
the two operands causes an overflow’, or maybe there’s an external test protocol that has to be fulfilled (e.g. an ISO norm for medical software), and this results in
the need of testing against additional values. In all these scenarios we only have to add another Row attribute to
the test.
Remember that
the argument values are written to
the test report, so as a side-effect this produces valuable documentation. (This can become especially important if
the fulfillment of some sort of external requirements has to be proven).
So your test method might look something like that in
the end:
[Test, Description("Arguments: operand1, operand2, expectedResult")]
[Row(1, 1, 2)]
[Row(0, 999999999, 999999999)]
[Row(0, 0, 0)]
[Row(0, double.MaxValue, double.MaxValue)]
[Row(4, double.MaxValue - 2.5, double.MaxValue)]
public void TestAdd(double operand1, double operand2, double expectedResult)
{
ICalculator calculator = container.GetService<ICalculator>();
double result = calculator.Add(operand1, operand2);
Assert.AreEqual(expectedResult, result);
}
And this will produce
the following HTML report (with Gallio):
Not bad for
the amount of work we invested in it, huh? - There might be scenarios where reports like that can be useful for demonstration purposes during a Scrum sprint review…
The last requirement to fulfill is that
the LastResult property is expected to store
the result of
the last operation. I don’t show this here, it’s trivial enough and brings nothing new…
And finally: Refactor (for
the right reasons)
To demonstrate my way of going through
the refactoring portion of
the red-green-refactor cycle, I added another method to our Calculator component, namely Subtract(). Here’s
the code (tests and production):
// CalculatorTest.cs:
[Test, Description("Arguments: operand1, operand2, expectedResult")]
[Row(1, 1, 0)]
[Row(0, 999999999, -999999999)]
[Row(0, 0, 0)]
[Row(0, double.MaxValue, -double.MaxValue)]
[Row(4, double.MaxValue - 2.5, -double.MaxValue)]
public void TestSubtract(double operand1, double operand2, double expectedResult)
{
ICalculator calculator = container.GetService<ICalculator>();
double result = calculator.Subtract(operand1, operand2);
Assert.AreEqual(expectedResult, result);
}
[Test, Description("Arguments: operand1, operand2, expectedResult")]
[Row(1, 1, 0)]
[Row(0, 999999999, -999999999)]
[Row(0, 0, 0)]
[Row(0, double.MaxValue, -double.MaxValue)]
[Row(4, double.MaxValue - 2.5, -double.MaxValue)]
public void TestSubtractGivesExpectedLastResult(double operand1, double operand2, double expectedResult)
{
ICalculator calculator = container.GetService<ICalculator>();
calculator.Subtract(operand1, operand2);
Assert.AreEqual(expectedResult, calculator.LastResult);
}
...
// ICalculator.cs:
/// <summary>
/// Subtracts
the specified operands.
/// </summary>
/// <param name="operand1">The operand1.</param>
/// <param name="operand2">The operand2.</param>
/// <returns>The result of
the subtraction.</returns>
/// <exception cref="ArgumentException">
/// Argument <paramref name="operand1"/> is < 0.<br/>
/// -- or --<br/>
/// Argument <paramref name="operand2"/> is < 0.
/// </exception>
double Subtract(double operand1, double operand2);
...
// Calculator.cs:
public double Subtract(double operand1, double operand2)
{
if (operand1 < 0.0)
{
throw new ArgumentException("Value must not be negative.", "operand1");
}
if (operand2 < 0.0)
{
throw new ArgumentException("Value must not be negative.", "operand2");
}
return (this.LastResult = operand1 - operand2).Value;
}
Obviously,
the argument validation stuff that was produced during
the red-green part of our cycle duplicates
the code from
the previous Add() method. So, to avoid code duplication and minimize
the number of code lines of
the production code, we do an Extract Method refactoring. One more time, this is only a matter of a few mouse clicks (and giving
the new method a name) with R#:
Having done that, our production code finally looks like that:
using System;
using LinFu.IoC.Configuration;
namespace Calculator
{
[Implements(typeof(ICalculator))]
internal class Calculator : ICalculator
{
#region ICalculator
public double? LastResult { get; private set; }
public double Add(double operand1, double operand2)
{
ThrowIfOneOperandIsInvalid(operand1, operand2);
return (this.LastResult = operand1 + operand2).Value;
}
public double Subtract(double operand1, double operand2)
{
ThrowIfOneOperandIsInvalid(operand1, operand2);
return (this.LastResult = operand1 - operand2).Value;
}
#endregion // ICalculator
#region Implementation (Helper)
private static void ThrowIfOneOperandIsInvalid(double operand1, double operand2)
{
if (operand1 < 0.0)
{
throw new ArgumentException("Value must not be negative.", "operand1");
}
if (operand2 < 0.0)
{
throw new ArgumentException("Value must not be negative.", "operand2");
}
}
#endregion // Implementation (Helper)
} // class Calculator
} // namespace Calculator
But is
the above worth
the effort at all? It’s obviously trivial and not very impressive. All our tests were green (for
the right reasons), and refactoring
the code did not change anything. It’s not immediately clear how this refactoring work adds value to
the project. Derick puts it like this:
STOP! Hold on a second… before you go any further and before you even think about refactoring what you just wrote to make your test pass, you need to understand something: if your done with your requirements after making
the test green, you are not required to refactor
the code. I know… I’m speaking heresy, here. Toss me to
the wolves, I’ve gone over to
the dark side! Seriously, though… if your test is passing for
the right reasons, and you do not need to write any test or any more code for you class at this point, what value does refactoring add?
Derick immediately answers his own question:
So why should you follow
the refactor portion of red/green/refactor? When you have added code that makes
the system less readable, less understandable, less expressive of
the domain or concern’s intentions, less architecturally sound, less DRY, etc, then you should refactor it.
I couldn’t state it more precise. From my personal perspective, I’d add
the following:
You have to keep in mind that real-world software systems are usually quite large and there are dozens or even hundreds of occasions where micro-refactorings like
the above can be applied. It’s
the sum of them all that counts. And to have a good overall quality of
the system (e.g. in terms of
the Code Duplication Percentage metric) you have to be pedantic on
the individual, seemingly trivial cases.
My job regularly requires
the reading and understanding of ‘foreign’ code. So code quality/readability really makes a HUGE difference for me – sometimes it can be even
the difference between project success and failure…
Conclusions
The above described development process emerged over
the years, and there were mainly two things that guided its evolution (you might call it eternal principles, personal beliefs, or anything in between):
Test-driven development is
the normal, natural way of writing software, code-first is exceptional. So ‘doing TDD or not’ is not a question. And good, stable code can only reliably be produced by doing TDD (yes, I know: many will strongly disagree here again, but I’ve never seen high-quality code – and high-quality code is code that stood
the test of time and causes low maintenance costs – that was produced code-first…)
It’s
the production code that pays our bills in
the end. (Though I have seen customers these days who demand an acceptance test battery as part of
the final delivery. Things seem to go into
the right direction…).
The test code serves ‘only’ to make
the production code work. But it’s
the number of delivered features which solely counts at
the end of
the day - no matter how much test code you wrote or how good it is.
With these two things in mind, I tried to optimize my coding process for coding speed – or, in business terms: productivity - without sacrificing
the principles of TDD (more than I’d do either way…). As a result, I consider a ratio of about 3-5/1 for test code vs. production code as normal and desirable. In other words: roughly 60-80% of my code is test code (This might sound heavy, but that is mainly due to
the fact that software development standards only begin to evolve.
The entire software development profession is very young, historically seen; only at
the very beginning, and there are no viable standards yet. If you think about software development as a kind of casting process, where
the test code is
the mold and
the resulting production code is
the final product, then
the above ratio sounds no longer extraordinary…)
Although
the above might look like very much unnecessary work at first sight, it’s not. With
the aid of
the mentioned add-ins, doing all
the above is a matter of minutes, sometimes seconds (while writing this post took hours and days…).
The most important thing is to have
the right tools at hand. Slow developer machines or
the lack of a tool or something like that - for ‘saving’ a few 100 bucks - is just not acceptable and a very bad decision in business terms (though I quite some times have seen and heard that…). Production of high-quality products needs
the usage of high-quality tools. This is a platitude that every craftsman knows…
The here described round-trip will take me about five to ten minutes in my real-world development practice. I guess it’s about 30% more time compared to developing
the ‘traditional’ (code-first) way. But
the so manufactured ‘product’ is of much higher quality and massively reduces maintenance costs, which is by far
the single biggest cost factor, as I showed in this previous post: It's
the maintenance, stupid! (or: Something is rotten in developerland.). In
the end, this is a highly cost-effective way of software development…
But on
the other hand, there clearly is a trade-off here: coding speed vs. code quality/later maintenance costs.
The here described development method might be a perfect fit for
the overwhelming majority of software projects, but there certainly are some scenarios where it’s not - e.g. if time-to-market is crucial for a software project. So this is a business decision in
the end. It’s just that you have to know what you’re doing and what consequences this might have…
Some last words
First, I’d like to thank Derick Bailey again. His two aforementioned posts (which I strongly recommend for reading) inspired me to think deeply about my own personal way of doing TDD and to clarify my thoughts about it. I wouldn’t have done that without this inspiration. I really enjoy that kind of discussions…
I agree with him in all respects. But I don’t know (yet?) how to bring his insights into
the described production process without slowing things down.
The above described method proved to be very “good enough” in my practical experience. But of course, I’m open to suggestions here…
My rationale for now is: If
the test is initially red during
the red-green-refactor cycle,
the ‘right reason’ is: it actually calls
the right method, but this method is not yet operational. Later on, when
the cycle is finished and
the tests become part of
the regular, automated Continuous Integration process, ‘red’ certainly must occur for
the ‘right reason’: in this phase, ‘red’ MUST mean nothing but an unfulfilled assertion - Fail By Assertion, Not By Anything Else!
