TDD Is Great...Except When It Isn't

CraigBlog

Syndication

I've been a big fan of Test Driven Development (TDD) ever since I first started using it to write FlexWikiPad (FWP) about a year ago. In fact, that was one of the reasons I wrote FWP in the first place, as an excuse to see what the fuss about TDD was. It turned out to be a huge win for FWP, and I've been using it for everything I write since then. Until now.

Having spent a good chunk of my free time in the last two months writing RichTextEditor, my first real WinForms control, I slammed headlong into a problem that many others have had before me: writing a test-driven component that has a complex relationship with its container. You see, the main tenet of TDD is based on the idea that you simulate the client of a component before you write the component. These simulations usually take the form of method calls, since usually what you're testing is a simple algorithmic class.

Sometimes what you're testing is a little more complicated, like a form. In cases like that, testing gets a bit more complicated too, because now you have to simulate a user clicking buttons and entering text. But even that isn't that bad if you separate out the windowy bits of the application (the view) from the bits that react to it (the controller). Then you just test the controller by making calls that simulate events in the view. That's how FlexWikiPad is developed, and I've been very happy with it.

But sometimes, what you're testing is a lot more complicated, like a WinForms control. There, the thing we have to simulate is the windowing system itself, since that's really the client of my control: Windows sends events to my control, and my control sends painting commands back. Making matters worse, at the time I didn't even know what messages to expect, and what should happen in response to them. As a result, I found myself at a loss to write tests for my control.

So I didn't. I tested it the old fashioned way: manually. Was it more error-prone? Yes. Does it make bug regression more difficult? Yes. Did I have any other choice given my time and resource constraints? Not really.

Note, however, what I am not saying here:

  1. That TDD doesn't work. It actually works very well when simulating the container isn't too hard. Which is most of the time.
  2. That TDD doesn't work in GUI scenarios. I've used it very succesfully when building applications. Just not RichTextEditor.
  3. That TDD never works for control development. The limiting factor is being able to simulate the exchange of messages between Windows and your control. If this is complex, at some point simulating it will be too much work. But if it's simple, you might manage it just fine.
  4. That you can't use TDD at all if you're writing a complex control. I actually still used it in RichTextEditor to develop the undo/redo logic of my control, since that stuff is decoupled from the rest of the control, and has a simple interface. It was very helpful for that portion of the development.

Another thing to be aware of is that I never really explored automating my testing process in other ways. For example. a macro recorder could be used to send keystrokes to my control, and possibly to test the results in one way or another. NUnitForms has a Recorder that might help in this regard.

Overall, I'm still a huge fan of TDD, and still plan to use it whenever I can. It's just that now I have a better idea of what “whenever I can” means.


Posted Oct 17 2004, 11:18 AM by craig-andera

Comments

you've been HAACKED wrote TDD Is Great...Except When It Isn't
on 10-17-2004 8:00 PM
Paul Bartlett wrote re: TDD Is Great...Except When It Isn't
on 10-18-2004 1:58 AM
I think you hit the nail on the head when you mentioned other ways of testing - after all, it is TDD not UTDD (unit test driven development).

One approach I started to use in my previous project was to have lots of ASSERTs, a debug/trace listener that I could query against, and use a record/playback technique (in my case the John Robbins "Tester" utility that was presented over the course of a number of BugSlayer articles). The nominal case is tested by running and checking the specialised log listener recorded no errors or warnings, then as you go along you can test for error conditions by recording appropriate usage, then adding playback and checking into the test script.

This is clearly not unit testing (system? smoke? integration?), but I'd still say the development was test-driven.
Craig wrote re: TDD Is Great...Except When It Isn't
on 10-18-2004 4:47 AM
I think that's a great idea. One of the main reasons I didn't go this route was that when I started that particular project, I was just playing around. I could have gone back and retrofitted it, but I was lazy. :)

In any event, I still believe that there are cases where container-component interaction is so complex that any sort of TDD will start to get very expensive. For example, trying to test logic that ensures a particular piece of text got drawn at x, y location 25, 168 and has a red wavy underline with a solid, one-pixel border might get a little hairy. It's not that it can't be done, it's that it starts to be a net-zero or net-loss proposition.

But as I said, I still believe in testing as much as you can. You seem to have found a way to automate testing of a large part of your app, which is a good thing.
Pretty stupid for a smart guy... wrote Test Driven Development - a tough nut to crack?
on 10-18-2004 7:22 AM
Stephane Lajoie wrote re: TDD Is Great...Except When It Isn't
on 10-19-2004 1:34 PM
If the container is that difficult to simulate, why not use the real thing? Have your testing code load your control in a WinForm and play with it (causing events and calling methods to verify that everything stays consistent).

Obviously, that doesn't allow you to test whether the control sends the right events to its host control but at least you can get pretty extensive test coverage on your own code without tripping on your inevitably imperfect fake host. I say that's better than nothing.

I agree with you, when the container is complex it becomes very labor-intensive to test your component. Extreme example: my boss actually wants us to implement the entire MAPI specification in order to test our MAPI component (a MAPI Message Store Provider). Please don't laugh.
Craig wrote re: TDD Is Great...Except When It Isn't
on 10-19-2004 7:09 PM
Sure - when the container is amenable to automation, that can be a handy approach. But as you observe, sometimes it isn't.
Geek Noise wrote Geek Notes 2004-10-21
on 10-21-2004 8:17 AM
ripster wrote re: TDD Is Great...Except When It Isn't
on 10-22-2004 4:47 AM
You might want to check out a mock object solution. POCMock from http://www.prettyobjects.com can be used to generate a complete mock of any .NET dll, once you have the mock of windows forms you can then setup your various expectations and check whether your control does exactly as you would expect.
Craig wrote re: TDD Is Great...Except When It Isn't
on 10-22-2004 5:45 AM
I use mock objects extensively, so I know how useful they can be. However, when what you're trying to mock up is the Windows Operating System itself...mock object solutions aren't particuarly helpful.
Jonathan de Halleux wrote re: TDD Is Great...Except When It Isn't
on 10-23-2004 12:05 PM
Craig,

The only way you can *hope* to fully test a gui is to create a model (finite state machine) that represents the different possibilities of interactions and then execute that model.

There is an excellent article from Harry Robinson that explains the why and the how:
http://www.geocities.com/harry_robinson_testing/robinson.pdf
Craig wrote re: TDD Is Great...Except When It Isn't
on 10-24-2004 7:28 AM
Hmm. I've written fully tested GUIs without explicitly generating a state machine, so I'm not sure I buy your assertion.

The article is interesting, however, and clearly a model-drive approach would be useful when the application lends itself to it.
Jonathan de Halleux wrote re: TDD Is Great...Except When It Isn't
on 10-24-2004 10:56 AM
The assertion is a bit extreme I admit it :).

But imagine that you want to hit the long-run tests, the ones that appear after using 2hours an application or with a sick combination of key strokes. Ideally, you would build a state machine and let run until it crashes (or not).
Jens Lippmann wrote re: TDD Is Great...Except When It Isn't
on 10-24-2004 2:32 PM
TDD seems to be the topic last week. Just came across your posting after I wrote about TDD on the percs blog. (http://blogs.applanet.com/percs)

I do agree on automation, but I disagree with Jonathan's proposal. Well, I very much agree with the state machine approach as part of the QA process (the state machine would presumably be built of use cases) but that's not really the developer's unit testing anymoree, is it?

I think a possible answers is actually on one of Jonathan's blogs (http://blog.dotnetwiki.org/archive/2004/06/10/390.aspx)and it is going back to more of a white box testing scenarion on the control, by using a Reflector Addin to generate empty test case. I did something like this manually before NUnitForms came back from hibernation, and probably can go back to it using Reflector.

Craig wrote re: TDD Is Great...Except When It Isn't
on 10-24-2004 2:44 PM
I think Jens nailed it when he pointed out that there's a distinction between developer-written test, and test team-written tests. I think it's pretty well recognized that both are useful tools in developing quality software. Long-running tests are probably the domain of the test team, rather than the developers, as adding "two hours" to every developer build is not feasible.

State machine development seems to my inexperienced eyes to be a tool primarily useful for the test team, with the impact to the dev team being *perhaps* to modify their code to make it easier for the test team to inspect application state.

In other words, it sounds like a great approach as part of the overall process, but I don't think it's going to change the way I code.
Troy wrote re: TDD Is Great...Except When It Isn't
on 10-25-2004 8:55 AM
I've started reading the text of my program as a test. I check to see that I have made specific connections to framework elements by ensuring that specific interfaces have been coded.

Then I can keep my state testing within the model. I use an MVC pattern and pass the "container" interface into the model. The code-checking tests verify that the controller has been set up to the view, and that the view is effectively connected (by code) to the the framework.

I call these tests "Verifiers".
Craig wrote re: TDD Is Great...Except When It Isn't
on 10-25-2004 9:37 AM
I've written something similar for my clients, but rather than parsing code, I use FxCop to check properties of my assemblies. It seems to be a bit more flexible, as well as being language neutral. Plus FxCop brings so much other goodness to the table.
Rjae wrote re: TDD Is Great...Except When It Isn't
on 10-25-2004 1:00 PM
If the implication is that software assets can reach a level of complexity that necessitates removing it from the TDD process (lobbing it over to QA to test assertions), then I think that is a dangerous path to follow. I <a href="http://blogs.applanet.com/percs/PermaLink.aspx?guid=489c0fd9-aff0-403f-9077-536df835d593">write about this</a>, but more importantly I invite those with TDD experience to <a href="http://applanet.com/wiki/default.aspx/Rjae.SoftwareMethodology.TDDPrimer.TableOfContents">create a solution</a>.
Rjae wrote re: TDD Is Great...Except When It Isn't
on 10-25-2004 1:06 PM
Craig wrote re: TDD Is Great...Except When It Isn't
on 10-25-2004 3:39 PM
I agree: you should adopt a test-first mentality and hold on to it whenever you can. However, as was pointed out above, you might be doing test-driven development without doing *automated unit test* driven development. Automated unit tests imply that you have to simulate your client/caller, which can be quite complex. Complex enough to stop you from doing it? Depends on your situation.

I should also point out that I didn't mean to imply in any way that developers are absolved from writing all the tests they can. Rather, that we should remember that enlisting the test team to write simulations and automation for more complex scenarios - since they have the tools and expertise to do it - is a good idea. It's just about division of labor, not passing the buck.

As widely available tools mature (e.g. NUnitAsp, NUnitForms), the cost of automation by the developer will continue to drop, and the test team can shift their focus to even more complex scenarios.
Roni wrote re: TDD Is Great...Except When It Isn't
on 01-11-2005 2:57 AM
You might want to check out a mock type solution. TypeMock from http://www.typemock.com can be used to dynamicly mock of any concrete class, you can mock windows forms and setup your various expectations and check whether your control does exactly as you would expect.
Craig wrote re: TDD Is Great...Except When It Isn't
on 01-11-2005 4:33 AM
That doesn't work when the interaction is complex, with callback relationships and bi-directional message passing.

A control does not have a type relationship with Windows. It has a message-passing relationship. While that, too, could be mocked, it's a lot more complex...to the point of not being worth it.

As someone else pointed out, you can still be test driven in this case. Just not NUnit test driven.
Maruis Marais wrote re: TDD Is Great...Except When It Isn't
on 02-08-2006 11:21 AM
Yeah, it is difficult to do TDD in some instances. But look at TypeMock.NET from http://www.typemock.com, it is a really good mocking framework.

I've posted a article the other day on how I solved an issue with a external Web Service depandency. Have a look at http://exceptionz.blogspot.com/2006/01/using-typemocknet-to-mock-external.html.

Cheers,
Maruis
Craig wrote re: TDD Is Great...Except When It Isn't
on 02-10-2006 7:37 AM
I've certainly used hand-rolled mock objects to the same effect. In fact, I've done it with web services.

The fact that TypeMock is a commercial product pretty much guarantees that I'll never use it, though.

And there's a *huge* difference between mocking a component's interactions a web service and mocking a control's interactions with Windows.

Add a Comment

(required)  
(optional)
(required)  
Remember Me?