SigmondsMart
Test driven development

This article (only in English) describes experiences with using TDD The article was first published on IBM connections (intranet), but is not at all IBM specific. TDD is abbreviation for Test Driven Development. TDD is a very common term within modern system development, the methodology is getting more and more used across the globe, and many books exists about TDD. PDD is abbreviation for Prototype Driven Development. Prototype driven development is know as a method within system development. and not well known, but I will use it in this article.

I have written this entry because I have been programming in several languages using PDD ( C / C++ / Delphi / Java ), and lately using TDD (Java, Junit, Mockito) in a project for a Norwegian public sector customer. What I actually feel following the TDD guidelines for one year, is that the advantages of TDD is somewhat exaggerated; especially in the Java community. But I do not say we should not unit test to some extent.

    Advantages with TDD:
  • Automated tests gradually increase code quality. This is because most of the code are continuously tested and new errors will be picked up.
  • Test first forces the developer to think about program structure before coding. Developers get used to writing code that is easily testable.
  • Tests prevents development of unnecessary functionality. It is too much work to write tests for "nice to have" functionality.
  • Reuse and rerun of tests make it much easier to verify the code after additional development.
  • An error found in unit testing, is much cheaper to correct than an error found in system test or even worse in production.
  • It is simpler to measure code quality.
--> TDD can simplify maintenance and error detection in code.
    Disadvantages with TDD:
  • Writing unit test code first does not prove that the selected technically solution of collaborating components actually works. Focus is set on defining and testing implementations of functional interfaces, instead of prototyping the technical solution.
  • If a restructuring of the code is required later (e.g. choose a different open source component), most of the tests have to be restructured as well. The developers can hesitate to do a required restructuring of the code, due to the amount of work to rewrite the tests.
  • There is no focus on creative thinking and rapid prototyping in TDD. This can be required in certain situations.
  • More than the double amount of code to maintain. E.g.: If there is a requirement for high branch coverage, this demands the unit tests to run through all possible variants of branches in the code.
  • Coverage (code and branch) test goals, can result in testing the obvious to increase test coverage. A developer knows what parts of the code that really is critical code, and the coverage tool don't.
  • Unit tests does usually not test the environment or interacting components. (e.g. the database interface). Integration tests are required to test that. I often feel that more errors are detected testing interactions between components.

--> The productivity can decrease due to a large codebase.

I will not go any deeper in analyzing the advantages and disadvantages of TDD, and I will not attempt to conclude if TDD worth the effort or not. More scientific approach is needed to do any conclusions. Instead I will discuss certain aspects of test driven development.

PDD:

I have previously worked a lot with what I will call prototype driven development (PDD). PDD is well suited to develop new technical solutions, or where the components used are unknown or partly unknown to the development team/company. When the initial analysis phase (including some design) is completed, we start to develop an absolute minimal solution. The minimum solution is what that is required to make the system technically work. If the project detects that the technical solution need to be modified, we have to restructure the code. In addition it might be required to switch some of the 3rd part components. In this phase of programming "unit test first" is only unnecessary waste of time.

    Advantages with PDD:
  • Architectural choices will be verified quite early in project. Expensive restructuring of code at a later stage can be avoided.
  • The code will work through all layers. If the different components are put together late in the project, you will risk surprises. This issue I have observed several times in large development projects. Different teams will not always think the same way. Even if the interfaces are well defined, we risk both technical and functional problems at a detailed level.
  • The project will be able to produce a minimal solution that can be showed to the customer at an early stage as a proven concept.
  • It is easy to plan and execute early versions of integration tests.
--> By verifying design as soon as possible, a lot of work on the wrong solution can be avoided.
    Disadvantages with PDD:
  • It can be too easy to just start and try to write some code. It is important with analysis and design phase before start coding.
  • The code can easily be unstructured.
  • A prototype is not a production system. Developers can do shortcuts that need to be corrected later.
  • Unit tests is not prioritized. Too many noncritical but still important errors can be ignored. We cannot have too many of them when releasing the software.
--> PDD can result is too little focus on structure and functional details.

Discussion:

Most unit tests are written as black box testing (we can also test that methods in the system to be tested are used, and to some degree internal behavior). We put some parameters into the black box, and out comes a result from the box after calling public functions. Then we verify that the result is correct according to the test. If we write unit test first, we have to define the boxes without content first. The developers code the most important tests towards the box, both happy day scenarios and exception scenarios. At first there is no real contents in the box. As the developers continue to code, more and more detailed logic is added. It will be required to change some of the tests or write more tests later.

If the integration/system tests or code review or customer verification or other kind of analytical thinking shows that the code is not good enough, we have to do a re-factoring of the code. The time we used on the early unit tests can be a waste of time. It is possible to write a system where all the unit tests are passed, but the system still does not work as intended. This can be because we have forgotten to test the interactions between components through all layers (from GUI to database). And even if the system works as we thought it should, it was not what the customer really wanted.

Often I can find coding mistakes that is not picked up by the unit tests, simply because the unit tests I wrote myself; did not test for everything. So unit tests is not miracle cure for preventing errors (but it helps). For me it can be just as important to test the interaction between layers and components. Unit tests seldom reveal functional or usability weaknesses with a system. It is important that unit tests do not remove focus from other quality assurance activities like code review, integration and system tests and quality control of GUI design.

The project can make it a requirement that the code checked into the version control system is without errors. Without errors in this context means that the code at least can be compiled and deployed, is written following certain guidelines, and that all automated unit tests are passed. We have in addition used tools that check code quality that we should run before we commit the code. It can be an advantage to get rid of bad coding habits at once. But sometimes this can be somewhat disturbing to achieve the main goal: To get a working application that also are well designed for the end user. A remark on the often used requirement that all unit tests must be able to run without errors before checking the code in: A minimum requirement for developers working on the same code-base (branch) is that the code compiles and runs. But do we initially need more requirements than that? Perhaps this requirement should be relaxed in the initial phase of a project, for convenience and backup reasons. One simple solution can be to use code branches for initial (prototype) development, where we relax the requirements that it valid for normal maintenance of production code.

I am not always comfortable with writing the code for unit tests first. One situation is if I have to reuse code that is unknown to me as a developer. I would with like to get the code to run first through all layers, rather than doing the detailed unit test stuff. In essence I would like to make a working prototype of the new functionality. But I admit I have learned a lot about unknown code by running and studying the existing unit tests. TDD can also be executed with a "write most of the tests last" philosophy (at least not start to write the unit tests before the prototype is completed).

Most of the classes in test are dependent of other classes in and outside of the system. If these classes are not simulated (mocked) in some way, it is impossible to test a class in isolation. Many of the unit tests will include mocking code that simulates input from dependent classes. If we are interfacing a database, web service or other external component such mocking is absolutely required. My experience is that mocking tends to complicate the unit tests. It is not always easy to debug why a mock does not work if I have done a programming mistake. Certain collaboration classes can be programmed in a way that makes mocking difficult. A rewrite or wrap of those classes can be done, but can often be time consuming. The most common mocking techniques relies on inheritance mechanisms. E.g. if you use Mockito. If inheritance cannot be done (final classes, static members) you have a problem. Mockito cannot even mock a value returned from a constructor. Yes, in an ideal world the Java code should be written in a unit test friendly way to start with... It is not by the book, but developers can test several components or traverse through several layers by writing unit test code. Junit permits this. In Java we often have a recommended layering model where some of the layers in simple cases will be without much logic in it (the same object is just passed up and down through the layers). Testing each layer separately can be a waste of time.

If the developers work with an application and technology that they know well and are programming new functionality based on well known components, then TDD with test first seems to be a good philosophy. TDD can be very rewarding for typical maintenance projects. But the real word is not always like this.

In projects with technology (components or component libraries) unknown to developers in the team, or where the components are put together in a new way; I will recommend to start with PDD. In certain phases in the life-cycle of a computer system, components libraries have to be replaced. In that case PDD could also be applied. It is possible to combine TDD with PDD for different parts of the same project, or in different phases of the same project. We can prototype parts of the code, but not everything. When the prototype is verified to be OK, developers can start to think structure and write the unit tests.

Summary:

I would like to have seen more scientific studies of the consequence of using TDD. Does an optimum test coverage percentage exist? How much does maintenance and writing of automated unit tests cost, and how much is gained by fewer errors and increased software quality? Most software written in the world do not send rockets to the moon. We should write software with sufficient quality, but what is the definition of sufficient? (What the customer demands).