SigmondsMart
Fitnesse

Fitnesse is an integrated stand alone wiki and acceptance test framework, that can be used as a test and collaboration tool. This article (only in English) describes my experiences with the tool.

Using Fitnesse with Java Fixture for system integration testing:
I have used Fitnesse open source tool in a couple of projects for system integration test. It started with the Fit protocol, and in the latest project I used the Slim protocol. I really like the wiki Camel case way of doing it. At first the Fitnesse syntax looked quite weird, but after using it for a while I like it more and more. I would certainly recommend to use Slim, since it is the newest protocol and easier to write fixtures with it. It is really easy to build a test Suite, e.g. using Scenarios for reuse and Script Fixtures for the individual tests. As a programmer sometimes I wish that writing the tests in the Wiki could be more like a programming language. A Fitnesse test as written in the Wiki lack conditional statements and looping possibilities. This is done intentionally by design, so that it should be easier for non programmers to understand and write tests.

I have used Fitnesse for both synchronous integration testing and asynchronous integration testing. Typically in synchronous testing it is a request/response model with response within a few seconds. If an API is called like that from GUI, a response time of more than a few seconds really feels bad. Send a request, and check if the response is as it should be using Fitnesse. It is very easy to visually verify an OK result in Fitnesse when everything is marked green in the Wiki and red if the call failed.

But how can you test in Fitnesse if the process is asynchronous and a result may or may not be received e.g. within 30 seconds? A conditional statement to check response is impossible and looping is impossible. The answer is simple, this must be done in the Fixture code call from the Wiki (In Java or other language). So write a loop in Java that calls the API that terminates either if a result is received or if a time limit is exceeded. If the Java fixture function returns true, then the corresponding Fitnesse GUI row is marked green, else is will be marked red. Either positive result or negative result, a negative result might also be response not received yet. The time limit is the longest time it can take before a result is returned to Fitnesse. If the time required to verify a result from an asynchronous process can be more than one minute, I would not recommend to use Fitnesse.

I have used Fitnesse to write Fixtures that calls web services (SOAP and REST). I have used it to call message queuing software. I have also used it to call file transfer software (e.g. scp) and to call Linux commands. I have used Fitnesse to verify log statements, e.g. by using REST interface to Kibana log collection tool. This can sometimes be a recommended solution, because then the testing will to a certain degree not interfere with other users and organizations using the same test environment (e.g. when you read a message from a queue).

I have used Fitnesse for some kind of limited performance test. E.g. to send 5 messages at the same time and wait for result (synchronous or asynchronous response). Either the Wiki text have to be repeated 5 times in the test, or this must be done in the Fixture code. I have done both. To my surprise I detected several serious errors in the SUT (System under test) this way, so can be worth doing.

JUnit tests (exclude them when deploying code) can be used as integration tests. You can use tools like Cucumber, BDD style and annotations to do it. The big advantage of Fitnesse is the visual GUI, and that testers can write and reuse tests without learning Java or Java IDEs like Eclipse. Yes, it is a bit more work to set up Fitnesse than using JUnit directly, but it can be worth it.
Lessons I have learned from experience:
Fitnesse is really ideal for integration testing and system integration testing, but it requires some work to set it up to be usable in practice. For a small team I really recommend local installation on each developers/testers PC and use of a version control system like GIT to maintain the wiki files (in addition to the Java code). One problem with this is that you may need access to servers to call the SUT. I used e.g. ssh and scp and Rest for this. I have not used Fitnesse with large test teams or multiple teams, so I will not recommend any particular method for this. Either the wiki servers can be stored locally or centrally.

The main challenge with Fitnesse is the lack of reusable Fixtures out there. I have looked at other test tools like IBMs quite expensive RIT tool (Rational Integration Tester) and SOAP UI. If you are to test web service calls (SOAP or REST), these tools really do it for you. RIT can also do testing of message based software and a lot of other protocols WITHOUT any programming (but with some advanced configuration instead). There is one RestFixture (Smartrics) that can be used, you can google it.

If you like to or have to program the Fixures yourself, Fitnesse is ideal. But it can be some work to make e.g. a Rest call or SOAP call or message based call this way. If you are going to test some SUT with more than web services included, SOAP UI cannot do this. Programming the Fixtures is very easy in principle and easiest with the Slim protocol (Pojos). But using e.g. standard Java or 3rd party APIs is not always trivial...
Advantages with Fitnesse
  • Ideal for synchronous request/response testing
  • Asynchronous testing is possible if carefully written Fixtures and not too long response times.
  • Intuitive and easy test GUI, when you have got used to the basic concepts
  • Visually appealing when wiki text is written the correct way
  • Just push a button for regression tests (one test or an entire test suite).
  • Configuration of wiki at any level (test or test suite or sub test suite).
  • The test log (e.g. Java logging) can be inspected directly from the wiki.
  • Stand alone installation is very easy locally on a tester/developers PC
  • Reusable Wiki templates (e.g. scenarios).
  • There are many ways for a team to use Fitnesse, either centrally stored Wiki or locally
  • Set up the correct way, you can generate unit tests from Fitnesse with the Fitnesse Junit test runner. The main advantage with this is easier debugging of the Fixture code (in IDE, e.g. Eclipse).
  • Very flexible since you can program your Fixtures in several languages.
  • Fitnesse can be used together with GIT or other version control systems
  • Internal version control system really like undo button/local history in Eclipse, you can choose to use it or not.
  • Edit wiki pages after startup av Wiki server or as simple text files
Disadvantanges with Fitnesse
  • Initially getting used to Wiki markup
  • No syntax checking of wiki pages before test is run.
  • Verifying special characters can sometimes be tricky (due to wiki page syntax).
  • Fitnesse uses itself as documentation Wiki, but still I feel it could be more information there.
  • It is not that easy to just google and find valuable examples
  • How to best use Fitnesse as a group collaborating software is not very well explained in documentation. There are several alternatives.
  • Advanced configuration and setup can be quite tricky
  • It is not that many written fixures (within Fitnesse or elsewhere) that can be reused
  • It can be time consuming to program the Fixtures (depending on complexity)
  • Fitnesse and maven is not well integrated.
  • Typical errors caused by not finding Fixture code can be non intuitive
  • I have got some very annoying errors in Wiki text where changes are unexpectedly reverted, probably caused by version control issues *) But actually unsure if this is a problem related to a particular version of Fitnesse or if it is maven/Fitnesse setup used in that particular project.
Summary
So stay Fit. No better to stay Slim. (Even better, run in the forest instead of sitting in front of the computer and stay fit and slim). When I search for Fitnesse on internet, usually I find a lot more about physical exercise or training centres than this software. But I can absolutely recommend this open source software for system integration testing.