5 Commandments of Testing

After six months of calling myself a tester I am moving back into development for the forseeable future. However my time in testing has really opened my eyes to its importance in software development. The following ‘commandments’ are important lessons I have learnt about testing, in roughly the order that I learnt them.

1. Thou shalt not blame the tools
When I first started out my main question was “what is the best way to test”?  I figured out early on that this was not the right question to be asking. The point is that there is no right way to test, just as there is no right way to do anything really. All we can do is find a set of best practices, and then work to refine them as we find bits that don’t work so well and can be improved.

When I started testing, the main test tool we used was Concordion, an HTML based testing framework that allows the developer to plug Java code into a tester-written HTML page, which would produce a red or green HTML output based on whether the test passed or failed. There were a lot of under-the-surface problems with this. From a developer point of view, having to trawl through HTML to work out where to hook their code in was time consuming. From a tester point of view it was hard to work out what the code is actually doing as the relevant bit is hidden so far down the class structure that was needed for the tests. Writing the HTML pages in the first place was a pain, and this led to a lot of copy and pasting from old tests. This in turn led to massively long and complex tests that tested multiple things, and required so much set up that the tests took ages to run. It also did not encourage true agile collaboration. The tester would write an acceptance test, agree it with the analyst, pass it to the developer to implement, and then not see it again until the test was passing. By this point the test was likely to have changed as the developers had made changes during implementation. Luckily the general feeling was that this was not a good process and that something needed to change.

The tendency was to blame Concordion, that Concordion encouraged copy and pasting and asserting on everything.  However looking at the documentation shows that this is clearly not the case. In fact we were making most of the mistakes warned about in the technique page. It was not then the tool that was the problem, but the way we were using it. It was our process that needed to change.

We started by stripping complicated set up out of our tests and simplifying the language used. Because of the copy and paste nature of test writing it was hard to distinguish what a test was actually meant to be doing. Most importantly we made certain that every time a new test was written, it would test only one thing. In some cases this has meant more tests, but these are smaller and simpler and that can only be a good thing.

We are now trialling a new test tool but I maintain that it was not the tool that was the problem, and think that we made some big in-roads into improving our testing process while still using Concordion. The new tool we are adopting, Yatspec was developed by a guy at work who wanted to address the problems we were having with Concordion. Specifically we have moved away from HTML so that now everything is done in Java. And as I am moving into a development role in the same team this makes me happy that I won’t have to spend lots of time looking through raw HTML!

There is another test tool that we use for end-to-end testing which was dreamt up by some guys who used to work here but have since left. In the absence of their expert knowledge, the tool is flaky and inconsistent. It is worrying that we rely on successful tests run on this framework before we can sign off releases. The host of dependencies tests rely on here make it very different to a production system.

Yet here I encounter the problem that I am still blaming the tool. In reality it is my lack of knowledge of the tool and how it works that is the problem, and luckily this is something I can fix.

If we accept that there is no one correct way to test, then it stands to reason that we can only evolve the practices that we do have and see what works and what doesn’t. This is almost better than being told that there is a right way to do something; this way you are constantly learning and improving.

2. Thou shalt question everything
Testing is a frame of mind. It does not necessarily require copious amounts of technical knowledge (although that can help too!) It is the idea that you do not trust anything or anyone, and question it until you have done enough to satisfy yourself that it is true. This is a big part of the reason why I will never be a great tester: I am far too trusting. If someone tells me that the code is done and I can see that the test is passing, I believe them. As a tester I will try and confirm this myself as that is part of my job, but I would always expect it to be as they said. A great tester on the other hand, would not trust that the work had been done as expected. They would check every tiny detail until they were satisfied themselves that nothing had been missed. If a developers job is to build working software, then a testers role is to try and break it, and it is up to the two to make each others’ jobs as difficult as possible!

3. Thou shalt know that thy work is never finished
There will always be more bugs. Fact. Some things only come up after years, on certain days of the week, at four in the morning, and only when the customer is called Bob. This comes back to the unknown-unknowns idea that I’ve mentioned before.  You can’t test for everything, you can only test for the things you know about. So ready-to-release does not mean bug-free, it just mean that we have found all the bugs we can right now. We could not bother to release and just keep testing and finding more bugs, but that won’t get us anywhere. As a tester you have to accept that there will be bugs that you have not found, and that this does not mean you have failed. There is no such thing as bug-free software. There are an infinite number of configurations and environments and it is impossible to test in each one of them. Bug-free software might be the aim, but even if it is achieved we have no way of proving it. So your job as a tester is not to test that the software is bug-free, but that it is correct and does what it is meant to do.

4. Thou shalt always test from the user’s perspective
I’ve talked about this before, but I am still amazed at how long it took me to really appreciate the importance of testing the system can be used the way a real user is going to use it. I don’t think I have yet gone far enough to realising this goal, as I have not met my users. I want to know how they use the applications I test and what their biggest pain points are. As a tester I want to feel my users’ pain.

5. Thou shalt always see the bigger picture
The bigger picture
When I started out I was unsure about the role of the tester on the team. Especially when we were using Concordion, it seemed a lot like the tester acted as a translator between the analyst and the developer. But we have really smart analysts, who can write HTML and can talk to developers themselves. I was not sure how necessary the tester was (exploratory testing aside). As the six months went on however I began to see just how important the tester is. A lot of it comes down to the translator idea, but there is far more to this than I initially thought. The tester can see the whole picture of what is happening to the team. They communicate with the analysts who pick the stories which the team will play. They test around any assumptions made about external systems and relay the results back to the analysts. They write acceptance tests to define a complex requirement in as simple a way as possible, including any edge cases that need to be tested. Then they track the development work for that story through to completion, where they sit with the developer and analyst to look at the passing tests. And then they do any extra checks that are needed to satisfy them that the story has been completed and everything they can think of has been tested. This process happens for each story, and there are on average five stories in various stages of play throughout the iteration. So the tester has to know the detail and state of current stories being worked on by the developers, and upcoming stories from the analysts. For this wealth of knowledge alone testers are certainly crucial members of the team.

Further reading:


About RNewstead

I am learning every day. Sometimes I worry there are too many interesting things in the world and not enough time.
This entry was posted in Agile, Test and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s