The Unbearable Lightness of Testing
by Dima & Alena

Auto generated by ‘Deep Shift Forum2Play’ convertor from phpBB Topic ID: 15345 on 18 March 2009

One act play about testing and usefulness of brain storming

Any resemblance to actual persons, living or dead, is intentional and purely deliberate

CHARACTERS:

Dima – Java-hedge-hopper.
Alena – PHP-interceptor.
Igor – heavy bomber.

SCENE I

Office, fish tank on a drawer, gold fish of unknown kind in it. It is dark and cold outside. The wind is driving snow against the window, making a light tapping noise. Igor is sitting in front of a table and sketches something with a pencil on a piece of paper. Dima enters the room. He sits down on a corner of Igor’s desk taking a load of his feet, flicking away some eraser crumbs as he sits down.

Dima. Listen, we are going to write Recruiting tests…

Igor puts aside a complex and scary looking diagram.

Dima. Though I have already written a testing plan for the ‘Recruiting Web Site’ object, I still have some slippery questions. When I started, I thought that these slippery questions would find answers along the way, but they never did. So, I have written a plan that might not work until I’ve found the answers to these questions. I am still very uncertain… Do you mind if we talk about THIS?

Igor. THIS is also quite disturbing to me. THIS is one of the reasons why we spawned our ‘exercise’. But let’s call for Alena, it is well known fact that two heads are good, but three heads are worse than four.

Dima goes to fetch Alena for the discussion. Igor starts making coffee in a grimy Turkish coffee pot (a ‘Cezve’ as the Turkish call it).


SCENE II

Same and Alena. All are sitting around big round table and are sipping coffee, nearby to where they sit; there is a small white board on a wall. Also, there is a luminescent back light from a pair of large and quite flat monitors. Dima is keen to begin, and he starts by thinking aloud…

Dima. I still do not understand… What we wish to achieve by launching the ‘automatic test package’? In general, I consider two possible reasons to run such tests:

First, it is possible to find existing glitches (for example, by having an ability to look over different characters entered into input fields in automatic mode, or search through all links and check their validity again in automatic mode).

Second, it is possible to write tests which are mimicking all actions a user can do during application session.
This will also include invalid actions that the developers have already coded for, things including input content and type validation. So we are looking at all of the expected user interactions, and also testing for unexpected user input too. Our tests will emulate all such actions and decide if system behaved correctly or not. Such tests, to me, are not the best for finding bugs. These tests are good to validate the application after code changes are done.

Igor. I can’t see a difference between your first and second options. We have an application and we need a test suite to check it now and in the future when the inevitable change to the application occurs. As for me, the main problem is that we have only a book but not a specification. I think it is this missing bit that confuses you.

Igor sprinkles something into fish tank.

Failed test means one of:
- application has changed but tests were not adjusted (we can model this with a platform by adding a new compulsory element somewhere);
- application error(for example we can save form without entering required element);
- error in test itself (we will debug to make sure this will not happen).
It is obvious that first two reasons are equal from failed test point of view. Only conclusions they led to are different. In first case – code of tests has to be aligned with changed application requirements, in second – application has to be fixed.

All noisily sip coffee and sit in deep concentration, all of the team are in a meditative-like state wondering how they can resolve this.

Dima. So we need to do three things? Just like a thief matching picklocks, we need to find all or the combinations which will break the application and, we also need to check that all pages contain all elements, and we also need to check that those elements allow entering values according to spec… (Drops voice.) A spec that we don’t have.

Igor. I will have to be spec then.

All smile – even fish.

Alena. It will be handy, I think, to be able to run quick tests running through basic application logic and tell us if anything is wrong. We’ll also need to run tests like inserting something unexpected like ‘ζ(3)’ into a date field. The Quick tests will also try to do that but not as specifically and use common sense. For example, quick one will try to save date ’40 March 2009’. Difference between both types of tests will be distinguished by the terms – ‘hack’ and ‘typo’.

Dima. Getting better but I am not quite there yet. What if I offer you to try to approach same thing from another angle, and have us answer the following question – WHOSE ERRORS DO WE WANT TO FIND?
a) We are interested in SalesForce programmers’ errors.
b) We are interested in catching SalesForce users’ errors who used standard instruction to build Recruiting application (as we did a few posts ago).

Alena. This iiiis (drawn out) an interesting question. Platform testing is a bad idea, to me. We do not have access to source code and functional specification. Do I need to remind you that we do not have any spec?

Igor. OK, let us all define all that we have, and sort it out in a piecewise fashion together.

Dima and Alena are nodding ‘yes’ and passionate Igor continues to sing his song…

Igor. If we are interested in SalesForce programmers bugs, then we need to test a platform, but not Recruiting as it is just a special platform case most likely not even using all features of it. If we are to consider that the Platform testing will include testing interfaces we used when we were creating Recruiting application, it’s clear to see that this is not what we want to do.
If we are interested in checking how careful our users who built their ‘Recruiting’ application using a book are – we do not need to bother about basic element behaviours. Because those behaviours were tested before SalesForce wrote up the guide in the book for others to build it themselves. We only need to check that application was built precisely following a manual, just as the Lego ‘Ahsoka’s Starfighter’. It is not too hard to code but, again, is not what we want.

While Igor is continuing with his definitions, enthusiasm in eyes of his peers starts evaporating.

Alena (thoughtfully). What can we do than? Maybe we should consider this crazy idea that I have? (Speaks with escalating confidence.) Suppose Recruiting is an application having no platform behind it. We won’t see source code anyway and we will consider the book as a specification. We will define ourselves things which are not covered in a book. Let’s finish reading it – may be we will see some more things we will find interesting to test. Decompose application into logical chunks and start testing with simplest one.

Igor. Exactly! For us – test-writers, Recruiting is just a WAUT. We need platform to test our tests quality by changing application from inside and without having access to its source code. Furthermore – I have heard that you started analysing some parts of Recruiting I did not want to touch and related them to a platform. I made a picture-explanation, ‘seeing is believing’, hang on a minute…

With these words Igor exits place of action…

SCENE III

Same personages but Igor.

Alena. Do you want to check above all an existence of an element, its type (like ‘digital’) and format its format (like ‘two decimal points’) as well as its location, hints and error messages @#!? Better to be shot dead, seriously! Besides, stories about element sizes and quantity of checkboxes on pages will not interest our readers.

Dima. Yes, why? If we do not test platform and do not test SalesForce users work – we only have one option left and you said it earlier – test it as a web application. Taking into account these newly discovered circumstances, let’s think together about what we can test?

Dima stands up and walks toward white board.

First, I think it will be ‘quite hard to near impossible’ to check the location, and I can not see a reason to ‘walk into that forest’. It is populated by checks on whether the element is hidden or overlayed on a page. I am not sure that we can walk this forest with our level of automatic testing understanding without freezing on the spot when we will see a big scary wolf.

Writes on white board bottom – ‘- hidden?’, ‘- visible?’.

Field element existence test – easy peasy – just try to give element a value and catch for error. To test field type – assign a value and try to save. We can not avoid these types of checks, really. To check element format – again enter correct and incorrect one and try to save. Can’t avoid this type of test either.

Writes on top ‘- exists’, ‘- type’, ‘- format’.

If a given field allows different formats – we have no choice but to try them all. If we can invent a smart way to define formats for a test we will be able to sit back and relax while tests are working and do not worry too much on time spent to try them all.

This forest I would walk in. I will not envy to that wolf-o-bug who will meet me on my way – I will fold him up like an origami. (Smiles.)

As for an elements ‘maxsize’ – it is only for <input> fields. We know its value from a book – it is a sin not to try and squeeze just a little bit more into it?
By the way, what about <checkbox>? It is tiny but a creature from OUR forest and you, Alena, please do not offend it.

Dima’s voice has trembled and Alena, as though accidentally, has looked away, choosing to examine a monitor who’s screensaver picture has just changed.

Alena. Why did the SalesForce take a dislike for <radio>? Did you notice? (I have mastered the art of ‘flipping the script’ – Alena thought.)

Dima. <radio> is under a cloud – fact. What did you plan to test, if anything (smiles), if my plans are so painful for you to consider?

Alena picks up a blue marker. She is wearing a blue jeans and has only just opened her blue eyes to the situation.

Alena. I was thinking on the most interesting parts – checking for valid/invalid values for each element type on a page. I would agree now with you that we need to check if element exists. Format checking is also important, but… Is it’s implementation different from the validity checks? I feel that there is a difference but can not word it yet.
100% agree with you on this. (Alena crosses text written by Dima on a bottom of white board.) We won’t do them. (Both look at whiteboard.)

Fig. 1 Board from Scene III

Dima. I’m not sure if it will help, but I think the format tests have to be done with allowed values formatted correctly for an element. Allowed values tests – attempts to save not allowed values.

Alena. If correct value can be entered by many ways – format testing is an attempt to save it by all such ways. I think this idea has to be visible in our code.

Dima. With Igor’s love of using pictures to explain things… I suspect he is about to show us either one large picture or many small ones… (Faces cringe in a comical expression of pain.)

SCENE IV

Igor is coming back with a peace of paper, and a tray full of Danish pastry and Krispy Creme donuts, mugs and boiling kettle in his hands.
Same place, same people, but with a re-energised feeling of hope.

Igor. It is good that we have an open plan office as otherwise how I would knock the door? As I look around, I can’t see any broken furniture, so does this mean there is a chance for us all to find consensus?

Alena and Dima tell Igor a brief summary of Scene III. Strange stuff – he does not comment or ask anything – just nods, chewing a donut after donut sipping a hot tea from a small tank, he calls a mug, a giant piece of squashed lemon sits in the bottom of his mug.

Igor (dabs with his finger into a piece of paper he brought and finally interrupting his ‘big-mug-tea-ceremony’). Look!, all we are considering is a Recruiting application here. We don’t have time for rest. Let’s consider a blurred area a ‘Platform’ and ignore it in our tests, and consider the remaining areas as our Recruiting application.

Fig. 2 While Igor was bringing this picture all problems were solved (but thanks for donuts)

Igor. This will reduce amount of work but will not simplify it.

Alena & Dima. We – AGREE! (Say it all at once and stare at each other.)

Dima. I would also like to sound a good business case to have and use such tests.
I started this conversation because I did not quite understand what the goal was for our testing. Why will these tests be needed in real life?

I feel that such business case is needed not only for me, but also needed by anyone who wants to clearly plan and understand the activities of testing.

I will share with you now my ‘business story’ that I think is close enough to reality.

We are a software company. We have a web service we created and support, this being the Recruiting application. Every time new features are introduced (quite often) we check if the system does not break. As we never have enough time – we only check areas we believe could be affected by our changes. Next – step we upgrade live servers our application is hosted on.
Usually next day we, as a rule of thumb, get some errors in logs or, even worse, reported by our customers. Very often broken places are not the areas which undergo testing before release.
The conclusion is forced upon us – it is necessary to check everything with each new feature introduced. As we plan to be in Recruiting business in many years to come it will be extremely ineffective (time=money) to do it manually.
Solution – automatic testing with following goals:
1) check existence of all objects and all elements within;
2) etc., etc.

Igor (continues with a strange for this time of the day enthusiasm). On top of that – Recruiting is a stable application. We have lots of users and big plans to enhance it. We want to add Firefox and Safari support. We want to create a boxed version in addition to a working now SaaS which will support not only MySQL but PostgreSQL and may be even MS SQL. We want to integrate system with main recruiting web sites. To achieve such high goals without compromising quality we want to test our work with automatic tests and rest by hiring a Tester (experienced tester = Tester). We do not want to embarrass ourselves by offering a Tester to check if character ‘X’ can be saved in ‘Maximum pay’ field.

Alena. Phew – it looks like we are finally seeing a blink of the light at the end of this very dark and very long tunnel.

A previous unlit fluorescent light flickers into life, bringing a new clarity to the situation.

Igor. One more thing, while I was listening to your tests classification (I wasn’t only eating the donuts!), I realised what could be a good next step for us. I think we need to classify all types of elements and describe their allowed values and formats in our WAUT.

Dima. I have a petition please! Let’s include full lists of things we want to check into our ‘business story’?
I would offer that we find all possible types of checks and classify them in the following form:

1. Error: element is missing;
Test – attempt to assign element a value;

2. Error: incorrect element type;
Test:
a) digital field: allowed values list, not allowed values list, after record save is shown as this value;
b) string field: allowed values list, not allowed values list, after record save is shown as this value;

3. Error: required field is not mandatory;
Test: try to save the object without entering required element.

Will it be the next step, after classification and analysis?

Then our future plans will not swell from the repeating information overflow. (Comes passionate to the board and picks up a marker.) We will greatly simplify our plans by classifying all sorts of errors with samples like this (writes):

Object Candidates

Tests:

1) (exists): list of elements;
2) (type): list of elements by type;
3) (required): list of required fields.

That’s it. We will only have to cover things which are specific for objects and their dependencies. We can try to formalise them later too.

Alena (picks up her favourite color marker). If you don’t mind, I would suggest we can make it more coherent if we add into point (3):

(not required field): list of not required fields.

Igor. What if I’ll try to summarize, and please stop and correct me if anything doesn’t look right?
We have standard elements having some attributes – for example, date field. We will categorise all such types of elements and their allowed values and formats. It will be our step one. Elements are bricks. We can either sculpt a giant chimney pipe or start to build our own Great Wall of China.

We have standard checks like: exists, required, type, format. This is our step number two. Here we already know that we build a palace and bricks were delivered on site and not out-of-spec. They are feather-edge ones with a right amount of cells and color (format), checked the building foundation (field exists) and mixed up the right quantity and consistency of cement mortar (mandatory condition).

Somewhere here we need to have dependency checking between elements or their attributes. An example of this is from our WAUT where ‘Minimum salary’ value has to be less or equal to ‘Maximum salary’ value. Step three? Here we not only know that we have built a palace but we have the knowledge and tools to check the walls are straight and all the rows of bricks sit nice and horizontal (dependencies testing).

Igor scripts quickly his conclusions and asks to look at a result with a warm look of his eyes.

Fig. 3 Board look at the end of the play

Alena. I propose to make first two steps (deliver bricks and lay a first row) than look behind, check horizontality, tiptoe and look ahead and only than make a third step.

Igor. Deal?

Dima (trying to smooth a yawn). OK – tomoooorow, to be exact, already today we will start our analysis’s duties.

Behind a window the dawn breaks. A rhythmic snore emanates from inside the aquarium… Someone else is disturbed by the sounds of a key in the door lock from outside, as another colleague arrives at the office… A new day was about to begin. Luckily for the testing group, it’s another working day… So the plans can start to take shape!

“There is no means of testing which decision is better, be­cause there is no basis for comparison. We live everything as it comes, without warning, like an actor going on cold.”

Milan Kundera “The Unbearable Lightness of Being”



Print this post | Home

Comments are closed.