Review – What is a ‘fatal’ in tests?
by Dima

This post is answering a question “Which events should stop testing process?”.

This time we are giving our list at the beginning of the post and explain how it was built below.

Here it is a main list of fatal errors we compiled:

1. No Selenium server
2. Unsuccessful login
3. Missing element
4. Can’t save with known-good-values (KGV)
5. Wrong page

Most of these errors will occur at the very beginning of testing. If we will not face them – I think, tests will run just fine.
Despite of the fact that this list covers most of the situations when tests should be stopped I added two more types into additional list:

6. Unsuccessful attempt to assert a field value
7. Too many non fatal errors



Justifications

1. Self-explanatory – simply no Selenium found.

2. If application has authentication we cannot continue if we can not login

3. Missing element

Discussion:

Dima: Do we need to interrupt tests immediately or should we try to check other elements?

Igor: I think we MUST have a rule to stop testing for ALL objects if there is a missing element.

Dima: What if an element is not mandatory?

Alena: Why do you want to stop testing of ALL objects? We can skip it and continue.

Igor: It does not make any difference if it is a mandatory element or not because it is completely different requirement. If element presence will be not mandatory I will agree with you – we should continue testing.

Dima: But … even if it is missing and because it is not mandatory we can test all other elements even though object ALREADY screwed from tests point of view.

Alena: If we wouldn’t find, for example, an ‘Employment Websites’ tab on a page – this is a really fatal error!

Igor: Ok. Explain to me please, how will it work if the element ‘Max budget’ is missing and all tests are trying to tuck a value into it?

Dima: We will try, it will fail, we will record an error that we cannot assign a value and we continue.

Dima: Every time we touch ‘Max budget’ process repeats.

Igor: I think a missing element is a critical thing and it does NOT make sense to continue testing.

Igor: I will give an example – we are testing the element ‘Web Address’ and trying to insert KGV into ‘Max budget’ but it is missing. What’s next? What error message will there be?

Dima: … attempt to save record … element ‘Max budget’ not found while attempting to assign a value ….

Igor: And … we will see this message about fifty times while we are testing ‘Web Address’ than fifty times while testing ‘Price Per Post’. Yes, we can continue tests technically but I do not want to be a guy who will be looking at those logs and trying to figure out what has happened.

Dima: Of course, it is easier to stop testing after this type of error but we wanted to know if there is a reason to do so.

Igor: In this case it will be required to take it into account by all future tests, report it once and suppress all similar errors after.

Pluses and minuses to continue tests after the missing element was found:

+ tester will be able to see other errors and report/fix them if we will not stop testing. We save one test run.
- missing elements in different objects could interfere with the production of even more errors which will most likely cause an error avalanche effect. It could be possible to solve this problem by defining if the element is critical (all compulsory elements could be critical in this sense and some non compulsory could be made critical manually too)
- even if other tests can continue we, as user friendly folks, should suppress repeated errors. This is not hard to do – we need to mark the element as missing and do not assign values to it in other tests.
- in any way it is much more work with the only benefit to end user to save a test run

The unescapable conclusion is – one plus and three minuses. The tester, while analysing a report, will have to spot missing element messages and keep it in mind while looking at other errors and think if they could be triggered by a missing element.
An analogy would be if we need to test a car without a brake system. We decided it won’t be a problem. We will test the other car components and we will test the brake system after the mechanic will receive and install it from the supplier. The pilot gets in, starts the engine and drives. He approaches his first intersection, we can hear tyres squeal and car crashes into traffic light. Here we are analysing the reason. We needed to reduce the car’s speed before turning the steering wheel but we did not have breaks in the car so we didn’t. Next step was – we needed to turn the steering wheel and, we did. Can we judge if this step was done correctly or may be the steering wheel is not sensible enough or may be … the car’s speed was too high because we hadn’t used the brakes? Questions, questions…

Finally we decided to include this type of error into a list of fatal errors.

4. Can’t save with known-good-values (KGV)

Before testing an object we fill all elements with KGV and save it just to make sure we can do it. If we can’t – it could be at least two reasons:
- we have a mandatory element we are not aware of (message will be ‘Error: You must enter a value’).
- valid values we know are in fact invalid. We know that ‘Price Per Post’ element should ‘eat’ value $50.0 but it does not. Typical reasons – the application changed and tests were not adjusted or there is version mismatch between application and tests.

Even though an error is totally fatal it is hard to identify a reason. A human should look at fatal error 4, its screen shot and decide what exactly caused it. We can only flag about it by raising fatal error 4.

5. It is obvious we can’t continue testing if we can’t see the page we expected. It is possible to track incorrect pages with a few methods. We will compare expected and actual page titles for now.

During Save-Edit sequence of actions we compare the page title with the expected one.

Title checks are used:
- one time during login
- when we save object with all values set to KGV – checkIsRecordSavable() – two times, after ‘New’ and ‘Save’ buttons are clicked.
- standard sequences checks – checkSequence() – after each page change

6. “Selenium exception” only for now.

7. We can call it conditionally fatal. We decided to let testers decide if they want to use it and how. Each element has a counter of standard errors. You can set value when tests will fully stopped with fatal error number 7 once it is reached.
Why do you need such an error?
For example, we must have a currency element but in fact we have a text element. This element will pass basic checks without any problems. When specific tests will start we will get an error for each testing value. We decided that such big amounts of errors generated by one element can point to a fatal error and included it for convenience. It is conditionally fatal because you can switch it off.

Now let’s talk about errors, which wanted to be fatal but failed our screening process:

Object contains additional element (we know nothing about)

Java tests do not pick up this error and I do not even know how this can be done. Can we count elements and compare?
If such a new element (remember – from tests point of view it is invisible) is compulsory it will be caught by us and fatal error number 4 will be raised. We will fill the object with KGV’s without assigning a value to this invisible element and test will fail.

Compulsoriness error – compulsory element is not compulsory or vice versa

Discussion:

Dima: I do not think it is a fatal error and we can continue testing

Igor: We possibly can test this (Employment Web Sites) object even if one element became non mandatory, but what if this object depends on other one and we will expect values for this element while testing it? Nevertheless … in this case we will pre-create perfect ‘Employment Web Site’ records before testing that higher level object and it shouldn’t be an issue.

Dima: We can create a parameter called …. ‘continue_on_lost_compulsoriness’ giving tester an option to decide what he wants to do in this situation – stop fatal or continue testing.

Igor: Catching a situation when non compulsory elements become compulsory will require a special type of test for a non compulsory element which will attempt to save object with this element set to blank value while all others set to KGV. Even when caught, it should not raise a fatal error. If we want to give a tester control over this type of error we can use a twin parameter called – ‘continue_on_gained_compulsoriness’

Igor: I agree – not fatal

Testing values errors

We can’t have fatal errors here. If we saved object when in fact we were expecting an error message we will record test failure, delete created record and start next test. If we can’t save an object when we were expecting to be able to do so – again, we will record an error and start new test.

Field size errors

For example, we have element with maximum size 5 and testing value ‘12345’ but in fact element has maximum size 3. If we will assign a value, save a record and compare saved ‘123’ with assigned ‘12345’ we might decide that element failed this test. This will happen with all testing values we will use with size greater than 3. We can avoid this problem by testing field size first and toss aside testing values which are too big for the element under test.

Walk this way, walk this way
Walk this way, walk this way
Walk this way, walk this way


Print this post | Home

Comments are closed.