Review – It is all about logs
by Dima

Now that the classes that can test SalesForce Recruiting application mostly fulfil their duties what are we missing?
Basically, we answered this question at the end of this post:

Creation – Day One (evening)

1) we have no fatal errors – no matter what happened tests do continue rolling new errors as a snow ball;
2) logging – we have all information in logs but in a very raw form which is hard to process further;
3) it is possible to get lost in a code jungles – comment it;
4) we did not use a possibility to run tests in parallel – this is because we have only one test

and even managed to discuss first problem.

It’s the perfect time to begin with the second question – collection and processing of information which is gained during the testing.

The question, how should reports look, and therefore, what do logs contain has worried us for quite a while.
It is very pleasant to see that your own tests have been doing a whole bunch of routine jobs on testing elements, but who will know of this? It would help if tests could create a report on the work they have completed and the amount of errors they have found, not a text file that looks like gibberish.

In version 54, which we have looked at, logs contained nearly all of the information which could be used to analyse errors. But it was presented as a text file, and honestly speaking, it would be more likely to scare than to excite. Other than text logs, screenshots were made, and before the test there would be an option to set certain level of messages that would be written down in the log (info, warn, error). You can read more about it here.

We are going to answer some questions, we had since, related to logs today.


Who will use the reports?

We need to make the reports usable for different types of users.

Firstly, this might be a person who will simply have to check if tests complete without errors. All he needs to see is if there were any errors during the test or not.
Secondly, this might be a person who needs to be able to reproduce all the actions that led to an error so he can tell the developer.

For every type of user the report must look a little different. Our main goal now is not the creation of reports, but the most amount of information that can be saved in logs, with which these reports can be made.

How to present the order of actions that led to the error?

One more requirement for the reports, an understandable description of the actions that lead to an error. From the tests point of view an action (or an “event”) isn’t only, for example, the entry of data into element or the press of a key.

Each method of tests is related to its actions. For example a check that with KGV the record was successfully saved – the GenericObject.checkIsRecordSavable() method is an action. Counting the amount of missing elements – the missedElementsCount() method is also an action. For simplicity, if the action is related to a method, we will call the action by the name of the method.

So that mean actions can include other actions in themselves. For example, the action missedElementsCount for the Login page with two input elements consist of two checkPresence actions; and in each checkPresence the isElementPresent action is used, which is related to Selenium method with the same name.

Therefore, the order of actions has a hierarchical structure. And this must be taken in account while presenting test results.

Screenshots

Ideally, you should be able to view screenshots at the start of an action (entry of a value, press of a key) and straight after it. In version 54 screenshots are only made after an error.

Tests only gather information

It became clear that the generation of reports should happen after the end of testing. Only then we will have the information about the overall test results (how many errors, how many warnings, which actions and checks should be marked as errors). This will allow us to create the generation of different reports independent of actual test runs. So tests save raw information into logs and what we will do with them it is up to us.

This is why we think tests should save all the actions that were completed, not just errors or warnings.

Where will the information stockpile while testing is in progress?

We could save raw data in a text file (like in version 54), to the database, in XML. We wanted the chosen storage to allow the presentation of actions hierarchy and specific components for every action. From this point of view XML suites us.

But we decided to cut a corner and didn’t write any data to a file at all yet, but keep the actions (or “events”) in the form of a list (ArrayList) of objects of a special class Event. These objects will store all the information about the action: action name, action start time, action end time, action target, value applied to target (for example, the value entered into a field), the expected results of the action, the result of the action, status (ok, error, fatal), message about an error, exceptions (if an exception was generated during the action), the names of screenshots before and after an action.

When everything will settle down, we will output logs to XML file during testing and will add XML file parsing for reports construction.

Let’s move to the details of implementation of these ideas.

Common access to the list of actions

Counting the list of actions, we have three things which should exist for every test in a single instance: Selenium driver, log4j logger, list of actions.
As we do not want to pass three references to each Java object, all the work with these common objects will be interfaced via instance of seleniumActions. It contains a driver, a list of actions and a logger.

Work with actions

How do we write an action into a list? Very easily.
An action can be opened and closed at any time, but, as we’ve already said, usually an action is related to a method. Logically, an action should be opened at the start of a method and closed at the end.
That is how it happens now. For example let’s look at the GenericElement.fillByNull().


Figure 1

Opening an action/event – means creating a new Event type object and filling in a few of its’ attributes, particularly, the name.

The action.typeText method, which is called upon in the action above, is also an action. Here is its code:


Figure 2

Screenshots

In the version 54 of tests, which we have been talking about before, screenshots were made only in the case of an error. Errors may have been found on different levels (for example, on the lowest level – during the generation of an exception from a Selenium driver, or on a higher level – during the comparison of an expected and an actual values), and this is why screenshots are made on different levels too. With this approach we have a lot of duplicated screenshots because an error on a lower level can lead to an error on a higher level and screenshots are made in both cases.

It became logical to move the creation of screenshots to those few functions which can change the page with the help of the Selenium driver and be removed from all the other functions. Functions like typeText, Click, waitForPageToLoad should create a screenshot before the execution of an action and after the end of it. With this approach we create too many screenshots including the ones we don’t need because they aren’t linked to an error. In this case after the end of every action, if it’s successful, we can delete the excessive screenshots. The other solution will be to delete all unnecessary screenshots at the test end.

The first report

After it was decided to use Events each method that was looked at as an action was “taught” to write down information about itself in the actions list.

Now everything was ready for the generation of any report after the end of testing.
For now we have created only one type of report but we tried to make it universal. From one perspective it allows us to look at all the information which was gathered during a test, but from the other, it could hide anything unwanted if we wish to.

We have decided that our first report will be in HTML format because:

1. With the help of HTML and JavaScript almost anything we dream of can be done
2. A complex report can be made interactive: show or hide the hierarchy of actions, show or hide different types of errors.
3. Next to every action in the report there can be a link to a screenshot which can be viewed straight away.

This is how it looks.




Figure 3

Besides that, additional information contains messages about the results of an action and a hint, usually in the case of an error.

Figure 4

You can download this report here and examine it further.

This log is your log
This log is my log
When lightning struck it
It kicked the bucket!

This Log – Lisa The Treehugger


/spanspan style=”display: none;” id = “1″

Print this post | Home

Comments are closed.