Purpose by Igor

Our internet application software development has grown from something small and evolved into something larger than we ever imagined. With 400,000 lines of code today and this figure increasing every day, our initially elegent solution has grown beyond the testing process initially used to ensure that our software interfaces are both easy to use and secure for our valued internet users.

 There are significant costs in remediating internet applications when malicious users decide to attack, is costly to the developer, the company, and the customer alike, as the application is taken offline for repair, or rolled back to a previous release. With the customer and developer unhappy with these new bugs, there has to be a better way to test new features and interfaces.

 With significant cost and time investments required to design and tune an automatic testing system, many companies may prefer to use the people most familiar with the code to test and repair the code. We acknowledge that automatic testing systems can certainly deliver long term gains, and such systems may themselves become a product that can be resold to other companies who are also struggling with the same decision. Only the future holds the answer to how this challenge will be addressed.

 To start developing such automatic testing solutions, we are aiming to understand the common problems and as a collective, we can organize our thoughts and ideas into solutions. With these solutions, we can aim to start developing the framework and solutions to ensure any internet applications integrity in the demanding 24×7 internet environment.

 The aim of this blog is to document our journey of exploring automatic testing, and we invite everyone who is interested to contribute to our effort or even just share in this experience that we are about to undertake. This blog will document our journey from conception to solution. It will stand as a living document to where we’ve been and where we are headed.

 This project has not come easy, and we would like to sincerely thank the companies and individual contributors, who continue to help us achieve our goal. It is our aim that the blog will be published in both Russian and English, but be prepared for the possibility that they may not always be in sync.

Target by Igor

After question ‘To be or not to be?’ was answered positively we needed to select a target or WAUT (Web Application Under Test).

What application should we use as a demonstration of our Testing Environment? Should we write it ourselves? Should we use our existing Web applications as way of demonstrating the testing protocol? If we were to write something new, the aim would be write something a fair bit more complex than the traditional “Hello World”, and hopefully settle on something smaller than “War and Peace”. The key part of selecting any sample application, is that it needs to have the propensity to have faults. From simple input validation issues, through to back end data file locking and error recovery. The thought of moving our existing web application into the role of WAUT, presented hosting technicalities in that our application was developed under an IIS/SQL2000 environment, and that a porting of our application to the open-source LAMP environment would be required before we could consider that as an option. The added trouble of switching to a Windows based hosting provider also made this a more difficult option. At this point, we were really just thinking about ‘getting on with business’, rather than getting bogged down with the conversion requirements for the sample web app test case.

We started looking at Salesforce.com as a potential WAUT. Having already looking at SalesForce about 3 years ago, we knew that it had the ability to quickly create sample applications. These applications would have the functionality (and potential vulnerabilities) required to demonstrate the ability of the testing protocols we would subject the application to as our selected WAUT. SalesForce’s open marketing strategies allows developers to access their platform via a well featured developer account, enabling and encouraging developers to embrace their technology.

After taking a long look at SalesForce, we felt that we needed to make a fair assessment of the big players in the market. This search led us to look at BaseCamp by 37signals. BaseCamp is a Project Management application with features like Projects, To-Do list, Milestones, Writeboards and Message features, all of which made this a potential candidate. We also considered using Yahoo/GMail as potential WAUT, with a plan to test not only Mail but also the other featured apps like Notes, Calendar and Contacts. The big restriction on this sort of proprietary web application is that it would never give us good visibility into application failures during the testing cycle.

SalesForce has been selected, as it had a functionality and feel that is very familiar to our own application. The open development interface allows us to make the WAUT as complex of as simple as required. After a cross-company vote, we decided that using a SalesForce Application for our WAUT would quickly deliver exactly what we need with very little effort or cost.

The Set-Up by Igor

SalesForce Recruiting application can be created using instructions from this book on a platform with a registration page located here. SalesForce periodically revises Recruiting application book following changes in the platform and/or Recruiting application. We decided to upload version we use on our server just in case future revisions of the book will be different from current.

Application consists of three main components. Company needs to fill a ‘Position’.


Fig. 1 Position

‘Candidate’ is a human applying for a ‘Position’ by filing a ‘Job Application’.

Read the rest of this entry »

May the ‘Plan’ be with you
by Dima

“A man who does not plan long ahead will find trouble right at his door”


Let’s get moving! How do we get started on our goal of automatic testing of web applications?

1) CREATE the Recruiting application.
2) BUILD a test environment for test execution and results observation. It needs to be convenient.
3) DEFINE what we need to test and how to recognize it if test failed or succeeded
4) WRITE tests that have a clear pass/fail criteria in their results
5) CONCLUDE what our findings are, based on our results

Stage 1. Application

We will create the Recruiting application by following the SalesForce book precisely. This will allow others to follow my work with ease.

Stage 2. Environment

In this stage, we will perform the functional testing of web pages. By this, we mean that we will check if the application works correctly from an application user point of view. The user must be able to use the application for the purposes it was designed. Appropriate values can be entered, buttons can be clicked, error messages are produced and the expected results are achieved when a user enters data in an expected way.

We are going to use a tool called Selenium. We will use Selenium Grid. To control Selenium RC, by sending actions to execute in a browser, special class is used called ‘driver’. There are Selenium drivers for many programming languages – C++, Java, PHP, Python etc. This makes it very adaptable.

As I mentioned before I am in the Java team.
I will use the following tools for our testing purposes:
TestNG to write tests, group them and run in parallel streams.
log4j For logging and debugging.
Ant to help build and accumulate all of my treasured creations.
Mozilla Firefox as a basic browser.
TortoiseSVN as SVN client.

To summarise and if I haven’t forgotten anything, I will need:
Ant, TestNG, log4j, Selenium Grid (including RC, Hub and driver), and of course I will need JDK. The fresher it is the better!
(If I forgot something I will correct this post secretly from everybody :) )

The primary goal for Stage 2 is to build all these tools into a workable instrument.

Read the rest of this entry »

Environment installation by Dima & Alena

We created this instruction for Windows platform as we use it on our workstations. We are not trying to build a powerful web service here, we just trying to figure out how to auto-test web applications. We love Unix too, and if your plan is to configure this on Unix, we expect that you will be able to assemble a similar setup using this post as a guidance.

Table of content

1. Required
	1.1 Java Development Kit (JDK)
	1.2 Ant
	1.3 Selenium
	1.4 Браузер (Firefox, IE)

2. Install to start Java tests
	2.1 TestNG
	2.2 log4j

3. Install to start PHP tests
	3.1 PHP
	3.2 PEAR
	3.3 PHPUnit
	3.4 Testing_Selenium
	3.5 log4php

4. To access our SVN repository
	4.1 TortoiseSVN

5. Additional information
	Why did we install log4?

1. Let’s start with required components

1.1 JDK

Please load it from here and make sure you have selected JDK (not JRE). All you need to do after download is to start the installation.
If you are a regular Java developer, you will most likely see many different versions of Java installed on you machine.  We installed downloaded package into С:\program files\Java\jdk1.6.0_12

Read the rest of this entry »

Why did we install log4? by Alena

You may be wondering – “why do we need log4php?’
I will try to answer this question in this post in just a moment – cognition comes through comparison.

No matter what format of logs we will decide to use (JSON, Test Anything Protocol or XML) – PHPUnit will only report to us, one of the following return states :

. – Test succeeded
F – Assertion Fails
E – Error
S – Skipped
I – Incomplete

These may seem rather brief, but for a person running the tests through PHPUnit, this level of feedback should be enough.

But we are not the people running the tests, we are writers. :)

What does a writer need? A writer needs an effective output from each line of code. Did the variables initialize? Did a given function return the correct value? Which branch algorithm did it choose? All of these things are of interest to the writer. To fulfil such curiosity, the test writers of the PHP team (humbly, me) decided to use log4php.

For example – one needs to test a page. Testing starts with a check which counts all elements that should be there.
From logging point of view, this can be performed in two ways.

Read the rest of this entry »

First launch by Dima, Alena

To check if our application environment works fine we recommend that you start by running a simple test – login to your SalesForce account, read the header of a page and logout.

By this step you should have JDK, Ant, Selenium-Grid, Firefox and TortoiseSVN installed.
Before we will be able to launch a test on this system, we will need to start
Hub & RC.

To start the Hub, let’s open a command window (shell) – ‘cmd’ and run:

ant -f C:\selenium\build.xml launch-hub

To start RC, let’s open another command window and run:

ant -f C:\selenium\build.xml -Dport=5565 -Dhost=localhost -Denvironment="*chrome"
-DhubURL=http://localhost:4444 launch-remote-control

For IE, change to -Denvironment=’*iehta’.

In the two open windows we just opened, you will see something like this:

Fig. 1 Hub & RC started

Read the rest of this entry »

The Unbearable Lightness of Testing
by Dima & Alena

Auto generated by ‘Deep Shift Forum2Play’ convertor from phpBB Topic ID: 15345 on 18 March 2009

One act play about testing and usefulness of brain storming

Any resemblance to actual persons, living or dead, is intentional and purely deliberate


Dima – Java-hedge-hopper.
Alena – PHP-interceptor.
Igor – heavy bomber.


Office, fish tank on a drawer, gold fish of unknown kind in it. It is dark and cold outside. The wind is driving snow against the window, making a light tapping noise. Igor is sitting in front of a table and sketches something with a pencil on a piece of paper. Dima enters the room. He sits down on a corner of Igor’s desk taking a load of his feet, flicking away some eraser crumbs as he sits down.

Dima. Listen, we are going to write Recruiting tests…

Igor puts aside a complex and scary looking diagram.

Dima. Though I have already written a testing plan for the ‘Recruiting Web Site’ object, I still have some slippery questions. When I started, I thought that these slippery questions would find answers along the way, but they never did. So, I have written a plan that might not work until I’ve found the answers to these questions. I am still very uncertain… Do you mind if we talk about THIS?

Igor. THIS is also quite disturbing to me. THIS is one of the reasons why we spawned our ‘exercise’. But let’s call for Alena, it is well known fact that two heads are good, but three heads are worse than four.

Dima goes to fetch Alena for the discussion. Igor starts making coffee in a grimy Turkish coffee pot (a ‘Cezve’ as the Turkish call it).

Read the rest of this entry »

Analysis by Alena & Dima

Life of web-tester is uneasy. We direct plays, write in a blog and also participate in the test creation process which is both unstoppable and exciting for us. Sooner or later we started on Stage 3 (with a realistic and conscious attitude). We continue to concentrate on it, and consider what to look at next. What can we see? We can see our application – Recruiting, assembled by manual. It is time now to X-ray it and proceed to Step 1 which was written once on a dark cold windy night on white board surface by a simple black marker.
Let’s begin.


SalesForce allows specifying Text field size when it is created. It can be any size as long as it is within 1..255 characters :) It is easy to check – enter as many characters as specified for a particular Text field plus one more and test if it can not fit. We know that SalesForce (SF) validation for the length based on setting maxsize attribute for <input>.
For painters & hackers: Other applications (outside SF universe) may allow more characters and check size with JS and/or on server side but this is a completely different story.


Field Phone allows input up to 40 characters. That’s why it is enough to push 41 characters to test this requirement. Phone will accept ANY characters which looks quite strange. Even though we are surrounded by phone numbers like ‘1300 ORDER TESTER’ we did not see a case where such numbers were accepted on web forms (especially in business web apps domain).


URL field may contain up to 255 characters so we can apply Text filed rules to test its size. If URL does not start with ‘http://’ or ‘https://’, or ‘ftp://’ it will be prefixed with familiar ‘http://’ when saved. The displayed URL is shown as an HTML link. Double quotes and ‘< ’, ‘>’ will not be shown. Adjoined pictures will give you some idea how we can test this element.

Read the rest of this entry »

Element-ary tree by Alena

Our ‘Analysis’ tree has bloomed and blossomed. QA gurus and wanna be testers stop and relax in its shadow. Here you go – another excursion from Junior Tester Club is approaching. Let’s listen…

‘Wind of Test’ circa 1999-2009 by SalesForce
Donated by ‘Deep Shift Labs Still Pictures’

Read the rest of this entry »

Looking for something? Visit the archives.