Using Nerrvana – final notes

Using Nerrvana - final notes

Testing the application with two database types – configuration files ‘as is’ – future plans

Part 1 – Using Nerrvana – our setup
Part 2 – Using Nerrvana – SVN hooks to Jenkins
Part 3 – Using Nerrvana – deployment & Jenkins (part 1)
Part 4 – Using Nerrvana – deployment & Jenkins (part 2)
Part 5 – Using Nerrvana – Jenkins setup for Selenium testing
Part 6 – Using Nerrvana – final notes – this post

At this point, we’ve talked about how we test our application with Jenkins and Nerrvana. In our recount, some details have been omitted in order not to complicate the essence of the process.

It’s now time to show the real configuration files.

Here’s the script publish.sh. As you can see there is not one section, as shown earlier, but two. One deals with unpacking and setting up the Answers application on deployment host to work with MySQL, the other with PostgreSQL.

#!/bin/sh
 
WWW_MYSQL_DIR='/var/www/answers/answers_mysql'
WWW_PGSQL_DIR='/var/www/answers/answers_postgres'
DB_NAME='answers'
 
rm -Rf $WWW_PGSQL_DIR/*
unzip -o prj.zip -d $WWW_PGSQL_DIR
mv $WWW_PGSQL_DIR/config/Settings.class.php.pgsql $WWW_PGSQL_DIR/config/Settings.class.php
chmod -R 777 $WWW_PGSQL_DIR
psql -U postgres -f $WWW_PGSQL_DIR/install/crt-pgsql.sql
psql -U postgres -f $WWW_PGSQL_DIR/install/postgres.sql $DB_NAME
 
rm -Rf $WWW_MYSQL_DIR/*
unzip -o prj.zip -d $WWW_MYSQL_DIR
mv $WWW_MYSQL_DIR/config/Settings.class.php.mysql $WWW_MYSQL_DIR/config/Settings.class.php
chmod -R 777 $WWW_MYSQL_DIR
mysql -v -u root  < $WWW_MYSQL_DIR/install/crt-mysql.sql
mysql -v -u root  < $WWW_MYSQL_DIR/install/mysql.sql $DB_NAME

Here is our build.xml. In fact, the same as shown previously, but with similar actions to MySQL, for PostgreSQL (lines 13-14, 22-23 and 35-36).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
<?xml version="1.0" encoding="UTF-8"?>
<project name="${env.JOB_NAME}" default="build" basedir=".">
  <property environment="env"/>
  <property name="tests" value="${env.WORKSPACE}/tests"/>
  <property name="ci" value="${env.WORKSPACE}/ci"/>
  <property name="code" value="${env.WORKSPACE}/code"/>
 
 
  <target name="make_settings">
      <exec executable="sed" output="${code}/config/Settings.class.php.mysql">
        <arg 
            line="-f ${ci}/deployment/config/mysql/settings-mysql.sed ${code}/config/Settings.class.template.php"
        />
      </exec>
      <exec executable="sed" output="${code}/config/Settings.class.php.pgsql">
        <arg 
            line="-f ${ci}/deployment/config/pgsql/settings-pgsql.sed ${code}/config/Settings.class.template.php"
        />
      </exec>
    </target>
 
  <target name="make_sql">
    <exec executable="sed" output="${code}/install/mysql.sql">
      <arg 
           line="-f ${ci}/deployment/config/mysql/install-mysql.sed ${code}/install/mysql.sql"
      />
    </exec>
    <exec executable="sed" output="${code}/install/postgres.sql">
      <arg
          line="-f ${ci}/deployment/config/pgsql/install-pgsql.sed ${code}/install/postgres.sql" 
      />
    </exec>
  </target>
 
  <target name="move_files" depends="make_settings,make_sql">
    <!-- Move authentication files to WAUT root -->
    <move file="${ci}/deployment/misc/login.php" tofile="${code}/login.php" />
    <move file="${ci}/deployment/misc/logout.php" tofile="${code}/logout.php" />
 
    <!-- Move MySQL db creation files to install folder -->
    <move
         file="${ci}/deployment/config/mysql/crt-mysql.sql"
         tofile="${code}/install/crt-mysql.sql"
    />
 
    <!-- Move PGSQL db creation files to install folder -->
    <move 
         file="${ci}/deployment/config/pgsql/crt-pgsql.sql" 
         tofile="${code}/install/crt-pgsql.sql" 
    />
 
    <!-- Move publish script to the workspace root, so it will be beside prj.zip -->
    <move 
         file="${ci}/deployment/publish-over-ssh/publish.sh"
         tofile="${env.WORKSPACE}/publish.sh"
    /> 
  </target>
 
  <target name="save_commi
 
build-mysql.xml в свою очередь внутри себя делает:
 
t_info" depends="move_files">
      <!-- Temp fix for SVN client and SVN plugin incompatibility -->
      <exec dir="${env.WORKSPACE}" executable="svn" >
          <arg 
              line="--username jenkins --password 123456 upgrade"
          />
      </exec>
      <exec dir="${env.WORKSPACE}" executable="svn"  output="${env.WORKSPACE}/version.txt">  
          <arg line="--username jenkins --password 123456 log -r ${env.SVN_REVISION}" />
      </exec>
      <exec dir="${env.WORKSPACE}" executable="svn">  
          <arg line="--username jenkins --password 123456 log -r ${env.SVN_REVISION}" />
      </exec>          
  </target>	
 
  <target name="svn" depends="save_commit_info">
      <java
           dir="."
           jar="${ci}/deployment/scm-decorator.jar"
           fork="true"
           failonerror="true"
      >
 
          <arg value="svn"/>
          <arg value="${env.WORKSPACE}/version.txt"/> 
      </java>    	
  </target>
 
  <target name="zip" depends="svn" description="Compress project files">
      <jar destfile="${env.WORKSPACE}/prj.zip"
            basedir="${env.WORKSPACE}/code"
            excludes=".svn" />
  </target>
 
  <target name="build" depends="zip" />
</project>

The Jenkins job configuration page contains one extra Nerrvana plugin step. The idea is to test the application with MySQL, and then with PostgreSQL. What are the differences between them?

First, they use different Test Run names as well as executable files.

The names should be different. Here are the test results after the build completion, when an application is tested in such a way. Names of Test Run’s clearly tell what databases we tested on.

I did not mention how tests know what URL to knock. The URL is set directly in the tests Ant file. We just haven’t touched this topic. There are two Ant files. Each of them sets a URL to the appropriate virtual host for Answers working with MySQL and PostgreSQL respectively.

That is, for example, xbuild-mysql.sh used to run tests for an application working with MySQL. Its contents is:

ant -f build-mysql.xml all > build.log 2 >& 1

build-mysql.xml does this:

<copy file="${resources}/config-mysql.xml" tofile="${output}/config.xml"/>

And finally in the config-mysql.xml file we see what tests will run and what URL will be used.

<properties>
    <entry key="suite-title">Answers Test Suite</entry>
    <entry key="tests-disabled">
        com.deepshiftlabs.testui.AnswersStartyCo. TestLogin,
        com.deepshiftlabs.testui.AnswersStartyCo.TestAddRemoveQuestionByAnonymous,
        com.deepshiftlabs.testui.AnswersStartyCo.TestAddRemoveQuestionByUser,
        com.deepshiftlabs.testui.AnswersStartyCo.TestSearch,
        com.deepshiftlabs.testui.AnswersStartyCo.TestReply,
        com.deepshiftlabs.testui.AnswersStartyCo.TestFilterSort,
        com.deepshiftlabs.testui.AnswersStartyCo.TestTags,
        com.deepshiftlabs.testui.AnswersStartyCo.TestSmoke,
    </entry>
 
    <entry key="website">http://answers.dslabs.lan/mysql</entry>

You can see that all the tests are currently disabled except TestLogin.

The second difference between the two Nerrvana plugin calls in our job is that in the first call to the plugin we synchronize tests with Nerrvana, and we skip it in the second one, since it’s the same code. This saves us some time. Also you can see that we switched off message analyzer in the first step of testing the application with MySQL, and enabled it in the second. Thus, we let tests complete on both hosts and then analyze messages and set build status.

The report prepared by the plugin is based on messages received from both steps and contains two sections.

Finally, under the spoiler you will find an example of a console log of this double plugin invocation execution (I shortened some verbose outputs).

Started by an SCM change
Building in workspace /var/lib/jenkins/jobs/Answers/workspace
Cleaning local Directory .
Checking out http://192.168.3.97/repos/answers/trunk at revision 146
A         tests
----------- SOME LINES REMOVED ---------------------------------
A         ci/toolchain/build-toolchain.xml
At revision 146
[deployment] $ ant -file build.xml build
Buildfile: build.xml
 
make_settings:
 
make_sql:
 
move_files:
     [move] Moving 1 file to /var/lib/jenkins/jobs/Answers/workspace/code
     [move] Moving 1 file to /var/lib/jenkins/jobs/Answers/workspace/code
     [move] Moving 1 file to /var/lib/jenkins/jobs/Answers/workspace/code/install
     [move] Moving 1 file to /var/lib/jenkins/jobs/Answers/workspace/code/install
     [move] Moving 1 file to /var/lib/jenkins/jobs/Answers/workspace
 
save_commit_info:
     [exec] Upgraded '.'
----------- SOME LINES REMOVED ---------------------------------
     [exec] Upgraded 'ci/deployment/misc'
     [exec] Upgraded 'ci/toolchain'
     [exec] ------------------------------------------------------------------------
     [exec] r146 | igork | 2012-11-14 08:22:44 +0000 (Wed, 14 Nov 2012) | 1 line
     [exec] 
     [exec] "Fixed" robots.txt to initiate Jenkins build
     [exec] ------------------------------------------------------------------------
 
svn:
 
zip:
      [jar] Building jar: /var/lib/jenkins/jobs/Answers/workspace/prj.zip
 
build:
 
BUILD SUCCESSFUL
Total time: 0 seconds
SSH: Connecting from host [phpbb.deepshiftlabs.com]
SSH: Connecting with configuration [Answers test server] ...
SSH: EXEC: STDOUT/STDERR from command [chmod +x publish.sh
./publish.sh] ...
Archive:  prj.zip
   creating: /var/www/answers/answers_postgres/META-INF/
  inflating: /var/www/answers/answers_postgres/META-INF/MANIFEST.MF  
   creating: /var/www/answers/answers_postgres/config/
   creating: /var/www/answers/answers_postgres/include/
----------- SOME LINES REMOVED ---------------------------------
   creating: /var/www/answers/answers_postgres/utils/
  inflating: /var/www/answers/answers_postgres/ask.php  
----------- SOME LINES REMOVED ---------------------------------
  inflating: /var/www/answers/answers_postgres/utils/DBUtils.class.php  
  inflating: /var/www/answers/answers_postgres/utils/Utils.class.php  
  inflating: /var/www/answers/answers_postgres/utils/init.php  
DROP DATABASE
CREATE DATABASE
REVOKE
DROP ROLE
You are now connected to database "answers".
psql:/var/www/answers/answers_postgres/install/crt-pgsql.sql:7: NOTICE:  table "USERS_SOURCE_TABLE" does not exist, skipping
DROP TABLE
psql:/var/www/answers/answers_postgres/install/crt-pgsql.sql:17: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "users_source_table_pkey" for table "users_source_table"
CREATE TABLE
INSERT 0 1
INSERT 0 1
----------- SOME LINES REMOVED ---------------------------------
GRANT
Archive:  prj.zip
   creating: /var/www/answers/answers_mysql/META-INF/
  inflating: /var/www/answers/answers_mysql/META-INF/MANIFEST.MF  
   creating: /var/www/answers/answers_mysql/config/
   creating: /var/www/answers/answers_mysql/include/
----------- SOME LINES REMOVED ---------------------------------
   creating: /var/www/answers/answers_mysql/utils/
  inflating: /var/www/answers/answers_mysql/ask.php  
----------- SOME LINES REMOVED ---------------------------------
  inflating: /var/www/answers/answers_mysql/utils/DBUtils.class.php  
  inflating: /var/www/answers/answers_mysql/utils/Utils.class.php  
  inflating: /var/www/answers/answers_mysql/utils/init.php  
--------------
drop database answers
--------------
 
--------------
create database answers
--------------
 
--------------
DROP TABLE if exists USERS_SOURCE_TABLE
--------------
 
--------------
CREATE TABLE `USERS_SOURCE_TABLE` (
  USERS_ID_COLUMN int(11),
  username varchar(50),
  USERS_DISPLAY_NAME_COLUMN varchar(50),
  USERS_EMAIL_COLUMN varchar(50),
  CONSTRAINT USERS_SOURCE_TABLE_pkey PRIMARY KEY (USERS_ID_COLUMN)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
--------------
 
----------- SOME LINES REMOVED ---------------------------------
 
--------------
CREATE VIEW `USERS_VIEW_NAME` AS
  SELECT
    `USERS_ID_COLUMN` AS `USERS_ID_COLUMN`,
    `USERS_DISPLAY_NAME_COLUMN` AS `USERS_DISPLAY_NAME_COLUMN`,
    `USERS_EMAIL_COLUMN` AS `USERS_EMAIL_COLUMN`
  FROM
    `USERS_SOURCE_TABLE`
--------------
 
--------------
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES_PREFIX_users TO 'ANSWERS_DB_USER'@localhost
--------------
 
--------------
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES_PREFIX_answers TO 'ANSWERS_DB_USER'@localhost
--------------
 
--------------
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES_PREFIX_questions TO 'ANSWERS_DB_USER'@localhost
--------------
 
--------------
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES_PREFIX_tags TO 'ANSWERS_DB_USER'@localhost
--------------
 
--------------
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES_PREFIX_tags_to_quest TO 'ANSWERS_DB_USER'@localhost
--------------
 
--------------
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES_PREFIX_subscribers TO 'ANSWERS_DB_USER'@localhost
--------------
 
--------------
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES_PREFIX_votes_a TO 'ANSWERS_DB_USER'@localhost
--------------
 
--------------
GRANT SELECT ON USERS_VIEW_NAME TO 'ANSWERS_DB_USER'@localhost
--------------
 
SSH: EXEC: completed after 1,003 ms
SSH: Disconnecting configuration [Answers test server] ...
SSH: Transferred 2 file(s)
Build step 'Send files or execute commands over SSH' changed build result to SUCCESS
2012-11-14 08:22:49 
 
---BEGIN PLUGIN EXECUTION---
2012-11-14 08:22:49 ---INITIALIZATION STARTED---
2012-11-14 08:22:49 
---BEGIN PLUGIN SETTINGS---
	Nerrvana HTTP address: https://api.nerrvana.com
	Nerrvana API key: 2bf7c81-a031f-1ad2-fd3a6-f59b9e0668e
	Secret key:  wctfFwelygx3tXS4TasrsOS4oXV7YadcPppvEnx55WG7qPk6jrAHjJi1TRoLlhrarIlabte4H9zWROXVkLnXto2LlVC47EXx91Uu
	Space ID: 4144
	Space: Answers
	Space path[FTPS folder]: Answers/_files
	Selenium nodes per platform: 1
	Test run name: Answers TRUNK MySQL
	Nerrvana platforms:
		Opera 12.02 (WinXP)
		Firefox 15.0 (WinXP)
	Executable file: xbuild-mysql.sh
	Nerrvana FTPS address: ftp.nerrvana.com
	Nerrvana FTPS user: demo154
	Nerrvana FTPS pass: dem0l54
	Workspace folder: ./tests
	Max execution time: 3600
	Poll period: 20
	Parse user messages mode[results analyzer]: OFF
-----END PLUGIN SETTINGS---
2012-11-14 08:22:49 -----INITIALIZATION COMPLETED---
 
2012-11-14 08:22:49 Generated test run name: Answers TRUNK MySQL build #55
2012-11-14 08:22:49 Generated test run description:
Created by Nerrvana-Jenkins plugin
Revision: 146
Committer: igork
Date: 2012-11-14 08:22:44 +0000 (Wed, 14 Nov 2012)
"Fixed" robots.txt to initiate Jenkins build
 
2012-11-14 08:22:49 ---BEGIN UPLOADING TESTS TO NERRVANA FTPS---
[workspace] $ lftp -f upload-build-55-1352863447229969289.ftp
2012-11-14 08:23:22 -----END UPLOADING TESTS TO NERRVANA FTPS
 
2012-11-14 08:23:25 Creating and starting test run via Nerrvana HTTP API call...done.
2012-11-14 08:23:25 New test run ID#952.
2012-11-14 08:23:25 New execution ID#5769.
2012-11-14 08:23:25 ---BEGIN NERRVANA POLLING CYCLE (waiting for tests to complete)
2012-11-14 08:23:47 	Current execution status: run
 
2012-11-14 08:24:09 	Current execution status: run
 
----------- SOME LINES REMOVED ---------------------------------
 
2012-11-14 08:26:42 	Current execution status: run
 
2012-11-14 08:27:04 	Current execution status: ok
 
2012-11-14 08:27:04 -----END NERRVANA POLLING CYCLE---
2012-11-14 08:27:04 Saving execution results into /var/lib/jenkins/jobs/Answers/builds/2012-11-14_08-22-45/results.xml...
2012-11-14 08:27:04 Done.
2012-11-14 08:27:04 
-----END PLUGIN EXECUTION---
 
 
2012-11-14 08:27:04 
 
---BEGIN PLUGIN EXECUTION---
2012-11-14 08:27:04 ---INITIALIZATION STARTED---
2012-11-14 08:27:04 
---BEGIN PLUGIN SETTINGS---
	Nerrvana HTTP address: https://api.nerrvana.com
	Nerrvana API key: 2bf7c81-a031f-1ad2-fd3a6-f59b9e0668e
	Secret key:  wctfFwelygx3tXS4TasrsOS4oXV7YadcPppvEnx55WG7qPk6jrAHjJi1TRoLlhrarIlabte4H9zWROXVkLnXto2LlVC47EXx91Uu
	Space ID: 4144
	Space: Answers
	Space path[FTPS folder]: Answers/_files
	Selenium nodes per platform: 1
	Test run name: Answers TRUNK PostgreSQL
	Nerrvana platforms:
		Opera 12.02 (WinXP)
		Firefox 15.0 (WinXP)
	Executable file: xbuild-pgsql.sh
	Nerrvana FTPS address: ftp.nerrvana.com
	Nerrvana FTPS user: demo154
	Nerrvana FTPS pass: dem0l54
	Workspace folder: ./tests
	Max execution time: 3600
	Poll period: 20
	Parse user messages mode[results analyzer]: ON
	User message threshold: ERROR
-----END PLUGIN SETTINGS---
2012-11-14 08:27:04 -----INITIALIZATION COMPLETED---
 
2012-11-14 08:27:04 Generated test run name: Answers TRUNK PostgreSQL build #55
2012-11-14 08:27:04 Generated test run description:
Created by Nerrvana-Jenkins plugin
Revision: 146
Committer: igork
Date: 2012-11-14 08:22:44 +0000 (Wed, 14 Nov 2012)
"Fixed" robots.txt to initiate Jenkins build
 
2012-11-14 08:27:04 ---TESTS UPLOAD SKIPPED---
2012-11-14 08:27:06 Creating and starting test run via Nerrvana HTTP API call...done.
2012-11-14 08:27:06 New test run ID#953.
2012-11-14 08:27:06 New execution ID#5770.
2012-11-14 08:27:06 ---BEGIN NERRVANA POLLING CYCLE (waiting for tests to complete)
2012-11-14 08:27:28 	Current execution status: run
 
2012-11-14 08:27:49 	Current execution status: run
 
----------- SOME LINES REMOVED ---------------------------------
 
2012-11-14 08:30:03 	Current execution status: run
 
2012-11-14 08:30:26 	Current execution status: ok
 
2012-11-14 08:30:26 -----END NERRVANA POLLING CYCLE---
2012-11-14 08:30:26 Saving execution results into /var/lib/jenkins/jobs/Answers/builds/2012-11-14_08-22-45/results.xml...
2012-11-14 08:30:26 Done.
2012-11-14 08:30:26 ---BEGIN TEST EXECUTION RESULTS---
2012-11-14 08:30:26 At least 11 message(s) from Nerrvana side reach(es) or exceed(s) threshold level (ERROR).
Message analyzer marks execution as failure.
2012-11-14 08:30:26 -----END TEST EXECUTION RESULTS---
2012-11-14 08:30:26 
-----END PLUGIN EXECUTION---
 
 
Build step 'Nerrvana plug-in' marked build as failure
Finished: FAILURE

As you can see, the Selenium testing process took 8 minutes (8:22:49 – 8:30:26). Everything that happens before it is the deployment and takes only a few seconds. In this case, we use two platforms, and Answers was tested with MySQL and PostgreSQL sequentially.

We have 8 tests in total and each does a lot of checks, testing a particular piece of Answers’ functionality – search, tags, adding questions and answers, deleting and editing of questions and answers by the authors, moderators and administrators and so on. We can run tests in 4 threads with 2 tests in each thread. By using 4 threads and adding 4 more platforms we remain roughly within the same 8 minutes. By using 8 threads (one test per thread), we can add more platforms or reduce testing time.

Let’s look in more detail how these 8 minutes were spent in the logs that I have brought.

Initialization and the first Test Run name and description generation – 1 second.

2012-11-14 08:22:49 ---INITIALIZATION STARTED---
----------- SOME LINES REMOVED ---------------------------------
2012-11-14 08:22:49 -----INITIALIZATION COMPLETED---
2012-11-14 08:22:49 Generated test run name: Answers TRUNK MySQL build #55
2012-11-14 08:22:49 Generated test run description:
Created by Nerrvana-Jenkins plugin
Revision: 146
Committer: igork
Date: 2012-11-14 08:22:44 +0000 (Wed, 14 Nov 2012)
"Fixed" robots.txt to initiate Jenkins build

Syncing Selenium tests between Jenkins and Nerrvana – 33 seconds.

2012-11-14 08:22:49 ---BEGIN UPLOADING TESTS TO NERRVANA FTPS---
[workspace] $ lftp -f upload-build-55-1352863447229969289.ftp
2012-11-14 08:23:22 -----END UPLOADING TESTS TO NERRVANA FTPS

Creating the Test Runa in Nerrvana and launching it via API – 3 seconds.

2012-11-14 08:23:25 Creating and starting test run via Nerrvana HTTP API call...done.
2012-11-14 08:23:25 New test run ID#952.
2012-11-14 08:23:25 New execution ID#5769.

Running tests – 3 minutes 39 seconds.

2012-11-14 08:23:25 ---BEGIN NERRVANA POLLING CYCLE (waiting for tests to complete)
2012-11-14 08:23:47 	Current execution status: run
----------- SOME LINES REMOVED ---------------------------------
2012-11-14 08:27:04 	Current execution status: ok
2012-11-14 08:27:04 -----END NERRVANA POLLING CYCLE---
2012-11-14 08:27:04 Saving execution results into /var/lib/jenkins/jobs/Answers/builds/2012-11-14_08-22-45/results.xml...
2012-11-14 08:27:04 Done.
2012-11-14 08:27:04 
-----END PLUGIN EXECUTION---

The second Nerrvana plugin step is launched.

Initialization and the second Test Run name and description generation – 1 second.

---BEGIN PLUGIN EXECUTION---
2012-11-14 08:27:04 ---INITIALIZATION STARTED---
2012-11-14 08:27:04 
2012-11-14 08:27:04 -----INITIALIZATION COMPLETED---
2012-11-14 08:27:04 Generated test run name: Answers TRUNK PostgreSQL build #55
2012-11-14 08:27:04 Generated test run description:
Created by Nerrvana-Jenkins plugin
Revision: 146
Committer: igork
Date: 2012-11-14 08:22:44 +0000 (Wed, 14 Nov 2012)
"Fixed" robots.txt to initiate Jenkins build

We do not sync Selenium tests with Nerrvana as they were synced during the previous step.

2012-11-14 08:27:04 ---TESTS UPLOAD SKIPPED---

Creating the Test Run in Nerrvana and launching it via API – 2 seconds.

2012-11-14 08:27:06 Creating and starting test run via Nerrvana HTTP API call...done.
2012-11-14 08:27:06 New test run ID#953.
2012-11-14 08:27:06 New execution ID#5770.

Running tests – 3 minutes 20 seconds.

2012-11-14 08:27:06 ---BEGIN NERRVANA POLLING CYCLE (waiting for tests to complete)
2012-11-14 08:27:28 	Current execution status: run
----------- SOME LINES REMOVED ---------------------------------
2012-11-14 08:30:26 	Current execution status: ok
2012-11-14 08:30:26 -----END NERRVANA POLLING CYCLE---

Results parsing and report preparation – 1 second.

2012-11-14 08:30:26 Saving execution results into /var/lib/jenkins/jobs/Answers/builds/2012-11-14_08-22-45/results.xml...
2012-11-14 08:30:26 Done.
2012-11-14 08:30:26 ---BEGIN TEST EXECUTION RESULTS---
2012-11-14 08:30:26 At least 11 message(s) from Nerrvana side reach(es) or exceed(s) threshold level (ERROR).
Message analyzer marks execution as failure.
2012-11-14 08:30:26 -----END TEST EXECUTION RESULTS---
2012-11-14 08:30:26

Thus, we see that most of the time was spent on testing. Everything else, including the synchronization of tests between Jenkins and Nerrvana (remember that LFTP uses the ‘mirror’ command and only updates changed files) is about 45 seconds. These figures are for the configuration shown in the very first post – deployment and jenkins hosts are on my home network as VMWare virtual machines, I’m connected to the Internet with ADSL2 +, I am in Sydney and Nerrvana is in the USA. No super speed here or close physical proximity.

How to reduce the test execution time twofold in our case?

We can just test our application on MySQL and PostgreSQL not sequentially, but simultaneously. But in this case, as shown in the image below in our Jenkins Dashboard we will see not one job, but three (one upstream and two downstream)

or four (two upstream and two downstream).

Moreover, a report produced by the Nerrvana plugin and shown in the picture above, will be located inside two different jobs, that will complicate and slow down its analysis for us. Another hitch will be the timing of tests. As the two Nerrvana plugins will be running in parallel, they will both have to do synchronization. It will add erxtra33 seconds, but will reduce the test time in half. That is, we get the total time around 4.5 minutes.

We run tests in parallel using our framework which utilizes TestNG in a single job, as you may have guessed. However when we will implement reports aggregation in our framework (more about it below) we possibly move on to parallel testing with different databases. We most likely will add support for MS SQL in Answers soon and will have to test on three different database types. In this case, we can forget about the short report Nerrvana plugin creates in Jenkins, and use a single aggregate report from our framework.

The main disadvantage now is the need to view as many reports as platforms we tested our application on. Nerrvana plugin reports are already aggregated but they give very basic information. Sometimes it is enough, and sometimes it’s not. Therefore, we will improve our framework to generate the results in XML format. This information will be uploaded back to Jenkins and then our new generator/aggregator will use XMLs to create a single report. We have described previously how such reports will look. We are happy to share the details of the implementation when we get to it.

Schematically, it will work like this:

Even if you do not need to test different settings as we do with different databases and multiple Nerrvana plugin steps, you will still be able to aggregate all platforms into one report inside a single build by adding extra an aggregation step after Selenium testing.

Well, the last thing I would like to do, is to get the web server’s logs and PHP error logs (we use PHP) and integrate them to our Selenium reports. Then it would be easy to move from errors in the logs to a place in Selenium report and vice versa. Also we found this article and want to try to get code coverage data after Selenium testing to see which areas were not touched by tests. If it works it will make sense to integrate this info into our reports.

This ends the story of web applications Selenium testing in Deep Shift Labs with Jenkins and Nerrvana. We hope that you find it useful.

Our next blog story will be about exporting Selenium test results to HP Quality Center (HP QC) using the Bumblebee web service from Agiletestware.

Print this post | Home

Comments are closed.