Using Nerrvana – Jenkins setup for Selenium testing

Using Nerrvana - Jenkins setup for Selenium testing

Creating the Nerrvana plugin configuration
file – walking through it – running
our tests – checking test results

Part 1 – Using Nerrvana – our setup
Part 2 – Using Nerrvana – SVN hooks to Jenkins
Part 3 – Using Nerrvana – deployment & Jenkins (part 1)
Part 4 – Using Nerrvana – deployment & Jenkins (part 2)
Part 5 – Using Nerrvana – Jenkins setup for Selenium testing – this post

In the previous post, we have automated our web application deployment process and are now ready to continue improving our build and include Selenium testing into it. At this point Jenkins is able to react to the commit, prepare and install our application on the deployment host. We also learned how to extract commit information from SVN, parse and save it to version.txt file.

Today we will launch Selenium tests in Nerrvana with our Jenkins plugin. The Nerrvana plugin is available at http://your_jenkins_instance/pluginManager/available. The plugin uses LFTP to synchronize tests between Jenkins and Nerrvana. So, please install it (yum install lftp) on the same server running Jenkins.
We recommend that you run your tests from Nerrvana’s UI and verify that Nerrvana can run them before proceeding. The process for running tests manually is described on the ‘Get Started‘ page.

To create a configuration file for the Nerrvana plugin, go to the page ‘Add new test run’ or ‘Edit test run’. As shown below, select the desired Space (A), enter the Test Run name (B), and select an executable file (C). Note: Try not to use a long name for the test run since the plugin will add a Jenkins build number to it. We recommend that you leave the Description field (D) blank and allow the plugin to add commit information to it dynamically. Select desired platforms (E) and, if your tests can run in parallel, the number of nodes per platform (F). Nerrvana is now able to create a configuration file (G), where you will need to set a few parameters before using it in Jenkins. Please save the file.

Generate Nerrvana Jenkins config in UI

Section ‘api-params’ contains your Nerrvana API access keys. They are added automatically when you generate the configuration file. They are also available on the Settings page, where you can change them, if necessary.

<api-params>
        <!-- Address of the Nerrvana gateway. -->
        <gateway>https://api.nerrvana.com</gateway>
        <!-- User-specific key which identifies user on Nerrvana side. -->
        <!-- Available as an 'API public key' on Settings page 
            (https://cloud.nerrvana.com/user/editAccount) 
            in Nerrvana. -->
        <apikey>a6c171d-20eb5-61e3-60991-8a1e523ce</apikey>
        <!-- This key is used by the Nerrvana plug-in to create a checksum of API 
            call parameters to ensure their consistency. -->
        <!-- Available as an 'API private key' on Settings page in Nerrvana. -->
        <secretkey>z3rCWlLdAP35eCCYAmQZ2kPw9X0LByrcc3XGh</secretkey>
    </api-params>

Name of the Test Run is in the elemnt ‘test-run-name’. The plugin will add the Jenkins build number, thus generating a new name for each new Test Run.

<!-- Parameters related to Nerrvana-driven Selenium tests. -->
    <!-- Test Run name template, Jenkins build number will be added to the 
        end automatically. -->
    <test-run-name>Answers TRUNK MySQL build #51</test-run-name>

The Test Run description is defined by elemnt ‘test-run-descr’. You can leave it blank or add, for example, ‘Generated by Jenkins plugin’.

<!-- Test Run description. All Test Runs created by this Jenkins build 
        step will have this description. -->
    <test-run-descr></test-run-descr>

Comments for the element ‘test-run-descr-file’ explain how to use a file to add a description dynamically with each job build. We will change the file name to version.txt, because such a file is generated in the deployment part of our build. You can create your own file and include any information you want.

<!-- Content of this file, if not empty, will be added to a description,
        defined by ‘test-run-descr’ parameter above.
        File path is relative to the Jenkins job workspace.
        During deployment phase you can extract revision number, commiter name and a
        commit message from your version control system, put them into this file and
        use them as a description.
        You can extract and parse SVN information into info.txt file with our little
        tool - https://github.com/deepshiftlabs/nerrvana-plugin-for-jenkins-ci
        You can read how we do it with SVN in our blog. -->
    <test-run-descr-file>version.txt</test-run-descr-file>

Here’s how a Test Run created by the plugin will look if:

- Test-run-name – ‘Answers TRUNK MySQL’

- Test-run-descr – ‘Created by Nerrvana-Jenkins job’

- The contents of ‘test-run-descr-file’:

Revision: 142
Commiter: igork
Date: 2012-10-17 07:39:20 +0000 (Wed, 17 Oct 2012)

Dynamic description

The Jenkins build number (‘build # 51′) was added to the Test Run name. The description is a combination of ‘test-run-descr’ and the content of a file specified by the content ‘test-run-descr-file’ parameter.

‘executable-file’ parameter defines an executable file we selected – ‘xbuild-mysql.sh’.

<!-- Which executable file Nerrvana should use to start tests. -->
    <executable-file>xbuild-mysql.sh</executable-file>

This file should be able to launch your tests. For example:

ant -f build-mysql.xml all > build.log 2 >& 1

Here, under the spoiler, you can see our build-mysql.xml, but I will not talk about it today. If you have any questions – we are happy to answer.

<project name="dslabs-test" default="" basedir=".">
  <!-- BEGIN versions -->
    <property name="title"          value="Tests of http://answers.starty.co"/>
    <property name="version"          value="0.02.000"/>
  <!-- END versions -->
 
  <!-- BEGIN build properties -->
    <property name="build.compiler"       value="javac1.6"/>
    <property name="libdir"               value="${basedir}/lib"/>
    <property name="output"               value="${basedir}/output"/>
    <property name="sources"              value="${basedir}/src"/>
    <property name="resources"            value="${basedir}/res"/>
    <property name="dist"                 value="${basedir}/dist"/>
    <property name="logdir"               value="${basedir}/log"/>
    <property name="bindir"               value="${basedir}/bin"/>
    <property name="logs"            	value="${basedir}/logs"/>
    <property name="test.failonerror"     value="true"/>
    <property name="compile.debug"        value="true"/>
    <property name="compile.debuglevel"   value="lines,vars,source"/>
    <property name="compile.deprecation"  value="true"/>
  <!-- END build properties -->
 
    <taskdef resource="testngtasks" classpath="${libdir}/testng-5.8-jdk15.jar"/>
 
    <path id="classpath">
        <fileset dir="${libdir}">
            <include name="*.jar"/>
        </fileset>
    </path>
 
    <target name="clean">
        <delete dir="${output}"/>
        <delete dir="${logs}"/>
    </target>
 
    <target name="prepare" depends="clean">
        <mkdir dir="${output}"/>
        <mkdir dir="${logs}"/>
    <!-- !!!-->
        <mkdir dir="${logs}/html"/>
        <copy todir="${logs}/html/js">
            <fileset dir="${resources}/js" />
        </copy>
        <copy todir="${logs}/html/img">
            <fileset dir="${resources}/img" />
        </copy>
        <copy todir="${logs}/html/css">
            <fileset dir="${resources}/css" />
        </copy>
 
        <copy file="${resources}/log4j.xml" todir="${output}"/>
        <copy file="${resources}/testng.xml" todir="${output}"/>
        <copy file="${resources}/config-mysql.xml" tofile="${output}/config.xml"/>
    </target>
 
    <target name="bld" depends="prepare">
        <javac
            srcdir="${sources}"
            destdir="${output}"
            excludes=""
            encoding="UTF8"
            debug="${compile.debug}"
            deprecation="${compile.deprecation}"
            optimize="${compile.optimize}"
            debuglevel="${compile.debuglevel}">
            <classpath refid="classpath" />
        </javac>
    </target>
 
    <path id="tests_classpath">
        <fileset dir="${libdir}">
            <include name="*.jar"/>
        </fileset>
        <pathelement location="${output}"/>
    </path>
 
    <target name="run" description="${title}">
        <java classpathref="tests_classpath"
            fork="true"
            dir="${output}"
            classname="org.testng.TestNG"
            failonerror="false">
            <arg value="-d" />
            <arg value="${logs}/testng_reports" />
            <arg value="testng.xml"/>
        </java>
<!--delete dir="${output}"/-->
    </target>
 
  <!-- build all -->
    <target name="all" depends="bld,run" description="Clean, build, and deploy project"/>
</project>

Section ‘platforms’ lists the platforms on which the tests will run. All supported platforms are included, with the selected ones uncommented.

<!-- List of platforms to run tests against for this config. -->
    <platforms>
        <!-- List of available platforms. Uncomment to use. -->
        <!--platform>
            <code>centos_58_firefox_36</code>
            <name>Firefox 3.6 (CentOS)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_firefox_110</code>
            <name>Firefox 11.0 (WinXP)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_firefox_150</code>
            <name>Firefox 15.0 (WinXP)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_firefox_36</code>
            <name>Firefox 3.6 (WinXP)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_ie_8</code>
            <name>IE 8 (WinXP)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_opera_1162</code>
            <name>Opera 11.62 (WinXP)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_opera_1202</code>
            <name>Opera 12.02 (WinXP)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_safari_515</code>
            <name>Safari 5.1.5 (WinXP)</name>
        </platform-->
        <!--platform>
            <code>winxp_sp3_safari_517</code>
            <name>Safari 5.1.7 (WinXP)</name>
        </platform-->
 
        <platform>
            <code>winxp_sp3_chrome_2001132</code>
            <name>Chrome 20.0.1132 (WinXP)</name>
        </platform>
        <platform>
            <code>winxp_sp3_chrome_2101180</code>
            <name>Chrome 21.0.1180 (WinXP)</name>
        </platform>
    </platforms>

Next element ‘nodes-count’ allows you to configure the plugin to run tests in parallel. More information about parallel execution is available here. If your tests will not work in parallel, change the number of nodes here to 1.

<!-- How many Selenium nodes should be used for each platform. -->
    <nodes-count>2</nodes-count>

The next parameter ‘folder-with-tests’ points to the location of the tests in the Jenkins workspace.

<!-- Parameters related to the transfer of the tests from Jenkins 
        to Nerrvana. -->
    <!-- Folder in the workspace of Jenkins job where Selenium tests will be 
        located. It is assumed that the SCM build step, which always occurs 
        BEFORE other steps, will put tests there. -->
    <folder-with-tests>.</folder-with-tests>

Our Selenium tests are in the folder ‘tests’ in the same repository. The structure of the project was shown in the post “Using Nerrvana – deployment & Jenkins (part 1)”. As our tests are in the ‘tests’ folder, we set it this way:

<folder-with-tests>./tests</folder-with-tests>

Space ID, Space name and FTPS folder were specified previously during configuration file generation and should be left as they are in the element ‘space’.

<!-- Nerrvana space previously created by you through the Nerrvana UI. -->
    <space>
        <id>4144</id>
        <name>Answers</name>
        <ftp-path>Answers/_files</ftp-path>
    </space>

Section ‘ftp’ defines FTPS access credentials. It is necessary to add the password to your Nerrvana account.

<!-- Address and credentials of the Nerrvana FTPS connection. Note that 
        a system running Jenkins should have LFTP application installed. -->
    <ftp>
        <server>ftp.nerrvana.com</server>
        <!-- Your username -->
        <username>demo123</username>
        <!-- Replace this value with your password!! -->
        <password>[p a s s w o r d]</password>
    </ftp>

The parameter ‘skip-tests-sync’ allows you to speed up testing by skipping test synchronization when the Nerrvana plugin is called multiple times in a single build. Now we’ll leave the default value – false, as we call the plugin once. In the next post I will show how it is used by us when we call it twice to test our Answers application with MySQL, and then with PostgreSQL.

<!-- You can skip test sync when you call the Nerrvana plugin many times 
        in a single job and tests were synced in a previous step. -->
    <skip-tests-sync>false</skip-tests-sync>

The parameters shown below are relevant only if you are using messages, which Nerrvana can analyze. There is a detailed description of these parameters in the plugin configuration file.

<!-- Job status will be set to FAILED if test execution in Nerrvana is 
        unsuccessful (tests did not sync, executable file did not start your 
        tests, etc).
            If tests were executed successfully, you have a few options:
        1. Mark build as successful in Jenkins and analyse generated reports 
        for problems your tests discovered.
        2. If you want to mark a build as FAILED based on errors your tests 
        found, you can:
            (a) Add additional step, load results, parse them to analyse errors 
            your tests discovered, and mark build as FAILED based on it. Keep 
            this parameter as 'false' in this case.
            (b) Use Nerrvana "Messages" and let the Nerrvana plugin analyse 
            results, and change build status to FAILED based on it. Keep this 
            parameter as 'true' in this case and use additional parameter 
            'message-threshold' to add additional logic.
            If you use messages and invoke the Nerrvana plug-in several times 
            in the same build, keep this option as 'false' for all invocations 
            except the last one to let the build complete all tests and analyse 
            errors at the very end. -->
    <use-messages-to-set-build-status>false</use-messages-to-set-build-status>
 
    <!-- Defines the level at which a FAILED status is generated. If this 
        value is set to 'WARN', for example, and your tests generate one or more 
        'WARN' or higher severity messages (ERROR or FATAL), Nerrvana execution 
        status, and Jenkins build, will be FAILED. For the full list of levels, 
        visit http://www.nerrvana.com/docs/using-messages page. -->
    <message-threshold>ERROR</message-threshold>

As our tests generate messages, which Nerrvana can analyze, we set the parameters as follows:

<use-messages-to-set-build-status>true</use-messages-to-set-build-status>
    <message-threshold>ERROR</message-threshold>

In this case, the plugin will not only receive and store messages polling Nerrvana while tests are running, but also analyze whether there are messages with the level ERROR or higher (FATAL). If any are found, the build status will be set as FAILED in Jenkins.

If you do not use Nerrvana messages and still want to set the status of the Jenkins build based on test results (rather than whether the run was completed successfully), you can add an extra step in your build after the Nerrvana plugin. Get test results from Nerrvana via FTPS, analyze them, and change the status of the Jenkins build.

The parameter ‘max-execution-time’ will stop tests if, for example, they go into an infinite loop. If you run tests using the Nerrvana UI, you can set the real value with some extra time added.

<!-- Maximum execution time (in seconds). Defines how long the Nerrvana 
        plug-in will wait for tests to complete. Start by setting to a large 
        value and adjust accordingly after a few runs.-->
    <max-execution-time>3600</max-execution-time>

The last parameter ‘poll-period’ specifies how often the Nerrvana plugin will poll API about the status of test execution.

<!-- How often the Nerrvana plug-in will update test execution status 
        from Nerrvana (in seconds). -->
    <poll-period>20</poll-period>

The flowchart of the plugin, especially in the part relating to polling and setting the build’s status, depending on these parameters, shown in greater detail in the documentation.

Now we only need to add a step to our build …

Add Nerrvana plugin step

… and copy the contents of the configuration file into the plugin configuration window.

Add Nerrvana plugin step - just copy/paste

You can save changes in a Jenkins job, run the build, and watch your tests running in the Nerrvana cloud. First, I launched a single test on a single platform. In the UI you can see how the plugin creates a Test Run and it is immediately started.

Nerrvana Test Run started by Jenkins

The Jenkins build number was added to the Test Run name and the description contains information we specified in the plugin configuration with version.txt file content, which was previously parsed by our decorator. We can see that the test has been completed successfully with no errors found – we cannot see any messages.

Nerrvana Test Run completed

From Nerrvana we can open (arrow in the picture above) the report our test generated.

Our Selenium report for one test

In the first launch, I turned off all the tests, except one – a login test in order to quickly make sure that everything works. Now I’ll enable all the tests, commit and Jenkins will create and start a new Test Run. Oh, and I will also add one more platform to it.

Below, under the spoiler, is an example of a Jenkins build console log, which continues the console log we publicised in a previous post about deployment. My comments follow it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
---BEGIN PLUGIN EXECUTION---
2012-10-17 07:39:24 ---INITIALIZATION STARTED---
2012-10-17 07:39:24
---BEGIN PLUGIN SETTINGS---
   Nerrvana HTTP address: https://api.nerrvana.com
   Nerrvana API key: a6c171d-20eb5-61e3-60991-8a1e523ce
   Secret key: z3rCWlLdAP35eCCYAmQZ2kPw9X0LByrcc
   Space ID: 4144
   Space: Answers
   Space path[FTPS folder]: Answers/_files
   Selenium nodes per platform: 1
   Test run name: Answers TRUNK
   Nerrvana platforms:
      Opera 12.02 (WinXP)
      Firefox 15.0 (WinXP)
   Executable file: xbuild-mysql.sh
   Nerrvana FTPS address: ftp.nerrvana.com
   Nerrvana FTPS user: demo123
   Nerrvana FTPS pass: dem0I23
   Workspace folder: ./tests
   Max execution time: 3600
   Poll period: 20
   Parse user messages mode[results analyzer]: ON
   User message threshold: ERROR
-----END PLUGIN SETTINGS---
2012-10-17 07:39:24 -----INITIALIZATION COMPLETED---
 
2012-10-17 07:39:24 Generated test run name: Answers TRUNK MySQL build #51
2012-10-17 07:39:24 Generated test run description:
Created by Nerrvana-Jenkins plugin
Revision: 142
Committer: igork
Date: 2012-10-17 07:39:20 +0000 (Wed, 17 Oct 2012)
Launching all tests on two platforms
 
2012-10-17 07:39:24 ---BEGIN UPLOADING TESTS TO NERRVANA FTPS---
[workspace] $ lftp -f upload-build-51-8414449724631519802.ftp
 
2012-10-17 07:40:16 -----END UPLOADING TESTS TO NERRVANA FTPS
 
2012-10-17 07:40:18 Creating and starting test run 
                    via Nerrvana HTTP API call...done.
2012-10-17 07:40:18 New test run ID#884.
2012-10-17 07:40:18 New execution ID#5486.
2012-10-17 07:40:18 ---BEGIN NERRVANA POLLING CYCLE 
                       (waiting for tests to complete)
 
2012-10-17 07:40:40    Current execution status: run
 
2012-10-17 07:41:02    Current execution status: run
 
------ Some оutput removed ---------------------------------
 
2012-10-17 07:44:17    Current execution status: run
 
2012-10-17 07:44:38    Current execution status: ok
 
2012-10-17 07:44:38 -----END NERRVANA POLLING CYCLE---
2012-10-17 07:44:38 Saving execution results into results.xml...
2012-10-17 07:44:38 Done.
2012-10-17 07:44:38 ---BEGIN TEST EXECUTION RESULTS---
2012-10-17 07:44:38 At least 43 message(s) from Nerrvana side reach(es) 
                    or exceed(s) threshold level (ERROR).
Message analyzer marks execution as failure.
2012-10-17 07:44:38 -----END TEST EXECUTION RESULTS---
2012-10-17 07:44:38
-----END PLUGIN EXECUTION---
 
Build step 'Nerrvana plug-in' marked build as failure
Finished: FAILURE

In the first 26 lines, the plugin outputs the parameters parsed from its configuration file.

In lines 28-34, the plugin creates a new test run name and description.

In lines 36-39, you see the synchronization process between tests in the Jenkins workspace and tests stored in the Nerrvana space used for this job.

In lines 41-44, the plugin creates a new Test Run in Nerrvana, launches it and gets a Test Run ID and unique Test Run execution ID back from API. This will be used to monitor test status by the plugin later. The same Test Run can be launched several times, but each run will have its own unique internal execution ID. At this stage, the Test Run is visible in the Nerrvana UI. “Answers TRUNK MySQL build # 51″ Test Run is created and starts running.

Nerrvana Test Run started by Jenkins - all tests and two platforms

In lines 45-58, the plugin polls test status every 20 seconds (polling parameter), while reading Nerrvana messages, which become available in real time during tests’ execution.

Once the tests are completed (lines 59-60), the plugin analyzes the errors (lines 61-63) and on the basis of the analysis sets the build status (lines 64-70). If you’re not using Nerrvana messages, the build would be marked as successful.

Using Nerrvana’s UI we can see how long it took for all tests to complete.

Nerrvana Test Run completed - all tests and two platforms

This is what messages look like in the UI, that the plugin reads and analyzes.

Nerrvana messages can be analyzed by plugin

We see that the reports generated by our framework contain more tests than the previous run (left panel).

Our Selenium report for all test

Another advantage of using Nerrvana messages is that the plugin can create simple reports (1). Reports show the status of the build and message threshold level used by the plugin (2) and the description of the Test Run (3). In our example, the description contains information about the commit that triggered the tests. Platform names (4) are linked to the Nerrvana FTPS server, which will contain detailed reports generated by the tests. Finally, clicking on the last column (5) expands the table and shows the messages that your tests sent to Nerrvana, which were received and collected by the Jenkins plugin.

Very often, this provides enough information for a developer to understand where the errors are, especially when he or she also maintains tests :)

Nerrvana plugin report in Jenkins

We have described the whole process of setting up and running tests. However, as you know, we test our application with two databases, and therefore we use a more complicated configuration. We deploy twice, and call the Nerrvana plugin two times in the same Jenkins job.

We will talk about this in a future post. We also ponder how the process can be improved.

Print this post | Home

Comments are closed.