Part 1 – Using Nerrvana – our setup – this post
Part 2 – Using Nerrvana – SVN hooks to Jenkins
Part 3 – Using Nerrvana – deployment & Jenkins (part 1)
Part 4 – Using Nerrvana – deployment & Jenkins (part 2)
Victor completed our first public version of Jenkins plug-in for our Selenium testing cloud – Nerrvana. He set Jenkins and now our tests for Answers run automatically. While he was registering and learning how to publish the plug-in on GitHub which is the preferred method of announcing plug-ins for Jenkins, I started working on extending our User’s Guide. I could just describe how to set up the plug-in but it seemed to me that we can do it better. This is why I decided to describe not only how to set up our plug-in, but also show our approach to web application testing with Selenium at Deep Shift Labs, as well as how we assembled all of the components and why, what was discussed along the way. Of course it would be one of the possible ways to set up a Selenium testing environment but I hope it will be more useful for you even if you won’t plug Nerrvana into your own process but will use your own cloud of browsers or solutions provided by other companies.
After some discussions we came up with the following approach to Selenium testing automation:
1. Post-commit hook is set to notify Jenkins
2. SVN hook tells Jenkins what revision was committed
3. Jenkins deploys a web application or, simply, takes a committed revision from a repository and sets it up on a dedicated virtual machine in our office.
4. Jenkins uses Nerrvana plug-in to sync Selenium tests between code just extracted from SVN and code already stored on Nerrvana and launches tests
5. When tests are completed it would be nice to send the committer an email containing the status of the test run and how many errors (short summary) were detected. For now* – it will be enough just to notify by email with links to reports our tests generated for each platform we tested against.
* – in the future we are planning to move test results back to Jenkins and compile cross-platform aggregated reports and we will move email notifications functionality into this custom Jenkins aggregate_notify job. Without it it is kind of boring and time consuming to look at 5-6 separate reports generated on different platforms each time.
Now I will unveil a few topics we discussed.
How often to launch Selenium tests?
We can launch test periodically, not a problem, for example; at the end of the day or twice a day. However this approach has some negative sides:
- it won’t be clear whose commit broke a build or whose tests changes contained errors or both. When we launch after each commit errors can be found quicker as code changes are minimal between commits.
- we have days when there are no commits as we work simultaneously on a few projects switching quite often and some projects do not have Selenium tests yet, and some won’t have them at all. With periodic testing these test runs will eat valuable hardware resources on Nerrvana producing no value to us.
Should we use Jenkins ‘quiet period’ option before kicking off Selenium testing?
Originally we decided to wait 5 minutes with intention to collect a few consequent commits to test them all together.
With this option used, commits submitted within 5 minutes interval and creating revisions 12, 13 and 14 will trigger Jenkins to test revision 14 (last one). If you have a few logical commits in different places and you want to commit them separately you need to remember to commit them within 5 minutes. If you won’t meet the deadline – tests will start, and most likely fail. Sometimes you want results quickly, sometimes you found a stupid bug while reviewing your code after first commit or your wife called you in between commits to share her shopping experience. In our example there is no guarantee that someone else will unroll their own chain of commits. It is perfectly possible that someone else wants to commit a few things hoping to have 5 minutes and not knowing that Jenkins clocks are about to launch the testing process. As a result we will have more false positive errors and generally more mess and time to sort it out.
So we decided to use a special keyword inside the commit message – “#noselenium”. If it is not included, Selenium tests will run. This is a better way as you do not need to remember about 5 minutes delay. It is also good if you commit files having no relationship with Selenium testing – like documentation or Unit tests. You simply will put #noselenium into your commit message. It seems that here you are required to remember to add #noselenium but we found a simple solution. We all use the TortoiseSVN client and there is a way to add a default commit template for a particular repository which will come up by default when you commit. What one has to do is to keep it when no Selenium testing is required or more commits will follow. As you can see it is still not ideal but at least does not apply unconditional 5 min waiting time.
When to launch PHP Toolchain Jenkins job?
At the very beginning Victor set the following sequence of Jenkins jobs – a commit triggers PHP Toolchain. Next, an application is deployed for testing. PHP Tollchain and deployment were combined into one Jenkins job. Finally Nerrvana runs Selenium tests – second Jenkins job. Later we decided to remove PHP Toolchain and make it an independent job and run it on every commit (#noselenium inside commit has no impact on it). This way Selenium tests will finish quicker as they do not wait PHP Toolchain completion.
How to access Jenkins running in our office in Kharkiv, Ukraine from SVN repository located in Newark, NJ, USA ?
SVN hook should be able to access Jenkins via HTTP. We run Jenkins inside our office in Kharkiv. Our SVN server is in Linode facility in Newark. We could allocate a non-standard port on a public IP to a private one inside the office but we decided to use a workaround by creating a slave repository inside our office network. This was our systems administrator – Vadym’s, suggestion.
SVN master and slave work with Jenkins commit hook
(click to enlarge)
Now we can commit to both repos and commit into slave repo, in fact, redirects it to master. In master repo svnsync, which is triggered also as an SVN hook, synchronizes change with slave repo in Kharkiv and here finally Jenkins hook is triggered which will launch deployment job and prepare an application for testing. Thus, we have a read-only repository in the office but SVN hook is still triggered when the changes come during sync with the master SVN-server in the USA. More information about synchronizing SVN can be found in this article. With this approach we do not need to open Jenkins out, and the possibility that the SVN hook will not be able to access Jenkins is much smaller as they are both now in the same local network.
The described approach was implemented by Vadym. Victor wrote down for me the steps to configure and install all the components, but I knew that in order to write a detailed guide I will have to repeat this process again myself, complementing his instructions and rectifying mistakes. In our Ukrainian office hardware is used to the limits. There’s a Nerrvana test instance, and quite complicated development setups of systems of our customers, as well as Jenkins etc. We extensively use Xen for virtualization since we know it pretty well after Nerrvana’s creation. I didn’t want to mess up in such a complex environment in search of additional space and resources for virtual machines. But without having them, I thought, I won’t be able to write something really useful and neat. I looked at resources and technical possibilities at home (I beg your pardon – our Sydney office) and thought why not to build my own environment right here? To do this, I need 2 boxes. One will be running an SVN-server, and Jenkins, on the other – will be used to deploy and test WAUT (Web Application Under Test).
Of course, these tasks can be performed on the same machine, but then it will be different from our production setup, and this option does not scale well. The only difference between my environment and the production one is that the SVN server is located on the same machine as Jenkins, but it does not matter, since they communicate with each other via HTTP, and in this case are not even aware of its close proximity.
Everything you see in the picture is actually a Windows 7 PC with VMWare Workstation 8. Glory to the wonders of virtualization! SVN/Jenkins and WAUT – VM CentOS 6.2. By the way I blogged recently about minimal CentOS 6.2 on VMWare Workstatin and kickstart file. I also forwarded a public IP address assigned to my ADSL modem to the IP address of the VM running WAUT, thus providing access to it from Nerrvana. If you will try doing the same and your ISP does not give you a static address, you can always use DynDNS, as most modems support this service.
Upon completing this job, I was able to write about how the SVN/Jenkins/Selenium testing process was assembled and tuned as I completed all the steps, improved it along the way and stepped on a few rakes. Our next post will be too big if I were to unveil the entire process, so I had to split it into three posts – SVN server and Jenkins SVN hook, deployment and, finally, Selenium testing with Nerrvana.