Today I would like to introduce a feature that we have recently added, and have been testing and using it ourselves for several months. It is functional monitoring. Our system, unlike other cloud Selenium services, not only provides the browsers for on-demand testing, but also executes your Selenium code. It can run tests by schedule defined in Nerrvana and does not require any external systems, like CI server, to kick start them.
I should admit, we designed Nerrvana this way not because we planned to use it for monitoring, but because at the time, when we started to build Nerrvana, there were no mature CI servers to solve this problem. As Nerrvana is already able to execute code, it was logical and relatively easy for us to add the ability to do monitoring.
By the way, this is not the only application that this feature made possible. For example, you can check tickets availability, or track the delivery of your goods, but we’ll write about it another time. Mike Levin from Yandex wrote about reasons to do functional monitoring in production (with his permission, we even translated his post into English).
Thus, you could run monitoring tests from when we launched Nerrvana. However, there were two problems we wanted to solve. First, we would like to receive a single email with all of the errors found by tests on different platforms. Second – we did not want to change tests’ code to send email notifications. We, for example, run non-monitoring tests on Jenkins commit and Jenkins build emails errors to a committer automatically.
So, everything is very simple. You are testing your application and one day realise that with very little effort you can take some of your tests that check for key functionality of your application, and give it to operations department and system administrators to run on a schedule and receive notifications by mail, or parse results into a monitoring system. In this case, the monitoring system will decide what to do next – to ignore, email or wake on-call person by SMS. You want to help those employees of your company who are responsible for the availability and uptime of your website.
In Nerrvana’s Test Run UI scheduled launch section, we added an additional input field for the email address.
This could be a personal address or service, like PagerDuty. Next, we added the ability to view, modify and send a test message. We expected that if you use Selenium to test and understand the importance of a functional monitoring, then your alerts will be, or are, parsed automatically. Therefore, for ease of parsing of notifications that you receive, we have made this additional tuning feature. In addition, to ensure the delivery of undoubtedly important monitoring notifications, we began to use SendGrid.
Here is the resulting test email:
The structure of the letter is clearly defined and will not change so you can rely on its format for parsing. Note that the first line is a special string that contains a number of errors in, again, a convenient for a parsing format.
You may ask how Nerrvana knows of any errors in your tests? Errors aggregation from different platforms is possible only if you use Nerrvana messages in your tests. Otherwise, you will have to send an email from each platform and aggregate somewhere on your side.
We do not use automatic parsing now and simply send the results by email. If you already use a monitoring system, it is obvious that during conversion of the tests into monitoring scripts to make sure that your monitoring system will be able to make decisions based on the information contained inside emails.
If your tests catch only critical errors – you can send them to yourself by SMS, so if you are woken up at night you know that there is a problem that needs to be urgently addressed. The problem however is that the reality (the network between the monitoring system and your site, Selenium itself and browsers, which Selenium uses), does not give such a guarantee and you need to have a rechecking mechanism.
Wouldn’t you agree that a reaction to such rechecks will depend on what is being tested and the sequence of responses from those restarts:
- Broken, working, working, working
- Broken, working, broken, working
- Broken, broken, broken, broken
Therefore, the use of the monitoring system is preferable because it allows you not to react immediately, but to aggregate the results and to use such logic – if the error is not critical, and it was repeated in three consecutive runs – notify staff. For example, Yandex has built its own system called Terra (in Russian) to be able to analyze and aggregate results on the fly and make decisions while running a large amount of functional monitoring tests, including ability to analyse results from retries, based on Apache Camel.
Once we have completed implementation, our hands itched wanting to use it in practice. We weren’t phased by the fact that we were going to monitor Nerrvana with Nerrvana and we started discussing what tests are important to us.
The result was this list.
2. Log in.
3. Changing password.
4. An attempt to register with existing email
5. Demo tests launch (available by default in a new account. You see them completed and can re-launch again) and verification of their successful completion.
6. Creating a new Space.
7. Creating a new Test Run.
9. Deleting Account.
Monitoring tests always use the same account Nerrvana knows about(* see details below). The first thing each Nerrvana monitoring test does – checks if this account with a special email address exists. If it does not exist – the test completes registration. Then the test changes account’s password as otherwise we either needed to put another special password generation “fix” for a test account so we will know the password or to open IMAP email box to see what password was generated. If it exists, the test uses a special page where the unique password reset link is displayed in the browser, but does it only for this (!) Test account. Normally such a link is emailed to you when you use the ‘Forgot password’ option.
The test deletes the test account at the end of testing, but if it failed with a fatal error, the account may remain in the system.
To test Nerrvana we had to create two special pages. The first one is showing the generated user name instead of sending a welcome email*. The second page is needed in order to show a reset password link on a web page instead of sending it by email*. Final logic implemented in each tests looks like this:
1. We are trying to register.
2. If the test user is already in the system (and registration fails with email exists message), then:
a. We go to the reset password page and click the button ‘Reset’ b. We go to the second special page that shows a link, which is usually sent by email, and click the link shown.
c. We change the password to the one that tests operate with.
d. Login to Nerrvana and delete a test account Nerrvana’s user profile page.
3. If the user does not exist, we register and use the first special page to spy on new username generated, which is usually sent by email.
4. Test functionality.
Tests run every 2 hours, but, as you can imagine, changing schedules takes only a couple of minutes.