How to understand that it is high time to automate testing

How can I understand that testing automation would be useful?
How to identify what to automate?
How to bring automation into project smoothly?

If you ever thought about these questions, this article would be useful for you.

Automation objectives

• To Increase testing efficiency. Regular autotests free specialists time to explore new functionality instead of quality control of developed one (especially regression testing). You can configure the start of autotests at night and after working hours. This will shorten testing time and ensure the accuracy of the results, because the machines don’t make mistakes;
• To reduce testing time. The process of "bug detection - registration - correction - verification" is reduced. In the "manual" mode, it takes about a day, in an automated mode - 2 minutes. For example, if during a smoke test after an application build in the CI, one of the tests "failed", build will be marked as defective and the test logs will be sent to developers who are responsible for fixing;
• Testing process transparency. Full and regular defects reporting is available for all members of the team. Reports are generated automatically and contain all information about the passed and not passed steps;
• Testing cost reducement. Once designed and written autotests need minimal maintenance - in case of a change in functionality and / or interface in new versions. The adjustment of the scripts will take from 10 minutes to several hours, depending on the number of changes in the product.

When does implementation of automatization provides benefit: 5 signs

• The size of the project. The project is more than a year old, it consists of a lot of subsystems. The number of tests to be run is increasing. We started to automate the testing of the resource management system, when we got 600 "manual" test cases and it took a week for one person to check them. Specialists should test, not pass test cases;
• Big team. There are more than two developers in the team. The developer should be sure that his changes won’t break someone else's code. Without auto-tests he finds it out in a day or two at best, at worst - from users;
• Old versions support. If you support several versions of the product and produce patches and service packs for each of them. Testing on different configurations is a routine that can be automated;
• Test data generation takes a lot of time. If you develop a service for processing and transforming all sorts of data: ordering systems, accounting profiles, etc. Manual data input into a system and visual analysis of results or sending requests and analyzing answers is an unjustified waste of resources.
• Frequent releases. If you have Agile with short iterations and frequent releases. Spending a week for testing within a sprint is too long, when you can spend just 1 day.

If at least 1 sign is relevant to your project, automation will bring the following improvements: the test data will be generated automatically, frequent regressions won’t take time, and the gap between bug identification and its fixing will be reduced.

If your project is not like this, wait until the functionality or team grows. Do not use autotests for the sake of autotests.

How did we realize that we need automatization?

We are developing an application for the record keeping of applicants and HR staff: search, recruitment, assignments for the trial period, holidays, current tasks and projects, etc

Presuppositions for automation:

The product is 3 years old;

5 developers in the team;

Usually releases are every 3 weeks;

Lots of test data generation

Total: 4 signs out of 5. Before automation, all the test cases took 4 days to check.

Decision

We decided to partially automate the system, but not all 900 test cases covered with autotests, but only those which affect the functionality that does not change in each sprint: authorization, password recovery, user actions, notifications, event creation. As a result, we wrote smoke-self-tests for all modules that are run in continuous integration (CI) during each build. We could spend more time and automate the rest of the cases, but it would not be expedient: the project is developing, business logic and the interface are changing too often to write autotests. And we had to rewrite them every spirit.

Result

As a result, smoke-autotests are run in CI during every build and notify developers about bugs in real time. So developers immediately begin to fix them without switching to other tasks, and QA has more time to thoroughly test the newly implemented features instead of rechecking the work of the old ones.

We had an opposite situation on another project - the app for creating assessments and analytics.

Presuppositions for automation:

The decision to automate was an experiment. It is a young project and it doesn’t have so much functionality yet - only 61 test cases.

Decision

The project is notable for the fact that it doesn’t have any manual tester. Research manual testing is done by the automator at the stage of writing and debugging the autotest script, which is then simply run into CI, thus replacing manual labor.

Result

46 test cases for the backend and 14 for the frontend were automated by July 2017. We got 96% of coverage with autotests.

This project is a good example to show what will happen if start automating from the very beginning of the project: time spent on testing during the whole project is less than if we first tested manually and then implemented the automation.

Automation Action Plan

So, you reassured that testing automation will be a useful thing to do. What's next? Let’s develop an action plan:

• what to automate;
• how to automate;
• the choice of tools for automation.

What to automate?

1. Hard-to-reach places in the system: backend processes, file logging, inputs to databases;
2. Frequently used features, where the risks of mistakes are high: payment system, signing up etc. Automation of critical functionality check ensures quick bug detection - 1 test takes 2 minutes on average. If you detect it faster- you fix it faster too;
3. Routine operations: data enumerating, forms with a large number of input fields. Automate fields filling with data and their verification after saving;
4. Validation messages: automate fields filling with incorrect data and check for the occurrence of validation;
5. Long end-to-end scripts. For example, the script for CRM estate agency: signup- registration of the signup form - task creation - employ- assigning tasks to specialists - getting tasks back to work - registering the result;
6. Data check that requires accurate mathematical calculations: accounting or analytical software;
7. Accuracy of data search check.

How to automate

Not all test cases need automation primarily. For example, if you have 300 test cases for checking modules, 100 for components, 20 for subsystems, and only 5 user scripts (see Pic. 2), it's better to start automation with the tests that test single functions, i.e. with unit-tests. They are performed quickly, but there are a lot of them. When these cases are already covered with autotests, you can start to automate test cases for components checking, then-subsystems, etc.

Full automation

Presuppositions for automation:

An example of complete automation is a Jira add-on app for project planning and management. There are about 600 test cases on the project, all are covered with autotests. In case of changes in the logic or interface, you have to edit about 100 scripts. Nevertheless it takes less time than manual testing.

Decision

At the planning stage of automation, to determine the number of automated test cases for each level of architecture, we took the proportions from the test pyramid (Pic.). As a result, we got more than 300 unit-tests, 200+ for integration and API and 38 GUI-tests that repeat the scenarios of product usage by the end user.

Result

As a result most of all we got low-level tests that can detect bugs in individual functions and modules. They run faster than high-level UI tests of user scripts. Such ratio is optimal in terms of execution time and the amount of coverage - information about detected bugs appears quickly enough, autotests running doesn’t take much time and the percentage coverage of the code is large enough (you can be sure that the tests will detect all critical and significant bugs).

Here comes the question: why do we need such a variety of complex and long tests with custom scenarios, if we can write a lot of small, fast and easy to maintain unit tests?

Unit-tests test the code and give the developer confidence that a separate piece of his code works as intended, and doesn’t break the logic of his colleague’s code. UI tests test the entire system in a way that the user will use it. The creation of such tests takes from a couple of days to several weeks, however, it is important to have them on the project. They imitate the scenarios of using your product by real people - the impression people get of your product as a whole depends on them.

Here we have just one general recommendation: to have all kinds of autotests in proportions according to the pyramid at each level of the system architecture. Then it becomes possible to get an effective output from such tests. Functional tests, which testers conducted during the sprint: negative, positive, combinatorial, etc., should be placed on the API-tests level. That is the way a fast and stable package of regression tests is created.

Acceptance tests or user scripts of the web and mobile applications are taken to the level of UI-tests.

Following the recommendations of the pyramid, we get guaranteed functionality and at the same time save the team time. Just compare: regular costs of multi-day regression testing by 3-4 QAs and 2-3 weeks of development of auto-tests by one specialist and a couple of hours for their maintaining.

What automation tools to use

The choice of the tool depends on the specifics of the application and the requirements for the test scenarios. Most often several tools are used - separately for each level of the system architecture. For example, a GUI is tested with the help of Selenium, an API with java + restAssured, and a load with jMeter.

While choosing pay attention to the following facts:

1. We don’t consider beta versions and unstable tools building. This saves time in solving problems associated with the bugs of the tool itself. For example, problems with the encoding in RF HTTPLibrary;
2. It is desirable that the tool documentation is in Russian (as, for example, for Selenium);
3. Specialized forums for a specific tool help you to get answers to usage questions faster. For example, if you use the java TestNG + Selenium, you can find an answer to any question on the Internet. But if you take the Python based Robot Framework - the documentation will be in English only and without description of the keywords usage features and you have to analyze the library code while searching for this information;
4. Pay attention to the possibility of integration of testing tools with software that is used in the company;
5. The price of testing tools: if you plan a one-time test, buying expensive tools is inappropriate.

What tools are used for specific purposes?
To automate functional testing, support of a specific programming language is important, as well as the ability to create test reports for automatic logging of detected bugs with regular running of tests in CI.

Examples:

• Selenium, Appium, Winium
• Java-based libraries and tools: TestNG, jUnit, RestAssured, Allure;
• Based on the Python Robot Framework, Calabash (KDT), pytest, py.saunter etc.

Load testing tools require support of the protocol which is used by the application, monitoring server settings built-in tools during the test, flexible configuration and recording of load testing scenarios, means of analyzing results, and generating test reports.

Examples:

jMeter, Yandex.Tank, Siege, HP LoadRunner.

We prefer to work with popular and reliable testing tools: Selenium Webdriver, Java TestNG, jUnit, RestAssured, Allure, jMeter, although we have experience in using other tools (for example, Robot Framework).

Practically every task within one type of testing can be solved with the help of any tool, however the complexity and cost of the solution will differ. For example, if the tool doesn’t have results analysis and tools for reports creation (as in the case of testNg + Selenium without Allure usage), the time for analyzing the results of automated testing will be about the same as for manual testing. No benefits at all. The implementation of test automation will be carried out easily and quickly, only if at the very beginning the right tool for the tasks to be solved is chosen. Right tool choice can already become the key to success.

To sum up

Not all projects really need full automation. Sometimes, in order to speed up the work of the team - auxiliary scripts are enough.

However, if you plan to develop the product for years with a large team, you can’t do without automation. In our own experience there were cases when we implemented automation after 3-4 years of work on the product. And the result was visible to both the team and end users. In such cases, automation begins gradually, while the team gets used to new tools and the work process. Pay attention to the tools - it is important to choose those that are suitable for already used tools and libraries.

Automated testing is designed to help the team do the same tasks faster and to release new product versions more quickly. If testing usually takes 3-4 days, with automation this time will be reduced to several hours. This is especially true for large and constantly developing IT products.