In this article we’re going to tell you about how inDriver’s release team cut the duration of regression testing for the release build of a mobile app, and the release cycle, to one week, about the problems they came up against and about how they solved them.
Some background information
In 2018, the release team was significantly smaller than it is now, releases were not yet happening regularly and all of the company’s QA-engineers played a part in the pre-launch inspection. That was very time-consuming, though, and it slowed down the overall development process significantly. It was decided that the testing of each app released would be conducted solely by the release team.
Back then, the release cycles lasted 2 weeks. The team had 3 QA engineers, who weekly for 2–3 full-time days checked the release build of one of the alternating platforms. In addition to checking the release, the guys were busy streamlining the release process, writing documentation, onboarding newcomers, and updating test cases in the TMS.
There were a number of drawbacks with this approach:
Defects could only be discovered during the 2–3 days of the check, so more time was required to fix them and the Time-to-Market (TTM) increased.
If anyone in the team was unable to take part in the testing of the release, the workload on the other team members was significantly greater, and so too was the time required for the check.
Long release checks are tedious, and the likelihood of a defect being missed increases.
There is a low level of coverage across devices, meaning that specific defects were more likely to be missed.
During the check, you had to get to grips with all the new features and the ways of configuring the test environment to suit them. The test cases and other documentation therefore have to be perfectly written, before the point when the feature gets into the release. Otherwise, the inspection process would be slowed down: a lot of time would be wasted clarifying all the nuances.
Changing the release inspection process
At the end of 2021, the decision was taken to scale up the release check to the QA engineers from all teams. And it was decided that, from 2022 onwards, we would move to weekly checks of both the mobile platforms at the same time, and stop doing the TMS TestRail in favor of Qase, since the former was very slow and kept going down.
Since the automatic branch cuts and the forming of the release builds for the mobile app used to take place on a Friday evening, Monday was chosen as the most suitable day for checking the release. The tech leads in each team knew that, for 4 hours on a Monday, the QA engineers would be working solely on the release check. For those 4 hours, the tech leads had to remove, or at least minimize, any team activities in which the QA engineer was supposed to be involved.
In turn, by the time the check started, the release team would prepare a relevant test environment, form the test runs and distribute them to all the cases. The aforementioned problems were thereby solved:
All the bug reports are compiled in the first 4 hours of a Monday, and, accordingly, the fixes are put into the release build at a much earlier stage.
If a couple of QA engineers are unable to take part in the release check, this has practically no impact on how quickly the release check can be conducted.
The release check does not take more than 4 hours — and it’s much easier to maintain one’s concentration and alertness for that kind of time interval.
The level of coverage across devices goes up several times over.
Certain very specific new features are initially checked by their “owners”, and then gradually rolled out across the entire team.
The release team also writes UI self-tests, in conjunction with the QA Automation team, which develops various testing instruments. To achieve this, the following tools are used:
- Appium.
- Kotlin, JUnit 5.
- Selenoid.
- Allure.
The upshot
At this point in time, UI tests have covered about 30 percent of the total test cases in the acceptance set. They are launched immediately after the cut on the release branch, and the result of them, in the shape of an Allure-report, is sent to Slack by a bot. As this automated process unfolds, the test cases are taken out of the release run’s pool. During the release check, the QA engineer on duty goes through the fallen tests manually, and, where necessary, compiles a bug-report:
In order to keep the tests green and discover bugs ahead of time, they are launched on a dev-build every night. After that, the duty QA enters the new test update tasks or bug-reports on a special dashboard. A range of tests can also be launched by anyone who wishes to do so with the help of GitHub Actions:
As a result of the changes set out above, the duration of build release checks has fallen from around 72 hours to 4 hours, without sacrificing anything in terms of the quality of the product.
One of the problems that the release team encountered was that of time zones. Our staff are located in a wide range of different cities and countries, making it far from easy to conduct a simultaneous release check. However, given that most of the engineers are located in three time zones, it was decided that the time of the check should be made as close as possible, taking everyone’s work schedule into account. In Almaty, for instance, the release check starts first thing in the morning.
Another problem is related to the fact that, in order to get a large number of engineers involved in the same process, particular effort is required on the organizational front:
Drawing up lists of the people who are going to take part in the release, taking vacations and days off into account.
Redistributing cases in the release process, if anyone has had any problems getting through them.
Checking and preparing the test environment in good time, and doing the same thing with the methods of delivering the app build, so that there is no downtime.
Checking to make sure that the developers promptly set to work on any defects discovered in the release build.
Answering any questions that arise on the part of those involved in the process and dealing with any unexpected situations.
To solve this problem, a flow of weekly duties was introduced by one of the QA engineers in the release team, who assumes responsibility for all the duties referred to above and independently accompanies the release from beginning to end.
In the first instance, it was hard to work out how long the release check was going to take. To calibrate its duration, the progress on the cases it is going through is recorded at hourly intervals. It turned out that an interval of 4 hours was ideally suited to the task, and enabled everyone to go through the release at a comfortable pace.
Thereafter, the release team left in the flow a requirement to note the release’s progress. The table below shows the percentage of cases from the release run that have been completed, at a particular stage of the release check. It can be seen that the total time spent on the check gradually falls:
Now, the time required to check a release version of the app is gradually falling. With the transition to this flow, the TTM of product features was significantly reduced, and at the same time the quality of the check was enhanced. The release process, meanwhile, became more predictable and transparent for all the other teams.