How to: Run Android/iOS tests on multiple devices with Gitlab
To complete my education as a software developer it was required to realise a software development related project over a period of 10 days. My task was to expand the Continuous Integration System already existing at Smoca.
As a version control system Smoca uses Git and all the projects are hosted on a local Gitlab server. For Continuous Integration Gitlab's built-in tools are used. The goal was to create a system that runs UI and unit tests on multiple devices (not simulators) whenever a new version is pushed to the server. This should work for iOS and Android projects. In this blog post I will share how I achieved this goal so that anyone can recreate such a system.
Testing on real mobile devices
Testing on real devices has many advantages over testing in a simulator. There are many problems that can occur on a real device but are often difficult to simulate. For example a bad internet connection can dramatically impact the user experience in a negative way. Bugs could also occur depending on different hardware. This is a problem especially for Android Apps. Finally, different versions of operating systems might lead to a different behaviour within an app. These problems can be reduced by testing on the most commonly used phones.
Testing system setup
Here is a quick summary of all the steps necessary to setup a similar testing system. This guide works specifically for Gitlab though the concepts can be applied to other Continuous Integration systems.
First of all, you will need a Gitlab-Runner. That's a piece of software installed on a computer in your network. This will have to be Mac if you want to run iOS tests since the Apple build tools are only available for macOS. I would recommend a dedicated Mac mini. The guide lists the necessary software you will need to install on it. This mainly consists of all the software needed for building and testing iOS/Android Apps.
In addition to the Gitlab-Runner you will also need to prepare the project of the apps to be tested. In the root of the project at a gitlab-ci.yml file. In this file “jobs” are defined. In this example one for testing. In the end the job executes a bash command.
Technically it would be possible to use the standard build-tools provided by Apple and Google. This can be a bit cumbersome though, especially for iOS since the commands get very verbose quickly. Also code signing can be a huge pain. For these reasons I used a powerful tool named Fastlane, which is a cross platform build tool that provides many useful functions such as automated code-signing, building, testing, releasing and many more.
I have written three guides that provide detailed information about the setup of a Gitlab-Runner and the setup of iOS and Android projects. These guides can be found here:
- Gitlab-Runners Setup Guide: https://github.com/smoca-ag/ci-demo-app-readme
- iOS Project and Setup Guide: https://github.com/smoca-ag/ci-demo-app-ios
- Android Project and Setup Guide: https://github.com/smoca-ag/ci-demo-app-android
As a result of my individual practical work, smoca now has the opportunity to run automized app tests on several mobile devices. I expect that my efforts will help smoca to reach an even higher quality of app development.