Introduction to Xamarin Test Cloud
- PDF for offline use
- Related Links:
Let us know how you feel about this
last updated: 2016-10
This guide offers a quick introduction to Xamarin Test Cloud and Automated UI Acceptance Testing. It will discuss the underlying concepts and components behind Xamarin Test Cloud and the frameworks that allow developers to write automated UI tests.
Mobile users are perhaps among the most demanding consumers of software today – they expect mobile apps to be responsive, bug free, and low cost. Apps that don’t meet this expectation will be promptly uninstalled and given a very low rating.
The most effective way to validate the behavior of an application is by running it and using it. If it behaves as expected, without crashing or returning incorrect results, then the application is in good shape and can be released to users without worry that it will cause frustration or be unusable. This process, of testing the UI by actually using an application and interacting with it, is known as UI Acceptance Testing. Traditionally, however, UI Acceptance Testing has been extremely expensive due to its manual nature. It has historically required lots of manpower to deploy the application and then use it, verifying that it behaves correctly. Multiply that by each build that needs testing as well as each device and operating system that it needs testing on, and the cost can skyrocket exponentially, meaning that effective UI Acceptance Testing has been out of reach for nearly everyone but the most resource-laden organizations.
Xamarin Test Cloud is a cloud-based solution that provides tools which support automated UI Acceptance Testing of mobile applications across hundreds of different devices. This enables anyone to ensure that their application performs correctly and efficiently across a variety of devices with minimal effort. Additionally, because it’s cloud-based, the maintenance and procurement efforts are removed from the test consumer, who can then focus on verify that the app works correctly.
The Xamarin Test Cloud ecosystem consists of the following parts:
- Calabash – This is a framework that allows tests to be written with Cucumber and Ruby. Calabash tests are well suited to Behavior Driven Development – a methodology that focuses on creating executable specifications. The specifications are written in the every day language that business would use, and then developers write tests to automate a mobile app by parsing the specifications. Calabash is no longer under active development.
- Xamarin.UITest – This is a framework that allows tests to be written in C# using the popular NUnit testing library. This framework is well suited to teams that are already skilled with writing NUnit tests and/or already developing their mobile applications using Xamarin.
- Test Cloud – Test Cloud is a cloud based service consisting of thousands of physical mobile devices. Users upload their apps and tests to Test Cloud, which will install the apps on the devices and run the tests. When the tests are complete, Test Cloud, the results made available to users through an easy to use and informative web-based front end.
- Xamarin Test Recorder – This tool, still under development, can simplify creating tests and is great for someone who is new to Xamarin.UITest and unfamiliar with the APIs. Testers can start Test Recorder, connect it to device, simulator, or emulator, and then start using the mobile app. Test Recorder will capture the interactions between the user and the mobile app, and output a Xamarin.UITest in C# for that scenario.
The Xamarin Test Cloud system provides the libraries and tools that allow teams to create automated tests, run them on a multitude of devices and validate that the Behavior is correct. The mobile application and tests are uploaded to Xamarin Test Cloud, which will install the app and run the tests on hundreds of physical devices. This referred to as a test run. When the test run is complete, Test Cloud will send out a notification with the test results. This workflow can be seen in the following diagram:
The test run can be performed manually by a user or as part of a Continuous Integration (CI) workflow that automatically submits new application builds to Xamarin Test Cloud when code changes are checked in:
We shall see how to write these tests later, but it is helpful to understand a little bit about the underlying architecture that runs the tests on the device. Let's discuss, at a high level, some of the components involved with automated UI testing.
Phone is not supported by Xamarin Test Cloud or Calabash
The Anatomy of the Test Cloud Framework
Automated interactions with a mobile application require some sort of automation library that will simulate a user's action and allow the test to verify the state of the user interface to prove the application is working correctly. Both Android and iOS have their own proprietary UI automation frameworks. These two very different set of testing APIs would prove challenging to a tester who wants to write cross-platform tests. To help address is concern, one or two helper components will be installed on the mobile device along with the app:
- Xamarin Test Cloud Agent – This is a lightweight HTTP server that interacts with tests via JSON over HTTP. The Xamarin Test Cloud Agent is a middle-man that will take queries (and in some cases, actions) from the tests and perform them on the application being tested. The Xamarin Test Cloud Agent is required for both Android and iOS applications, and has a slightly different role on each platform.
- DeviceAgent – This is installed only for iOS applications that are built with Xcode 8. Logically, it is very similar to the Xamarin Test Cloud Agent – it is responsible for performing gestures and advanced queries on iOS views but does so using a different set of testing APIs. We will learn more about the DeviceAgent below.
Let's see what role each component plays on the respective platforms.
Xamarin Test Cloud Agent on Android
In Android, The Xamarin Test Cloud Agent is responsible for using the Android automation APIs to control the user interface and to locate views so that a test may interact with them. It is bundled into a separate APK and runs as a separate application, which has permission to automate the application under test. This is possible, because when the test is deployed to the mobile device, Calabash or UITest will sign both application packages with the same key.
Xamarin Test Cloud Agent on iOS
The Test Cloud Agent has a slightly different role in iOS applications – by itself it does not automate the app under test. Instead, the Test Cloud Agent will interrogate the active window and retrieve information about the views for to the user. The view information is returned to the test script. The test script will then automate the iOS application with the help of another component called the DeviceAgent. The DeviceAgent will simulate the gestures and actions for the test (using the automation API's provide with Xcode 8) and if necessary return the result of those interactions to the test:
The Xamarin Test Cloud Agent is available via a NuGet Package, and must be include in the App Bundle before a test can be executed. The Xamarin Test Cloud Agent should only be included in Debug builds of the application. Apple will reject apps that are submitted with the Xamarin Test Cloud Agent linked into the App Bundle.
The latest versions of Calabash and UITest support testing with both Xcode 7 and Xcode 8. The verison of Xcode used to build the app will determine which native automation API is used to automate the app. The Testing Framework and iOS compatibility for Calabash is determined by the version of Xcode that is used. See the table below:
|iOS Version||Local with Xcode 7||Local with Xcode 8|
For example, a Mac with Xcode 8 installed will be able to test on devices running iOS 10 or iOS 9 using DeviceAgent, but would not be able to test on iOS 8 or below.
The Xamarin Test Cloud Agent NuGet package is only for Xamarin.iOS apps; it is not available (or required) for Xamarin.Android applications.ℹ️
This section assumes that Xcode 8 is being used to develop iOS apps. iOS apps built using Xcode 7 will only use the Xamarin Test Cloud Age Agent.
Running Tests in Xamarin Test Cloud
Running Tests in Xamarin Test Cloud is conceptually similar to running tests locally, except that Xamarin Test Cloud hosts the tests and will execute them on selected devices:
To facilitate this, tests are uploaded along with the app when it’s pushed to Xamarin Test Cloud for testing. Xamarin Test Cloud will then reset the device to a clean state, install the app, and run the tests.
Test runs that are submitted to Xamarin Test Cloud are added to a queue and will be run as devices become available. Once a test run has begun, they are subject to the following time constraints on a device:
- Test runs for a paid subscription are limited to 360 minutes.
- Test runs for a trial account are limited to 90 minutes.
- Test runs for a Xamarin University account are limited to 30 minutes.
- Individual Xamarin.UITests may not exceed 30 minutes each.
- Individual Calabash steps may not exceed 10 minutes each.
- Trial and University accounts are limited to 10 test runs per day.
The above times do not include waiting for a given device to become available, heavily utilized devices may require a long wait.
It not possible for a test run in Xamarin Test Cloud to perform, simulate/emulate, or control the following:
- Network throttling
- Started the app in specific device orientation
- VPN into corporate network instead of opening up ports to the firewall
- Integration with other apps installed on the device
Xamarin Test Cloud is not able to support the following hardware features:
- Throttling WiFi
- Physically rotating the device
- Simulating different battery conditions
In this guide we discusses the concepts behind Xamarin Test Cloud. We learned about Calabash and Xamarin.UITest, testing frameworks for Ruby and C# respectively. We discussed the Xamarin Test Cloud Agent and the role it plays in testing Android and iOS applications. We were also introduced to the DeviceAgent, an app that will is necessary to automate an iOS application that is compiled and packaged with Xcode 8.