VISUAL STUDIO   Windows Mac

Submitting UITests at the Command Line

PDF for offline use
Related Articles:

Let us know how you feel about this

Translation Quality


0/250

last updated: 2017-08

Testing on physical devices is an important part of creating a mobile application. Calabash allows developers to create automated tests for mobile applications and Xamarin Test Cloud provides a service that will run these tests on hundreds of physical devices. This guide will discuss how to submit UITests to Xamarin Test Cloud.

Overview

This guide discusses how to submit your UITests tests to Xamarin Test Cloud using the command line. This scenario is appropriate for continuous integration servers or builds that performed at the command line using a script. It will also discuss strategies for signing your Android APK’s with custom keystores.

Submitting tests to Test Cloud at the command line requires some preliminary work to generate two crucial pieces of information:

  • Test Cloud Team API Key – The Team API Key is an API key that ties a test run to a given Test Cloud team within a specific Organization. Please see the Organizations & Teams document for more details.
  • Test Cloud DEVICEID – This is a hash value that represents the collection of device/operating system combinations that were selected when the test run was created. It is not necessary to create a new DEVICEID for each test run. It is possible to use the DEVICE_ID for as long as the device/operating system combination is supported by Test Cloud.

ℹ The Team API Key and DEVICE_ID are not related and are independent of each other.

Limitations

Test runs that are submitted to Xamarin Test Cloud are added to a queue and will be run as devices become available. Once a test run has begun, they are subject to the following time constraints on a device:

  • Test runs for a paid subscription are limited to 360 minutes.
  • Test runs for a trial account are limited to 90 minutes.
  • Test runs for a Xamarin University account are limited to 30 minutes.
  • Individual Xamarin.UITests may not exceed 30 minutes each.
  • Individual Calabash steps may not exceed 10 minutes each.
  • Trial and University accounts are limited to 10 test runs per day.

ℹ The above times do not include waiting for a given device to become available, heavily utilized devices may require a long wait.

It not possible for a test run in Xamarin Test Cloud to perform, simulate/emulate, or control the following:

  • Network throttling
  • Started the app in specific device orientation
  • VPN into corporate network instead of opening up ports to the firewall
  • Integration with other apps installed on the device

Xamarin Test Cloud is not able to support the following hardware features:

  • Bluetooth
  • Throttling WiFi
  • Camera
  • Physically rotating the device
  • Simulating different battery conditions

Requirements

Familiarity with the command line is required.

It is assumed that UITests have been written and are running in your local environment, and that you have the test-cloud.exe application installed via NuGet.

In order to submit tests to Test Cloud:

  • Tests must be submitted by a email address that has been invited to a Test Cloud Team that is part of a valid Test Cloud Organization. Please see the Organizations & Teams document for more details.
  • The Xamarin Test Cloud Team API key must be provided
  • A test Run has been created, in particular the DEVICE_ID that was generated for the combination of device/operating system has been captured.

Android applications must request the INTERNET permission in AndroidManifest.XML.

Firewall Configuration

In order for tests to be submitted to Xamarin Test Cloud, the computer submitting the tests must be able to communicate with the Test Cloud servers. Firewalls must be configured to allow network traffic to and from the servers located at testcloud.xamarin.com on ports 80 and 443. This endpoint is managed by DNS and the IP address is subject to change.

In some situations, a test (or a device running the test) must communicate with web servers protected by a firewall. In this scenario the firewall must be configured to allow traffic from the following IP addresses:

  • 195.249.159.238
  • 195.249.159.239

Submitting UITests

Tests are submitted to the using test-cloud.exe command line application, which is distributed as a part of the Xamarin.UITest NuGet package. This is a multi-purpose utility that will submit both Android and iOS tests, and it works on both OS X and Windows.

The following table describes some of the more important command line parameters for test-cloud.exe, which may be found in the packages directory as shown in the screenshot below. It is not a complete list of all parameters.

Parameter Description
--assembly-dir <PATH> The path to the folder holding the test assemblies.
--devices <DEVICE ID> The device ID that was provided in the Test Cloud Upload dialog.
--user <EMAIL> The e-mail address of the team member submitting the tests.
--nunit-xml <FILENAME> The filename to which test results are exported, formatted as NUnit results XML. Optional.
--sign-info <FILENAME> Android only. Supply a signing information file that will be used to sign the Test Server APK. See the section below for more details. Optional.
--dsym <FILENAME> iOS only. Will upload the dSYM files along with the application and tests. This allows for more detail in the log files. Optional.
--fixture <NUNIT-FIXTURE> NUnit fixture / namespace to run. (Can be used multiple times)
--include <CATEGORY-NAME> Identifies the NUnit test categories that should only be included in the test run.
--exclude <CATEGORY-NAME> Identifies the NUnit test categories that should be excluded from the test run.
--test-chunk Run tests in parallel by method
--fixture-chunk Run tests in parallel by fixture
--data <FILE OR DIRECTORY> Uploads file or directory along with assemblies. (Can be used multiple times).

It is possible to see all the command line parameters available by passing the help command to test-cloud:

$ mono packages/Xamarin.UITest.X.X.X/tools/test-cloud.exe help submit
packages\Xamarin.UITest.X.X.X\tools\test-cloud help submit

It is important that all commands be run from the solution directory. As an example, the following screenshot of the CreditCardValidation folder on Github identifies the solution project for the CreditCardValidation-UITest project:

Limitations

Test runs that are submitted to Xamarin Test Cloud are added to a queue and will be run as devices become available. Once a test run has begun, they are subject to the following time constraints on a device:

  • Test runs for a paid subscription are limited to 360 minutes.
  • Test runs for a trial account are limited to 90 minutes.
  • Test runs for a Xamarin University account are limited to 30 minutes.
  • Individual Xamarin.UITests may not exceed 30 minutes each.
  • Individual Calabash steps may not exceed 10 minutes each.
  • Trial and University accounts are limited to 10 test runs per day.

ℹ The above times do not include waiting for a given device to become available, heavily utilized devices may require a long wait.

It not possible for a test run in Xamarin Test Cloud to perform, simulate/emulate, or control the following:

  • Network throttling
  • Started the app in specific device orientation
  • VPN into corporate network instead of opening up ports to the firewall
  • Integration with other apps installed on the device

Xamarin Test Cloud is not able to support the following hardware features:

  • Bluetooth
  • Throttling WiFi
  • Camera
  • Physically rotating the device
  • Simulating different battery conditions

Submitting iOS Tests

The following snippet is the simplest command line that could be used to submit an iOS application and tests to Test Cloud:

test-cloud.exe submit <PATH-TO-IPA> <TEAM API KEY> --devices=<DEVICES> --assembly-dir=<PATH-TO-TEST-ASSEMBLY-DIR> --user=<EMAIL>

Submitting Android Tests

The following snippet is the simplest command line that could be used to submit an Android application and tests to Test Cloud:

test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> --devices=<DEVICES> --assembly-dir=<PATH-TO-TEST-ASSEMBLY-DIR> --user=<EMAIL>
ℹ️

Only release builds of an Android application should be submitted to Test Cloud.

test-cloud.exe will sign the application APK and the test server APK with the same keystore before uploading to Test Cloud. It is possible to control this signing step as described in the next section.

Signing Android APK's

test-cloud.exe will use one of three algorithms to sign the APKs prior to submission:

  1. Use Build-in Keystore. This is the default behaviour. test-cloud.exe will sign the application and the test server APK using it’s own keystore.
  2. Provide your own Keystore. The second option is to provide your own keystore (along with credentials) that test-cloud.exe will use to sign the application and test server APK. The following snippet shows the additional command line parameters:

    test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> keystore <PATH-TO-KEYSTORE> <KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIAS-PASSWORD>

This option allows for complete control over what key will be used to sign the application, but it presents a potential security risk because the application keystore must be included in source code and the password for the keystore will be in clear text in build scripts.

  1. Use a signing information file. This option involves using test-cloud.exe to extract a few files from a signed test server APK, which are then stored in a small zip file, known as a signing information file. The main reason for using a signing information file is so that it can be added to source code control. This ensures that the application and test server are consistently signed with the same key information.

The following snippet shows how to create the signing information file:

test-cloud.exe gen-sign-info <PATH-TO-SIGNED_APK> <PATH_TO_KEYSTORE> <KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIAS-PASSWORD> --dir <DIRECTORY-TO-SAVE-SIGN-INFO>

When run, test-cloud.exe will create a file called testserver.si in the directory specified by the --dir parameter.

**Note**: The signing information file must be recreated each time the Xamarin.UITest package is updated.

Summary

This guide discussed how to use test-cloud.exe to upload Xamarin.UITests and applications to Xamarin Test Cloud for testing. It also discusses several options for how to sign Android APKs prior to submitting test runs to Test Cloud.

Xamarin Workbook

If it's not already installed, install the Xamarin Workbooks app first. The workbook file should download automatically, but if it doesn't, just click to start the workbook download manually.