Submitting Calabash Tests to Xamarin Test Cloud
- PDF for offline use
- Related Articles:
Let us know how you feel about this
last updated: 2017-01
Testing on physical devices is an important part of creating a mobile application. Calabash allows developers to create automated tests for mobile applications and Xamarin Test Cloud provides a service that will run these tests on hundreds of physical devices. This guide will discuss how to submit tests to Xamarin Test Cloud.
This guide discusses how to submit your Calabash tests to Xamarin Test Cloud.
Test runs that are submitted to Xamarin Test Cloud are added to a queue and will be run as devices become available. Once a test run has begun, they are subject to the following time constraints on a device:
- Test runs for a paid subscription are limited to 360 minutes.
- Test runs for a trial account are limited to 90 minutes.
- Test runs for a Xamarin University account are limited to 30 minutes.
- Individual Xamarin.UITests may not exceed 30 minutes each.
- Individual Calabash steps may not exceed 10 minutes each.
- Trial and University accounts are limited to 10 test runs per day.
ℹ The above times do not include waiting for a given device to become available, heavily utilized devices may require a long wait.
It not possible for a test run in Xamarin Test Cloud to perform, simulate/emulate, or control the following:
- Network throttling
- Started the app in specific device orientation
- VPN into corporate network instead of opening up ports to the firewall
- Integration with other apps installed on the device
Xamarin Test Cloud is not able to support the following hardware features:
- Throttling WiFi
- Physically rotating the device
- Simulating different battery conditions
Android applications must request the internet permission in AndroidManifest.XML.
Xamarin Test Cloud uses Ruby 2.3.1p112. For maximum compatibility with tests run locally and tests run in Test Cloud it is recommended that you use the same version while developing tests.
It is assumed that Calabash tests have been written and are running in your local environment, and that you have the xamarin-test-cloud gem installed.
It is assumed that a Test Cloud project has been created, your account has been added to that team, and you have the Team API Key. Please see the Organizations & Teams document for more details.
In order for tests to be submitted to Xamarin Test Cloud, the computer submitting the tests must be able to communicate with the Test Cloud servers. Firewalls must be configured to allow network traffic to and from the servers located at testcloud.xamarin.com on ports 80 and 443. This endpoint is managed by DNS and the IP address is subject to change.
In some situations, a test (or a device running the test) must communicate with web servers protected by a firewall. In this scenario the firewall must be configured to allow traffic from the following IP addresses:
Submitting Calabash Tests
Tests are submitted using the command line tool test-cloud, which is distributed via the xamarin-test-cloud gem. test-cloud is a multi-purpose utility that will upload iOS and Android tests. It works on OS X and Windows.
The following table describes some of the more important command line parameters for test-cloud. It is not a complete list of all parameters.
|--devices=<DEVICES>||The device ID that was provided in the Test Cloud Upload dialog.|
|--user=<EMAIL>||The e-mail address of the team member submitting the tests.|
|--config=<CONFIG>||The path to the Cucumber configuration file. i.e. **./config/cucumber.yml**|
|--profile=<PROFILE>||This is the Cucumber profile to use when running the tests. Optional, but recommended.|
|--dsym=<DSYM-FILE>||iOS only. Will upload the dSYM files along with the application and tests. This will allow for more detail in the log files. Optional|
It is important that all commands be run from the project directory that holds the config and features directory. As an example, consider the following screenshot of the TaskyPro-Calabash sample project:
Calabash scripts may only reference local gems that are installed on your machine. It is not possible to use Gems that are referenced with
:path=>. At the time of submission, test-cloud will package up those gems and submit them along with the test scripts to Xamarin Test Cloud.
As a part of the submission process, test-cloud will run some quick validation checks on the project before submitting the test to Xamarin Test Cloud. It is also possible to use Cucumber's dry run feature to validate that the tests are correct using either of the two following command line snippets in your project directory:
cucumber --profile <your XTC profile> --config <path to cucumber.yml> --dry-run
Cucumber will not load the features/support/env.rb class when the
--dry-run parameter is specified. This may cause problems for more complex test projects that rely on this feature.
Creating A Gemfile
If you are using Bundler to maintain the gems in use by your scripts, then you may ignore this section as you already have a Gemfile. If you are not using Bundler, then you should create a Gemfile along with your tests. A Gemfile is a text file that tells Test Cloud what version of the gems your project uses.
Create a text file in the directory that holds your Calabash features folder, and name it Gemfile. A sample Gemfile is shown below:
source 'https://rubygems.org' gem 'calabash-cucumber', '0.16.3' gem 'calabash-android', '0.5.14' gem 'xamarin-test-cloud', '1.1.2'
The gems and their version number listed should match the version number of the gems you have installed.
Submitting iOS Applications to Test Cloud
iOS tests and applications are submitted at the command line using the test-cloud application:
test-cloud submit <PATH-TO-IPA> <TEST-CLOUD-API-KEY> --user=<EMAIL> --devices <DEVICE-ID> --profile=<PROFILE> --dsym=<PATH-TO-DSYM>
Submitting Android Applications to Test Cloud
Submitting Android applications to Test Cloud requires the following two steps:
Build the Android Test Server for the application. If you do not build the test server for the application, test-cloud will not be able to submit the tests to Test Cloud. You can build the Android Test Server using the following command:
$ calabash-android build <PATH TO YOUR APK>
If this step is omitted, test-cloud will halt the upload and display an error message.
Run test-cloud to upload the application and tests. The following snippet is an example of a command line that should be used to submit tests to Test Cloud:
$ test-cloud submit <PATH-TO-APK <TEST-CLOUD-API-KEY> --user=<EMAIL> --devices <DEVICE-ID> --profile=<PROFILE>
Signing Android APK's
An Android APK must be signed before test-cloud will upload the tests to Xamarin Test Cloud. test-cloud will use it’s own keystore for resigning the APK and the test server APK. It is possible to change the default keystore used for signing these two APK’s by running calabash-android setup as shown below:
$ calabash-android setup Please enter keystore information to use a custom keystore instead of the default Please enter keystore location ./creditcardvalidation-example.keystore Please enter the password for the keystore password1 Please enter the alias uitest_sample Saved your settings to calabash_settings. You can edit the settings manually or run this setup script again
This guide discussed how to use test-cloud to upload Calabash tests and applications to Xamarin Test Cloud for testing.