Submitting UITests to Xamarin Test Cloud
Uploading Tests using Xamarin Studio or Visual Studio
Let us know how you feel about this
last updated: 2017-01
Testing on physical devices is an important part of creating a mobile application. Calabash allows developers to create automated tests for mobile applications and Xamarin Test Cloud provides a service that will run these tests on hundreds of physical devices. This guide will discuss how to submit UITests to Xamarin Test Cloud.
Once the automated tests have been written, developers will want to upload the tests and receive feedback about the test runs. There are two ways to accomplish this:
- Using the IDE – Both Xamarin Studio and Xamarin for Visual Studio can submit tests directly to Test Cloud.
- At the command line – It is also possible to upload tests and applications to Xamarin Test Cloud at the command line. This scenario is more suited for continuous integration or when the build process is started via a command line script.
This guide will focus on how to upload UITests and mobile applications to Xamarin Test Cloud using Xamarin Studio or Visual Studio. There are two possible scenarios to be considered:
- A complete Xamarin solution with a mobile app project and a Xamarin.UITest project – This will discuss how to submit a tests when the mobile app is written using Xamarin and the Xamarin.UITest project is a part of the solution.
- A Xamarin.UITest project and an external IPA or APK – In this case Xamarin.UITests are written for a mobile app that was created outside of the Xamarin platform.
Test runs that are submitted to Xamarin Test Cloud are subject to the following time constraints:
- Test runs for a paid subscription are limited to 360 minutes.
- Test runs for a trial account are limited to 90 minutes.
- Test runs for a Xamarin University account are limited to 30 minutes.
- Individual Xamarin.UITests may not exceed 30 minutes each.
- Individual Calabash steps may not exceed 10 minues each.
- Trial and University accounts are limited to 10 test runs per day.
It not possible for a test run in Xamarin Test Cloud to perform, simulate/emulate, or controlling the following:
- Network throttling
- Started the app in specific device orientation
- VPN into corporate network instead of opening up ports to the firewall
- Integration with other apps installed on the device
Xamarin Test Cloud is not able to support the following hardware features:
- Throttling WiFi
- Physically rotating the device
- Simulating different battery conditions
IDE support for Xamarin Test Cloud requires Xamarin Studio 5.9 or Xamarin for Visual Studio 3.11.
It is assumed that a Test Cloud team has been created, that your e-mail address has been added to the team, and you have the Team API Key. Please see the Organizations & Teams document for more Details.
Android applications must request the INTERNET permission in AndroidManifest.XML.
In order for tests to be submitted to Xamarin Test Cloud, the computer submitting the tests must be able to communicate with the Test Cloud servers. Firewalls must be configured to allow network traffic to and from the servers located at testcloud.xamarin.com on ports 80 and 443. This endpoint is managed by DNS and the IP address is subject to change.
In some situations, a test (or a device running the test) must communicate with web servers protected by a firewall. In this scenario the firewall must be configured to allow traffic from the following IP addresses:
Submitting UITests From a Xamarin Solution
Both Visual Studio and Xamarin Studio can submit UITests to Xamarin Test Cloud when the solution contains both a mobile Xamarin project and a UITest project. When UITests are submitted to Xamarin Test Cloud, the IDE will rebuild the application and the tests and upload the binaries, placing them in a queue for execution.
It is not possible to select an NUnit test
Categorywhen submitted tests to Xamarin Test Cloud through the IDE. Running tests that belong to an NUnit
Categoryis only supported when submitting tests by using the command line.
Prior to submitting an Android project, you must ensure that the Android project is set as the startup project:
Before submitting an iOS project, you must ensure that your build configuration is set to Debug > iPhone and that the iOS project is set as the startup project:
This will cause the Run in Test Cloud dialog to appear. Select which application you would like to run the tests for:
The upload will begin once you click the Upload and Run button. You can see the progress in the Tests Cloud pane inside the IDE:
When you do so, Visual Studio will present you with the following dialog informating you that the workflow is being handed off to the web browser:
When you click the OK button, the application and tests will be uploaded to Test Cloud.
When the upload is finished, the IDE will hand off the workflow to the the web browser to complete the test run. If necessary, you will be prompted to log into the Test Cloud website.
First, it will be necessary to select an existing team for the test run:
After the team has been choosen, selected the devices to run the test on:
The final dialog will allow you to provide some final information about the test run itself:
Once you click Done the test run will be queued in Test Cloud. When the test run is finished, Test Cloud will send out an e-mail with a summary of the test results:
Click on View Results in order to see details about the test run on the Test Cloud website.
Submitting UITests for an External Mobile App
It is possible to write Xamarin.UITests for non-Xamarin mobile applications. In this situation, Xamarin Studio can submit tests for a solution that only has single Xamarin.UITest project. Open up the Unit Test pad, and right click on the UITest project. Select Run in Test Cloud from the context menu that appears:
The <Run In Test Cloud dialog will appear:
Select the APK or the IPA to submit along with the tests, and then click the Upload and Run button. Xamarin Studio will then proceed to compile the UITest project and upload the test assembly along with the APK or IPA to Xamarin Test Cloud.
Although it is possible to write Xamarin.UITests for an existing IPA or APK in Visual Studio, it is not possible to submit through Visual Studio.
This guide discussed how to submit tests to Xamarin Test Cloud using Xamarin Studio or Visual Studio.