VISUAL STUDIO   Windows Mac

Submitting UITests to Xamarin Test Cloud with Visual Studio

PDF for offline use
Related Articles:
Related Links:

Let us know how you feel about this

Translation Quality


0/250

last updated: 2017-08

Testing on physical devices is an important part of creating a mobile application. Calabash allows developers to create automated tests for mobile applications and Xamarin Test Cloud provides a service that will run these tests on hundreds of physical devices. This guide will discuss how to submit UITests to Xamarin Test Cloud.

Overview

Once the automated tests have been written, developers will want to upload the tests and receive feedback about the test runs. There are two ways to accomplish this:

  • Using the IDE – Visual Studio 2015 and Visual Studio for Mac can submit tests directly to Test Cloud. Developers using Visual Studio 2017 must submit tests to Test Cloud using the command line.
  • At the command line – It is also possible to upload tests and applications to Xamarin Test Cloud at the command line. This scenario is more suited for continuous integration or when the build process is started via a command line script.

This guide will focus on how to upload UITests and mobile applications to Xamarin Test Cloud using Visual Studio. There are two possible scenarios to be considered:

  1. A complete Xamarin solution with a mobile app project and a Xamarin.UITest project – This will discuss how to submit a tests when the mobile app is written using Xamarin and the Xamarin.UITest project is a part of the solution.
  2. A Xamarin.UITest project and an external IPA or APK – In this case Xamarin.UITests are written for a mobile app that was created outside of the Xamarin platform.

Limitations

Test runs that are submitted to Xamarin Test Cloud are added to a queue and will be run as devices become available. Once a test run has begun, they are subject to the following time constraints on a device:

  • Test runs for a paid subscription are limited to 360 minutes.
  • Test runs for a trial account are limited to 90 minutes.
  • Test runs for a Xamarin University account are limited to 30 minutes.
  • Individual Xamarin.UITests may not exceed 30 minutes each.
  • Individual Calabash steps may not exceed 10 minutes each.
  • Trial and University accounts are limited to 10 test runs per day.

ℹ The above times do not include waiting for a given device to become available, heavily utilized devices may require a long wait.

It not possible for a test run in Xamarin Test Cloud to perform, simulate/emulate, or control the following:

  • Network throttling
  • Started the app in specific device orientation
  • VPN into corporate network instead of opening up ports to the firewall
  • Integration with other apps installed on the device

Xamarin Test Cloud is not able to support the following hardware features:

  • Bluetooth
  • Throttling WiFi
  • Camera
  • Physically rotating the device
  • Simulating different battery conditions

Requirements

IDE support for Xamarin Test Cloud is available in Visual Studio for Mac and Visual Studio 2015. Visual Studio 2017 does not support submitting UITests to Test Cloud.

It is assumed that a Test Cloud team has been created, that your e-mail address has been added to the team, and you have the Team API Key. Please see the Organizations & Teams document for more Details.

Android applications must request the INTERNET permission in AndroidManifest.XML.

Firewall Configuration

In order for tests to be submitted to Xamarin Test Cloud, the computer submitting the tests must be able to communicate with the Test Cloud servers. Firewalls must be configured to allow network traffic to and from the servers located at testcloud.xamarin.com on ports 80 and 443. This endpoint is managed by DNS and the IP address is subject to change.

In some situations, a test (or a device running the test) must communicate with web servers protected by a firewall. In this scenario the firewall must be configured to allow traffic from the following IP addresses:

  • 195.249.159.238
  • 195.249.159.239

Submitting UITests From a Xamarin Solution

Both Visual Studio and Visual Studio for Mac can submit UITests to Xamarin Test Cloud when the solution contains both a mobile Xamarin project and a UITest project. When UITests are submitted to Xamarin Test Cloud, the IDE will rebuild the application and the tests and upload the binaries, placing them in a queue for execution.

ℹ️

It is not possible to select an NUnit test Category when submitted tests to Xamarin Test Cloud through the IDE. Running tests that belong to an NUnit Category is only supported when submitting tests by using the command line.

Prior to submitting an Android project, you must ensure that the Android project is set as the startup project:

Prior to submitting an Android project, you must create the APK first and ensure that the Android project is set as the startup project:

Before submitting an iOS project, you must ensure that your build configuration is set to Debug > iPhone and that the iOS project is set as the startup project:

Next, right click on the project in the Test Pad, and select Run in Test Cloud:

This will cause the Run in Test Cloud dialog to appear. Select which application you would like to run the tests for:

The upload will begin once you click the Upload and Run button. You can see the progress in the Tests Cloud pane inside the IDE:

Next, right click on the the UITest project in the Solution Explorer, and select Run in Test Cloud from the context menu:

When you do so, Visual Studio will present you with the following dialog informating you that the workflow is being handed off to the web browser:

When you click the OK button, the application and tests will be uploaded to Test Cloud.

When the upload is finished, the IDE will hand off the workflow to the the web browser to complete the test run. If necessary, you will be prompted to log into the Test Cloud website.

First, it will be necessary to select an existing team for the test run:

After the team has been choosen, selected the devices to run the test on:

The final dialog will allow you to provide some final information about the test run itself:

Once you click Done the test run will be queued in Test Cloud. When the test run is finished, Test Cloud will send out an e-mail with a summary of the test results:

Click on View Results in order to see details about the test run on the Test Cloud website.

Submitting UITests for an External Mobile App

It is possible to write Xamarin.UITests for non-Xamarin mobile applications. In this situation, Visual Studio for Mac can submit tests for a solution that only has single Xamarin.UITest project. Open up the Unit Test pad, and right click on the UITest project. Select Run in Test Cloud from the context menu that appears:

The <Run In Test Cloud dialog will appear:

Select the APK or the IPA to submit along with the tests, and then click the Upload and Run button. Visual Studio for Mac will then proceed to compile the UITest project and upload the test assembly along with the APK or IPA to Xamarin Test Cloud.

Although it is possible to write Xamarin.UITests for an existing IPA or APK in Visual Studio, it is not possible to submit through Visual Studio.

Summary

This guide discussed how to submit tests to Xamarin Test Cloud using Visual Studio for Mac or Visual Studio.

Xamarin Workbook

If it's not already installed, install the Xamarin Workbooks app first. The workbook file should download automatically, but if it doesn't, just click to start the workbook download manually.