Visual Studio App Center Test is the next generation of Xamarin Test Cloud! Read the blog post.


PDF for offline use

Let us know how you feel about this

Translation Quality


last updated: 2017-02

This guide will discuss some of the issues that can occur when trying to run Calabash or UITests on local devices.


This guide will discuss some of the issues that can occur when trying to run UITests on local devices.


Xamarin Test Cloud and Xamarin.UITest only support Android and iOS applications. The Windows Phone and Windows versions of Xamarin.Forms apps cannot be tested in Test Cloud.

It is important to understand the concepts described in the Introduction to Xamarin Test Cloud guide.

It is assumed that the most recent stable version of the Xamarin.UITest NuGet Package is installed in the UITest project. It is assume that iOS projects have the most recent version of the Xamarin Test Cloud Agent installed.

iOS devices must be configured with a valid development provisioning profile.

To run Xamarin.UITests with Visual Studio for Mac, the following dependencies must be met:
  • NUnit 2.6.x – Xamarin.UITest is not compatible with NUnit 3.x.
  • Android SDK – Only if testing Android apps.
  • Java Developers Kit – Only if testing Android apps.
  • Xcode Command Line Tools – Only for testing iOS apps.

To run Xamarin.UITests with Visual Studio, the following dependencies must be met:

  • NUnit 2.6.x – Xamarin.UITest is not compatible with NUnit 3.x.
  • A Test Runner for Visual Studio – A 3rd party test runner, such as the NUnit Test Adapter for NUnit 2 or Resharper from Jetbrains, is required for Visual Studio to be able to run the NUnit tests. The NUnit3TestAdapter is not compatible with Xamarin.UITest.
  • Android SDK – Only if testing Android apps. Windows requires that the ANDROID_HOME environment variable is set with the path to the Android SDK.
  • Java Developers Kit – Only if testing Android apps.

It is not possible to run UITests for iOS apps on Windows.

Troubleshooting Topics

The Apple App Store Rejected My iOS App Because of "Non-public API Usage"

If you are using Xamarin Test Cloud to test your application and it is rejected by the App Store for a reason that resembles the following explanation:

The app references non-public selectors in YOUR_PROJECT: _addRecorder:, _playbackEvents:atPlaybackRate:messageWhenDone:withSelector:, _removeRecorder:, applicationWillTerminate, ignore:, terminateWithSuccess If method names in your source code match the private Apple APIs listed above, altering your method names will help prevent this app from being flagged in future submissions. In addition, note that one or more of the above APIs may be located in a static library that was included with your app. If so, they must be removed.

The issue could be the Calabash.DLL is included in your final IPA that is submitted. The Calabash DLL makes uses of non public Apple API's which will cause your app to be rejected by the App Store. The Xamarin.iOS linker will remove the Calabash DLL from the final IPA if it not explicitly referenced anywhere by the code. By default a UITest project created by the Xamarin templates for Release builds do not have the ENABLE_TEST_CLOUD compiler variable, which will cause the Calabash DLL to be removed from app bundle. Debug builds will have the compiler directive defined, preventing the linker from removing the Calabash DLL.

To solve this problem, ensure that the ENABLE_TEST_CLOUD compiler variable is set for your Debug builds, and is not present for Release builds. Then add the following snippet to the FinishedLaunching method of the AppDelegate class to ensure that the Calabash DLL is excluded from Release builds:

#region Code for starting up the Xamarin Test Cloud Agent

// Newer version of Visual Studio for Mac and Visual Studio provide the
// ENABLE_TEST_CLOUD compiler directive to prevent the Calabash DLL from
// being included in the released version of the application.

Running UITests at the Command Line Fail with Little Diagnostic Information

Although it is possible to run Xamarin.UITests with a version of the NUnit console runner older than 2.6.3, it is possible that some tests may fail with no meaningful information. If the UITest all pass when run via the IDE, but fail when run at the command line, you should download NUnit 2.6.3 (but less than NUnit 3.0) and use that to run your tests.

How to Obtain Test Results

To diagnose this issue, examine the XML output from the test runner. By default nunit-console creates a file called TestResult.xml with more details. It is also possible to specify an XML file with the nunit-console.exe command line parameter -xml, for example:

$ nunit-console -xml=MyTestResults.xml ./MyUiTest.UITests/bin/Release/MyUiTest.UITests.dll

Parameterized NUnit Test Fixtures Fail When Run Locally

The parameterized TestFixtureAttribute was introduced in NUnit 2.5.3. It is possible to run Xamarin.UITests with any NUnit compatible test runner, however older NUnit test runners will fail when they try to run the parameterized tests with very little diagnostic information.

To correct this issue, run the UITest using the nunit-console test runner from NUnit 2.6.3 (NUnit 3.0 is not compatible with Xamarin.UITest) or higher.

Instruments failed to start the app while running UI Test on iOS device

One cause of this error is when a Calabash or Xamarin.UITest is run on an iOS device that does not have UIAutomation enabled. Please see Running Calabash Tests on Local Devices or Device Configuration for details on how to enable UIAutomation on an iOS device.

Enable Development After Upgrading Devices to iOS 8.x

The Settings > Developer table view row disappears after updating to iOS 8.1.1. The device will not be ready for Calabash testing until it has been re-enabled for Development. Open Xcode and navigate to the Devices window (Shift + Command + 2). Wait for Xcode to finish copying symbols and then check the device's for the Developer settings.

Once this is done, reboot the device.

Does not contain a compatible architecture for target device (RunLoop::IncompatibleArchitecture)

Symptoms: When targeting a Simulator, the app launches, then quits several times in rapid succession.

The problem is that the binary in incompatible with the simulator version.

For example, if you build an app iPhone 6 Simulator it will contain an x86_64 instruction slice. You can install this binary on an iPhone 4s simulator, but it will not launch because the iPhone 4s Simulator because that simulator requires an i386 instruction set. Simply put, you must build an app that is compatible for the simulator you are targeting.

Update to the Latest Calabash Framework

The Calabash tests must use the same version of the Calabash server that the application is linked with. This section covers how to update the Calabash framework that will be used by Ruby.


This section does not apply if you are using the Xamarin Test Cloud Agent. This section is applicable only to iOS applications that have the Xamarin Test Cloud Agent manually linked in.

  1. First, ensure that the most recent version of calabash-cucumber is installed:
$ gem install calabash-cucumber
Fetching: run_loop-1.1.0.gem (100%)
Successfully installed run_loop-1.1.0
Fetching: calabash-cucumber-0.11.4.gem (100%)
Successfully installed calabash-cucumber-0.11.4
Parsing documentation for run_loop-1.1.0
Installing ri documentation for run_loop-1.1.0
Parsing documentation for calabash-cucumber-0.11.4
Installing ri documentation for calabash-cucumber-0.11.4
Done installing documentation for run_loop, calabash-cucumber after 2 seconds
2 gems installed
  1. Next, download the Calabash server binary:
$ calabash-ios download
caution: excluded filename not matched:  \__MACOSX/*
caution: excluded filename not matched:  calabash.framework/.DS_Store
This will download the most recent Calabash server binary to your working folder.

Clear the Simulator Content & Settings:

The following command line will reset all simulators to their factory state:

$ bundle exec calabash-ios sim reset

Building Compatible Binaries using xcodebuild:

Here is an example of a xcodebuild command that will build an .app that is compatible for all simulators:

$ xcrun xcodebuild \
    -SYMROOT=build \
    -derivedDataPath build \
    ARCHS="i386 x86_64" \
    VALID_ARCHS="i386 x86_64" \
    -project ./path/to/your/.xcproject directory \
    -scheme "The Name of your Calabash Scheme" \
    -sdk iphonesimulator \
    -configuration DEBUG \
    clean build

The .app can be found in this directory: ./build/Build/Products/Debug-iphonesimulator/ Use this path as your APP path.

Building Compatible Binaries using Xcode

Adjust the build settings of your Calabash target to always build a compatible binary.


Calabash 0.12.0 or newer will raise an error if the .app is an incompatible architecture for the target simulator.

Invoking UIAutomation APIs cause a test to fail

Starting in Xcode 8, the UIAutomation APIs are no longer available for testing. Any aps that are built using Xcode 8 or targeting iOS 10 will not have the UIAutomation APIs available to them and will fail with a deprecation warning.

The latest versions of Calabash and UITest support testing with both Xcode 7 and Xcode 8. The verison of Xcode used to build the app will determine which native automation API is used to automate the app. The Testing Framework and iOS compatibility for Calabash is determined by the version of Xcode that is used. See the table below:

iOS Version Local with Xcode 7 Local with Xcode 8
iOS 7 UIAutomation n/a
iOS 8 UIAutomation n/a
iOS 9 UIAutomation DeviceAgent
iOS 10 n/a DeviceAgent

For example, a Mac with Xcode 8 installed will be able to test on devices running iOS 10 or iOS 9 using DeviceAgent, but would not be able to test on iOS 8 or below.

For more details on how to transition your test suite away from uia methods, please refer to the Calabash Wiki page on the DeviceAgent.

To use UIAutomation APIs, the app must be built using Xcode 7.

Xamarin Workbook

If it's not already installed, install the Xamarin Workbooks app first. The workbook file should download automatically, but if it doesn't, just click to start the workbook download manually.