Categorizing Tests

PDF for offline use

Let us know how you feel about this

Translation Quality


0/250

last updated: 2016-10

Overview

UITests may be grouped into logical categories by adorning them with the CategoryAttribute from the NUnit framework. This provides some flexibility on what tests are to be run.

For example, an application may have one set of tests that are specifically for tablets, and another set that are for phones. This makes it possible to run the tablet specific tests seperate from the phone tests. Another popular scenario is to segregate slow tests from fast tests. The fast tests are run more frequently, perhaps at each commit to source code control. The slow tests are run less frequently, for example at the end of the day.

Requirements

Running tests by category is not supported in either Xamarin Studio or Visual Studio. Running tests according to category can only be performed at the command line.

Categorizing Tests

UITests can be categorised either by text fixture or by test by adding the CategoryAttribute to the class or method. It is possible to assign more than one category. The following class shows an example of categorization:

[TestFixture]
[Category("nerp")]
public class Tests
{
    iOSApp app;

    [SetUp]
    public void BeforeEachTest()
    {
        app = ConfigureApp.iOS.StartApp();
    }

    [Test]
    [Category("derp")]
    [Category("erp")]
    public void CreditCardNumber_TooShort_DisplayErrorMessage()
    {
        app.WaitForElement(c=>c.Class("UINavigationBar").Marked("Simple Credit Card Validator"));
        app.EnterText(c=>c.Class("UITextField"), new string('9', 15));
        app.Tap(c=>c.Marked("Validate Credit Card").Class("UIButton"));
        app.WaitForElement(c => c.Marked("Credit card number is too short.").Class("UILabel"));
    }


    [Test]
    [Category("flerp")]
    public void CreditCardNumber_TooLong_DisplayErrorMessage()
    {
        app.WaitForElement(c=>c.Class("UINavigationBar").Marked("Simple Credit Card Validator"));
        app.EnterText(c=>c.Class("UITextField"), new string('9', 17));
        app.Tap(c=>c.Marked("Validate Credit Card").Class("UIButton"));
        app.WaitForElement(c => c.Marked("Credit card number is too long.").Class("UILabel"));
    }

}
⚠️

Xamarin Test Cloud does not honor the ExplicitAttribute; tests marked as Explicit will still be run.

Running Tests According to Category

There are two ways to run UITests according to category:

  • test-console.exe – this is the command line utility for submitting tests to Xamarin Test Cloud.
  • nunit-console.exe – this is the command line runner for NUnit tests. It is also used to run UITests locally.

Submitting Tests to Test Cloud by Category

You can instruct Test Cloud to run a subset of your tests using the --category parameter.

test-cloud.exe <APK or IPA> <TEAM_API_KEY> --user <EMAIL> --category flerp

It may be specified multiple times to run the union of several test categories, for example:

test-cloud.exe <APK or IPA> <TEAM_API_KEY> --user <EMAIL> --category flerp --category erp

Running Tests Locally by Category

Running tests locally is accomplished using nunit-console.exe, NUnit's command line runner. The command line switch -include identifies the test categories to run, while the switch -exclude specifies the test categories to ignore.

$ nunit-console ./CreditCardValidator.iOS.UITests/bin/Debug/CreditCardValidator.iOS.UITests.dll -include=flerp

Running tests locally is accomplished using nunit-console.exe, NUnit's command line runner. The command line switch /include identifies the test categories to run, while the switch /exclude specifies the test categories to ignore.

nunit-console .\CreditCardValidator.iOS.UITests\bin\Debug\CreditCardValidator.iOS.UITests.dll /include:flerp

Please the see the NUnit console documentation for more information on how to use category expressions to include or exclude tests according to category combinations.

ℹ️

When running tests locally, IApp must be configured with the path to the application and the application bundle.

Summary

In this guide we discussed how to categorize tests and run them by category at either the command line or in Test Cloud.

Xamarin Workbook

If it's not already installed, install the Xamarin Workbooks app first. The workbook file should download automatically, but if it doesn't, just click to start the workbook download manually.