Submitting the APK and Test

PDF for offline use

Let us know how you feel about this


0/250

last updated: 2016-10

Submitting the Tests

Tests are submitted at the command line using the test-cloud application that was installed by the Calabash installer script. You will need following four pieces of information:

  1. The path the APK that was created in the previous section.
  2. The Android device ID when the test run was created.
  3. The API key for the team.
  4. The e-mail address of a user who is a part of the Test Cloud team.

The Test Run wizard will provide you with the command line that you can use to submit your tests. For the sake of reference this following snippet is the simplest command line that you can use to submit your tests:

$ test-cloud submit <PATH TO IPA> <XTC TEAM API KEY> \
    --devices  <IOS DEVICE ID> \
    --user <XTC USER ID>

test-cloud will display progress to the console while it is running. You can see a sample of this output below:

### Checking for Gemfile ###
2016-09-27 09:36:58 -0600 
2016-09-27 09:36:58 -0600 Gemfile missing.
2016-09-27 09:36:58 -0600 You must provide a Gemfile in your workspace.
2016-09-27 09:36:58 -0600 A Gemfile must describe your dependencies and their versions
2016-09-27 09:36:58 -0600 See: http://bundler.io/
2016-09-27 09:36:58 -0600 
2016-09-27 09:36:58 -0600 Warning proceeding with default Gemfile.
2016-09-27 09:36:58 -0600 It is strongly recommended that you create a custom Gemfile.
2016-09-27 09:36:58 -0600 Proceeding with Gemfile: /var/folders/61/bhbtr4_51tx9bl3l5lqzl87m0000gn/T/d20160927-50438-e6ibrx/Gemfile
source 'http://rubygems.org'
gem 'calabash-android', '0.7.3'
2016-09-27 09:36:58 -0600 

### Packaging ###
Fetching gem metadata from http://rubygems.org/
Fetching version metadata from http://rubygems.org/
Resolving dependencies...
Using awesome_print 1.7.0
Using builder 3.2.2
Using gherkin 4.0.0
Using cucumber-wire 0.0.1
Using diff-lcs 1.2.5
Using multi_json 1.12.1
Using multi_test 0.1.2
Using escape 0.0.4
Using httpclient 2.8.2.4
Using json 1.8.3
Using retriable 1.4.1
Using thor 0.19.1
Using rubyzip 1.2.0
Using bundler 1.12.5
Using cucumber-core 1.5.0
Using luffa 2.0.0
Using cucumber 2.4.0
Using slowhandcuke 0.0.3
Using calabash-android 0.7.3
Bundle complete! 1 Gemfile dependency, 19 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Updating files in vendor/cache
  * awesome_print-1.7.0.gem
  * builder-3.2.2.gem
  * gherkin-4.0.0.gem
  * cucumber-core-1.5.0.gem
  * cucumber-wire-0.0.1.gem
  * diff-lcs-1.2.5.gem
  * multi_json-1.12.1.gem
  * multi_test-0.1.2.gem
  * cucumber-2.4.0.gem
  * escape-0.0.4.gem
  * httpclient-2.8.2.4.gem
  * json-1.8.3.gem
  * retriable-1.4.1.gem
  * thor-0.19.1.gem
  * luffa-2.0.0.gem
  * rubyzip-1.2.0.gem
  * slowhandcuke-0.0.3.gem
  * calabash-android-0.7.3.gem

### Verifying dependencies ###

### Calculating digests ###

### Negotiating upload ###

### Gathering files based on negotiation ###

### Uploading negotiated files ###

### Test enqueued ###
User: tom.opgenorth@xamarin.com
Team: Simple Credit Card Validator

Running on Devices:
HTC Desire 510 (4.4.3)
LG G3 s (4.4.2)
LG Nexus 5 (6.0.1)

2016-09-27 09:37:14 -0600 Validating
2016-09-27 09:37:25 -0600 Validating
2016-09-27 09:37:35 -0600 Validating
2016-09-27 09:37:46 -0600 Validation completed. Waiting for devices
2016-09-27 09:37:57 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:38:08 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:38:18 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:38:29 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:38:40 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:38:50 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:39:01 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:39:12 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:39:23 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:39:33 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:39:44 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:39:55 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:40:05 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:40:18 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:40:29 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:40:40 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:40:50 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:41:01 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:41:12 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:41:23 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:41:33 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:41:44 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:41:55 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:42:06 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:42:16 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:42:29 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:42:40 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:42:51 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:43:01 -0600 Running on 3 devices (0 / 3 completed, 0 pending) 
2016-09-27 09:43:12 -0600 Running on 1 device (2 / 3 completed, 0 pending) - Report 0% done
2016-09-27 09:43:23 -0600 Tests completed. Processing data.
2016-09-27 09:43:34 -0600 Tests completed. Processing data.
2016-09-27 09:43:44 -0600 Done!
2016-09-27 09:43:44 -0600 Total scenarios: 1
2016-09-27 09:43:44 -0600 1 passed
2016-09-27 09:43:44 -0600 0 failed
2016-09-27 09:43:44 -0600 Total steps: 4
2016-09-27 09:43:44 -0600 
2016-09-27 09:43:44 -0600 Test Report:
2016-09-27 09:43:44 -0600 < URL FOR VIEWING TEST RESULTS ON XAMARIN TEST CLOUD />

When the test run is finished, you will be able to check the test results in one of two ways:

  • test-cloud will provide you with a URL to view the results of the test run. Copy and paste that URL into to your browser to see the test results.
  • Xamarin Test Cloud will send you an e-mail with a summary of the test run and a link to to view the test result details.

You might see an error/warning about requiring a Gemfile. For the purposes of this quick start, it is safe to ignore the error and warning, as a Gemfile is created for the test run. It is strongly recommended that a Gemfile be used for production level testing.

Firewall Configuration

In order for tests to be submitted to Xamarin Test Cloud, the computer submitting the tests must be able to communicate with the Test Cloud servers. Firewalls must be configured to allow network traffic to and from the servers located at testcloud.xamarin.com on ports 80 and 443. This endpoint is managed by DNS and the IP address is subject to change.

In some situations, a test (or a device running the test) must communicate with web servers protected by a firewall. In this scenario the firewall must be configured to allow traffic from the following IP addresses:

  • 195.249.159.238
  • 195.249.159.239

Xamarin Workbook

If it's not already installed, install the Xamarin Workbooks app first. The workbook file should download automatically, but if it doesn't, just click to start the workbook download manually.