UITest, Calabash, Test Recorder and Xamarin Test Cloud FAQ

Glenn.WilsonGlenn.Wilson USXamarin Team Xamurai
edited July 2016 in Xamarin Test Cloud

Documentation

The main entry point for the Xamarin Test Cloud Documentation is at:
https://developer.xamarin.com/testcloud/

Here are a few common issues to consider when things don't go as expected.

UITest

Why did my tests fail?

Old NuGet packages

We’re always working to make UITest better, adding features, fixing bugs and adding support for new versions of iOS, XCode and Android and the operating systems our tools run on.

If something doesn’t work or seems wrong the first thing to check is that you are using the latest, stable Xamarin.UITest NuGet package.

For an iOS app you should also ensure that you have the latest, stable Xamarin.TestCloud.Agent NuGet package if you are creating it with Xamarin. If you are building your iOS app in XCode then make sure you have the latest, stable Calabash framework.

Because the UITest NugGet packages are updated more frequently than the IDEs, the first thing to do when creating a new UITest project is to update the Xamarin.UITest NuGet package. Do not update the NUnit NuGet package beyond 2.6.4 – version 3.x is not currently supported. If you accidentally update it to 3.x just delete it and install NUnit 2.6.4. See: https://kb.xamarin.com/customer/en/portal/articles/2378153-how-do-i-downgrade-a-nuget-package-

Why do My Tests work locally but fail in the Xamarin Test Cloud?

Here are some common reasons why your UITest might work locally but fail in the test cloud.

Missing Test Data

Are your UITests using local data? Are you using an app.config for data for your UITest? They will not be automatically available in the Xamarin Test Cloud. See:
https://forums.xamarin.com/discussion/comment/189113/
and
https://forums.xamarin.com/discussion/64796/test-cloud-upload-app-config

Renamed screenshot filenames

The app.Screenshot(“Title”) method returns a FileInfo and so you can access, rename or move these files. That’s ok if you are running locally. But, if you do that in the Xamarin Test Cloud that prevents the Test Reports from being generated. If you must rename them locally you can protect that with

 if (!TestEnvironment.IsTestCloud)

to make sure you don’t rename screen shot files in the cloud.

Inappropriate DLLs uploaded with the test

Isn’t it cool that you can create cross-platform code that runs on Macs, on Windows, on iOS and Android? These different platforms often have DLL files that are named the same, do the same basic things and enable this cross-platform magic. But, don’t try to run native Android code natively on Mac, or iOS code on Windows. When you upload your test suite to the test cloud it should only include the DLLs necessary to run your tests and not the DLLs used in your app. The wrong platform DLLs in your test suite can mask the right method calls and lead to bizarre behavior.

Calabash

Why did my tests fail?

Out of date gems

We’re always working to make Calabash better, adding features, fixing bugs and adding support for new versions of iOS, XCode and Android and the operating systems our tools run on.

If something doesn’t work or seems wrong the first thing to check is that you are using the latest, stable gems.

Why do My Tests work locally but fail in the Xamarin Test Cloud?

Missing Test Data

Are your Calabash tests using local data files? Are they somewhere under the features folder? If not, they are not being uploaded as part of your Xamarin Test Cloud submission. Move your data files used in your test to a folder under the features folder, and make corresponding changes to your test code to find them at that location.

Xamarin Test Recorder

Where can I find the API Key?

View API Key on Xamarin Test Cloud

To use the Test Recorder you need a Xamarin Test Cloud API Key. To view your API Key visit your account settings on the Xamarin Test Cloud. Select Teams & Apps in the left sidebar, then select the team to use, then click on show API key.

Xamarin Test Cloud

Why did my tests fail?

View the Device and Test Logs

From a Test Run Overview, click on one of the test steps. Then click on one of the device images. On the right of the image are links for the Device Log and Test Log for the entire run for that device. Above the device image are links for the Device Log, and possibly Test Failures and Stack Traces for the current step for that device.

This discussion has been closed.