Extremely slow to run Xamarin.Android UITests = $$ expensive

AndrewBTAndrewBT GBMember ✭✭
edited May 2017 in Xamarin Test Cloud

Hi there

We are using Xamarin Test Cloud to smoke test a native iOS & Android control library with Xamarin.iOS and Xamarin.Android bindings. Our goal is in UITests to simply run the demo / examples app and to click on each example. We check for runtime crashes as a simple smoke test 'Does it work' before deploying the latest binaries to NuGet. This allows us to quickly rule out runtime errors that result from xamarin bindings.

Anyway, we are finding that Xamarin Test Cloud is extremely slow to run tests. We're not sure if this is our tests, or the infrastructure of Xamarin as yet. We have 25 Xamarin.Android tests so far which we want to run on one device only, and they seem to take half an hour to run. That's 1 minute per test. We want to add a couple of hundred tests and are pretty worried about the escalating costs and how we'll soon be bumped into the 'Enterprise' tier just for a small number of smoke tests for one app on one device ...

Questions:

  • Is there any way to determine why our tests are slow and to improve them?
  • Is there any reason why Xamarin Test Cloud will take a long time to run a test (a minute for one test is rediculous).

Edit

See supplementary information below. One specific test takes

  • 20 seconds to run on an emulator locally
  • 30 seconds to run on a device locally
  • 78 seconds to run on xamarin test cloud: 2.6x longer

Xamarin Test Cloud performance Image

Obviously at this speed it kind of makes XTC unusable as we cannot add more than a small number of tests before we a.) run out of money b.) our build server times out with hanging builds or c.) our developers go insane from waiting :)

Any help appreciated!!

Best regards,
Andrew

Posts

  • AndrewBTAndrewBT GBMember ✭✭

    Oh my gosh, our iOS tests take 1 device minute, but Android 30 device minutes. Both run a similar amount of tests (not very many)...

    There seems to be a bad performance problem with Xamarin Test Cloud and Android. Unfortunately this impacts us as we're already pushed into the $385/month tier for a single app, just because of bad performance on Android :neutral:

  • xmax-axmax-a RSMember ✭✭

    @AndrewBT Did you found a solution for bad performance on Android, or it's still the same?

  • xmax-axmax-a RSMember ✭✭
    edited November 2017

    Duplicate post.

    Please delete this.

Sign In or Register to comment.