Why does a test run in Test Cloud take a really long time for Android?

Recently, we noticed some of our UITests were timing out because a test was taking more than 30 minutes to complete.
After researching more into it, we realized that running tests in Test Cloud for Android takes significantly longer than when running locally.

For starters, we thought maybe screenshots were slowing down the test time, and so we did the following test runs to get the times for each. We ran the same suite of tests for each of the times/test runs below:

ANDROID (Nexus 6P, Nougat 7.1.1)

Locally -- no screenshots
Emulator —> 4:35
Physical Device —> 5:31

Locally -- screenshots enabled
Emulator —> 7:04
Physical Device —> 7:38

Test Cloud -- screenshots always enabled
Device —> 44:00

Does anyone know why tests run in Test Cloud for Android are ~6x slower than running locally, even on a physical device?
Thanks!

Answers

  • AndrewBTAndrewBT GBMember ✭✭

    We have the same problem on Android. A simple set of 25 tests takes half an hour to run! See https://forums.xamarin.com/discussion/96233/extremely-slow-to-run-uitests-expensive/p1?new=1

  • AndrewBTAndrewBT GBMember ✭✭

    Hmm whaddya know. My forum post above got deleted.

    Anyway here is the info again.

    • One test on Xamarin Android takes 78 seconds on Xamarin Test cloud.
    • On local device it takes 30 seconds
    • On local emulator it takes 20 seconds.

    Basically from emulator -> xamarin test cloud we are looking at a performance difference of 3.9x. A simple set of 25 tests takes 30 minutes to run, which means we cannot scale our testing without incurring significant time & money cost. Even waiting an hour for a build is pretty unproductive ...

    Any help much appreciated.

  • AndrewBTAndrewBT GBMember ✭✭

    Update:

    We have 'solved' this by workaround by running all our tests in one test.

    When we run 25 tests in one single UITest the whole test-run takes 3 minutes 25s on local device. We are going to check the test run time on Xamarin.

    That would be an acceptable workaround for us, but for others maybe not

    @XamarinTestCloud team: Maybe there is a huge static overhead for loading test APKs in Android?

  • boazlachmanboazlachman USMember

    Hi,
    I have also encountered the same issue, but unfortunately putting all the tests into one is not feasible in my situation.

    Could someone please help or at least explain why a step which takes 3 seconds on emulator and local device takes 24 seconds in the test cloud(!)

    Any help would be greatly appreciated

  • CanBilginCanBilgin USMember

    same issue here... it looks from the log that screenshots are taking major testing time:

    18-08-2017 16:57:36.084 +02:00 - 24690 - Took screenshot. { Counter: 1, Title: "App Launches", FileName: "/Volumes/Data/xamarin/workspaces/b02a086c-46c4-48b4-a819-75432d457e5c/workspace/screenshot-1-bcc2563a9ed6433686331ba33cba495d.png", ElapsedMilliseconds: 6816 }

    Considering that each test has 5-6 steps on average. Only screenshots took about half a minute for each test.

    For us running 24 tests with multiple steps takes around 40 minutes and half of this is gone to screenshots.

  • xmax-axmax-a RSMember ✭✭

    Way too slow for a couple of simple commands even for me within 2 tests. But I am in a trial version, and so far this not looks like a good option to buy.

Sign In or Register to comment.