Forum Xamarin Test Cloud (Read Only)

Long running tests from VSTS

I thought I would start a discussion on the topic of continuous integration, we originally had the idea of creating a build job in VSTS in which the app is built and the final step would be to ship it up to test cloud to run.

The issue is we have around 80 tests per platform (Android/iOS) and take around 2 hours to complete on test cloud, the issue is while the test is running the VSTS build is just sat doing nothing waiting for a response and burning through Azure credits.

As our apps grow, our azure bill could start to become very expensive. Which seems pointless as 90% of the time it's just sat waiting for test results to be sent back to it.

The perfect solution would be the build runs, post the tests up to test cloud and then ends. Once the tests are finished they are sent back to the build and updates it with results.

Has anyone thought about this issue and take a different approach?

Any help would be appreciated.

Posts

  • ShantimohanElchuriShantimohanElchuri USMember ✭✭✭✭✭

    Your observation is scaring me. I am just in the process of using Test Cloud from VSTS. If we have to follow 'Cloud first' strategy of Microsoft then they have to make these amends. In this promotion local testing is completely ignored.

    If a post to git can trigger a build automatically, accomplishing what you are asking is not an impossible task.

    Did you check if you are billed even if you logout of VSTS while the tests are running in Test Cloud? At least they should avoid billing for such 'idle time'.

  • brijmitchellbrijmitchell USMember ✭✭

    I would need to check with my administrator to see if we are billed for such time, I believe if we turned the option on we would be.
    From what I have seen so far, it is as the 30 min option is turned on and I can see it has used up credits for the time it was waiting/idle.

    I believe by default is set to 30 mins, if you turn this setting off it will run and use as many minutes/credits as you have allowed it.

    Free: 240 minutes (shared with Build)
    30-minute maximum single job duration

    Or

    $40 / Agent
    360 minute maximum single job duration

    For now, I have decided to take a smoke approach and reduce the number of tests we are running each night, I have tagged a number of tests with the word smoke and these will run each night. When we do a release the full suite will be kicked off manually.

    it's not perfect, but if we really are being billed for the Azure host to just sit and wait for results then its needs must, unfortunately.

Sign In or Register to comment.