iOS Unit Tests do not run

I am trying to use Xamarin UITest locally on an iOS app, running an iOS Simulator, but I cannot get any tests to run on the iOS Simulator.

When I run the tests, they do fail (even my "Assert.Pass()" test)

What is meant by the "important properies" message? When I tried to add ".PreferIdeSettings()" to the ConfigureApp chain, it doesn't compile.

Also, on the first TODO, there is mention of "right click Test Apps, select Add App Project", but I cannot find this in my Xamarin Studio Unit Tests windows (which only lets me run, stop and view test results)

    [SetUp]
    public void BeforeEachTest ()
    {
        // 
        // TODO: If the iOS app being tested is included in the solution then open
        // the Unit Tests window, right click Test Apps, select Add App Project
        // and select the app projects that should be tested.
        app = ConfigureApp
            .iOS
        // TODO: Update this path to point to your iOS app and uncomment the
        // code if the app is not included in the solution.
        //  .AppBundle ("../../../iOS/bin/iPhoneSimulator/Debug/UITester.iOS.app")
            .AppBundle ("../../../Touch.Container/bin/iPhoneSimulator/Debug/MyContainer.exe")
            .DeviceIdentifier("<deviceID>")
            .EnableLocalScreenshots()
            .StartApp ();           
        app.Repl ();
           }

01-06-2015 09:16:21.724 -05:00 - 4 - iOS test running Xamarin.UITest version: 0.7.1
01-06-2015 09:16:21.792 -05:00 - 72 - Skipping IDE integration as important properies are configured. To force IDE integration, add .PreferIdeSettings() to ConfigureApp.

Tagged:

Posts

  • mattwardmattward Xamurai GBMember Xamurai

    Not sure why you cannot see Test Apps in the Unit Tests window. Xamarin Studio will look at the referenced assemblies and show the Test Apps item if it finds Xamarin.UITest is referenced. You should be able to workaround this by adding a normal reference from the UITest project to your main iOS project. Then you will need to edit the .csproj file and set the property on that reference to false.

    <ProjectReference Include="..\MyiOS\MyIOS.csproj">
        <Project>{GUID...}</Project>
        <Name>MyIOS</Name>
        <ReferenceOutputAssembly>False</ReferenceOutputAssembly>
        <Private>False</Private>
    </ProjectReference>
    

    The Skipping IDE integration as important properies are configured message is just saying that the UITest startup code in your BeforeEachTest method is overriding Xamarin Studio's settings. If you create a new iOS solution you will see a UITest project already configured to use the main iOS project. In this UITest the startup code will be something similar to:

    app = ConfigureApp.iOS.StartApp ();
    

    With the above code Xamarin Studio will use the currently selected iOS simulator when running the UITests. You should also be able to do the same thing with the PreferIdeSettings () method.

  • SteveRawlinsSteveRawlins ✭✭ USMember ✭✭

    Thank you, Matt. Unfortunately, when I try to add a reference to my iOS app, the project is "greyed out" with:

    (Incompatible target framework: Xamarin.iOS, Version=1.0)

  • mattwardmattward Xamurai GBMember Xamurai

    Ah, yes. The UITest project is targeting .NET framework whilst the iOS project does not so you cannot do that from within Xamarin Studio. So if you cannot do that from the Unit Tests window then you would have to manually edit the .csproj file which is a bit of a pain.

    Not sure why you are not seeing a Test Apps tree item in the Unit Tests window. With Xamarin Studio 5.9.1 installed and an iOS project open that does not have a UITest project I can add a new UITest project to the solution. Then the Unit Tests window will show a Test Apps item once all the tree nodes are expanded. Before select the Add App Project I would add the Xamarin.TestCloud.Agent NuGet package to the original iOS project, then you should be able to add a reference to it using the Add App Project.

    With a new UITest project created the Test Apps item should be available in the Unit Tests window. Xamarin Studio will show the Test Apps item if the project has a reference to the Xamarin.UITest assembly, which it should by default if you have created a new UITest project from one of the project templates.

  • SteveRawlinsSteveRawlins ✭✭ USMember ✭✭
    edited June 2015

    Thank you so much for your analysis and guidance. I started over with a different iOS app.

    I am now seeing the "Test Apps" item in the Unit Tests Window (In Xamarin Studio 5.9.2, build 2)

    Furthermore, following your instructions to add the Xamarin Test Cloud Agent to the iOS app, I was able to "Add App Project" (prior to putting the Test Cloud Agent on my app "container" project, the iOS project was not available as a checkbox, so I did, indeed, need that Test Cloud Agent in the target app)

    Now, when I run tests my Mac prompts for admin pw ("Instruments" wants access), which I granted. Do you get that everytime?

    I seem to be making progress (I am new to Xamarin UITesting and trying to get some iOS tests with Tap, Wait, EnterText etc. to help me learn)

    My goal for this stage was to keep it local (not use the TestCloud) and also to use C# and not Calabash/Ruby.

    When I run my tests now I get:

    SetUp : System.Exception : Unable to contact test backend running in app. A common cause is that the app is not properly linked with Calabash. Please verify that it includes the Calabash component.

    Does this mean it's trying to run on TestCloud?

  • mattwardmattward Xamurai GBMember Xamurai

    I do not get a prompt from Instruments asking for my admin credentials on my Mac.

    The unable to test backend running in app message is probably because the iOS app was deployed to the simulator at some point in the past when Xamarin.TestCloud.Agent was not being referenced by. There is a limitation with the version of Xamarin.UITest that is installed out of the box with Xamarin Studio 5.9 where it does not replace the iOS app in the simulator. To fix this you can either reset the simulator, one of the menu options from the iOS simulator is Reset Content and Settings, or alternatively you can update your NuGet packages for Xamarin.UITest and TestCloud.Agent. With Xamarin.UITest 0.8.0 the application should be re-deployed each time to the simulator so you do not need to reset the simulator.

    You will only be using Test Cloud if you have selected Run in Test Cloud and that will try to upload the app and then open a browser on Test Cloud for the final configuration.

  • SteveRawlinsSteveRawlins ✭✭ USMember ✭✭

    I tried both suggestions (re-set my iOS Simulator/update the Xamarin.UITest package to 0.8.0 which is where it currently sits). Still getting these errors when I run my tests:

    SetUp : System.Exception : Unable to contact test backend running in app. A common cause is that the app is not properly linked with Calabash. Please verify that it includes the Calabash component.

    and

    SetUp : System.Exception : The app bundle in /Users/steverawlins/Source/TRUNK/Samples/ViewChoices/Touch.Container/bin/iPhoneSimulator/Debug/TouchContainer.app does not seem to be properly linked with Calabash. Please verify that it includes the Calabash component.

    Must I install Calabash as a prerequisite? I'm installing the Calabash gems now, per Xamarin/Calabash

  • mattwardmattward Xamurai GBMember Xamurai

    No you do not need to install Calabash separately. If the main iOS app project is referencing the Xamarin.TestCloud.Agent NuGet package then that will have provide the needed Calabash.dll.

  • SteveRawlinsSteveRawlins ✭✭ USMember ✭✭

    I added Xamarin.Calabash.Start(); to my iOS App's AppDelegate FinishedLaunching

    And now the repl launches.

  • mattwardmattward Xamurai GBMember Xamurai

    Oh, yes, sorry I forgot about that. The project templates in Xamarin Studio automatically add that startup code. They add it with an #ifdef so that it is only included in Debug builds. Apple will reject the app if it uses Calabash so normally it is disabled for release builds.

  • SteveRawlinsSteveRawlins ✭✭ USMember ✭✭

    Thank you, Matt. My UITests are running now, locally on Simulator.

    Because the target-app (the project I'm testing) was not created by a template, it didn't have the test hook. I understand now that the main prog needs to call Calabash (and that it can be conditionally compiled).

    Urk... I think I even read that somewhere in Xamarin doc.

    Anyway, THANK YOU for your help. My UI unit-tests are now executing using Tap, Text, Clear, WaitFor, Flash, Back() etc. and that's exciting because it's taken me some effort to get this far.

  • thedigitalseanthedigitalsean ✭✭ USUniversity ✭✭

    Just commenting that this is still a bug. I am using Xamarin Studio 5.9.7 (build 22) and my unit test project is Xamarin UITest 1.1.1 and NUnit Version 2.6.4. iOS app and unit test projects created with templates. I do not see "Test Apps" in my Unit Test window and cannot add references to by iOS project from unit test project.

  • mattwardmattward Xamurai GBMember Xamurai
    edited October 2015

    @thedigitalsean - My only guess is that the project was created in Visual Studio so the reference to Xamarin.UITest. You might be hitting a bug in Xamarin Studio 5.9.7 where the Test Apps folder is not displayed due to the reference being created in Visual Studio. The Test Apps folder is displayed if the project has a reference to Xamarin.UITest.

    If you have a reference that starts with "Xamarin.UITest, Version=..." then Xamarin Studio does not detect Xamarin.UITest and does not display the Test Apps folder. An example is shown below:

    <Reference Include="Xamarin.UITest, Version=1.1.1.0, Culture=neutral, PublicKeyToken=null">
      <HintPath>packages\Xamarin.UITest.1.1.1\lib\Xamarin.UITest.dll</HintPath>
    </Reference>
    

    The workaround is to modify the Include attribute so it looks like:

    <Reference Include="Xamarin.UITest">
      <HintPath>packages\Xamarin.UITest.1.1.1\lib\Xamarin.UITest.dll</HintPath>
    </Reference>
    

    This bug is fixed in Xamarin Studio 5.10.

  • thedigitalseanthedigitalsean ✭✭ USUniversity ✭✭

    Thanks for the help. However, I develop exclusively on a Mac so this cannot be Visual Studio related. My reference to UITest is already in the format you suggest:

    ....\packages\Xamarin.UITest.1.1.1\lib\Xamarin.UITest.dll

    The project was originally created in July/August with an older version of Xamarin Studio. I deleted the UITest project due to this bug and am now trying to re-add it. The manual adding of the reference that you suggested at the beginning of this post is the only thing that has worked. I still don't get the "Test Apps" in the Unit Test window.

  • mattwardmattward Xamurai GBMember Xamurai

    Not sure what problem you are hitting. Xamarin Studio checks the project to see if it references Xamarin.UITest and then will display Test Apps in the Unit Test window.

    If you create a new iOS project does it display?
    Does removing and then re-adding the Xamarin.UITest NuGet package cause it to be displayed?
    Any exceptions in the IDE log (Help - Open Log Directory)?

  • thedigitalseanthedigitalsean ✭✭ USUniversity ✭✭
    edited October 2015
    • Created a new iOS project. Then added new UITest project to that. Still nothing in my Unit Tests Pad and cannot add reference to the iOS project from the UITest project.
    • Removing and re-adding Xamarin.UITest has no effect
    • I don't see anything that looks related to this in my logs. But let me know the best way to send the log files.
  • thedigitalseanthedigitalsean ✭✭ USUniversity ✭✭
    edited October 2015

    Here is some additional data. Not sure if it helps. When I add the reference manually in the csproj file with

    <ProjectReference Include="path to csproj"> <Project> {GUID} </Project> <ReferenceOutputAssembly>False</ReferenceOutputAssembly> <Private>False</Private> </ProjectReference>

    The References show up in the References directory but it is unable to resolve any namespaces defined in the project. So my project does not compile.

    If I remove the ReferenceOutputAssembly line, then a Red X appears beside the project name in the References directory. However, the namespaces will resolve and the project will compile. I got excited for a second but then all my tests fail with:

    System.IO.FileNotFoundException : Could not load file or assembly 'Xamarin.iOS, Version=0.0.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065' or one of its dependencies.
    
  • mattwardmattward Xamurai GBMember Xamurai

    Can you open a bug on bugzilla and attach a simple example project that you can reproduce the problem, the IDE logs and the information from Xamarin Studio - About Xamarin Studio - Show Full Details.

  • thedigitalseanthedigitalsean ✭✭ USUniversity ✭✭

    Since we have a enterprise license, the info has been sent along with 2 sample projects to Xamarin support.

  • NicholasTurnerNicholasTurner ✭✭ USMember, University ✭✭

    @thedigitalsean I do know from experience that the call to repl() screws everything up. It causes a big delay as I believe this to be a blocking call and my test stays in the virtual void. You might try putting the repl calling into a test

  • thedigitalseanthedigitalsean ✭✭ USUniversity ✭✭
    edited October 2015

    I appreciate the help @mattward and @NicholasTurner . After further investigation it turns out I was the issue. The Test Apps menu was in fact showing up. I have also learned that the run time problems I had was because I was trying to use Xamarin.UITest to run traditional NUnit unit tests in addition to the UI integration tests. To that end, I was trying to instantiate classes defined in my iOS project and unit test them. Unfortunately even though Xamarin.UITest runs in NUnit, you cannot do that.

    My reason for trying this is because it is more than a little disappointing that there is no built-in way to run iOS/Android NUnit tests with Xamarin that can be automated and integrates with Xamarin Studio. The current NUnit project created with Xamarin Studio template requires the developer to manually click through a UI in the simulator for testing and debug and the results are not reported back in xml or integrated into Xamarin Studio.

    Since I work in a regulated industry, I do not have the option of just running UI integration tests and skipping unit testing. So I was able to dig up some old blogs and an open source project called Touch.Unit (and, more specifically, Touch.Server) and put something together that at least allows me to run the unit tests from the command line and get my results back in xml. I'll leave the links here for anyone that may come across this post in the future:

    This post gave the big picture and was especially helpful to me since we use Atlassian products:
    http://lipsky.me/blog/2014/1/29/r1x6nswctliiwqngg0zu63u2713cnp

    I had to fork Touch.Unit because there is an issue where with iOS 9 I needed to specify the sdk version. I have a pull request to get it integrated but here is the base project:
    https://github.com/spouliot/Touch.Unit
    my modifications:
    https://github.com/spouliot/Touch.Unit/pull/27

    Next, I added these lines in the AppDelegate of my test project.

    runner = new TouchRunner (window);
    
    // register every tests included in the main application/assembly
    runner.Add (System.Reflection.Assembly.GetExecutingAssembly ());
    runner.AutoStart = true;
    try{
        runner.Writer = new NUnitOutputTextWriter (
            runner,
            new MonoTouch.NUnit.TcpTextWriter ("localhost", 16384),  //Make sure this port matches what specify when running Touch.Server
            new NUnitLite.Runner.NUnit2XmlOutputWriter(DateTime.Now));
    
        runner.TerminateAfterExecution = true;
    } catch (Exception) {
        //Do nothing.  This should only occur when running locally.
        Console.WriteLine("Cannot set NUNit Runner Writer");
    }
    

    Then I compile and run TouchServer with:
    mono --debug /path/to/Touch.Unit/Touch.Server/bin/Debug/Touch.Server.exe -launchsim=path/to/ios/unit/test/project/bin/iPhoneSimulator/Debug/your-project.app -autoexit -logfile=TestResults.xml -sdk=9.0 -port=16384

    Other helpful links:
    https://spouliot.wordpress.com/2011/10/13/touch-unit-automation/
    https://spouliot.wordpress.com/2011/10/08/touch-unit-network-logger/
    https://spouliot.wordpress.com/2012/04/02/touch-unit-automation-revisited/

    After removing some of the unneeded header text, I can integrate the TestResults.xml in my Continous Integration system and we're happy. Now, my only wish was that Xamarin could set this up in Xamarin Studio and these results could be integrated more into the development environment when ran locally. Or at least Xamarin could provide a general guide on how to do.

Sign In or Register to comment.