nunit command line to run only Android or only iOS tests?

This question has two parts:

The default UITest project setup template in Xamarin Studio sets up a Tests.cs class and an AppInitializer.cs class. The Tests.cs is a test fixture that is parameterized by an enum Platform.iOS and Platform.Android. Say I want to use the nunit command line to run only the iOS instance of the fixture. If I try all possible combination of -fixture or -run options with fully qualified class names and fully qualified Platform parameters, nothing works. Everything resulted in not finding the fixture to run. I decided that perhaps using an enum for the parameter was causing trouble so I changed it to a simple string of "iOS" or "Android". I then tried the -fixture option and it still did not work. I finally tried the -run option and it finally worked! So that means either specifying parameterized fixtures on the nunit command line when the parameter is an enum doesn't work or I could not find the proper option string. Any help would be appreciated.

Now lets say that over time, I create many test fixture .cs files each one having both iOS and Android instances. How do I say to run all test fixtures but still specify the parameter? I don't want to have to list out each test fixture class explicitly.

Best Answers

Answers

  • ShawnCastrianni.5092ShawnCastrianni.5092 US ✭✭✭
    edited January 2016

    @GlennWilson thanks. I guess this was a permutation that I didn't try.

    I am trying to come up with a way for me to control which platforms tests get run for my local CI build since I build each platform separately on their own schedule while still allowing me to submit to Test Cloud later on.

    for example, I can skip the fixture parameter mechanism and just use env variables. I can control which env variables I set when invoking the nunit command line from my CI build. but I am not sure the env variable technique will work on Test Cloud so I am hesitant to go that way.

  • @GlennWilson Thanks. I used TestEnvironment combined with local environment variables and successfully configured my CI build to run simulator tests while still being compatible with Test Cloud runs.

  • NuninzNuninz USMember ✭✭

    Hi @ShawnCastrianni.5092 could you let us know specifically how did you solve this plz?

  • ShawnCastrianni.5092ShawnCastrianni.5092 US ✭✭✭
    edited June 2016
    1. I created an AbstractTextFixture that all my test fixtures extend. It contains a Platform platform property that is set as follows

      public Platform GetPlatform()
      {
          Platform retVal;
          switch (TestEnvironment.Platform) {
          case TestPlatform.TestCloudiOS:
              retVal = Platform.iOS;
              break;
          case TestPlatform.TestCloudAndroid:
              retVal = Platform.Android;
              break;
          default:
              //Check env
              string uitestPlatform = Environment.GetEnvironmentVariable (UITEST_PLATFORM);
              if (!Enum.TryParse<Platform> (uitestPlatform, out retVal))
                  throw new Exception (UITEST_PLATFORM + " env variable is not defined correctly");
              break;
          }
          return(retVal);
      }
      
    2. This detects platform automatically if in TestCloud, otherwise, it expects an environment variable set called UITEST_PLATFORM to determine the platform.

    3. In order to run my tests in my IDE, I have setup two custom run configurations: iOS UITests, Android UITests. These special run configurations simply just define the UITEST_PLATFORM to either iOS or Android.
    4. To run my tests in my IDE, I just right click inside the Unit Test panel or right click the UITest project and do a "Run with..." or "Run test with..." and then choose 1 of my two custom run configurations.
    5. Disadvantage with this is that Xamarin Studio (and maybe Visual Studio) does NOT have a "Debug test with..." right click menu option so I can't debug my tests without having to hardcode my platform in code rather than use the custom run configurations.
    6. This also works on the command line for my automated build since I can set the UITEST_PLATFORM to whatever I need from outside the command line.
  • AlexeyShikovAlexeyShikov UAMember ✭✭

    I faced the same issue and found alternative solution.

    The idea is to change fixture's Ignored property whenever environment variable specified (for example CI=true).

    So that there is no need to hack configurations and/or preprocessor commands.

    Here is dirty code that ignores fixtures for Platform.Android whenever they are run on Bitrise service (which has env property CI=true).

        public class PlatformTestFixtureAttribute: TestFixtureAttribute
        {
            public PlatformTestFixtureAttribute()
            {
            }
    
            public PlatformTestFixtureAttribute(params object[] arguments)
                : base (arguments)
            {
                IgnoreAndroidOnCI (arguments);
            }
    
            private void IgnoreAndroidOnCI(object[] args)
            {
                foreach (var arg in args) 
                {
                    if (arg is Platform) 
                    {
                        if (Platform.Android == (Platform)arg) 
                        {
                            IgnorePlatformOnCI (Platform.Android);
                        }
                    }
                }
            }
    
            protected void IgnorePlatformOnCI(Platform platform)
            {
                // Do not ignore on TestCloud
                if (TestEnvironment.IsTestCloud) {
                    return;
                }
    
                // TestFixture already ignored by developer
                if (Ignore != null || IgnoreReason != null) {
                    return;
                }
    
                var ciValue = Environment.GetEnvironmentVariable ("CI");
    
                bool isCI;
                if (!bool.TryParse(ciValue, out isCI))
                {
                    isCI = false;
                }
    
                // CI=true
                if (isCI)
                {
                    IgnoreReason = $"Running in Continuous Integration mode (usualy build machine)";
                }
            }
        }
    
  • MimiChowMimiChow USUniversity
    edited July 2016

    I also faced this issue and use preprocessor directives to solve this issue.

    All my test fixtures extends from a common base class (AbstractSetup). In that class, I have the following directives.

    #if !RUN_IOS_ONLY
    [TestFixture(Platform.Android)]
    #endif

    #if !RUN_DROID_ONLY
    [TestFixture(Platform.iOS)]
    #endif
    public abstract class AbstractSetup
    {
    ...
    }

    I don't define RUN_IOS_ONLY and RUN_DROID_ONLY in the test code. So from the IDE, I can run both platforms. From the command line, I define the platform when building the test project, as follow with the /p:DefineConstants option

    xbuild /p:DefineConstants=RUN_IOS_ONLY ...

    Then run the tests as follow (no need to use -run option to explicitly pass in the test fixture and platform).
    mono /path-to/NUnit-2.6.4/bin/nunit-console.exe MyApp.UITests.dll

  • m_jerom_jero USMember ✭✭
    edited October 2016

    Following the idea of the solution proposed by Alexey, I made a few changes:

    public PlatformTestFixtureAttribute(params object[] arguments)
                : base(arguments)
            {
                foreach (var arg in arguments)
                {
                    if (arg is Platform)
                    {
                        if (Platform.Android == (Platform)arg)
                        {
                            AddAndroidCategory();
                        }
                    }
                }
            }
    
            protected void AddAndroidCategory()
            {
                var onlyIOSValue = Environment.GetEnvironmentVariable("REMOVE_ANDROID_TEST");
    
                bool isOnlyIOS;
                if (!bool.TryParse(onlyIOSValue, out isOnlyIOS))
                {
                    isOnlyIOS = false;
                }
    
                // REMOVE_ANDROID_TEST=true
                if (isOnlyIOS)
                {
                    Category = "AndroidTest";
                }
            }
    

    Instead of ignoring the Android test (what modifies the results output displaying all the Android test as ignored), I add a categry for them.
    Then when running them:

    nunit-console.exe MyTest.dll -exclude=AndroidTest
    
  • 15mgm1515mgm15 USMember ✭✭✭✭

    @ShawnCastrianni Interesting way to differentiate between platforms, it is been 7 months or so, did you manage to find another solution for this?

  • ShawnCastrianniShawnCastrianni USBeta ✭✭✭

    @15mgm15 No, still using the above.

  • NickMartinNickMartin USMember

    My approach to this problem is similar to @m_jero and @AlexeyShikov:

        public class PlatformTestFixtureAttribute : TestFixtureAttribute
        {
            public PlatformTestFixtureAttribute(Platform platform, params object[] arguments)
                : base(MergeArguments(platform, arguments))
            {
                if (Categories == null)
                {
                    Category = platform.ToString();
                }
                else
                {
                    Categories.Add(platform.ToString());
                }
            }
    
            private static object[] MergeArguments(Platform platform, object[] arguments)
            {
                if (arguments == null)
                {
                    return new object[] { platform };
                }
                var mergedArguments = new object[arguments.Length + 1];
                mergedArguments[0] = platform;
                arguments.CopyTo(mergedArguments, 1);
                return mergedArguments;
            }
        }
    

    Now you can include or exclude by platform via command line - no need to do anything with environment variables.
    Added bonus: In Visual Studio's Test Explorer, you can now group by "Traits" and select to run the environment of your choice! My team loves that capability.

    Also, I think this is a nice interface since it forces the specification of supported platforms on a test fixture.

Sign In or Register to comment.