Regression Bugs and their impact on development - An open discussion/review

ChaseFlorellChaseFlorell CAInsider, University mod

Hey Xamarin Team.

First off, I really want to thank you for the great work in both the Xamarin eco-system and in Xamarin.Forms in general. The mountain you chose to climb is a big one, and not an easy one. The concept for X.Forms is one that many of us want to get behind full-on, and with that I'd like to give you some constructive feedback. Please take it as that, and not an attack (sometimes my words come across in a way that I don't intend).

In our small team, when we saw V1 of Xamarin.Forms released, we checked it out, ran some tests, and decided that it is absolutely the direction that we wanted to go. Seeing a "V1" to us indicated a "Stable Release", not Beta, and so we implemented it 100% in our current project.

Since the initial launch, there have been 3 substantial releases (that I've paid attention to)...

  1. the initial release that came in the Xamarin update and exists in the Xamarin templates
  2. the failed (unpublished) release last friday
  3. the latest 1.2.1.6229

Of these three releases, I feel that we've been seeing more regressions than progression in the library. As a subscriber paying $1k / platform / developer annually, honestly I expect more. I don't want to pay to be your beta tester.

Here's a point form list of expectations that I have (and I'm sure others do as well).

  • Open and transparent communication (you're actually doing pretty good at this).
  • Fast turn-around hotfixes on regression bugs that impact our ability to deliver a product to our clients.
  • A consistent release cycle in line with realistic developer expectations (we know you can't deliver the world, and especially overnight).
  • Open and published bug tracker (similar to that at Jetbrains).
    • And acknowledgement that a bug has been received. I have several bug reports where I don't even know if they've been looked at.
  • Bug fixes must come first, and similar to the second point, published hotfix every time a bug gets squished.

As it stands right now there are two "deal breaker" bugs that are preventing me from delivering a product to a client. This therefore delays me getting paid... I like getting paid :)

  1. NullReferenceException on SetActionBarTextColor
  2. Can't add multiple items to Grouped ListView

Anyhow, I just wanted to open a discussion on this, get your feedback, and hopefully spark a little change... we'll see how it goes.

Again, you're all doing great things, and with a little refining, there's no reason Xamarin can't stay on top of the cross platform mobile wars :-D

Posts

  • LauraGrossoLauraGrosso ITMember, University
    edited July 2014

    +1. Blank app doesn't compile anymore for issue number 1.
    We take decisions every day about the platform to go to, for our projects, present and future.
    In the next days, we will start a cross-platform project.
    I think we will stay with cross-platform MVVM but NOT forms because we also see bugs are not being acknowledged in bugzilla anymore.

    And I have another question: maybe the support of Forms is going to degrade the support of the "classic" platforms which had worked so well so far?

  • BMKyleBMKyle USMember

    +1 - Bugs come first.

  • What - in my humble opinion - is now missing is a dynamic (managed) list of known "high-priority" bugs with workarounds and expected timeframe for solution.

    This is very important for newer technologies. It can make the difference between using them or not.

    The bug mentioned by ChaseFlorell (http://forums.xamarin.com/discussion/20481/v-1-2-1-nullreferenceexception-at-platform-android-platform-setactionbartextcolor) needs particular attention at the moment.

  • nodoidnodoid GBMember, Beta ✭✭✭

    The bug list is already there - it's Bugzilla. As to the others, my suggestion would be to separate out the releases as the main branch are with alpha getting it first -> beta -> release, that way those who are reliant on the release in stable only get it once the testers have but it through the paces and show stoppers fixed and it avoids the issue that happened last week with the aborted release of 1.2.0

    Xam.Forms is an amazingly complex technology. Think of it this way, the Xamarin team have developed a layer that achieves an almost impossible job; a single framework that all of the major mobile platforms can work with and on. If you think of even something as simple as an ImageView and how it varies between the three platforms, you can see the job it has to do - add onto that the ability to have a different UX between the platforms based on a condition within a constructor and then multiply that up for every piece of UI it covers and then you may get an idea of the complexity.

    Just my 2c worth....

  • LauraGrossoLauraGrosso ITMember, University

    @nodoid‌, we reason why we want to choose Xamarin Forms is very well represented by your words.
    Nonetheless, important bugzilla entries for .forms are being neglected. This had never happened with "xamarin.ios" or ".android".
    This scares us, slows us down, makes us think we may not want to follow the xamarin.forms solution that has all of the advantages you mention. As of now, they are more theoretical than tangible.

  • EricGroverEricGrover USMember ✭✭

    I agree that Xamarin Forms is an amazing technology, and I love the direction that Xamarin is going. I too am frustrated by the seeming lack of testing that goes into production releases. I just installed 1.2.1 and it broke my xaml layouts. In a previous Xamarin Forms release, toolbar buttons on the start page was broken on Android. I have run into numerous bugs that I think should have been discovered during reasonable testing.

    We are all developers and realize how difficult writing quality software is, and do not expect defect free releases. But I don't think its unreasonable to ask for a more thorough and rigorous set of regression tests be applied before releasing a "stable" release of the framework.

    Again, I love the fact that I can write MVVM, C# mobile apps on iOS and Android! I think the work that has been done so far has been fantastic. Just wanting to provide feedback so that these pain points can be avoided in the future.

  • ChaseFlorellChaseFlorell CAInsider, University mod

    @nodoid‌ I can appreciate everything you said, and no one is disputing the fact that it's a tough row to hoe... in fact I think I began the discussion saying just that. But the remaining points are still very much valid.

  • LauraGrossoLauraGrosso ITMember, University

    Thanks for this Jason.

    We are with you in this giant effort.

    May I just point out that at the moment the Blank Application crashes if you update the nuget packages to 1.2.1.
    This made me very embarassed when I wanted to show one of my colleagues that another bug (Editor text) had been corrected.
    I suggest the boilerplate apps are part of the testing apps next time around :)

    Keep up the good work, we're sure you'll do it.

  • andypattandypatt GBMember

    Out of curiosity do you you use the Xamarin Test Cloud for regression testing the Xamarin Forms Releases? Test cloud sounds like it would be ideal for helping catch the differences in platform targets (such as the Actionbar bug on Android 4.4).

  • GeoffArmstrongGeoffArmstrong CAMember ✭✭

    @JasonASmith‌,

    Regarding your comment about bug reports being marked private, I just looked at the bugs that I submitted on Bugzilla and they're marked private, but I certainly didn't intend them to be. I recall not checking the "private" box, and I made a sample project specifically so it could be a "public bug".

    Interestingly, when I go visit the bugs now (e.g. 21177), it's marked as private (there is a check box next to "Xamarin Team (Private Bug)") and I'm unable to uncheck that box since the control is disabled.

    So, if most bugs are private it may be due to a UI issue or some sort of bug in the tracking system itself. :)

  • TheRealJasonSmithTheRealJasonSmith USXamarin Team Xamurai

    We do in fact use test cloud! As to the blank app, yeah it (obviously) should have been part of the regression suite. We realize now we never had one without a navigation page in it... DOH.

  • ChaseFlorellChaseFlorell CAInsider, University mod

    I was experiencing the exact same behavior as @GeoffArmstrong‌ with regards to the "Private" bug reports. I see locks beside all of my reports, and I specifically remember intentionally not having that box checked, yet it was still marked as private. I figured it was a setting in Bugzilla that you guys were using.

  • ChaseFlorellChaseFlorell CAInsider, University mod

    @JasonASmith‌, this sounds like a simple case of growing pains. Thanks so much for being open and honest about these bits, we really appreciate it.

  • RiccardoMoschettiItalyRiccardoMoschettiItaly USMember
    edited July 2014

    +1 for private bugs not intentionally private

    I just submitted a bug (https://bugzilla.xamarin.com/show_bug.cgi?id=21396)
    I did not mark it as "private"
    Now I see it private in "my bugs". The bug's URL, in fact, cannot be accessed without authentication.

    PS: and I cannot see bug no. 21177 by @GeoffArmstrong‌, so I wouldn't have noticed the dupe

  • TheRealJasonSmithTheRealJasonSmith USXamarin Team Xamurai
    edited July 2014

    We are reviewing the private bug issue now, if its a configuration issue on our side we will get it fixed (obviously) before we make any prioritization changes.

  • EricMaupinEricMaupin USXamarin Team Xamurai

    So, if most bugs are private it may be due to a UI issue or some sort of bug in the tracking system itself. :)

    It did seem odd that so many were marking theirs as private. Took a look at bugzilla and sure enough it was set wrong. I've corrected this.

  • ChaseFlorellChaseFlorell CAInsider, University mod

    Nice!

  • DanielLDanielL PLInsider ✭✭✭✭

    @ermau Is it possible to change previously submitted bugs as public? I don't see that functionality.

  • EricMaupinEricMaupin USXamarin Team Xamurai

    @ermau Is it possible to change previously submitted bugs as public? I don't see that functionality.

    @DanielL‌ Uncheck "Xamarin Team (Private Bug)" on the right side.

  • ChaseFlorellChaseFlorell CAInsider, University mod
  • DanielLDanielL PLInsider ✭✭✭✭
  • EricMaupinEricMaupin USXamarin Team Xamurai

    Ah that's a shame, PM me links to issues you want opened and I will open them.

  • WarrenStephensWarrenStephens USMember ✭✭

    we checked [Forms] out, ran some tests, and decided that it is absolutely the direction that we wanted to go...

    My desire (as someone completely new to Xamarin, and attracted specifically by Forms) is that I would rather start out with a limited, but highly portable app as my first effort (i.e. Forms) – rather than begin with a powerful, but partially-portable app implementation (i.e. Xamarin.iOS and .Android).

Sign In or Register to comment.