Shared HttpClient instance fails after cancellation

Although the thread safety of HttpClient was recently improved, I've found a scenario where the state of a HttpClient instance can become irrevocably corrupt. After more than a year of suffering from this obscure problem, I think I finally have a simple project in which it is reliably reproducible (attached with code below for easy reference).

The flow is as follows:

  1. Start 10 HTTP requests for the same resource on individual background threads
  2. Start 10 more HTTP requests for the same resource on individual background threads
  3. Cancel the first 10 requests
  4. Wait for all the threads to finish

The expectation is that the first set of requests terminate quickly with a TaskCancelledException, then the second set run to completion. The request logic only actually waits for the header to be returned and doesn't bother downloading the content, so under favourable network conditions we would expect this to all complete within 10 seconds or so.

However, in the sample project the second set of requests never return and in fact the HttpClient timeout of 30 seconds eventually kicks in to kill the tasks off.

There are four adjustments to the sample project that stop this problem happening (the first three have NOTE comments in the code):

  1. Increasing the DefaultConnectionLimit to 100
  2. Removing the task cancellation step
  3. Using an instance of HttpClient per request rather than using a single shared instance
  4. Targeting a different domain for the second set of requests

Change any one of these things and the code will run as expected.

Based on this evidence, and tinkering with the target URL, I think the likely culprit has something to do with DNS resolution. Perhaps cancelling a task when a DNS query is being processed somehow corrupts the state of the HttpClient object? It is a natural area for lock protection, but perhaps the cancellation logic forgets the unlock?

There is still a bit of non-determinism involved, so I would very much appreciate a reproduce if any of you has the time. If you use a single instance of HttpClient I would say it is in your interest even if you haven't seen the bug in the wild.

The sample code is adapted from a @MihaMarkic bug report about a related (or possibly identical) problem.

Here is the sample code for easy reference:

using System;
using Android.App;
using Android.Widget;
using System.Threading;
using System.Threading.Tasks;
using System.Collections.Generic;
using System.Net.Http;
using HttpsBug;

namespace HttpClientBug {

    [Activity(Label = "Http Client Bug", MainLauncher = true, Icon = "@drawable/icon")]
    public class MainActivity : Activity {

        HttpClient commonClient;

        protected override void OnCreate(Android.OS.Bundle bundle) {
            base.OnCreate(bundle);

            SetContentView(Resource.Layout.Main);

            // NOTE 1: Uncomment this line to stop HttpClient request hang

//            System.Net.ServicePointManager.DefaultConnectionLimit = 100;

            var button = FindViewById<Button>(Resource.Id.myButton);
            button.Text = "Start";
            button.Click += async delegate {

                Console.WriteLine("---------Running Test");

                button.Text = "Running...";
                button.Enabled = false;

                commonClient = new HttpClient() { Timeout = TimeSpan.FromSeconds(30) };

                // Start download tasks that should be cancelled
                var ctsCancel = new CancellationTokenSource();
                var downloadTaskCancel = MultiDownload(ctsCancel.Token, "Cancel");

                // Start download tasks that should not be cancelled
                var ctsDoNotCancel = new CancellationTokenSource();
                var downloadTaskDoNotCancel = MultiDownload(ctsDoNotCancel.Token, "Do Not Cancel");               

                // NOTE 2: Remove the cancel operation to stop the hang

                // Cancel first set of tasks
                ctsCancel.Cancel();

                // Wait for cancelled tasks to complete
                await downloadTaskCancel;

                // Wait for non-cancelled tasks to complete
                await downloadTaskDoNotCancel;

                button.Enabled = true;
                button.Text = "Start";

                Console.WriteLine("---------Test Complete");
            };
        }

        async Task MultiDownload(CancellationToken ct, string tag) {
            var tasks = new List<Task>();
            for (int i = 0; i < 10; i++) {
                tasks.Add(Download(ct, i, tag));
            }
            await Task.WhenAll(tasks);
        }

        async Task Download(CancellationToken ct, int i, string tag) {
            Console.WriteLine($"Starting '{tag}' {i}");
            try {

                // NOTE 3: Swap localClient for commonClient in the request to stop the hang

                using (var localClient = new HttpClient() { Timeout = TimeSpan.FromSeconds(30) })
                using (var response = await commonClient.GetAsync("https://download.mozilla.org/?product=firefox-30.0-SSL&os=linux&lang=ach", HttpCompletionOption.ResponseHeadersRead, ct)) {
                    if (response.IsSuccessStatusCode) {
                        Console.WriteLine($"Finished '{tag}' {i}");
                    } else {
                        Console.WriteLine($"Failed '{tag}' {i}");
                    }
                }
            } catch (TaskCanceledException) {
                Console.WriteLine($"Task Cancelled '{tag}' {i}");
            } catch (Exception ex) {                
                Console.WriteLine($"Exception '{tag}' {i} - {ex.Message}");
                Console.WriteLine(ex.StackTrace);
            }
        }

    }
}
Tagged:

Posts

  • RupertRawnsleyRupertRawnsley GBMember ✭✭✭

    Raised a bug report here: https://bugzilla.xamarin.com/show_bug.cgi?id=35779

    Note to Xamarin: I'm disappointed you didn't pick this up from the forum. I included a detailed description, a reproduce project, and even speculation as to cause - what more could you ask for?

  • BrendanZagaeskiBrendanZagaeski USForum Administrator, Xamarin Team Xamurai
    edited April 2017

    Many thanks for the bug report. Bug reports, emails to the support team via "Business & Enterprise Support", or comments on the latest Stable release thread are all quite helpful to ensure visibility of potential bugs to the Xamarin team.

    The Support Team has some tentative plans for 2016 to increase direct monitoring of the forums, but for now forum threads outside of the announcements are primarily only monitored by community members, so emails and bug reports are the best approach to get direct attention from the Xamarin team on potential bugs. Thanks again!


    I will now close this forum thread to help direct any further updates about this issue onto the corresponding bug report:
    https://bugzilla.xamarin.com/show_bug.cgi?id=35779

    (As always, I'm also happy to reopen the thread upon request within the next month or two. Just send a quick email to [email protected] with a link to this thread. For follow-up after the next couple of months, if the remaining related open bug reports or forum threads do not cover the desired scenario, please follow the recommendations to create a new bug report, forum thread, or Stack Overflow question. Thanks!)

This discussion has been closed.