Unpredictable MethodAccessException deep in HttpClientHandler
GBMember ✭✭✭
I have a very hard to track down occasional error and I would welcome suggestions on likely causes and how to proceed to resolve it.
I'm getting several errors of the following form:
System.MethodAccessException: Method `System.Net.WebHeaderCollection:AddValue (string,string)' is inaccessible from method `System.Net.Http.HttpClientHandler/<SendAsync>c__async0:MoveNext ()'
...and...
System.MethodAccessException: Method `System.Net.ServicePointManager:CloseConnectionGroup (string)' is inaccessible from method `System.Net.Http.HttpClientHandler:Dispose (bool)'
This appears bogus as both methods are internal and are being accessed legitimately by classes from within their library. My suspicion is that there is some flaw in the logic in AsyncMethodBuilderCore, but it is an area of Mono with which I am unfamiliar.
It doesn't happen all the time (order of 1% chance) and often goes away if the app is restarted. If it is going to happen, it happens straight away and keeps happening. I get it in Release (standalone) and Debug (shared library) modes equally. It is more likely to happen after an installation, which suggests some sort of library / timing issue.
I can occasionally trap it in the IDE, but I'm at a loss as how to proceed to debug it. Here is a full stack trace when one of the exceptions was thrown:
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<System.Net.Http.HttpClientHandler.<SendAsync>c__async0> (stateMachine={System.Net.Http.HttpClientHandler}) in
System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Net.Http.HttpResponseMessage>.Start<System.Net.Http.HttpClientHandler.<SendAsync>c__async0> (stateMachine={System.Net.Http.HttpClientHandler}) in
System.Net.Http.HttpClientHandler.SendAsync (request=, cancellationToken=) in
System.Net.Http.HttpMessageInvoker.SendAsync (Parameters=) in
System.Net.Http.HttpClient.SendAsyncWorker (Parameters=) in
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<System.Net.Http.HttpClient.<SendAsyncWorker>c__async0> (Parameters=) in
System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Net.Http.HttpResponseMessage>.Start<System.Net.Http.HttpClient.<SendAsyncWorker>c__async0> (Parameters=) in
System.Net.Http.HttpClient.SendAsyncWorker (Parameters=) in
System.Net.Http.HttpClient.SendAsync (Parameters=) in
System.Net.Http.HttpClient.SendAsync (Parameters=) in
System.Net.Http.HttpClient.PostAsync (Parameters=) in
Log.ReportAsync (Parameters=) in
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<Log.<ReportAsync>c__async0> (Parameters=) in
System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.Start<Log.<ReportAsync>c__async0> (Parameters=) in
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Log.<ReportAsync>c__async0> (Parameters=) in
Log.ReportAsync (Parameters=) in
LearnPad.Droid.Client.LearnPadActionBarActivity.<OnOptionsItemSelected>c__AnonStorey13.OnOptionsItemSelected () in /Users/rupertrawnsley/LearnPad/Source/Mono/LearnPad.Android.Client/View/LearnPadActionBarActivity.cs:260
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<LearnPad.Droid.Client.LearnPadActionBarActivity.<OnOptionsItemSelected>c__AnonStorey13.<OnOptionsItemSelected>c__async2B> (Parameters=) in
System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.Start<LearnPad.Droid.Client.LearnPadActionBarActivity.<OnOptionsItemSelected>c__AnonStorey13.<OnOptionsItemSelected>c__async2B> (Parameters=) in
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<LearnPad.Droid.Client.LearnPadActionBarActivity.<OnOptionsItemSelected>c__AnonStorey13.<OnOptionsItemSelected>c__async2B> (Parameters=) in
LearnPad.Droid.Client.LearnPadActionBarActivity.OnOptionsItemSelected.AnonymousMethod__2 (Parameters=) in /Users/rupertrawnsley/LearnPad/Source/Mono/LearnPad.Android.Client/View/LearnPadActionBarActivity.cs:258
System.Threading.Tasks.Task<System.Threading.Tasks.Task>.InnerInvoke (Parameters=) in
System.Threading.Tasks.Task.Execute (Parameters=) in
System.Threading.Tasks.Task.ExecutionContextCallback (Parameters=) in
System.Threading.ExecutionContext.Run (Parameters=) in
System.Threading.ExecutionContext.Run (Parameters=) in
System.Threading.Tasks.Task.ExecuteWithThreadLocal (Parameters=) in
System.Threading.Tasks.Task.ExecuteEntry (Parameters=) in
System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem (Parameters=) in
System.Threading.ThreadPool.AnonymousMethod__0 (Parameters=) in
The code that is throwing the error is as follows:
...
using (var response = await client.PostAsync(new Uri(IoC.Resolve<IPortal>().DevicesUrl, "com/Logs.cfc?method=reportV2"), parameters).ConfigureAwait(false)) {
if (response.IsSuccessStatusCode) {
Log.Info("Contacted log server successfully");
} else {
throw new RecoverableException("Failed to send logs: " + response.ReasonPhrase + " (" + await response.Content.ReadAsStringAsync() + ")");
}
serverResponseData = JObject.Parse(await response.Content.ReadAsStringAsync());
}
...
But the errors occur in various places that PostAsync is called.
The indeterminate nature of the problem makes a simple standalone reproduce very difficult without further clues as to the cause.
Posts
I had the same issue, it seemed to be fixed with an include;
private void IncludeServicePointManager() { System.Net.ServicePointManager.EnableDnsRoundRobin = true; }@ToinedeBoer.8215 Did this work on Windows, because in Mono it doesn't seem to be implemented yet: https://bugzilla.xamarin.com/show_bug.cgi?id=31764
@RupertRawnsley I did it for Android, and it stil seems to work for now.
Curious. It should throw an exception according to the code: https://github.com/mono/mono/blob/master/mcs/class/System/System.Net/ServicePointManager.cs#L233
Its in the Include, so I will never get invoked for real. (maybe I choose the wrong property to use in include)
I just wanted to make sure all was included to prevent this:
System.MethodAccessException: MethodSystem.Net.ServicePointManager:CloseConnectionGroup (string)' is inaccessible from method `I'm still looking into the issue, because it hapend again but much less then before.
(when I find something I will let you know)
I am also starting an investigation on a very similar problem. I get the exception
System.MethodAccessException: Method
System.Net.WebHeaderCollection:AddValue (string,string)' is inaccessible from methodSystem.Net.Http.HttpClientHandler/c__async0:MoveNext ()'Once I start getting the exception there is no way to work around it, I have to restart my app. I would be very interested in any progress anyone has done on that.
I'm seeing the same crash in HockeyApp for my prod users, 145 crashes in the current release so far... Any progress on solving this?
md52ce486a14f4bcd95899665e9d932190b.JavaProxyThrowable: System.MethodAccessException: Method
System.Net.ServicePointManager:CloseConnectionGroup (string)' is inaccessible from methodSystem.Net.Http.HttpClientHandler:Dispose (bool)'at System.Net.Http.HttpMessageHandler.Dispose () [0x00000] in :0
at System.Net.Http.DelegatingHandler.Dispose (Boolean disposing) [0x00000] in :0
at System.Net.Http.HttpMessageHandler.Dispose () [0x00000] in :0
at System.Net.Http.DelegatingHandler.Dispose (Boolean disposing) [0x00000] in :0
at System.Net.Http.HttpMessageHandler.Dispose () [0x00000] in :0
at System.Net.Http.DelegatingHandler.Dispose (Boolean disposing) [0x00000] in :0
at System.Net.Http.HttpMessageHandler.Dispose () [0x00000] in :0
at System.Net.Http.HttpMessageInvoker.Dispose (Boolean disposing) [0x00000] in :0
at System.Net.Http.HttpClient.Dispose (Boolean disposing) [0x00000] in :0
at System.Net.Http.HttpMessageInvoker.Dispose () [0x00000] in :0
at RPR.Mobile.Shared.Http.WebApiHttpClient.RPR.Mobile.Shared.Contracts.IWebApiHttpClient.Dispose () [0x00000] in :0
at RPR.Mobile.Shared.Services.BaseDataService+c__AnonStorey1
2[RPR.Mobile.Shared.Entities.Common.Http.EmptyBody,RPR.Mobile.Shared.Entities.Common.Http.EmptyBody].<>m__1 (System.Threading.Tasks.Task1 a) [0x00000] in :0at System.Threading.Tasks.ContinuationResultTaskFromResultTask
2[RPR.Mobile.Shared.Entities.Common.Http.QueryResponse1[RPR.Mobile.Shared.Entities.Common.Http.EmptyBody],RPR.Mobile.Shared.Entities.Common.Http.EmptyBody].InnerInvoke () [0x00000] in :0at System.Threading.Tasks.Task.Execute () [0x00000] in :0
at dalvik.system.NativeStart.run(Native Method)
I really recommend using the ModernHttpClient handler. So much more reliable. Also couple it with Polly for retry logic.
@Cheesebaron I've now switched to ModernHttpClient and I get the same problem. It must be happening outside of the MessageHandler code.
At the outer level, how are you invoking your task?
@Cheesebaron Usually something like this:
var url = new Uri("http://google.com"); var formParameters = new Dictionary<string, string>() { { "paramA", "valueA" }, { "paramB", "valueB" }, }; var content = new FormUrlEncodedContent(formParameters); using (var httpClient = new HttpClient(new ModernHttpClient.NativeMessageHandler())) using (var response = await httpClient.PostAsync(url, content).ConfigureAwait(false)) { if (response.IsSuccessStatusCode) { // Process response } }We have always created a new HttpClient because until recently there was a threading bug in Mono that stopped you reusing the same client. I've actually switched to a shared instance of HttpClient now for most requests in case it gets around this bug somehow.
So where would this code go usually? In an event handler? Maybe something else?
I can't say that I've ever seen these issues in my apps. I usually invoke that kind of code from ICommand implementations in my ViewModel, which in turn calls a Service, which is basically a singleton helper class, which does all the heavy lifting.
Yes, typically some sort of event handler would initiate the call (onRefresh for instance), often on the UI thread, but obviously context will have switched to a background thread (using async/await) by the time the HTTP code executes.
I don't use a particular view model component, but the structure is similar to what you describe: almost all HTTP requests are channeled through a small number of singleton functions that abstract away the complexities of the request and merely return the requested resource.
In case it's relevant: I forgot to mention in my example that I always set the Timeout property of the HttpClient to avoid an indefinite hang when making the request.
I don't think I'm doing anything particularly unusual, although my app does can download tens of resources at once over a short space of time. The fact that this problem is not experienced by many other users is surprising and probably relevant.
It smells like a threading problem deep in the async/await marshalling code. It must cause some sort of stateful corruption because once it happens, it continues to happen with every request.
I'd agree that you're probably not doing something unusual, as we're seeing the same issue still and are using ModernHttpClient as well. I double-checked and everywhere we instantiate a HttpClient we're passing in a ModernHttpClient handler.
Hi @RupertRawnsley and @JustinToth
Do either of you have a method of reproducing this? As mentioned by Rupert, this issue seems to happen very seldom and goes away after a fresh instance.
Ideally we'd be interested in a way to replicate this(Even if it's rare) with a sample project so that we can officially get this reported in our Bugzilla tracking tool. That way one of our platform engineers can investigate this further.
If either of you have a method of reproducing this in a sample repository, please feel free to reply to this forum post or start a bug report. I will personally move forward with a bug report once I can find a means of reproducing this.
Thanks!
Hi @JonDouglas,
Thanks for looking at this one. I'm afraid I haven't found a reliable reproduce for it yet, but I will certainly share it if I do.
There is a slim possibility it is related to this other rare bug for which I think I have found a reproduce. They are both deep in the heart of the HTTP code and manage to corrupt the stack irrevocably. Perhaps they are different presentations of a problem with a common cause?
Having the same issue once a week.
Any ideas on how to solve the issue ?
Polly could be an option, but once we get the exception, even 3 retry attempts will not fix the issue
I am having the same issue
Having the same issue ;(
I am having the same issue.
With android, I do something like: call a webservice -> timeout (in 30s) -> kill app -> launch again -> and i meet this issue
When i get this issue, i can't call any webservice, must re-install app to do that.
Hope we can get it done as soon as!!
var requestMsg = requestMsgFunc(); Debug.WriteLine("Real API: " + requestMsg.RequestUri); var response = await Task.Run(() => { var cancelSource = new CancellationTokenSource(); var reqTask = httpClient.SendAsync(requestMsg, cancelSource.Token); if (Task.WaitAny(new Task[] {reqTask}, TimeSpan.FromSeconds(TIME_OUT_IN_SECOND)) < 0) { cancelSource.Cancel(); // attempt to cancel the HTTP request throw new NetworkException(); } return reqTask.GetAwaiter().GetResult(); }).ConfigureAwait(false); var responseString = await response.Content.ReadAsStringAsync(); requestMsg.Dispose();That's quite annoying. I have to choose not to call dispose. But what I can I do for ADAL lib?
@RupertRawnsley Did you find the solution to this issue? Could you mind sharing it here?
@WingChan.5109 Sorry, nothing yet. I've failed to find a simple and reliable reproduce to submit to Xamarin.
JavaProxyThrowable: System.MethodAccessException: Method
System.Net.WebHeaderCollection:AddValue (string,string)' is inaccessible from methodSystem.Net.Http.HttpClientHandler/c__async0:MoveNext ()'I'm my case this exception is caused by the network switching connections, for example going from 3G to Wi-Fi during an IO request.
Error happens in ADAL,
OS version: Lollipop
Device: samsung SM-A300H
Error: System.MethodAccessException: Method
System.Net.ServicePointManager:CloseConnectionGroup (string)' is inaccessible from methodSystem.Net.Http.HttpClientHandler:Dispose (bool)'at System.Net.Http.HttpMessageHandler.Dispose () [0x00000] in :0
at System.Net.Http.HttpMessageInvoker.Dispose (Boolean disposing) [0x0001c] in :0
at System.Net.Http.HttpClient.Dispose (Boolean disposing) [0x00023] in :0
at System.Net.Http.HttpMessageInvoker.Dispose () [0x00000] in :0
at Microsoft.IdentityModel.Clients.ActiveDirectory.HttpClientWrapper+d__1.MoveNext () [0x0045e] in :0
Here are some devices have the problem:
Android version: JellyBeanMr1 Device: Sony C2305
OS version: LollipopMr1 Device: samsung SM-J110M
Android version: Lollipop Device: samsung SM-T805Y
OS version: Kitkat Device: samsung GT-I9060I
Android version: Lollipop Device: samsung SM-A500G
Android version: M Device: motorola XT1068
Android version: Kitkat Device: Xiaomi HM NOTE 1S
Hello to everybody,
we also came across this problem (1% of our clients).
System.MethodAccessExceptionMethod
System.Net.WebHeaderCollection: AddValue (string, string) 'is inaccessible method fromSystem.Net.Http.HttpClientHandler / c__async0: MoveNext ()'raw
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start (ref TStateMachine Statemachine)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder .start (ref TStateMachine Statemachine)
System.Net.Http.HttpClientHandler.SendAsync (HttpRequestMessage request, CancellationToken cancellationToken)
System.Net.Http.HttpMessageInvoker.SendAsync (HttpRequestMessage request, CancellationToken cancellationToken)
System.Net.Http.HttpClient. __ BaseCallProxy0 (HttpRequestMessage request, CancellationToken cancellationToken)
System.Net.Http.HttpClient.SendAsyncWorker ()
We already created a bug in Bugzilla Xamarin on this issue.
https://bugzilla.xamarin.com/show_bug.cgi?id=41133
When I look at the date of forum posts, it is quite sad that there is such a serious problem in one of most important part of today's applications which is the communication with the web. The worst part of it all is, that there is no way to make it functioning again, except of restarting the application. It is even worse, because even creating new HttpClient does not help there. So how we tell user, that he must restart an application/phone to let it function again? Many users even do not know, how to restart an app (they only know how to switch/minimize it).
If it was an issue in our code, I would definitely reserve maximum possible number of people for solving it. Unfortunately, we do not see another way to solve the problem except of code analysis. However, we are not much of help for searching through your code, with our team size of 3 programmers. As well as others, we do not have much chance to find out when the problem starts to occure.
I could not help, but the situation comes to me so, that Xamarin company is waiting until someone brings a problem on a silver platter and sends a test project. There is so much possitive energy during presenting new features everywhere and you are not able able to fix the problem in such an important part as HttpClient?
We are using your products for a long time and appreciate a work you do. However, I apologize for criticism, but in this case you simply deserve it.
^ What he said...
Same issue - Added to your bug report.
Clear project, build, deploy.
I had old version of dll causing this (with fastdeployment checked).
I've run into this problem a few times, but it happens so rarely for me, so very hard to debug.
Does this still happen if you use the AndroidClientHandler with HttpClient?
We never had this problem before, but now it started occuring (at low rate).
Out of my head, only related thing changed was adding second server connection service that 'could' potentially be called during, or right after first service is complete with server api calls.
Could someone confirm similar scenario possibility?
This is my observation as well. I can reproduce it sometimes by starting a task with a HttpClient request, and immediately starting another task with another HttpClient request (different HttpClient objects).
There could be other problems as well, but this seems to be one of them at least.
One question.
Are you guys using NTLM Authentication aswell?
For us it seems we hit into couple HttpClient bugs that shouldn't be occuring, and some are not even to be found on reports.
NTLM auth might not be widely used in Android, means it gets less feedback.
Just for example :
@Jurij not in our case.
Coincidentally I was trying the new AndroidClientHandler yesterday and got a repeatable BadRequest on one particular resource immediately. You might want to experiment; there's a useful blog post here: http://kerry.lothrop.de/httpclient-flavors/
The problem is within the Http client initialization. Bug may occur only within initial requests.
Multiple requests can potentially build up till Http Client is ready to process them upon internal initialization (then executed at same moment, sometimes causing it to bug internal Http Client state).
Our current workaround is to let only one request to be sent at start. Once sent, all rest awating requests can be released to run.
So far, problem is solved.
@Jurij That's consistent with my experience and a great clue. Do you have an example project you can share? As soon as we get a reproduce I'm sure they will be able to fix it.
@RupertRawnsley
We are currently busy with Android app release, and there's other projects on side, so we don't have time to reproduce.
If your working on example project, this might help.
Event order, time order, threads.
Hi,
we start using AndroidClientHandler after months of waiting. We coul'd confirm that problem dissaper.
But we have another problems, because there is missing implementaion of Timeout and CancellationToken.
More info is there
https://bugzilla.xamarin.com/show_bug.cgi?id=44673
Hopefully it will help others
We've gotten 2,000 of these so far in our current release build, and for many releases this has been our highest frequency crash. It's sad that a year and a half later Xamarin still isn't paying attention to this egregious bug.
You can use workaround from my earlier post above, we haven't had any problems since.
And I agree. We have to find bugs in their own source code and find workarounds, meanwhile they leave bugs untouched.
We aren't even allowed to update our Xamarin version for last few months, just because they still haven't decided to fix newly introduced bugs with newer versions. For example, the SQLite datetime bug which literally corrupts all our collected database data, luckily we found bug before we let it out in wild. Served for them on silverplatter, and yet ignored.