Deserializing large stream to object causes out of memory exception

EvgenyVolynskyEvgenyVolynsky USMember
edited November 2 in Xamarin.Android

I cannot solve the following problem since last week. I have a typical client server architecture with WCF restful service on the server side and xamarin pcl + xamarin native android on the other side. The requirement is to synchronize the server database with the android sqlite database if the client has an old version.

I return a stream of ca 120 MB from server, that is being consumed by httphandler, that I have implemented myself to report the progress. When the stream was consumed, it is being deserialized by JsonDeserializer and here the journey begins.

I have at this place out of memory crashes.

What I have tried sofar:

  1. I created an environment.txt with: MONO_GC_PARAMS=bridge-implementation=tarjan,nursery-size=256m,soft-heap-limit=1024m and tried several configurations
  2. I set this option to manifest
    android:largeHeap="true"
    android:hardwareAccelerated="false">

  3. I tried to use lazy objects without success
  4. The last thing was the serialization of the stream with the jsonserialzer directly in small portions, hovewer I cannot report the progress.
    var response = await client.GetStreamAsync(queryString);

                        if (response != null)
                        {
                            // Ensure that the response was successful
                            response.EnsureSuccessStatusCode();
                            using (StreamReader sr = new StreamReader(response))
                            using (JsonReader reader = new JsonTextReader(sr))
                            {
    
                                JsonSerializer serializer = new JsonSerializer();
    
                                //read the json from a stream
                                // json size doesn't matter because only a small piece is read at a time from the HTTP request
                                result = serializer.Deserialize<ServerSynchronizationDataDTO>(reader);
    
                            }
                        }
    

Here is the http handler code:

        protected async override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
                {
                    DownloadProgressEventArgs args = new DownloadProgressEventArgs();
                    HttpResponseMessage response = null;
                    ServerSynchronizationDataDTO result = null;

                    try
                    {
                        response = await base.SendAsync(request, cancellationToken);
                        var totalBytes = Convert.ToInt32(response.Content.Headers.ContentDisposition.DispositionType);
                        var canReportProgress = totalBytes != 0;
                        args.CanReportProgress = canReportProgress;
                        args.TotalBytes = totalBytes;

                        using (var stream = await response.Content.ReadAsStreamAsync())
                        {
                            var bytesTransferred = 0L;
                            int read;
                            var buffer = new byte[4096];
                            using (MemoryStream ms = new MemoryStream())
                            {
                                while ((read = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
                                {
                                    bytesTransferred += read;
                                    ms.Write(buffer, 0, read);
                                    args.BytesTransferred = bytesTransferred;
                                    args.ProgressPercentage = bytesTransferred / (decimal)totalBytes;
                                    DownloadProgressChanged?.Invoke(args);
                                }
                                args.DownloadComplete = true;
                                DownloadProgressChanged?.Invoke(args);
                                ms.Position = 0;

                                var serializer = new JsonSerializer();
                                await Task.Run(() =>
                                {
                                    using (var sr = new StreamReader(ms))
                                    using (var jsonTextReader = new JsonTextReader(sr))
                                    {
                                        result = serializer.Deserialize<ServerSynchronizationDataDTO>(jsonTextReader);
                                    }
                                    response.Content = new ObjectContent<ServerSynchronizationDataDTO>(result, new JsonMediaTypeFormatter(), "application/json");
                                });
                            }
                        }
                    }
                    catch (Exception ex)
                    {
                        throw new Exception(ex.Message);
                    }

                    return response;
                }

Any help would be greate, because we have to deliver the product to customer and the major feature does not work.

Thank You in advance.

Sign In or Register to comment.