Send a lot of images to a webapi and the app crashes

EnricoRossini

HI guys,
I have a problem with my app. Users take a lot of picture for a job about 3000 pictures and when the job is finished, they send all images to the server. I send all picture across WebAPI.

The app crashes after about 400 pictures and I can't understand why.

Basically I have a list with the picture to send and with a foreach I call the function that sends an image. I tried to use SemaphoreSlim but it doesn't useful for this case.

JulienRosen

There is probably a problem with your code.

Amar_Bait

Probably out of memory exception (app or server). And also sending 3000 pictures is a lot of web api load (and bandwidth). You need to have few things in your app to make this work correctly.

1. Make sure you're disposing the stream of each picture you're sending, to avoid (lack of) memory exceptions.
2. 3000 pictures is A LOT of bandwidth. It means you're probably sending (ten+) gigabytes of data to your server. You'll probably need to make the images as small as possible (with keeping an acceptable resolution/quality)
3. Make sure the web server/web api you're sending pictures to is enough powerful to handle all that data.
4. I think you'll need to parallelize the job (sending 2-3 pictures a time is enough, more than that and you'll make things slower than sending a picture a time)
5. You need to implement a good retry and resume strategy if a problem happens (and it will probably always happen from time to time: connectivity problem, battery out, server unavailability, Android/iOS tomb-stoning your app...etc) during the transfer of images. You can't afford to have an entire collection of 3000 pictures resent, if the upload failed in the 2723th picture.
6. Showing us the code for the app (sending function) and web api (receiving action) would help us understand your code and try to find the problem(s). Also try to catch the exception that makes the app crash and share it with us.

VisionsInCode

Hey,
How about another approach? Why not use dropbox?
Here is a good post on how to use the dropbox in Xamarin:
http://xamariniac.hlinteractive.se/index.php/2016/10/02/xamarin-forms-dropbox-integration-app-folder/

EnricoRossini

Thanks @nadjib

I have a list of files to send from my device to my server.

List<myFiles> list = new List<myFiles>() { "[long list of files...]" };

For that I want to create a list of tasks: for each file I should invoke a function that sends to my webapi that file via PUT

public async Task<bool> UploadPhoto(byte[] photoBytes, int PropertyId, string fileName)
{
    bool rtn = false;

    if (CrossConnectivity.Current.IsConnected)
    {
        var content = new MultipartFormDataContent();
        var fileContent = new ByteArrayContent(photoBytes);
        fileContent.Headers.ContentType = 
                            MediaTypeHeaderValue.Parse("multipart/form-data");
        fileContent.Headers.ContentDisposition = 
                            new ContentDispositionHeaderValue("attachment")
        {
            FileName = fileName + ".jpg"
        };
        content.Add(fileContent);

        string url = RestURL() + "InventoriesPicture/Put";

        try
        {
            using (var client = new HttpClient())
            {
                client.DefaultRequestHeaders.Add("authenticationToken", SyncData.Token);
                HttpResponseMessage response = await client.PutAsync(url, content);
                if (response.IsSuccessStatusCode)
                {
                    rtn = true;
                    Debug.WriteLine($"UploadPhoto response {response.ReasonPhrase}");
                }
                else
                {
                    Debug.WriteLine($"UploadPhoto response {response.ReasonPhrase}");
                }
            }
        }
        catch (Exception ex)
        {
            Debug.WriteLine($"UploadPhoto exception {ex.Message}");
        }

        Debug.WriteLine($"UploadPhoto ends {fileName}");
    }

    return rtn;
}

In a function I have a foreach that calls UploadPhoto. I think there are too many tasks at the same time then I want to send a file, wait the result from the webapi and then send next file and so on.

Amar_Bait

Show me your foreach loop?

Amar_Bait

And also the part the get bytes from a file

Amar_Bait

Also it will be a (lot) better idea to send Stream instead of byte[]. The reason is that when you use byte[] to load a file content, all the file content will be loaded to memory, while Stream will load into memory only small chunk of the file in a reusable buffer. Think of it like you're streaming to your web server.

EnricoRossini

@VisionsInCode said:
Hey,
How about another approach? Why not use dropbox?
Here is a good post on how to use the dropbox in Xamarin:
http://xamariniac.hlinteractive.se/index.php/2016/10/02/xamarin-forms-dropbox-integration-app-folder/

This article is interested but unfortunately I can't use Dropbox in my case: all pictures are protected by privacy and I don't want to have copies of them somewhere else apart from my server.

VisionsInCode

Ah I see, Hmm how about ftp?
Check out this discussion:
https://forums.xamarin.com/discussion/90148/upload-image-to-a-ftp-server-using-pcl-xamarin-forms

EnricoRossini

Everything starts from Send that receives the list of Images (Images is a model with info about each image). picture.IsExist and picture.GetImageAsByte check\read directly in the file system the image (or file). The procedure is working well if you send few images (less then 400)

    public async Task Send(List<Images> list)
    {
        var tasks = new List<Task<bool>>();

        int i = 0;
        int nImages = list.Count;
        foreach (Images img in list)
        {
            tasks.Add(Task.Run(async () => await SendPictureIfExists(img.PropertyId, img.FileName, img.Section)));
        }

        var combinedTask = Task.WhenAll(tasks);
        var successfulContinuation = combinedTask.ContinueWith(task => 
                                        CombineResult(task.Result), TaskContinuationOptions.OnlyOnRanToCompletion);
        var failedContinuation = combinedTask.ContinueWith(task => 
                                        HandleError(task.Exception), TaskContinuationOptions.NotOnRanToCompletion);
        await Task.WhenAny(successfulContinuation, failedContinuation);
    }

    void CombineResult(bool[] result)
    {
    }

    void HandleError(AggregateException ex)
    {
    }

    /// <summary>
    /// Sends the picture if exists.
    /// </summary>
    /// <returns>The picture if exists.</returns>
    /// <param name="PropertyId">Property identifier.</param>
    /// <param name="FileName">File name.</param>
    /// <param name="Section">Section.</param>
    public async Task<bool> SendPictureIfExists(int PropertyId, string FileName, InventorySection Section)
    {
        bool isExist = false;

        try
        {
            var picture = DependencyService.Get<IPicture>();
            isExist = await picture.IsExist(PropertyId, FileName, Section);
            if (isExist)
            {
                byte[] imageAsByte = await picture.GetImageAsByte(PropertyId, FileName, Section);
                if (imageAsByte.Length > 0)
                    await UploadPhoto(imageAsByte, PropertyId, FileName);
            }
        }
        catch (Exception ex)
        {
        }

        return isExist;
    }

EnricoRossini

@VisionsInCode said:
Ah I see, Hmm how about ftp?
Check out this discussion:
https://forums.xamarin.com/discussion/90148/upload-image-to-a-ftp-server-using-pcl-xamarin-forms

Ok, interesting too. Do you know how many images can it send?

Amar_Bait

Yeah. The problem is you're having too much tasks executed at the same time. And there's a little bit of overcomplication in code, like:

tasks.Add(Task.Run(async () => await SendPictureIfExists(img.PropertyId, img.FileName, img.Section)));

You could simplify it to:

tasks.Add(SendPictureIfExists(....));

or:

tasks = list.Select(img=>SendPictureIfExists(img.PropertyId, img.FileName, img.Section)).ToList();

Perhaps you should batch the tasks by 3 a time:

for (var i=0; i<batchesTotal; i++) 
{
     var nextBatchTasks = GetTasksBatch(i)
     await nextBatchTasks .WhenAll();
 }

public IEnumerable<Task<bool>> GetTasksBatch(int batchNumber)
{
    return Tasks.Skip(batchNumber* 3).Take(batchNumber);
}

EnricoRossini

Thanks @nadjib I'm trying...

EnricoRossini

Now the app sends images...

One thing I don't understand is why a ThreadPool starts or end randomly. Now when the app sent 380 pictures it seems blocks.

EnricoRossini

I changed the code, now it is

    public async Task Send(List<Images> list)
    {
        nImages = list.Count;
        nSentImages = 0;
        foreach (Images img in list)
        {
            tasks.Add(SendPictureIfExists(img.PropertyId, img.FileName, img.Section));
            tasks.Add(SendPictureIfExists(img.PropertyId, img.FileName + "_original", img.Section));
            tasks.Add(SendPictureIfExists(img.PropertyId, img.FileName + "_155x155", img.Section));
            tasks.Add(SendPictureIfExists(img.PropertyId, img.FileName + "_80x80", img.Section));

            var combinedTask = Task.WhenAll(tasks);
            var successfulContinuation = combinedTask.ContinueWith(task => 
                                          CombineResult(task.Result), TaskContinuationOptions.OnlyOnRanToCompletion);
            var failedContinuation = combinedTask.ContinueWith(task => 
                                          HandleError(task.Exception), TaskContinuationOptions.NotOnRanToCompletion);
            await Task.WhenAny(successfulContinuation, failedContinuation);

            tasks.Clear();
        }
    }

With this code I can send 481 pictures...

Amar_Bait

We need to get the exception. Can you manage to get the exception?

Amar_Bait

Also you could perhaps assign null to bytes[] variables after you finish uploading, then each 20 pictures uploads or so call GC.Collect()

EnricoRossini

@nadjib said:
We need to get the exception. Can you manage to get the exception?

No really or I don't know what you mean

EnricoRossini

public async Task Send(List<Images> list)
{
    foreach (Images img in list)
    {
        tasks = new List<Task<bool>>();
        tasks.Add(SendPictureIfExists(img.PropertyId, img.FileName, img.Section));

        var combinedTask = Task.WhenAll(tasks);
        var successfulContinuation = combinedTask.ContinueWith(task =>
                                CombineResult(task.Result), TaskContinuationOptions.OnlyOnRanToCompletion);
        var failedContinuation = combinedTask.ContinueWith(task =>
                                HandleError(task.Exception), TaskContinuationOptions.NotOnRanToCompletion);
        await Task.WhenAny(successfulContinuation, failedContinuation);
        successfulContinuation = null;
        failedContinuation = null;
        combinedTask = null;

        tasks.Clear();
        tasks = null;

        GC.Collect();
    }
}

With this code I can send 509