Getting the average redness of an image from a camera on IOS

VincamVincam NLMember ✭✭

We've already asked this on Stackoverflow but since we're not getting a lot of attention we'll put this here as well.
There are some discussions found on these forums however none of the solutions found there work for us (assuming we implemented them correctly).

We're currently working on an app for both Android and IOS using Xamarin. For this app we need a heartbeat monitor that scans how 'red' your finger is (or rather how the colour fluctuates because of the heart) after putting it on the camera for a while.

In order to do this on iOS, we want to find the average redness of each frame while sampling with the camera. After we have the redness values our algorithm will find the BPM.

Right now we're struggling to get the colour of each pixel from a frame. Obviously when we can get the colour of each pixel we can just average the red component, however getting the colour seems difficult.

This is the code we have right now, though we have tried many options found on different forums (they all look similar).

public override void DidOutputSampleBuffer (AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
{           
        // Grab an image from the buffer
        var image = GetImageFromSampleBuffer (sampleBuffer);

        var resolution = Width * Height;

        Width = (int)image.CGImage.Width;
        Height = (int)image.CGImage.Height;

        int bytesPerPixel = 4;
        int bytesPerRow = bytesPerPixel * Width;
        int bitsPerComponent = 8;

        var rawData = new byte[bytesPerRow * Height];
        var handle = GCHandle.Alloc (rawData);  

        double red_avg = 0;
        double green_avg = 0;
        double blue_avg = 0;

        try {
            using (var colorSpace = CGColorSpace.CreateDeviceRGB ()) {
                using (var context = new CGBitmapContext (rawData, Width, Height,
                                         bitsPerComponent, 
                                         bytesPerRow, 
                                         colorSpace, 
                                         CGBitmapFlags.PremultipliedLast)) {
                    context.DrawImage (new RectangleF(0, 0, Width, Height), image.CGImage);
                }
            }

            for (int x = 0; x < Width; x++)
                for(int y = 0; y < Height; y++)
                {
                    var index = bytesPerRow * y + bytesPerPixel * x;

                    byte red   = rawData[index];
                    byte green = rawData[index + 1];
                    byte blue  = rawData[index + 2];
                    byte alpha = rawData[index + 3];

                    red_avg += red;
                    green_avg += green;
                    blue_avg += blue;
                }

            //Console.WriteLine("V1: " + red_avg / resolution + "  V2: " + blue_avg / resolution + "  V3: " + green_avg / resolution);
        }
        finally {

            handle.Free ();
            sampleBuffer.Dispose ();
        }
    }
}

The above code is run every frame. When we run our program the average red, green and blue values all roughly become the same value every frame, however they in- and decrement in the way you would expect when exposing the camera to more or less light.

We feel we might not understand fully how we should read the colour values from the byte array we construct.

What are we doing wrong?

Best Answer

Answers

  • adamkempadamkemp USInsider, Developer Group Leader mod

    I don't understand exactly what the problem is. That is, you have this code, and it runs, and you're testing it somehow. What are you expecting when you test it, and what actually happens?

    In my experience using APIs like the CGBitmapContext constructor you're using the most likely source of problems is the last argument (the CGBitmapFlags argument). You may have to play with that until you get the right options. It uses that to determine how to format the pixels, and if it's wrong then the colors will be out of order or the wrong size or channels may be missing.

  • VincamVincam NLMember ✭✭

    We're sorry if we weren't clear enough on what the problem is.

    This code runs without crashing. What we'd like to see is an output that makes sense.
    This means that when we hold the camera in front of something that is almost RGB(255,0,0) we would see that the output would be higher in the red_avg, and smaller in the green_avg and the blue_avg. What it gives us now is an output where the red_avg, green_avg and the blue_avg all give a similar value (e.g. they all give some value like 80.453232 (differs) instead of R: high, G: low, B: low). The reason why we think we are still somewhat in the right direction is because the value all colour outputs give do react on more or less light. This means that red_avg, green_avg and blue_avg all output a lower value when we let very little light come to the camera, and ouput a higher value when we let more light to the camera.

    Fiddling with the flags could work as we've seen a lot of those discussions on the internet. We have however chosen for the combination we've seen as the solution to our problem. At the same time can changing the flags cause the process to crash.
    I guess playing some more with the flags could help.

  • adamkempadamkemp USInsider, Developer Group Leader mod

    It looks like you're starting from this example, which makes me think you're already doing something pretty inefficient. What that example does is grab a frame from the buffer (as a byte array) and then render that frame into an image. Then your code is creating a new byte array, constructing an image context from it, drawing the image into that buffer, and inspecting the bytes. But you already had the bytes (inside GetImageFromSampleBuffer). You should be doing this work in there, which would be much more efficient.

  • VincamVincam NLMember ✭✭
    edited August 2015

    First if all I apologize responding only after long periods of time. We're not working on this full time.

    On the matter, we suspect (realize) that the way we're doing this is indeed inefficient, though it is just the way our code developed off of the example you posted during our trying out new things. We think that given that this is inefficient it does not cause the problematic behavior we're experiencing as we're basically converting our data to and image and back, unless something does go wrong during these conversions. We will test if this goes wrong though we feel as if the problem lies somewhere else...

  • adamkempadamkemp USInsider, Developer Group Leader mod
    edited August 2015

    I'm glad you figured it out!

  • SOULDEVSOULDEV INMember

    Hey Vincam
    I was making a Heart rate App for my school project. Would you mind sharing the code for your app so I can get some help.
    Thanks

Sign In or Register to comment.