SkiaSharp draw on image -> handle orientation change

Basserd · November 2018

Hi!

I am trying to implement SkiaSharp in order to be able to draw on an image, which works great, also with the ability in picking different colours.
But when I change the orientation, the drawn lines move out of position until I change back the orientation to the orientation in which the drawing is made (see attachments).
I think understand why this is happening, but I can't figure out how to fix this.
I tried setting the canvas.Scale() and also transforming the paths with SKMatrix.MakeScale() but I didn't get the correct results.

I am still experimenting, but this is the piece of code I am working on at the moment.

SKImageInfo info = args.Info;
SKSurface surface = args.Surface;
canvas = surface.Canvas;

canvas.Clear();

float scale = Math.Min((float)info.Width / resourceBitmap.Width, (float)info.Height / resourceBitmap.Height);
float x = (info.Width - scale * resourceBitmap.Width) / 2;
float y = (info.Height - scale * resourceBitmap.Height) / 2;

destRect = new SKRect(x, y, x + scale * resourceBitmap.Width, y + scale * resourceBitmap.Height);

canvas.ClipRect(destRect);

using (new SKAutoCanvasRestore(canvas))
{
    BitmapExtensions.DrawBitmap(canvas, resourceBitmap, destRect, BitmapStretch.AspectFit, BitmapAlignment.Center, BitmapAlignment.Center);
}

foreach (FingerPaintPolyline polyline in completedPolylines)
{
    paint.Color = polyline.StrokeColor.ToSKColor();
    paint.StrokeWidth = polyline.StrokeWidth;

    canvas.DrawPath(polyline.Path, paint);
}

foreach (FingerPaintPolyline polyline in inProgressPolylines.Values)
{
    paint.Color = polyline.StrokeColor.ToSKColor();
    paint.StrokeWidth = polyline.StrokeWidth;

    canvas.DrawPath(polyline.Path, paint);
}

I am hoping to get some tips / advise.

Thanks in advance.

Best Regards,

Bas

ShawnCastrianni.5092 · December 2018

Your BitmapExtensions.DrawBitmap is drawing the image to fit the canvas which causes its dimensions and location to change regarding raw pixel x,y coordinates. I am also guessing you are storing your drawing annotations in raw pixel x,y coordinates which are NOT changing when you rotate. Therefore, the do not align when you rotate since image changed but drawing annotations did not change.

One way to do this is to NOT store the drawing annotations points in raw pixel x,y coordinates, but instead as percentages of the underlying image. So a dot drawn in the upper left corner of the image would be stored as 0%,0% and a dot drawn in the lower right corner of the image would be stored as 100%,100%. Then before you draw the annotations on top of the image, you have to convert the percentages back to raw pixel x,y values based on the NEW image location. So for example, a dot stored as 50%,75% would be converted to raw pixel x,y (assuming destRect is the exact Rect the image fills up, which it may not be, not sure):

float ptX = destRect.Left + destRect.Width * 0.5f;
float ptY = destRect.Top + destRect.Height * 0.75f;

I am not suggesting that this is the best way to do it nor the only way to do it, but it is one way to do it.

ShawnCastrianni.5092 · December 2018

Here is some sample code. I show in comments how to manually convert each point on the path from a percentage point to a canvas point. However, I did NOT take care to maintain the separate segments of the SKPath so the conversion links them altogether. Therefore, I switch to use a SKMatrix and then calling Transform on the SKPath which DOES maintain the separate segments. I have also attached the screenshots of portrait and landscape to show the annotation stays on top of the image in the correct location.

public class AnnotateFacePage : ContentPage
{
    private bool _isDrawMode;
    public bool IsDrawMode {
        get { return (_isDrawMode); }
        set {
            _isDrawMode = value;
            OnPropertyChanged();
        }
    }
    private Command _drawCommand = null;
    public ICommand DrawCommand {
        get {
            _drawCommand = _drawCommand ?? new Command(DoDrawCommand);
            return (_drawCommand);
        }
    }
    private Command _clearCommand = null;
    public ICommand ClearCommand {
        get {
            _clearCommand = _clearCommand ?? new Command(DoClearCommand);
            return (_clearCommand);
        }
    }

    private SKCanvasView _canvasV = null;
    private bool _isDrawing = false;
    private SKMatrix _m = SKMatrix.MakeIdentity();
    private SKMatrix _im = SKMatrix.MakeIdentity();
    private SKBitmap _bitmap = null;
    private SKRect? _bitmapRect = null;
    private SKPoint _lastPanPt = SKPoint.Empty;
    private SKPath _sketchPath = new SKPath();

    public AnnotateFacePage()
    {
        BindingContext = this;
        Title = "Annotate Face";
        ToolbarItem drawTBI = new ToolbarItem();
        ToolbarItems.Add(drawTBI);
        ToolbarItem clearTBI = new ToolbarItem {
            Text = "Clear"
        };
        ToolbarItems.Add(clearTBI);
        _canvasV = new SKCanvasView();
        _canvasV.PaintSurface += HandlePaintCanvas;
        Grid mainG = new Grid();
        mainG.Children.Add(_canvasV, 0, 0);
        MR.Gestures.BoxView gestureV = new MR.Gestures.BoxView();
        mainG.Children.Add(gestureV, 0, 0);
        Content = mainG;
        //Load assets
        using (var stream = new SKManagedStream(ResourceLoader.GetEmbeddedResourceStream(this.GetType().GetTypeInfo().Assembly, "face.jpg"))) {
            _bitmap = SKBitmap.Decode(stream);
        }
        //Interaction
        gestureV.LongPressing += HandleLongPressed;
        gestureV.Panning += HandlePanning;
        gestureV.Panned += HandlePanned;
        gestureV.Pinching += HandlePinching;
        gestureV.Pinched += HandlePinched;
        //Bindings
        drawTBI.SetBinding(ToolbarItem.TextProperty, nameof(IsDrawMode), converter: new BoolDrawModeValueConverter());
        drawTBI.SetBinding(ToolbarItem.CommandProperty, nameof(DrawCommand));
        clearTBI.SetBinding(ToolbarItem.CommandProperty, nameof(ClearCommand));
    }

    private void HandlePaintCanvas(object sender, SKPaintSurfaceEventArgs e)
    {
        SKImageInfo info = e.Info;
        SKCanvas canvas = e.Surface.Canvas;
        canvas.Clear();
        if(!_bitmapRect.HasValue)
            _bitmapRect = CalculateBitmapAspectRect(_bitmap);
        canvas.SetMatrix(_m);
        canvas.DrawBitmap(_bitmap, _bitmapRect.Value);
        using (SKPaint p = new SKPaint { IsAntialias = true, StrokeWidth = 5, StrokeCap = SKStrokeCap.Round, Color = SKColors.Red, Style = SKPaintStyle.Stroke }) {
            canvas.DrawRect(_bitmapRect.Value, p);
            SKPath canvasPath = FromPercentPath(_sketchPath);
            canvas.DrawPath(canvasPath, p);
        }
    }

    protected override void OnSizeAllocated(double width, double height)
    {
        base.OnSizeAllocated(width, height);
        _bitmapRect = null;
    }

    private SKRect CalculateBitmapAspectRect(SKBitmap bitmap)
    {
        if (bitmap == null)
            return (SKRect.Empty);
        SKSize imgSize = new SKSize(bitmap.Width, bitmap.Height);
        return (SKRect.Create(_canvasV.CanvasSize.Width, _canvasV.CanvasSize.Height).AspectFit(imgSize));
    }

    private SKPath FromPercentPath(SKPath percentPath)
    {
        SKMatrix destM = SKMatrix.MakeIdentity();
        SKMatrix.PostConcat(ref destM, SKMatrix.MakeScale(_bitmapRect.Value.Width, _bitmapRect.Value.Height));
        SKMatrix.PostConcat(ref destM, SKMatrix.MakeTranslation(_bitmapRect.Value.Left, _bitmapRect.Value.Top));
        SKPath retVal = new SKPath(percentPath);
        retVal.Transform(destM);
        //SKPath retVal = new SKPath();
        //for (int i = 0; i < percentPath.PointCount; i++) {
        //    SKPoint percentPt = percentPath.GetPoint(i);
        //    float canvasX = percentPt.X * _bitmapRect.Value.Width + _bitmapRect.Value.Left;
        //    float canvasY = percentPt.Y * _bitmapRect.Value.Height + _bitmapRect.Value.Top;
        //    SKPoint canvasPt = new SKPoint(canvasX, canvasY);
        //    if (i == 0)
        //        retVal.MoveTo(canvasPt);
        //    else
        //        retVal.LineTo(canvasPt);
        //}
        return (retVal);
    }

    private void HandleLongPressed(object sender, MR.Gestures.LongPressEventArgs e)
    {
        _m = SKMatrix.MakeIdentity();
        _im = SKMatrix.MakeIdentity();
        _canvasV.InvalidateSurface();
    }

    private void HandlePanning(object sender, MR.Gestures.PanEventArgs pea)
    {
        if(IsDrawMode) {
            float drawX = (float)pea.Center.X;
            float drawY = (float)pea.Center.Y;
            SKPoint drawPt = ToUntransformedCanvasPt(drawX, drawY);
            SKPoint canvasPt = _im.MapPoint(drawPt);
            //Convert to percentages for storage
            float percentX = (canvasPt.X - _bitmapRect.Value.Left) / _bitmapRect.Value.Width;
            float percentY = (canvasPt.Y - _bitmapRect.Value.Top) / _bitmapRect.Value.Height;
            SKPoint percentPt = new SKPoint(percentX, percentY);
            if (!_isDrawing) {
                _sketchPath.MoveTo(percentPt);
                _isDrawing = true;
            } else {
                _sketchPath.LineTo(percentPt);
            }
        } else {
            float deltaX = (float)pea.DeltaDistance.X;
            float deltaY = (float)pea.DeltaDistance.Y;
            SKPoint deltaTran = ToUntransformedCanvasPt(deltaX, deltaY);
            SKMatrix deltaM = SKMatrix.MakeTranslation(deltaTran.X, deltaTran.Y);
            SKMatrix.PostConcat(ref _m, deltaM);
        }
        _canvasV.InvalidateSurface();
    }

    private void HandlePanned(object sender, MR.Gestures.PanEventArgs pea)
    {
        if(IsDrawMode)
            _isDrawing = false;
        else
            _m.TryInvert(out _im);
    }

    private void HandlePinching(object sender, MR.Gestures.PinchEventArgs pea)
    {
        float pivotX = (float)pea.Center.X;
        float pivotY = (float)pea.Center.Y;
        float deltaScale = (float)pea.DeltaScale;
        SKPoint pivotPt = ToUntransformedCanvasPt(pivotX, pivotY);
        SKMatrix deltaM = SKMatrix.MakeScale(deltaScale, deltaScale, pivotPt.X, pivotPt.Y);
        SKMatrix.PostConcat(ref _m, deltaM);
        _canvasV.InvalidateSurface();
    }

    private void HandlePinched(object sender, MR.Gestures.PinchEventArgs pea)
    {
        _m.TryInvert(out _im);
    }

    private SKPoint ToUntransformedCanvasPt(float x, float y)
    {
        return (new SKPoint(x * _canvasV.CanvasSize.Width / (float)_canvasV.Width, y * _canvasV.CanvasSize.Height / (float)_canvasV.Height));
    }

    private void DoDrawCommand()
    {
        IsDrawMode = !IsDrawMode;
    }

    private void DoClearCommand()
    {
        _sketchPath.Reset();
        _canvasV.InvalidateSurface();
    }
}

jezh · December 2018

@Basserd
Have you got the solution ? If yes ,could you please share it with us?

Xamarin

SkiaSharp draw on image -> handle orientation change

Answers

Xamarin

Quick Links

SkiaSharp draw on image -> handle orientation change

Answers