Processing real-time camera frames with SkiaSharp

I'm trying to come up with some cross-platform logic to process video frames in real-time. I already have a platform specific implementation working on UWP using Win2D; I'm trying to see if I can now replicate that (or do something similar) for Android using SkiaSharp. If this works, I can do away with a Win2D approach and hopefully engineer something that works across all platforms.

To illustrate, I'm trying to create something that would be equivalent to what is done in UWP video processing using the IBasicVideoEffect interface. See the below real-time effect as an example:

public void ProcessFrame(ProcessVideoFrameContext context)
{
using (CanvasBitmap inputBitmap =
CanvasBitmap.CreateFromDirect3D11Surface(_canvasDevice, context.InputFrame.Direct3DSurface))
using (CanvasRenderTarget renderTarget =
CanvasRenderTarget.CreateFromDirect3D11Surface(_canvasDevice, context.OutputFrame.Direct3DSurface))
using (CanvasDrawingSession ds = renderTarget.CreateDrawingSession())
{
_invertEffect.Source = inputBitmap;
ds.DrawImage(_invertEffect);
}
}

In my application, I want to be able to apply filter effects similar to invert and also draw on the frame. The output of this activity needs to be available both for the live preview and any recorded output such as video recordings or photos just like what IBasicVideoEffect enables in UWP.

I think the way to do this is to create an SKSurface from an Android SurfaceTexture. Camera samples seem to use TextureView and I can see that SkiaSharp 1.60.0 backs a SKGlView with TextureView on Android as well. I believe to be able to create such an SKSurface, I need to create/acquire a GRContext from the TextureView and then create a suitable GRBackendRenderTargetDesc. However, I can't seem to figure out how to create the GRBackendRenderTargetDesc and I'm not sure that the way I create the GRContext is correct:

var grContext = GRContext.Create(GRBackend.OpenGL, GRGlInterface.CreateNativeGlInterface());

I am hoping that if I am able to create an SKSurface from the TextureView, after hooking into the FrameAvailable event of the Android SurfaceTexture, I'll be able to apply filters to and draw on the frames using SkiaSharp.

I would really appreciate comments, suggestions and pointers on the following:

  • Does this appear to be a viable approach for conducting real-time frame processing? If not, what approach would you recommend?
  • How can I create an SKSurface from the Android TextureView / SurfaceTexture, etc?
  • Any samples and instructions regarding GRBackendRenderTargetDesc and GRContext would be most welcome.

Thanks for any help.

Answers

Sign In or Register to comment.