Forum Xamarin.iOS

Problem with Custom Render Frame on Arkit

Hi all,
i have a problem in public override void DidUpdateFrame(ARSession session, ARFrame frame) method.
My goal is to draw grid over the frame that arkit give me.

my code:

public override void DidUpdateFrame(ARSession session, ARFrame frame) {

     using (var s = session) {
        using (var frm = frame) {
           try {

              using (var currentBackgroundFrameImage = s.CurrentFrame?.CapturedImage) {

                 using (var i = Convert22(currentBackgroundFrameImage, frm)) {
                    using (CIContext temporaryContext = CIContext.FromOptions(null)) {
                       CGImage cgImage = temporaryContext.CreateCGImage(i, new CGRect(0, 0, i.Extent.Width, i.Extent.Height));
                       using (UIImage ni = UIImage.FromImage(cgImage)) {
                          var context = UIGraphics.GetCurrentContext();
                          ni.Draw(new CGPoint());
                          context.MoveTo(0, 0);
                          context.AddLineToPoint(ni.Size.Width, ni.Size.Height / 2);
                          context.SetStrokeColor(new CGColor(255, 0, 0));
                          var newImage = UIGraphics.GetImageFromCurrentImageContext();
                          this.control.SceneView.Scene.Background.Contents = newImage;
           catch (Exception e) {

Well, my convert method is used to create and apply transform to crop and scale correctly the image from PixelBuffer:

public CIImage Convert22(CVPixelBuffer pixelBuffer, ARFrame frame) {
     var imageSize = new CGSize(pixelBuffer.Width, pixelBuffer.Height);
     var sizT = this.control.SceneView.CurrentViewport;
     var viewport = this.control.SceneView.Bounds;
     var viewportSize = viewport.Size;
     var interfaceOrientation = this.control.SceneView.Window.WindowScene.InterfaceOrientation;
     var image = new CIImage(pixelBuffer);
     var normalizeTransform = CGAffineTransform.MakeScale(1.0f / imageSize.Width, 1.0f / imageSize.Height);

     var portraitScale = CGAffineTransform.MakeScale(-1.0f, -1.0f);

     portraitScale = CGAffineTransform.Translate(portraitScale, -1, -1);
     var flipTransform = interfaceOrientation.IsPortrait() ? portraitScale : CGAffineTransform.MakeIdentity();

     var displayTransform = frame.GetDisplayTransform(interfaceOrientation, viewportSize);

     var toViewPortTransform = CGAffineTransform.MakeScale(viewportSize.Width, viewportSize.Height);
     var finalT = normalizeTransform * flipTransform * displayTransform * toViewPortTransform;
     return image.ImageByApplyingTransform(finalT).ImageByCroppingToRect(viewport);

Application run...but after few seconds is blocked.. Why?
I have found into the Iphone log this "Error":

this error appears when i try to use pixel buffer data into ** var image = new CIImage(pixelBuffer);** but no exception handled....
Personally i don't know how to solve problem.

CIImage is basic to apply transform to obtain the correct UIImage cropped for Viewport. :neutral:


Sign In or Register to comment.