Finding UI Element on a given X,Y coordinates

I am working on a Xamarin form with a grid. There are absolute layouts containing labels in the grid. I have to move labels from one grid to another. The drag function is working properly, however, I want to find the drop location based on a set of (X,Y) coordinates. I have tried various methods but no luck so far. I have added my code below. Can anyone please check it and let me know how can I find the UI control which is grid and its children based on the X and Y coordinates. The attached image shows the label in the first row and column of the "gridTest" grid layout is being dragged and dropped in the first column of the "gridDrop" grid layout (check the attached screenshots for more details). Additionally, as the dragged label for some reasons is not being dropped so I created a new label and dropped in the second grid.

GridDragDrop.xamlfirst
second

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"
xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
xmlns:d="http://xamarin.com/schemas/2014/forms/design"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d"
x:Class="TouchTrackingEffect.GridDragDrop">
<ContentPage.Content>
<AbsoluteLayout x:Name="absMain" BackgroundColor="Beige"><!--YProportional,WidthProportional,HeightProportional-->
<!--Main Grid-->
<Grid x:Name="gridTest" AbsoluteLayout.LayoutBounds=".1,0,.5,.5" AbsoluteLayout.LayoutFlags="All">
<Grid.RowDefinitions>
<RowDefinition Height="100"></RowDefinition>
<RowDefinition Height="100"></RowDefinition>
<RowDefinition Height="100"></RowDefinition>
<RowDefinition Height="100"></RowDefinition>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*"></ColumnDefinition>
<ColumnDefinition Width="*"></ColumnDefinition>
<ColumnDefinition Width="*"></ColumnDefinition>
<ColumnDefinition Width="*"></ColumnDefinition>
<ColumnDefinition Width="*"></ColumnDefinition>
</Grid.ColumnDefinitions>
<!--Grid First Row-->
<AbsoluteLayout x:Name="AL1" Grid.Row="0" Grid.Column="0" BackgroundColor="Accent">
<Label x:Name="wordLabel" Text="Drag this label"></Label>
</AbsoluteLayout>
<AbsoluteLayout  x:Name="AL2" Grid.Row="0" Grid.Column="1" BackgroundColor="AliceBlue"></AbsoluteLayout>
<AbsoluteLayout  x:Name="AL3" Grid.Row="0" Grid.Column="2" BackgroundColor="AntiqueWhite"></AbsoluteLayout>
<AbsoluteLayout  x:Name="AL4" Grid.Row="0" Grid.Column="3" BackgroundColor="Aqua"></AbsoluteLayout>
<AbsoluteLayout  x:Name="AL5" Grid.Row="0" Grid.Column="4" BackgroundColor="Aquamarine"></AbsoluteLayout>
<!--Grid Second Row-->
<AbsoluteLayout x:Name="AL31" Grid.Row="1" Grid.Column="0" BackgroundColor="Accent"></AbsoluteLayout>
<AbsoluteLayout  x:Name="AL32" Grid.Row="1" Grid.Column="1" BackgroundColor="AliceBlue">
<Label x:Name="passLbl" Text="another"></Label>
</AbsoluteLayout>
<AbsoluteLayout  x:Name="AL333" Grid.Row="1" Grid.Column="2" BackgroundColor="AntiqueWhite"></AbsoluteLayout>
<AbsoluteLayout  x:Name="AL34" Grid.Row="1" Grid.Column="3" BackgroundColor="Aqua"></AbsoluteLayout>
<AbsoluteLayout  x:Name="AL35" Grid.Row="1" Grid.Column="4" BackgroundColor="Aquamarine"></AbsoluteLayout>
<!--Grid Third Row-->
<!--Grid Fourth Row-->
<Label x:Name="descLabel" Grid.Row="3" Grid.Column="0" Grid.ColumnSpan="5" VerticalOptions="CenterAndExpand" HorizontalOptions="CenterAndExpand" />
</Grid>
<!--Drop Grid-->
<Grid x:Name="gridDrop" AbsoluteLayout.LayoutBounds=".9,.8,.4,.4" AbsoluteLayout.LayoutFlags="All" BackgroundColor="Azure">
<Grid.RowDefinitions>
<RowDefinition Height="100"></RowDefinition>
<RowDefinition Height="100"></RowDefinition>
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*"></ColumnDefinition>
<ColumnDefinition Width="*"></ColumnDefinition>
</Grid.ColumnDefinitions>
</Grid>
<AbsoluteLayout x:Name="wordAbsLayout" AbsoluteLayout.LayoutBounds="1,0,.3,.4" AbsoluteLayout.LayoutFlags="All" BackgroundColor="Bisque">
<Label x:Name="word1Label" Text="Hat" AbsoluteLayout.LayoutBounds="10,10,25,50" VerticalOptions="CenterAndExpand" HorizontalOptions="CenterAndExpand" />
<Label x:Name="word2Label" Text="Cat" AbsoluteLayout.LayoutBounds="50,10,25,50" VerticalOptions="CenterAndExpand" HorizontalOptions="CenterAndExpand" />
<Label x:Name="word3Label" Text="Mat" AbsoluteLayout.LayoutBounds="100,10,25,50" VerticalOptions="CenterAndExpand" HorizontalOptions="CenterAndExpand" />
</AbsoluteLayout>
</AbsoluteLayout>
</ContentPage.Content>
</ContentPage>

GridDragDrop.xaml.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using TouchTracking;
using Xamarin.Forms;
using Xamarin.Forms.Xaml;

namespace TouchTrackingEffect
{
    [XamlCompilation(XamlCompilationOptions.Compile)]
    public partial class GridDragDrop : ContentPage
    {
        class DragInfo
        {
            public DragInfo(long id, Point pressPoint)
            {
                Id = id;
                PressPoint = pressPoint;
            }

            public long Id { private set; get; }


            public Point PressPoint { private set; get; }

        }
        // dictionary for Label
        Dictionary<Label, DragInfo> lblDragDictionary = new Dictionary<Label, DragInfo>();

        Random random = new Random();

        public GridDragDrop(String bagToken)
        {
            InitializeComponent();

            // adding touch effects to first label
            TouchEffect touchEffect = new TouchEffect();
            touchEffect.TouchAction += OnTouchEffectAction;
            wordLabel.Effects.Add(touchEffect);
        }

        void OnTouchEffectAction(object sender, TouchActionEventArgs args)
        {

            word1Label = sender as Label;
            switch (args.Type)
            {
                case TouchActionType.Pressed:
                    // Don't allow a second touch on an already touched BoxView
                    if (!lblDragDictionary.ContainsKey(word1Label))
                    {
                        lblDragDictionary.Add(word1Label, new DragInfo(args.Id, args.Location));

                        // Set Capture property to true
                        TouchEffect touchEffect = (TouchEffect)word1Label.Effects.FirstOrDefault(e => e is TouchEffect);
                        touchEffect.Capture = true;
                    }
                    break;

                case TouchActionType.Moved:
                    if (lblDragDictionary.ContainsKey(word1Label) && lblDragDictionary[word1Label].Id == args.Id)
                    {
                        Rectangle rect = AbsoluteLayout.GetLayoutBounds(word1Label);
                        Point initialLocation = lblDragDictionary[word1Label].PressPoint;
                        rect.X += args.Location.X - initialLocation.X;
                        rect.Y += args.Location.Y - initialLocation.Y;
                        AbsoluteLayout.SetLayoutBounds(word1Label, rect);

                        descLabel.Text = " Rect X = " + rect.X + " Rect Y = " + rect.Y;
                    }
                    break;

                case TouchActionType.Released:
                    if (lblDragDictionary.ContainsKey(word1Label) && lblDragDictionary[word1Label].Id == args.Id)
                    {

                        descLabel.Text += gridTest.Bounds.Width + " " + Bounds.Height;

                        // create new label from word1Label and add to grid

                        Label lbl2 = new Label { Text = word1Label.Text};

                        lblDragDictionary.Remove(word1Label);

                        gridDrop.Children.Add(lbl2, 0, 1); //(label, col, row)

                    }
                    break;
            }
        }
    }
}

Answers

  • JarvanJarvan Member, Xamarin Team Xamurai

    Try to use AutomationElement.FromPoint(Point) method to retrieve the control like below. FromPoint returns the element in the logical tree that is closest to the root element.

    private AutomationElement ElementFromCursor()
    {
        // Convert mouse position from System.Drawing.Point to System.Windows.Point.
        System.Windows.Point point = new System.Windows.Point(Cursor.Position.X, Cursor.Position.Y);
        AutomationElement element = AutomationElement.FromPoint(point);
        return element;
    }
    

    Tutorial:
    https://docs.microsoft.com/en-us/dotnet/api/system.windows.automation.automationelement.frompoint?view=netframework-4.8

  • junejo_aishajunejo_aisha Member ✭✭

    @Jarvan Many thanks for your reply. How can I use this method in Xamarin forms because when I put this code in the page. I am getting several errors.

  • JarvanJarvan Member, Xamarin Team Xamurai
    edited September 11

    Sorry for my mistake, the System.Windows.Automation provides support for Windows Presentation Foundation (WPF) UI Automation clients.

    To get UI Element on coordinates, you may need to get the coordinates of the controls on page and make a comparison with the given coordinates.
    Tutorial:
    https://www.andrewhoefling.com/Blog/Post/the-coordinate-system-in-xamarin-forms-and-android
    https://forums.xamarin.com/discussion/66386/how-to-get-the-coordinates-where-there-is-a-control-on-the-screen

  • junejo_aishajunejo_aisha Member ✭✭

    @Jarvan actually I have checked these tutorials before but I do not know for some reason I am not getting the screen coordinates after translation. For example, if I translate the y-position of a label, it is indeed moving but not giving me the new coordinates after translation. How can I achieve that?

    private void TapGestureRecognizer_Tapped(object sender, EventArgs e)
    {
    lblNew1.TranslationY -= 25;
    Rectangle rectTrans = lblNew1.Bounds;
    hdrLbl.Text += "\n new bounds =" + rectTrans.X + rectTrans.Y;
    hdrLbl.Text += ScreenCoords.GetScreenCoordinates(lblNew1);

        }
    

    public static (double X, double Y) GetScreenCoordinates(this VisualElement view)
    {
    // A view's default X- and Y-coordinates are LOCAL with respect to the boundaries of its parent,
    // and NOT with respect to the screen. This method calculates the SCREEN coordinates of a view.
    // The coordinates returned refer to the top left corner of the view.
    var screenCoordinateX = view.X;
    var screenCoordinateY = view.Y;

            var parent = (VisualElement)view.Parent;
            while (parent != null && parent.GetType().BaseType == typeof(View))
            {
                screenCoordinateX += parent.X;
                screenCoordinateY += parent.Y;
                parent = (VisualElement)parent.Parent;
            }
            return (screenCoordinateX, screenCoordinateY);
        }
    

Sign In or Register to comment.