Initializing Handtracker

Install the Ur package and import it to ^ConjureKitManager^.

using Auki.Ur;

Also import ^ARFoundation^ to be able to use the ^ARRaycastManager^.

using UnityEngine.XR.ARFoundation;

Create a private ^HandTracker^ variable, as well as serializable ^ARSession^ and ^ARRaycastManager^ variables.

private HandTracker _handTracker;

[SerializeField] private ARSession arSession;
[SerializeField] private ARRaycastManager arRaycastManager;

Attach the ^ARRaycastManager^ component to AR Session Origin GameObject. Then drag the ^ARSession^ and the ^ARRaycastManager^ components to coressponding fields on the ^ConjureKitManager^ GameObject.

Get the ^HandTracker^ instance and initialize the AR system in ^ConjureKitManager^'s ^Start()^ function.

_handTracker = HandTracker.GetInstance();
_handTracker.SetARSystem(arSession, arCamera, arRaycastManager);

Start the ^HandTracker^ by calling

_handTracker.Start();

Call ^_handTracker.Update()^ every frame to continuously track the hand while moving.

private void Update()
{
    _handTracker.Update();
}

Hand Interaction

Now we want to position a sphere with a collider on our hand's index fingertip so it can interact with the cube.

To begin, create a 3d sphere, rename it to FingertipLandmark, and scale it down to 0.3. On the collider component, tick the ^isTrigger^ checkbox.

Create a new material, change its color to something more noticeable, and drag it to the sphere mesh renderer.

Add a new tag in Project Settings -> Tags and Layers named hand or any other name you choose and add this tag to the FingertipLandmark we just created.

Create a ^Renderer^ variable for the FingertipLandmark.

[SerializeField] private Renderer Fingertip Landmark;

Populate it with the sphere we just created. And in Start function, set the fingertip landmark as a child of our camera transform.

fingertipLandmark.transform.SetParent(arCamera.transform);

To get triggers from other colliders, the cube should have a ^Rigidbody^ component. Add it and tick the ^Is Kinematic^ checkbox to make sure the cube doesn't fall.

Create a new C# script called ^TouchableByHand.cs^ that will handle trigger events on the cube. If the cube is triggered with an object tagged with hand its color will change to a random color. If the trigger exits the cube, it will return to white.

public class TouchableByHand : MonoBehaviour
{
    private void OnTriggerEnter(Collider other)
    {
        if (other.tag == "hand")
        {
            gameObject.GetComponent<Renderer>().material.color = Random.ColorHSV();
        }
    }

    private void OnTriggerExit(Collider other)
    {
        if (other.tag == "hand")
        {
            gameObject.GetComponent<Renderer>().material.color = Color.white;
        }
    }
}

Add this script to the cube prefab.

To get our landmark positions in real-time, we will use the Ur callback - ^OnUpdate^ that is invoked when a new hand pose is received. This callback will pass 4 data types:

1. landmarks: landmark positions relative to the hand position
2. translations: the positions of each hand in camera space
3. isRightHand: indicates whether the right or left hand was detected
4. score: is the hand confidence score. If 0, no hand was detected.

The landmark and translation arrays contain consecutive floats representing the x, y & z-components of a 3D vector.

Once we get the landmarks and translations, we can place the fingertip landmark on landmark 8, the tip of the index finger (see the Ur documentation for a diagram of all hand landmarks). If the hand tracker detects a hand, we should see the fingertip landmark. If not, meaning our hand is not in camera sight, we can disable the fingertip landmark renderer.

_handTracker.OnUpdate += (landmarks, translations, isRightHand, score) =>
{
    if (score[0] > 0)
    {
        var handPosition = new Vector3(
            translations[0],
            translations[1],
            translations[2]);

        var pointerLandmarkIndex = 8 * 3; // Index fingertip
        var pointerLandMarkPosition = new Vector3(
            landmarks[pointerLandmarkIndex + 0],
            landmarks[pointerLandmarkIndex + 1],
            landmarks[pointerLandmarkIndex + 2]);

        fingertipLandmark.enabled = true;
        fingertipLandmark.transform.localPosition = handPosition + pointerLandMarkPosition;
    }
    else
    {
        fingertipLandmark.enabled = false;
    }
};

Visualization of Hand Landmarks

The Ur package allows us to visualize the hand landmarks in two simple steps.

Start by creating a private boolean.

landmarksVisualizeBool = true;

Then create a toggle method that uses the hand tracker methods ^ShowHandMesh^ and ^HideHandMesh^.

public void ToggleHandLandmarks()
{
    landmarksVisualizeBool = !landmarksVisualizeBool;

    if (landmarksVisualizeBool)
    {
        _handTracker.ShowHandMesh();
    }
    else
    {
        _handTracker.HideHandMesh();
    }
}

Now it can be toggled using a UI toggle or any other method you choose.