In this chapter we'll cover how to get started with using and visualizing Ur for hand tracking.
Install the Ur package and import it to ^ConjureKitManager^
.
using Auki.Ur;
Also import ^ARFoundation^
to be able to use the ^ARRaycastManager^
.
using UnityEngine.XR.ARFoundation;
Create a private ^HandTracker^
variable, as well as serializable ^ARSession
^ and ^ARRaycastManager^
variables.
private HandTracker _handTracker;
[SerializeField] private ARSession arSession;
[SerializeField] private ARRaycastManager arRaycastManager;
Attach the ^ARRaycastManager^
component to AR Session Origin GameObject. Then drag the ^ARSession^
and the ^ARRaycastManager^
components to coressponding fields on the ^ConjureKitManager^
GameObject.
Get the ^HandTracker^
instance and initialize the AR system in ^ConjureKitManager^
's ^Start()^
function.
_handTracker = HandTracker.GetInstance();
_handTracker.SetARSystem(arSession, arCamera, arRaycastManager);
Start the ^HandTracker^
by calling
_handTracker.Start();
Call ^_handTracker.Update()^
every frame to continuously track the hand while moving.
private void Update()
{
_handTracker.Update();
}
Now we want to position a sphere with a collider on our hand's index fingertip so it can interact with the cube.
To begin, create a 3d sphere, rename it to FingertipLandmark, and scale it down to 0.3. On the collider component, tick the ^isTrigger^
checkbox.
Create a new material, change its color to something more noticeable, and drag it to the sphere mesh renderer.
Add a new tag in Project Settings -> Tags and Layers named hand
or any other name you choose and add this tag to the FingertipLandmark we just created.
Create a ^Renderer^
variable for the FingertipLandmark.
[SerializeField] private Renderer Fingertip Landmark;
Populate it with the sphere we just created. And in Start function, set the fingertip landmark as a child of our camera transform.
fingertipLandmark.transform.SetParent(arCamera.transform);
To get triggers from other colliders, the cube should have a ^Rigidbody^
component. Add it and tick the ^Is Kinematic^
checkbox to make sure the cube doesn't fall.
Create a new C# script called ^TouchableByHand.cs^ that will handle trigger events on the cube. If the cube is triggered with an object tagged with hand
its color will change to a random color. If the trigger exits the cube, it will return to white.
public class TouchableByHand : MonoBehaviour
{
private void OnTriggerEnter(Collider other)
{
if (other.tag == "hand")
{
gameObject.GetComponent<Renderer>().material.color = Random.ColorHSV();
}
}
private void OnTriggerExit(Collider other)
{
if (other.tag == "hand")
{
gameObject.GetComponent<Renderer>().material.color = Color.white;
}
}
}
Add this script to the cube prefab.
To get our landmark positions in real-time, we will use the Ur callback - ^OnUpdate^ that is invoked when a new hand pose is received. This callback will pass 4 data types:
The landmark and translation arrays contain consecutive floats representing the x, y & z-components of a 3D vector.
Once we get the landmarks and translations, we can place the fingertip landmark on landmark 8, the tip of the index finger (see the Ur documentation for a diagram of all hand landmarks). If the hand tracker detects a hand, we should see the fingertip landmark. If not, meaning our hand is not in camera sight, we can disable the fingertip landmark renderer.
_handTracker.OnUpdate += (landmarks, translations, isRightHand, score) =>
{
if (score[0] > 0)
{
var handPosition = new Vector3(
translations[0],
translations[1],
translations[2]);
var pointerLandmarkIndex = 8 * 3; // Index fingertip
var pointerLandMarkPosition = new Vector3(
landmarks[pointerLandmarkIndex + 0],
landmarks[pointerLandmarkIndex + 1],
landmarks[pointerLandmarkIndex + 2]);
fingertipLandmark.enabled = true;
fingertipLandmark.transform.localPosition = handPosition + pointerLandMarkPosition;
}
else
{
fingertipLandmark.enabled = false;
}
};
The Ur package allows us to visualize the hand landmarks in two simple steps.
Start by creating a private boolean.
landmarksVisualizeBool = true;
Then create a toggle method that uses the hand tracker methods ^ShowHandMesh^
and ^HideHandMesh^
.
public void ToggleHandLandmarks()
{
landmarksVisualizeBool = !landmarksVisualizeBool;
if (landmarksVisualizeBool)
{
_handTracker.ShowHandMesh();
}
else
{
_handTracker.HideHandMesh();
}
}
Now it can be toggled using a UI toggle or any other method you choose.
Apply for a grant of AUKI tokens to get your project off the ground, and work directly with the Auki Labs team to get your creation to market. Successful applicants may be granted up to 100k USD worth of AUKI tokens, and development and marketing support from the Auki Labs team.