Hand gesture detection

In the Animator Controller, create a new Trigger parameter and name it after the action you want the pet to do when it detects a specific hand gesture.

Add this animation and make a transition to it from "Any State." In this transition, add a condition that checks if the Trigger parameter is true.

In ^Main.cs^, add these variables:

public bool hasPlayedDead = false;
private GameObject raccoonObject;
private Animator raccoonAnimator;

Add a new function that finds the spawned pet and triggers the animation:

public void PlayDead()
{
    hasPlayedDead = true;
    raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
    raccoonAnimator = raccoonObject.GetComponent<Animator>();
    raccoonAnimator.SetTrigger("PlayDead");
}

Create a new ^Vector3^ called ^handLandmarksPositions^:

private Vector3[] handLandmarksPositions;

Define it in the ^Start()^ function:

handLandmarksPositions = new Vector3[HandTracker.LandmarksCount];

In ^_handTracker.OnUpdate^, replace ^var landMarkPosition^ and ^landMarkPosition^ with ^handLandmarksPositions[l]^:

for (int l = 0; l < HandTracker.LandmarksCount; ++l)
{
    handLandmarksPositions[l] = new Vector3(
        landmarks[handLandmarkIndex + (l * 3) + 0],
        landmarks[handLandmarkIndex + (l * 3) + 1],
        landmarks[handLandmarkIndex + (l * 3) + 2]);

    // Update the landmarks position 
    _handLandmarks[l].transform.localPosition = handPosition + handLandmarksPositions[l];
}

In order to detect a gesture, measure the distances between relevant hand landmarks and call the function that triggers the animation when conditions for that hand gesture are met:

var indexPalmDistance = Vector3.Distance(handLandmarksPositions[8], handLandmarksPositions[0]);
var middlePalmDistance = Vector3.Distance(handLandmarksPositions[12], handLandmarksPositions[0]);
var ringPalmDistance = Vector3.Distance(handLandmarksPositions[16], handLandmarksPositions[0]);
var pinkyPalmDistance = Vector3.Distance(handLandmarksPositions[20], handLandmarksPositions[0]);

if (indexPalmDistance > 0.1f && middlePalmDistance < 0.08f && ringPalmDistance < 0.08f && pinkyPalmDistance < 0.08f)
{
    if (!hasPlayedDead)
    {
        PlayDead();
    }
}

Now when you build and run the project, the animation should be triggered when the correct hand gesture is detected.

Hand interaction

To have the pet perform a different animation when the hand interacts with it, add a second Trigger to the Animator Controller and name it after the action. Like before, add this animation and make a transition to it from "Any State." In this transition, add a condition that checks if the second Trigger parameter is true.

In ^Main.cs^, create a Renderer called ^FingertipLandmark^. This will be used to detect when the hand touches the pet.

[SerializeField] private Renderer fingertipLandmark;

In ^_handTracker.OnUpdate^, set ^fingertipLandmark^’s position to be at the tip of the index finger:

fingertipLandmark.transform.localPosition = handPosition + handLandmarksPositions[8];

Below ^_handTracker.OnUpdate^, set ^fingertipLandmark^’s parent to be ^arCamera.transform^:

fingertipLandmark.transform.parent = arCamera.transform;

Now create a second function that triggers the second animation:

public void GetUp()
{
    hasPlayedDead = false;
    raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
    raccoonAnimator = raccoonObject.GetComponent<Animator>();
    raccoonAnimator.SetTrigger("GetUp");
}

In Unity, create a sphere GameObject which will be the fingertip landmark. Scale it down to 5cm and check “Is Trigger” in the collider component. Drag this fingertip landmark sphere into its field in the Main GameObject.

In the digital pet prefab, add a Rigidbody component so it can get triggers from other colliders, and check “Is Kinematic”.

Now create a new C# script to handle trigger events on the digital pet. Give it a name, for example ^TouchableByHand.cs^. Make sure it’s in the same namespace as ^Main.cs^ and delete the ^Start()^ and ^Update()^ functions.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace AukiHandTrackerSample
{
    public class TouchableByHand : MonoBehaviour
    {
        
    }
}

Create the following variables:

public Main mainScript;
private GameObject raccoonObject;
private Animator raccoonAnimator;

In the ^Awake() ^function, find object of type ^Main^ and assign it to ^mainScript^:

private void Awake()
{
    mainScript = FindObjectOfType<Main>();
}

Define the ^OnTriggerEnter()^ function which will trigger the second animation when the fingertip landmark touches the pet:

private void OnTriggerEnter(Collider other)
{
    if (mainScript == null)
    {
        mainScript = FindObjectOfType<Main>();
        if (mainScript == null)
        {
            Debug.LogError("Main script not found");
            return;
        }
    }

    if (mainScript.hasPlayedDead)
    {
        mainScript.GetUp();
    }
}

In Unity, add ^TouchableByHand.cs^ to the digital pet prefab.

Now when you build and run the project, the second animation should be triggered when the fingertip landmark touches the pet.

Finishing touches

Hide the hand landmarks by opening the hand landmark prefab and unchecking it. Similarly, hide the sphere on the index finger by unchecking its mesh renderer.

To add occlusion culling, go to the AR Camera and add an AR Occlusion Manager component. Set both of the Human Segmentation settings to "Fastest."

Lastly, remove the spawn button after the pet has spawned by adding this line at the end of the ^CreateRaccoon()^ function:

GameObject.Find("SpawnButton").SetActive(false);

That’s it! Now you have an AR pet that responds to hand gestures and hand interactions.