Hand gesture detection

In the Animator Controller, create a new Trigger parameter and name it after the action you want the pet to do when it detects a specific hand gesture.

Add this animation and make a transition to it from "Any State." In this transition, add a condition that checks if the Trigger parameter is true.

In Main.cs, add these variables:

public bool hasPlayedDead = false;
private GameObject raccoonObject;
private Animator raccoonAnimator;

Add a new function that finds the spawned pet and triggers the animation:

public void PlayDead()
{
    hasPlayedDead = true;
    raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
    raccoonAnimator = raccoonObject.GetComponent<Animator>();
    raccoonAnimator.SetTrigger("PlayDead");
}

Create a new Vector3 called handLandmarksPositions:

private Vector3[] handLandmarksPositions;

Define it in the Start() function:

handLandmarksPositions = new Vector3[HandTracker.LandmarksCount];

In _handTracker.OnUpdate, replace var landMarkPosition and landMarkPosition with handLandmarksPositions[l]:

for (int l = 0; l < HandTracker.LandmarksCount; ++l)
{
    handLandmarksPositions[l] = new Vector3(
        landmarks[handLandmarkIndex + (l * 3) + 0],
        landmarks[handLandmarkIndex + (l * 3) + 1],
        landmarks[handLandmarkIndex + (l * 3) + 2]);

    // Update the landmarks position 
    _handLandmarks[l].transform.localPosition = handPosition + handLandmarksPositions[l];
}

In order to detect a gesture, measure the distances between relevant hand landmarks and call the function that triggers the animation when conditions for that hand gesture are met:

var indexPalmDistance = Vector3.Distance(handLandmarksPositions[8], handLandmarksPositions[0]);
var middlePalmDistance = Vector3.Distance(handLandmarksPositions[12], handLandmarksPositions[0]);
var ringPalmDistance = Vector3.Distance(handLandmarksPositions[16], handLandmarksPositions[0]);
var pinkyPalmDistance = Vector3.Distance(handLandmarksPositions[20], handLandmarksPositions[0]);

if (indexPalmDistance > 0.1f && middlePalmDistance < 0.08f && ringPalmDistance < 0.08f && pinkyPalmDistance < 0.08f)
{
    if (!hasPlayedDead)
    {
        PlayDead();
    }
}

Now when you build and run the project, the animation should be triggered when the correct hand gesture is detected.

Hand interaction

To have the pet perform a different animation when the hand interacts with it, add a second Trigger to the Animator Controller and name it after the action. Like before, add this animation and make a transition to it from "Any State." In this transition, add a condition that checks if the second Trigger parameter is true.

In Main.cs, create a Renderer called FingertipLandmark. This will be used to detect when the hand touches the pet.

[SerializeField] private Renderer fingertipLandmark;

In _handTracker.OnUpdate, set fingertipLandmark’s position to be at the tip of the index finger:

fingertipLandmark.transform.localPosition = handPosition + handLandmarksPositions[8];

Below _handTracker.OnUpdate, set fingertipLandmark’s parent to be arCamera.transform:

fingertipLandmark.transform.parent = arCamera.transform;

Now create a second function that triggers the second animation:

public void GetUp()
{
    hasPlayedDead = false;
    raccoonObject = GameObject.Find("Raccoon Cub PA(Clone)");
    raccoonAnimator = raccoonObject.GetComponent<Animator>();
    raccoonAnimator.SetTrigger("GetUp");
}

In Unity, create a sphere GameObject which will be the fingertip landmark. Scale it down to 5cm and check “Is Trigger” in the collider component. Drag this fingertip landmark sphere into its field in the Main GameObject.

In the digital pet prefab, add a Rigidbody component so it can get triggers from other colliders, and check “Is Kinematic”.

Now create a new C# script to handle trigger events on the digital pet. Give it a name, for example TouchableByHand.cs. Make sure it’s in the same namespace as Main.cs and delete the Start() and Update() functions.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace AukiHandTrackerSample
{
    public class TouchableByHand : MonoBehaviour
    {
        
    }
}

Create the following variables:

public Main mainScript;
private GameObject raccoonObject;
private Animator raccoonAnimator;

In the Awake() function, find object of type Main and assign it to mainScript:

private void Awake()
{
    mainScript = FindObjectOfType<Main>();
}

Define the OnTriggerEnter() function which will trigger the second animation when the fingertip landmark touches the pet:

private void OnTriggerEnter(Collider other)
{
    if (mainScript == null)
    {
        mainScript = FindObjectOfType<Main>();
        if (mainScript == null)
        {
            Debug.LogError("Main script not found");
            return;
        }
    }

    if (mainScript.hasPlayedDead)
    {
        mainScript.GetUp();
    }
}

In Unity, add TouchableByHand.cs to the digital pet prefab.

Now when you build and run the project, the second animation should be triggered when the fingertip landmark touches the pet.

Finishing touches

Hide the hand landmarks by opening the hand landmark prefab and unchecking it. Similarly, hide the sphere on the index finger by unchecking its mesh renderer.

To add occlusion culling, go to the AR Camera and add an AR Occlusion Manager component. Set both of the Human Segmentation settings to "Fastest."

Lastly, remove the spawn button after the pet has spawned by adding this line at the end of the CreateRaccoon() function:

GameObject.Find("SpawnButton").SetActive(false);

That’s it! Now you have an AR pet that responds to hand gestures and hand interactions.