UI not responding to gaze and tap

timhsmith79

I must be missing something as I've been scouring the forums to no avail. I followed the instructions at: https://community.unity.com/t5/Hololens/Unity-UI-on-the-HoloLens/td-p/2597979

So here is the issue. Started a new project, removed the existing camera and added in the Camera from the HoloToolKit under Utilities. I then added the CursorWithFeedback prefab, created an empty game object called "Managers", and dumped in Gaze Manager, Gesture Manager, and Hands Manager.

I then follow the instructions noted in the above link, focusing on adding a single button. When clicked the button is to instantiate a cube (though I added billboard and tagalong to the Canvas). Test it in the Editor, works great with a mouse, but I get nothing with gaze and tap. I've got the latest HTP (5.4.0f3) and even tried in the latest Beta (5.5.0b4).

To test this out, I just added in the Cube I've been trying to instantiate, then add in the SpatialMapping prefab and add TapToPlace on my Cube. All works fine. I can select my cube and place it on the spatial mesh created via the hololens in my room, so not sure why I can't use the UI button.

I even tried adding a box collider to the buttons as other have mentioned, but nada. any help would be greatly appreciated.

ahillier

Hi @timhsmith79,
I have only used the GUI controls a few times, so there is probably a better way of accomplishing this, but here is what I did to get your scenario to work:

1) Set-up layers.
I added the billboard and tagalong script to the Canvas object. I then created a new layer and called it 'TagAlong'. I set the Canvas object (but not it's children) to be on the 'TagAlong' layer. I then set the 'Raycast Layer Mask' on the GazeManager to ignore the 'TagAlong' layer. I kept the button on the 'UI' layer and made sure that the GazeManager would still respond to that layer.

2) Create/rescale colliders.
When you add the 'TagAlong' component to your canvas, it will automatically add a box collider to the object. However, this collider is usually too big or too small. I rescaled the collider (larger collider in the image below) until it fit around the UI elements that I always want visible in the scene. I also added a box collider to my Button control (smaller collider in the image below) and then rescaled the collider until it fit around the button (it started super-tiny, but my final scale matched the Width/Height values of the button itself).

3) Respond to messages from the GazeManager (OnGazeEnter/Leave) and GestureManager (OnSelect).
I created this basic script for responding to button input. It will change color when focused and will spawn an object (whatever prefab is provided) where the user is looking when they press the button. I hooked up the 'OnSelect' function from this script to the OnClick() event on the 'Button' script:

using UnityEngine;
using UnityEngine.UI;
using HoloToolkit.Unity;

public class ButtonBehavior : MonoBehaviour
{
    public GameObject objectToSpawn;

    public void OnGazeEnter()
    {
        GetComponent<Button>().OnPointerEnter(null);
    }

    public void OnGazeLeave()
    {
        GetComponent<Button>().OnPointerExit(null);
    }

    public void OnSelect()
    {
        if (objectToSpawn != null)
        {
            GameObject.Instantiate(objectToSpawn, GazeManager.Instance.Position, Quaternion.Euler(Vector3.zero));
        }
    }
}

Here is what the Inspector looks like for my canvas:

Here is what the Inspector looks like for my button:

I hope this helps,
~Angela

timhsmith79

I did just review the color picker example in the HoloToolkit, which seems to use a game object as the item being clicked on in the Canvas hierarchy. I haven't stumbled upon an example yet that actually shows gesture interaction with UI elements.

ahillier

Hi @timhsmith79,
I have only used the GUI controls a few times, so there is probably a better way of accomplishing this, but here is what I did to get your scenario to work:

1) Set-up layers.
I added the billboard and tagalong script to the Canvas object. I then created a new layer and called it 'TagAlong'. I set the Canvas object (but not it's children) to be on the 'TagAlong' layer. I then set the 'Raycast Layer Mask' on the GazeManager to ignore the 'TagAlong' layer. I kept the button on the 'UI' layer and made sure that the GazeManager would still respond to that layer.

2) Create/rescale colliders.
When you add the 'TagAlong' component to your canvas, it will automatically add a box collider to the object. However, this collider is usually too big or too small. I rescaled the collider (larger collider in the image below) until it fit around the UI elements that I always want visible in the scene. I also added a box collider to my Button control (smaller collider in the image below) and then rescaled the collider until it fit around the button (it started super-tiny, but my final scale matched the Width/Height values of the button itself).

3) Respond to messages from the GazeManager (OnGazeEnter/Leave) and GestureManager (OnSelect).
I created this basic script for responding to button input. It will change color when focused and will spawn an object (whatever prefab is provided) where the user is looking when they press the button. I hooked up the 'OnSelect' function from this script to the OnClick() event on the 'Button' script:

using UnityEngine;
using UnityEngine.UI;
using HoloToolkit.Unity;

public class ButtonBehavior : MonoBehaviour
{
    public GameObject objectToSpawn;

    public void OnGazeEnter()
    {
        GetComponent<Button>().OnPointerEnter(null);
    }

    public void OnGazeLeave()
    {
        GetComponent<Button>().OnPointerExit(null);
    }

    public void OnSelect()
    {
        if (objectToSpawn != null)
        {
            GameObject.Instantiate(objectToSpawn, GazeManager.Instance.Position, Quaternion.Euler(Vector3.zero));
        }
    }
}

Here is what the Inspector looks like for my canvas:

Here is what the Inspector looks like for my button:

I hope this helps,
~Angela

Jesse_McCulloch

Try adding the "HoloLens Input Module" to the EventSystem object that is created when you add the UI component....

timhsmith79

@ahillier,

Thank you for your detailed instructions above. That did it! Worked great. I think the only thing I didn't do in my testing was write a new script to call onSelect(). I maybe foolishly expected it would "just work", as it does with mouse input.

@Jesse_McCulloch,

Your suggestion is in the instructions for the link I provided above. So I tried that in my initial testing with no success. So if the "Hololens Input Module" is supposed to make that functionality work, It did not in my case.

Thanks,
Tim