Moving holograms using gaze and gesture

felsiska

I've been trying to move holograms using gaze and gesture:

• airtap object
• gaze to move object
• airtap again to place object

I followed the instructions posted earlier on but still could not get it to move.

By @HoloSheep :

After you have correctly copied the HoloToolkit into a new unity project and made all of the appropriate HoloLens project setting adjustments you would then do the following to get the gaze and gesture manager scripts working:

•add empty game object to root of your scene hierarchy (name it ModelBase)
•add 3D model (like a cube) as child of ModelBase
•add 3D box collider as component of ModelBase
•size 3D box collider to slightly larger than 3D model (in all dimensions) be sure to include the models origin inside 3D box collider area
•add Cursor prefab from HTK to root of your scene hierarchy
•add another empty game object to root of your screen hierarchy (name it Managers)
•add GazeManager script from HTK as component of Managers
•add GestureManager script from HTK as component of Managers
•add SpatialMapping prefab from HTK to root of hierarchy (turn off Draw Visual Meshes checkbox)
•add TapToPlace script as component to ModelBase

Build and deploy.
You should be able to gaze at your 3D object and Airtap on it to engage the spatial mapping and move the object around. The next Airtap should place your object where you were gazing at (on the spatial mapped surface) and the spatial mapping should turn off.

I've also made all appropriate setting adjustments.
Appreciate if anyone could help.

HoloSheep

@felsiska the origin point (aka pivot point, but useful to think of it as the origin of a 3D model in this discussion) of the 3D game objects and its relationship to the invisible skin of the collider is what is important here.

Since Unity isn't really a model editing tool (like 3DS Max, Maya or SketchUp etc..) it turns out that editing the pivot point of a 3D model is not an option in Unity (in the traditional sense of editing anyway).

Since something like a default 3D Cube in Unity has its origin point in the center, and since you can't edit that origin point, when you move that 3D cube (by its origin) it will get placed at whatever location you specify, but the origin will be at that location not the bottom of the cube as you might prefer in this case, so half of the cube is above the surface and half below.

This is one of the two common gotchas that people run into when using the TapToPlace approach to moving objects (which moves the object by its origin) both of which can be solved by using the wrapper empty gameobject parent approach to faking a new origin.

1st common gotcha: If the origin of the object is not on the corner or surface that you would like to lock onto the destination point the object does not end up sitting where you want it.

2nd common gotcha: If the origin point is located outside of the main 3D Object and you size your collider just around the main 3D Object the detection logic never fires and you can't move the 3D object. (as described in this post)

Cube with Center Origin

So for the 1st case what you need is a way to get the 3D object to act like its origin is somewhere on the bottom face, bottom edge or even the bottom corner of the cube. The way to achieve that is through the relationship between the empty gameobject (that we called ModelBase) that parents the cube and the cube itself.

Both the parent and the child have their own Transform positions. The Parent is relative to the world origin (in this case the camera position) and the child is relative to the parent. So, in the case of an empty gameobject that has no width or height of its own, you can think of it pretty much as a point in space and as it turns out the origin of an empty gameobject is in exactly the middle of that same point. Since I have choosen to position the parent at X=0 Y=0 Z=2 (so that it is 2 meters in front of the camera to start, its origin is exactly at the same point in space.

So if I put a cube that is 1 unit wide inside and empty game object parent. And then I move the child cube in the x direction by half its width and in the y direction by half its height and the z direction by half its depth... the corner of the cube ends up at the origin point of the parent.

Parent's details (displaying the parent's origin)

Child's details (displaying the child's local origin)

The end result is that the parent becomes a wrapper for the child object. If we place our colliders and scripts (like the TapToPlace script) on the parent "ModelBase" (which you did above, and this is one of the reasons why) then we are really moving that wrapper or container around and since it is really just a point in space... we can position the child game object anywhere we like relative to that point in space. That point in space then effectively becomes the child game objects new origin (in an abstract but effective kind of way).

Does that help?

DavidKlineMS

@felsiska,
You can hide the bounding cube by commenting out the following line in the Update method in Placeable.cs:

DisplayBounds(canBePlaced);

Thanks!
David

HoloSheep

@felsiska please describe what you are seeing and what is and isn't working for you.

I would be happy to try and help work you through it.

felsiska

the cursor is reacting to the cube surface correctly. but nothing happen when I airtap the cube. I've also attached cursor feedback to cursor so that I can see the change of cursor shape when I airtap.

is there any part of the script that I need to edit?

Joe_R

did u check the SpatialPerception? its need to be checked when you use spatial mapping

felsiska

Yes I've checked it. just realized that my GestureManager script is not updated. Its working now after updating them. will explore further.

Thx for the help

felsiska

• Using the TapToPlace script, how do I move the hologram while locking it to the surface mesh to prevent it from "falling over" the mesh?
• I understand that all objects with TapToPlace script within the same project will move at the same time when manipulation is detected. Any way to only move the object that is tapped at during manipulation?

HoloSheep

@felsiska

• not sure I understand what you are referring to when you say "falling over the mesh"

• The issue of individual single versus multi object response to existing TapToPlace script boils down to the object hierarchy that you place your objects in and the associated colliders. To get multiple object to move independently you could simply repeat the steps of:

•add empty game object to root of your scene hierarchy (name it ModelBase)
•add 3D model (like a cube) as child of ModelBase
•add 3D box collider as component of ModelBase
•size 3D box collider to slightly larger than 3D model (in all dimensions) be sure to include the models origin inside 3D box collider area

•add TapToPlace script as component to ModelBase

and use a unique name for each empty game object (ModelBase1, ModelBase2..) with each having its own child object (3D model)

Alternatively there are multiple options that are often more complex but involve prefabs and programmatic generation of objects that could also achieve the same results by creating the necessary arrangement of scripts, models and colliders.

felsiska

@HoloSheep

• regarding issues of objects "falling over the mesh" is similar to how an object go beyond and behind the spatial meshes.
• Thx for the advice. Managed to move them individually. however it seems that the center of collider is locked to the mesh instead of the corners. this will result in having the object "sinking" below ground level.
• I've re-used Placeable script from Holograms 230 and managed to lock the object to a surface during manipulation. but object will sink halfway into the surface mesh when placed. how do I change the script so that it will lock to the surface of collider and not the center?

HoloSheep

@felsiska ah, okay I see. Sounds like you are running into the issue of fixed model origin (aka pivot point).

See this thread and some of its reference links for a description of the issue and ways to work around it.

felsiska

@HoloSheep

I am currently only using unity 3d game objects for trial. tried to move and resize the collider so that the center touches the surface of object nicely, but it still sinks below mesh surfaces. is there a way to move the pivot point of the collider? or will it always lock to the center?

thx so much for helping btw. really appreciate it.

HoloSheep

@felsiska the origin point (aka pivot point, but useful to think of it as the origin of a 3D model in this discussion) of the 3D game objects and its relationship to the invisible skin of the collider is what is important here.

Since Unity isn't really a model editing tool (like 3DS Max, Maya or SketchUp etc..) it turns out that editing the pivot point of a 3D model is not an option in Unity (in the traditional sense of editing anyway).

Since something like a default 3D Cube in Unity has its origin point in the center, and since you can't edit that origin point, when you move that 3D cube (by its origin) it will get placed at whatever location you specify, but the origin will be at that location not the bottom of the cube as you might prefer in this case, so half of the cube is above the surface and half below.

This is one of the two common gotchas that people run into when using the TapToPlace approach to moving objects (which moves the object by its origin) both of which can be solved by using the wrapper empty gameobject parent approach to faking a new origin.

1st common gotcha: If the origin of the object is not on the corner or surface that you would like to lock onto the destination point the object does not end up sitting where you want it.

2nd common gotcha: If the origin point is located outside of the main 3D Object and you size your collider just around the main 3D Object the detection logic never fires and you can't move the 3D object. (as described in this post)

Cube with Center Origin

So for the 1st case what you need is a way to get the 3D object to act like its origin is somewhere on the bottom face, bottom edge or even the bottom corner of the cube. The way to achieve that is through the relationship between the empty gameobject (that we called ModelBase) that parents the cube and the cube itself.

Both the parent and the child have their own Transform positions. The Parent is relative to the world origin (in this case the camera position) and the child is relative to the parent. So, in the case of an empty gameobject that has no width or height of its own, you can think of it pretty much as a point in space and as it turns out the origin of an empty gameobject is in exactly the middle of that same point. Since I have choosen to position the parent at X=0 Y=0 Z=2 (so that it is 2 meters in front of the camera to start, its origin is exactly at the same point in space.

So if I put a cube that is 1 unit wide inside and empty game object parent. And then I move the child cube in the x direction by half its width and in the y direction by half its height and the z direction by half its depth... the corner of the cube ends up at the origin point of the parent.

Parent's details (displaying the parent's origin)

Child's details (displaying the child's local origin)

The end result is that the parent becomes a wrapper for the child object. If we place our colliders and scripts (like the TapToPlace script) on the parent "ModelBase" (which you did above, and this is one of the reasons why) then we are really moving that wrapper or container around and since it is really just a point in space... we can position the child game object anywhere we like relative to that point in space. That point in space then effectively becomes the child game objects new origin (in an abstract but effective kind of way).

Does that help?

felsiska

understood. its working now. thanks so much for the detailed explanation.

felsiska

currently I have a room model, with furniture, in fbx format. if I want to move the furniture within the room space, will it be better to use the Placeable script instead of TapToPlace? the only problem I have about Placeable is that it will show the collider box instead of just the game object itself while during placing. is it also possible to have the furniture new position to be anchored to the walls/floors of room model instead of the real spatial surfaces?

AylinCopoglu

@felsiska said:
currently I have a room model, with furniture, in fbx format. if I want to move the furniture within the room space, will it be better to use the Placeable script instead of TapToPlace? the only problem I have about Placeable is that it will show the collider box instead of just the game object itself while during placing. is it also possible to have the furniture new position to be anchored to the walls/floors of room model instead of the real spatial surfaces?

From where do you have the PlaceAble Script?

DavidKlineMS

@AylinCopoglu,
I believe @felsiska is using Placeable.cs from the Holograms 230 course, which displays visual queues, in the form of a translucent cube, around the object and performs several raycasts to determine if the object will fit on a surface.

Thanks!
David

AylinCopoglu

@DavidKlineMS said:
@AylinCopoglu,
I believe @felsiska is using Placeable.cs from the Holograms 230 course, which displays visual queues, in the form of a translucent cube, around the object and performs several raycasts to determine if the object will fit on a surface.

Thanks!
David

Thanks David!

felsiska

@DavidKlineMS said:
@AylinCopoglu,
I believe @felsiska is using Placeable.cs from the Holograms 230 course, which displays visual queues, in the form of a translucent cube, around the object and performs several raycasts to determine if the object will fit on a surface.

Thanks!
David

Yes, its from Holograms 230. currently the collider size is 2x of the object so that object will sit nicely on the surface. however this will make the object look not to scale during placement where the collider itself will appear and hover across the surface. any ways to hide the collider box and only show the 3d object itself during placement?

DavidKlineMS

@felsiska,
You can hide the bounding cube by commenting out the following line in the Update method in Placeable.cs:

DisplayBounds(canBePlaced);

Thanks!
David

felsiska

@DavidKlineMS said:
@felsiska,
You can hide the bounding cube by commenting out the following line in the Update method in Placeable.cs:

DisplayBounds(canBePlaced);

Thanks!
David

thanks so much!

John_Smith1

In order to move a hologram from one position to another, is it possible to drag (move) it only using gaze - without hand movement. That is,

• Gaze at the hologram
• Airtap to select it
• Move your gaze vector (WITHOUT moving hand)
• Again airtap at the destination, so as to position the object there.

Is there any source explaining this?

Thanks and regards,
John

HoloSheep

@John_Smith1

The Origami demo does exactly that.
Check out the Holograms 101 tutorial.

That project allows you to air tap on the holographic pad of paper, gaze around the room and the pad will move to the closest available surface that intersect with your gaze and when you air tap again the origami pad of paper gets repositioned to the new location. (some may object to calling that a "dragging" operation but that is essentially what it is)

HTH.

John_Smith1

Thank you for your quick response.

codeholo

Hi, you can try my full tutorial on how to move, resize or rotate objects in your HoloLens apps using MR-Toolkit-Unity 2017.2 version (as latest as it gets) https://codeholo.com/2018/05/06/move-resize-and-rotate-objects-in-your-hololens-apps/