Display-locking content on the near clip plane

trzy

Hi,

I'm trying to implement a targeting "reticle" that appears over certain game objects when the user is looking at them. The reticle should be confined to a 2D plane in front of the user, and in front of all holograms. I had this effect working reasonably well in a non-holographic 3D application in Unity using Camera.main.WorldToViewportPoint() to project bounding boxes onto the viewport, and then Camera.main.ViewportToWorldPoint() (after having forced the z coordinate to Camera.main.nearClipPlane) to place them back in world space in order to render a mesh with Graphics.DrawMeshNow().

This apparently does not work on HoloLens. The reticles seem to get locked into some other plane that does not update with the user's position and orientation. The effect is hard to describe and I'm not clear as to what is actually happening.

Here is some code that attempts to project an object's bounds.max point onto the near clipping plane. As the object moves in 3D space, the reticle would be constrained to the XY plane, without any scaling, which is the desired effect:


Vector3 max = Camera.main.WorldToViewportPoint(bounds.max);
max.z = Camera.main.nearClipPlane;
Graphics.DrawMeshNow(m_reticle_mesh, Camera.main.VIewportToWorldPoint(max), Quaternion.identity);


I read another thread where it was recommended to simply position a display-locked object at camera.transform.position + distance*camera.transform.forward but that's not quite what I'm looking for. Rather than displaying a single flat object at the center of the user's gaze, I'm actually displaying 4 objects (4 corners of a reticle), which strongly depends on orientation.

Attached image shows the desired effect.

Thank you,

Bart

spt

Can't do that in stereoscopic space. The emulator (and any non-stereoscopic display) will fool you into thinking that this is possible because there actually is a single near plane, its not true 3d.
However the device gives a different parallax view to each eye. If you attempt to put up a reticle on the near plane, your eyes will be crossed and you will get double vision. So everything you display must be located in 3 dimensions, and should not be located too close to the viewer since they will have to cross their eyes to focus on it.
Take a look at how RoboRaid does their targetting reticle. Notice it appears to float many feet away, not close up.

spt

You are going to have to play around with it.
I see no reason you would want to write a shader, or use some HoloToolkit specific shader. Unity has several built in and the standard one can do just about anything you'd want without any need to write code.

spt

Can't do that in stereoscopic space. The emulator (and any non-stereoscopic display) will fool you into thinking that this is possible because there actually is a single near plane, its not true 3d.
However the device gives a different parallax view to each eye. If you attempt to put up a reticle on the near plane, your eyes will be crossed and you will get double vision. So everything you display must be located in 3 dimensions, and should not be located too close to the viewer since they will have to cross their eyes to focus on it.
Take a look at how RoboRaid does their targetting reticle. Notice it appears to float many feet away, not close up.

trzy

That makes sense.

For drawing bounding volumes around objects on 2D plane some distance in front of the user, I suppose I need to do a line/plane intersection between a plane facing perpendicular to the camera forward direction and the bounding volume corners I am interested in?

If I recall correctly (at work now and don't have the device handy), RoboRaid's reticle still appeared over all targets. I'm not very familiar with writing shaders in Unity yet. Are there any shaders in the HoloToolkit that work well for this (i.e., disable depth testing and draw over all geometry onto a UI layer?) There is a cursor shader, which is what I'm using, but it appears to perform depth testing against scene geometry using unity_GUIZTestMode, whatever that is.

spt

You are going to have to play around with it.
I see no reason you would want to write a shader, or use some HoloToolkit specific shader. Unity has several built in and the standard one can do just about anything you'd want without any need to write code.

trzy

Just a follow-up in case anyone stumbles upon this thread:

I tried a few things and found what looks nicest is to display the reticles in a plane perpendicular to the camera located at the centroid of the object being gazed at. The reticles are a fixed size in world space and so will scale with distance but this effect is, surprisingly, not very noticeable. To do this, I generate all 8 possible corners of the axis-aligned bounding box in world space, rotate them to local camera view space (Camera.main.transform.worldToLocalMatrix), project onto a plane located z=|centroid_point - (0,0,0)| away, and then find the min/max x and y coordinates to create a new, 2D AABB consisting of 4 points in this space. To render using DrawMeshNow, I have to rotate them back into world space, which seems rather wasteful but is simple enough.

I briefly tried (unsuccessfully) to maintain perceived size constant but as with what I was originally attempting to do, the projection math doesn't quite work out.

I also tried projecting onto a plane just slightly beyond the clip distance. This turned out to be very uncomfortable, especially as objects moved further away, with the reticles and object fighting for my eyes' focus.