Hello everyone.

We have decided to phase out the Mixed Reality Forums over the next few months in favor of other ways to connect with us.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

The plan between now and the beginning of May is to clean up old, unanswered questions that are no longer relevant. The forums will remain open and usable.

On May 1st we will be locking the forums to new posts and replies. They will remain available for another three months for the purposes of searching them, and then they will be closed altogether on August 1st.

So, where does that leave our awesome community to ask questions? Well, there are a few places we want to engage with you. For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality. If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers. And always feel free to hit us up on Twitter @MxdRealityDev.

How to get 2D buttons appearing on a 3D App?

I have looked at past questions on this issue and found that this is still work in progress unless someone can enlighten me please. Having been through Tutorial 101E quite thoroughly, I thought to try to write a simple little app myself based on what I saw, Using one of the triangular meshes given I wondered if I could add a UI object like a button in the 3D view then modify it so that the asset script would move the button so that it would appear in the viewport at the place I wanted. For example I wanted the button to appear in the top left corner of the viewport. Using Unity to add my Asset script shown below:-

`public class buttonmover : MonoBehaviour
{
Vector3 v = new Vector3(100, Screen.height-33, 0.4f);
// Use this for initialization
void Start ()
{
}

// Update is called once per frame
void Update ()
{
    Vector3 worldPosition = Camera.main.ScreenToWorldPoint(v);
    transform.position=worldPosition;
}

}`

Compiling up the code on VS15 then running it in release mode gave me the top left corner of the scene to look like this

However when I move the cursor around the button gets rotated, scaled and distorted

You can also get this

What I want is to control the upward normal of the UI button as well as the position so that it is always pointing directly toward the observer. Despite using Euler angles or enforcing values as I did for the position it does not seem to improve this situation. Is it obvious what the problem is or am I using the wrong method or am I just a moron?:smile:

Best Answers

Answers

  • wintermootwintermoot ✭✭
    edited May 2016

    Are these controls designed to be used by players wearing the device or debug? I'd avoid pinning UI statically to the viewport if you intend players to interact with the controls. In the least, distance the controls from the camera. For instance, on the side of a plane/cube/mesh placed at least 2m from the camera. Ideally, this plane 'follows along' with the camera to avoid discomfort. Even better, the content is world locked to a location in the scene and maintains its position using a stationary frame of reference that is not the camera.

    In the design docs, check out the difference between "head-locked" vs "body-locked" vs content. Sorry if this is already obvious and the buttons are for debug use in the emulator exclusively :)

    https://developer.microsoft.com/en-us/windows/holographic/coordinate_systems

  • Jarrod1937Jarrod1937 ✭✭✭
    Interactions are done via gaze controlled by head movement. This means if you truly lock an interactive element to the camera (head movement) the user can never interact with it. You should look into the Holotoolkit for Unity, and look into the tagalong script.
  • Dear Wintermoot and Jarrod1937,

    Thank you for your comments and please excuse my ignorance of the processes as I am still in the learning part. As you both said, there is a 'tag along' script or 'follow along' process which I should adopt. What I did not mention earlier is that I still get a problem when I use the tag-along script which is vulnerable to occlusion when you move your 'gaze' around. I added in the Holotoolkit Assets as part of my test code and use that in conjunction with the original Assets as described in the Tutorial notes 101E. I also found it difficult to differentiate between more than one button as they merge to the same place and dutifully follow the 'Gaze' but now are overlapped. You are correct that buttons should not be necessary as you can dictate what you want by voice control however I needed to explore the use of other gestures using and selecting buttons and pull downs to do a combination of tasks interactively. Perhaps I ask too much here and I admit I am new to this area of development.

    I shall look at the "head-locked" vs "body-locked" vs content next as you suggested wintermoot.

    Thanks

  • @wheelchairman Have you tried looking at the Galaxy Explorer source code? I believe they have a user facing UI. Maybe worth a look.

    https://github.com/Microsoft/GalaxyExplorer

  • @wheelchairman
    You may also want to look at the Billboard script in the HoloToolkit. This script, when attached, will pivot the object to always face the user (even when they are not looking at it).

    To handle occlusion, you can write a script to always place your object in front of any hologram, including the Spatial Mapping mesh. In this scenario, holograms may come very close to the user and may move at unexpected speeds.

    I highly recommend user testing multiple options and selecting the one that best serves the experience.

    Thanks!
    David

  • @mtycholaz said:
    @wheelchairman Have you tried looking at the Galaxy Explorer source code? I believe they have a user facing UI. Maybe worth a look.

    https://github.com/Microsoft/GalaxyExplorer

    Sir, I shall certainly do I you suggest.
    Many thanks

  • @DavidKlineMS said:
    @wheelchairman
    You may also want to look at the Billboard script in the HoloToolkit. This script, when attached, will pivot the object to always face the user (even when they are not looking at it).

    To handle occlusion, you can write a script to always place your object in front of any hologram, including the Spatial Mapping mesh. In this scenario, holograms may come very close to the user and may move at unexpected speeds.

    I highly recommend user testing multiple options and selecting the one that best serves the experience.

    Thanks!
    David

    Dear David, Yes I tried the Billboard too and it still had a problem I think with Occlusion. I'll just check this again in case I have this wrong.
    Thank you

  • @wheelchairman, Correct, Billboard does not work around occlusion. You will need to reposition your hologram within your application in front of any other holograms to eliminate occlusion.

    Thanks!
    David

  • Due to another question that has come up currently, I am unable to progress this issue further. However I now understand from the comments made by others that it is difficult to work in both 2D and 3D together. From what I further understand is the plane that my buttons ought to be on should be the nearest objects to my camera eye otherwise they may get partially or completely hidden. As also pointed out using buttons is not really desirable as you can get the same functionality by using alternative methods like gaze, gesture and speech. I still think however that gazing then speech selecting the object and choosing a button is a rich and effective way of controlling the application.

    If anyone has any further comments to make on this issue, I would be most grateful as I just don't quite feel this matter it completely answered.
    Many thanks

  • @wheelchairman I have gone back and forth on a few style interfaces for hololens. I have tried a Unity UI 2D interface with tagalong and billboarding, which works quite well, as well as metro style and 3d styles as well. User comfort has been a focus, taking into account Microsofts best practices. I think I have landed on a mixture. 3d for a menu systems and 2d for a HUD or realtime info updates. Here is a video I made with a few of the styles(forgive the placemarker art elements)

  • @wheelchairman I wondering if the TagAlong.cs script from the HoloToolkit-Unity would help you solve the problem of placing UI panels in front of spatial mapping mesh. Please give it a try if you get a chance.

    The panels will follow you around and will place themselves always in front of the mesh. You can also adapt this script to your cases.

Sign In or Register to comment.