The Mixed Reality Forums here are no longer being used or maintained.
There are a few other places we would like to direct you to for support, both from Microsoft and from the community.
The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.
For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.
If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.
And always feel free to hit us up on Twitter @MxdRealityDev.
How to get 2D buttons appearing on a 3D App?
I have looked at past questions on this issue and found that this is still work in progress unless someone can enlighten me please. Having been through Tutorial 101E quite thoroughly, I thought to try to write a simple little app myself based on what I saw, Using one of the triangular meshes given I wondered if I could add a UI object like a button in the 3D view then modify it so that the asset script would move the button so that it would appear in the viewport at the place I wanted. For example I wanted the button to appear in the top left corner of the viewport. Using Unity to add my Asset script shown below:-
`public class buttonmover : MonoBehaviour
{
Vector3 v = new Vector3(100, Screen.height-33, 0.4f);
// Use this for initialization
void Start ()
{
}
// Update is called once per frame void Update () { Vector3 worldPosition = Camera.main.ScreenToWorldPoint(v); transform.position=worldPosition; }
}`
Compiling up the code on VS15 then running it in release mode gave me the top left corner of the scene to look like this
However when I move the cursor around the button gets rotated, scaled and distorted
You can also get this
What I want is to control the upward normal of the UI button as well as the position so that it is always pointing directly toward the observer. Despite using Euler angles or enforcing values as I did for the position it does not seem to improve this situation. Is it obvious what the problem is or am I using the wrong method or am I just a moron?
Best Answers
-
Options
Sorry I have been a little remiss in not answering neerajwadhwa about using the TagAlong.cs script from the HoloToolKit library. Also I looked at subere23's funky video. I must admit I want to do something completely boring but probably useful for me getting data through in a set area of the screen which tells me something usefl. I must admit I did try the TagAlong.cs script amongst others but in the end got something else working which gave me an instant debug comment on the Hololens Emulator. Not sure yet how it would work on a real HoloLens. Here is a little sample of what it looked like when I tried monitoring the gaze position and where I plonked down my spheres and recorded the position of last one I put down. All units are in millimetres:-
In this example, I have 3 comment messages. All are similar in construction except that the first two comments gets their content from the position of the cursor and the position of the last sphere I plonked down respectively, The last comment is just a note to help me debug my program. The last comment I wrote the following script for:-
using UnityEngine; using UnityEngine.UI; public class addcomment : MonoBehaviour { private Text textobject; static public string comment = ""; void Start() { textobject = this.gameObject.GetComponentInChildren<Text>(); } // Update is called once per frame void Update() { if (posmarker.markers.Count > 0) { textobject.text = comment; textobject.SetAllDirty(); } } }
I am sure some stalwarts out there will pick me up for being such a bad programmer but I found it ok to read or write static public strings. However, I guess it is safest when writing a static string in a multithreaded environment to write it this way:-
lock(addcomment.comment) { addcomment.comment = "Kilroy was here"; }
This ensure in light of multithreading there might be no conflicts with other attempts to update the value of this string from another thread running simultaneously. Reading it apparently requires no lock condition to be made.
This code I called Addcomment.cs and placed it as a script in the Inspector of a GameObject I called FPSDisplay (2):-
Now I found the heirarchy of FPSDisplay (2) important to have the following items to ensure I got the effect I wanted. Here is the heirarchy I used:-
From my game object FPSDisplay(2), I had to UI Canvas on which to render a white rectangle box on which to write a text script. So a child of FPSDisplay(2) is a UI Canvas.
If you are attempting to do the same pleas ensure that your render mode and render camera are chosen just as I did above.
Next, I created an FPSBackground GameObject which renders the containing 2D rectangle for my text comment.
Please note the important attributes here.
1) The transform origin is at the bottom left hand corner.
2) The Pos (Position) of the start of the rectangle is 600 units (pixels I believe) from the left hand edge of the HoloLens Emulator screen. The position maybe anywhere but this is where I chose to put it to follow on from other display items.
3)The Width and Height I chose to fit the size of font I selected.Finally the child of FPSBackground is a object I defined as FPSText. Please Note: This is a UI Text which I renamed and not a 3D Text.
I chose an Arial font of size 16 to keep in the background box size of 300 x 40 pixels.
Please note I am no expert on this and I got this by just juggling a lot until I found this basic build principle which I found was easy to duplicate and modify for other labels too. I hope you find this of some use. I am sure this is far from perfect and more experienced coders will correct me on several points.
5 -
Optionszhzeshu ✭
@subere23 said:
@wheelchairman I have gone back and forth on a few style interfaces for hololens. I have tried a Unity UI 2D interface with tagalong and billboarding, which works quite well, as well as metro style and 3d styles as well. User comfort has been a focus, taking into account Microsofts best practices. I think I have landed on a mixture. 3d for a menu systems and 2d for a HUD or realtime info updates. Here is a video I made with a few of the styles(forgive the placemarker art elements)It looks pretty good, could you put the surface of the window on Horizontal direction. The original direction you designed is vertical direction.
5
Answers
Are these controls designed to be used by players wearing the device or debug? I'd avoid pinning UI statically to the viewport if you intend players to interact with the controls. In the least, distance the controls from the camera. For instance, on the side of a plane/cube/mesh placed at least 2m from the camera. Ideally, this plane 'follows along' with the camera to avoid discomfort. Even better, the content is world locked to a location in the scene and maintains its position using a stationary frame of reference that is not the camera.
In the design docs, check out the difference between "head-locked" vs "body-locked" vs content. Sorry if this is already obvious and the buttons are for debug use in the emulator exclusively
https://developer.microsoft.com/en-us/windows/holographic/coordinate_systems
UX Designer @ Anki
https://www.twitter.com/wintermoot
Dear Wintermoot and Jarrod1937,
Thank you for your comments and please excuse my ignorance of the processes as I am still in the learning part. As you both said, there is a 'tag along' script or 'follow along' process which I should adopt. What I did not mention earlier is that I still get a problem when I use the tag-along script which is vulnerable to occlusion when you move your 'gaze' around. I added in the Holotoolkit Assets as part of my test code and use that in conjunction with the original Assets as described in the Tutorial notes 101E. I also found it difficult to differentiate between more than one button as they merge to the same place and dutifully follow the 'Gaze' but now are overlapped. You are correct that buttons should not be necessary as you can dictate what you want by voice control however I needed to explore the use of other gestures using and selecting buttons and pull downs to do a combination of tasks interactively. Perhaps I ask too much here and I admit I am new to this area of development.
I shall look at the "head-locked" vs "body-locked" vs content next as you suggested wintermoot.
Thanks
@wheelchairman Have you tried looking at the Galaxy Explorer source code? I believe they have a user facing UI. Maybe worth a look.
https://github.com/Microsoft/GalaxyExplorer
@wheelchairman
You may also want to look at the Billboard script in the HoloToolkit. This script, when attached, will pivot the object to always face the user (even when they are not looking at it).
To handle occlusion, you can write a script to always place your object in front of any hologram, including the Spatial Mapping mesh. In this scenario, holograms may come very close to the user and may move at unexpected speeds.
I highly recommend user testing multiple options and selecting the one that best serves the experience.
Thanks!
David
Sir, I shall certainly do I you suggest.
Many thanks
Dear David, Yes I tried the Billboard too and it still had a problem I think with Occlusion. I'll just check this again in case I have this wrong.
Thank you
@wheelchairman, Correct, Billboard does not work around occlusion. You will need to reposition your hologram within your application in front of any other holograms to eliminate occlusion.
Thanks!
David
Due to another question that has come up currently, I am unable to progress this issue further. However I now understand from the comments made by others that it is difficult to work in both 2D and 3D together. From what I further understand is the plane that my buttons ought to be on should be the nearest objects to my camera eye otherwise they may get partially or completely hidden. As also pointed out using buttons is not really desirable as you can get the same functionality by using alternative methods like gaze, gesture and speech. I still think however that gazing then speech selecting the object and choosing a button is a rich and effective way of controlling the application.
If anyone has any further comments to make on this issue, I would be most grateful as I just don't quite feel this matter it completely answered.
Many thanks
@wheelchairman I have gone back and forth on a few style interfaces for hololens. I have tried a Unity UI 2D interface with tagalong and billboarding, which works quite well, as well as metro style and 3d styles as well. User comfort has been a focus, taking into account Microsofts best practices. I think I have landed on a mixture. 3d for a menu systems and 2d for a HUD or realtime info updates. Here is a video I made with a few of the styles(forgive the placemarker art elements)
https://www.youtube.com/watch?v=1-lbjKWwIpE
@wheelchairman I wondering if the TagAlong.cs script from the HoloToolkit-Unity would help you solve the problem of placing UI panels in front of spatial mapping mesh. Please give it a try if you get a chance.
The panels will follow you around and will place themselves always in front of the mesh. You can also adapt this script to your cases.
Sorry I have been a little remiss in not answering neerajwadhwa about using the TagAlong.cs script from the HoloToolKit library. Also I looked at subere23's funky video. I must admit I want to do something completely boring but probably useful for me getting data through in a set area of the screen which tells me something usefl. I must admit I did try the TagAlong.cs script amongst others but in the end got something else working which gave me an instant debug comment on the Hololens Emulator. Not sure yet how it would work on a real HoloLens. Here is a little sample of what it looked like when I tried monitoring the gaze position and where I plonked down my spheres and recorded the position of last one I put down. All units are in millimetres:-
In this example, I have 3 comment messages. All are similar in construction except that the first two comments gets their content from the position of the cursor and the position of the last sphere I plonked down respectively, The last comment is just a note to help me debug my program. The last comment I wrote the following script for:-
I am sure some stalwarts out there will pick me up for being such a bad programmer but I found it ok to read or write static public strings. However, I guess it is safest when writing a static string in a multithreaded environment to write it this way:-
This ensure in light of multithreading there might be no conflicts with other attempts to update the value of this string from another thread running simultaneously. Reading it apparently requires no lock condition to be made.
This code I called Addcomment.cs and placed it as a script in the Inspector of a GameObject I called FPSDisplay (2):-
Now I found the heirarchy of FPSDisplay (2) important to have the following items to ensure I got the effect I wanted. Here is the heirarchy I used:-
From my game object FPSDisplay(2), I had to UI Canvas on which to render a white rectangle box on which to write a text script. So a child of FPSDisplay(2) is a UI Canvas.
If you are attempting to do the same pleas ensure that your render mode and render camera are chosen just as I did above.
Next, I created an FPSBackground GameObject which renders the containing 2D rectangle for my text comment.
Please note the important attributes here.
1) The transform origin is at the bottom left hand corner.
2) The Pos (Position) of the start of the rectangle is 600 units (pixels I believe) from the left hand edge of the HoloLens Emulator screen. The position maybe anywhere but this is where I chose to put it to follow on from other display items.
3)The Width and Height I chose to fit the size of font I selected.
Finally the child of FPSBackground is a object I defined as FPSText. Please Note: This is a UI Text which I renamed and not a 3D Text.
I chose an Arial font of size 16 to keep in the background box size of 300 x 40 pixels.
Please note I am no expert on this and I got this by just juggling a lot until I found this basic build principle which I found was easy to duplicate and modify for other labels too. I hope you find this of some use. I am sure this is far from perfect and more experienced coders will correct me on several points.
It looks pretty good, could you put the surface of the window on Horizontal direction. The original direction you designed is vertical direction.