Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

How to switch to 3D Unity app from a 2D Xaml view and back?

Our app needs to expose a filepicker or a listbox and eventually a WebView control and we would like to be able to start with a 2D slate and transition into a 3D Unity app to display a 3D model. Then switch back to 2D to pick another model and repeat.

I'm able to do this with a Direct3D app but have been unsucessful with Unity so far. Any help would be appreciated.

Thank you!

Tagged:

Answers

  • With some of the examples apps i have seen an on screen keyboard. I have tried to call the UnityEngine.TouchScreenKeyboard but i cant get it to pop. Can you guys point me in the direction of getting an on screen keyboard to pop?

    Thanks
    Calzaretta

  • @Calzaretta which kind of application are you making? (2D, 3D Unity or 3D DirectX)

    In a 2D UWP that I am experimenting with, the keyboard automagically comes up when I put focus into a TextBox control at runtime (in the emulator, I asume it will be the same on a real device).

    The docs also give details on how to enable the keyboard in unity.

    but the keyboard input docs on DirectX don't seem to mention enabling the system keyboard, so I am guessing that might not be easily available in DirectX apps?

    Windows Holographic User Group Redmond

    WinHUGR.org - - - - - - - - - - - - - - - - - - @WinHUGR
    WinHUGR YouTube Channel -- live streamed meetings

  • 3D Unity

  • After a couple of quick experiments I would agree that the docs could be a little more clear and more accurate about how to get the keyboard to appear.

    Here is what I eventually arrived at to get it to show up...

    As suggested in the docs I created a new unity app from scratch that was of "XAML" build type. NOTE: I ignored "step 7" since HoloLens does not appear for me in the Virtual Reality Devices list, but that didn't seem to be a problem.

    I tried adding a Unity UI Input field to my scene thinking that would be the way to go, but experimenting with that approach makes me think that doing so is unnecessary and a distraction. However, for anyone using the Unity UI in HoloLens, you should probably note that there is a known issue and should refer to this.

    Ultimately, I just added a 3D object to my scene and created a C# script with the following content:

    using UnityEngine;
    using System.Collections;
    
    public class InputField : MonoBehaviour {
    
        TouchScreenKeyboard keyboard;
        public static string keyboardText = "";
    
        void Start () {
            keyboard = new TouchScreenKeyboard("Sample text that goes into the textbox", TouchScreenKeyboardType.Default, false, false, false, false, "sample prompting text that goes above the textbox");
        }
    
        void Update () {
    
            if (keyboard != null && keyboard.active == false)
            {
                if (keyboard.done == true)
                {
                    keyboardText = keyboard.text;
                    keyboard = null;
                }
            }
       }
    }
    

    and dragged it on to my 3D object to associate it to something in the app so that it would fire at runtime.

    Then when I run the app the Start method invokes the keyboard.

    With gaze and tap I set the focus in the textbox and use gaze and tap to modify the text in the textbox.

    If you put a breakpoint inside the If condition in the Update routine, it should break after you dismiss the keyboard and you can inspect the KeyboardText variable to see if it equals the text you entered with the system keyboard.

    How you really invoke it in your app (instead of from the Start routine) I leave up to you.

    HTH

    Windows Holographic User Group Redmond

    WinHUGR.org - - - - - - - - - - - - - - - - - - @WinHUGR
    WinHUGR YouTube Channel -- live streamed meetings

  • Jarrod1937Jarrod1937 ✭✭✭
    edited April 2016

    Take note though that the new technical preview of Unity that's pinned to the top of these forums did away with the loose script needed for the UI in Unity. You still need to follow the steps in the link, but the script is not part of the Unity build and you select it from the internal list.

    Edit:

    "- HoloLensInputModule no longer a loose script
    - If users downloaded the loose script from a previous build:

    - Delete the loose HoloLensInputModule.cs script from your project
    - Remove the now-defunct component for HoloLens Input Module from the "Event System" GameObject
     - Re-add the HoloLens Input Module component to that same object (now it will be the source-built version)
     - Add the Standalone Input Module to Event System if you want to be able to use the UI in play mode in-editor (the old version was botched in that you should be able to have different input modules side by side - the new version does not contain Standalone Input Module functionality, but now, each will only activate when needed if you have them both attached to Event System)
    

    "

  • Ah good catch @Jarrod1937!

    Windows Holographic User Group Redmond

    WinHUGR.org - - - - - - - - - - - - - - - - - - @WinHUGR
    WinHUGR YouTube Channel -- live streamed meetings

  • I am not building in xaml. I will try this but i am using D3D when i build. I thought that was unity 3d. Sorry if my answer was not correct. I am really new to unity.

    I will try this in an empty project as maybe something i have done is causing me issues.

    Thanks
    Calzaretta

  • @Calzaretta said:
    I am not building in xaml. I will try this but i am using D3D when i build.

    I believe that is your problem.
    The documentation specifically calls out:

    The system's on screen keyboard is unable to overlay on top of a volumetric view so Unity has to create a secondary 2D XAML view to show the keyboard then return back to the volumetric view once input has been submitted.

    which is why I think it is necessary to build your unity application with the XAML option if you want to use the system on screen keyboard.

    Windows Holographic User Group Redmond

    WinHUGR.org - - - - - - - - - - - - - - - - - - @WinHUGR
    WinHUGR YouTube Channel -- live streamed meetings

  • Right now i am looking to create a room spatial mapper class and i want the user to be able to enter the room name. Just like fragments does.
    :( if i can only use xaml that will be no good.. I will keep digging to see if i can find something

  • @Calzaretta so do you have a dependency on having the UWP Build Type in your Unity app being set to D3D?

    If you change that one setting to XAML instead of D3D does something in your app not work or do you give up/trade off something?

    Just in cast there is any confusion, I was not suggesting that you needed to create the entire app as a XAML 2D app. I was only referring to a particular Build Settting in your Unity 3D project.

    My first assumption was that you knew that based on my initial read of this thread, but since I haven't been able to find much info or documentation on the need for setting the UWP Build Type value to D3D (except perhaps performance, or keeping the runtime light weight), I thought I should double check and find out if you knew of another benefit of that setting.

    I have found that after building with it set one way (or the other) that it typically required deleting the previous App build directory after changing the setting to clear out the old other type of build files. But Unity will rebuild those (as well as the Library folder which sometimes needs to be removed) as I understand it.

    Windows Holographic User Group Redmond

    WinHUGR.org - - - - - - - - - - - - - - - - - - @WinHUGR
    WinHUGR YouTube Channel -- live streamed meetings

  • When using Unity, be sure to select XAML as the UWP build type.

    Switching between app states (2D and 3D) is done with the UAP application view switching model APIs. See this page for more information.

    A store app can create XAML/2D views which will be displayed in the “mixed world” as a slate (like other 2D apps) and switch between these and their 3D or “holographic” view. This will cause the system to switch out of the mixed world into the holographic view of the app (and back out again when a 2D view is activated).

  • I was just using what i learned from build 2016. They told us to create d3d app for the best user experience on the hololens.

    Should i not be using the d3d build setting?
    @neerajwadhwa Do you know how to make the onscreen keyboard pop in a d3d app?

  • @_Kyle we merged your question with a similar question asked in the past. If you think this is not the answer you were looking for please respond appropriately.

  • @ahillier, do you have an example of mixed mode app?

  • @ddobric sorry we don't have one currently to share out. But the feedback has been noted.

  • :) Thanks. Think, this is very importan feature. It could dramatically increase productivity, if we can reuse existing code and existing know-how.

  • @HoloSheep said:
    After a couple of quick experiments I would agree that the docs could be a little more clear and more accurate about how to get the keyboard to appear.

    Here is what I eventually arrived at to get it to show up...

    As suggested in the docs I created a new unity app from scratch that was of "XAML" build type. NOTE: I ignored "step 7" since HoloLens does not appear for me in the Virtual Reality Devices list, but that didn't seem to be a problem.

    I tried adding a Unity UI Input field to my scene thinking that would be the way to go, but experimenting with that approach makes me think that doing so is unnecessary and a distraction. However, for anyone using the Unity UI in HoloLens, you should probably note that there is a known issue and should refer to this.

    Ultimately, I just added a 3D object to my scene and created a C# script with the following content:

    using UnityEngine;
    using System.Collections;
    
    public class InputField : MonoBehaviour {
    
        TouchScreenKeyboard keyboard;
        public static string keyboardText = "";
    
        void Start () {
            keyboard = new TouchScreenKeyboard("Sample text that goes into the textbox", TouchScreenKeyboardType.Default, false, false, false, false, "sample prompting text that goes above the textbox");
        }
      
        void Update () {
            
            if (keyboard != null && keyboard.active == false)
            {
                if (keyboard.done == true)
                {
                    keyboardText = keyboard.text;
                    keyboard = null;
                }
            }
       }
    }
    

    and dragged it on to my 3D object to associate it to something in the app so that it would fire at runtime.

    Then when I run the app the Start method invokes the keyboard.

    With gaze and tap I set the focus in the textbox and use gaze and tap to modify the text in the textbox.

    If you put a breakpoint inside the If condition in the Update routine, it should break after you dismiss the keyboard and you can inspect the KeyboardText variable to see if it equals the text you entered with the system keyboard.

    How you really invoke it in your app (instead of from the Start routine) I leave up to you.

    HTH

    Great answer! I tried it out and it brought up the 2D keyboard at start. However, I need a bit of hand holding on how to switch back to the 3D view, because my screen was totally empty after I submitted the text on keyboard, but it was supposed to still contain the Origami objects, plus a 3D TextMesh showing what I just typed.

  • Hi @Main,

    So first thing I would suggest is to make sure that your TextMesh object and your 3D objects from your scene are showing up in your camera view in the Unity editor.

    Since I started from scratch instead of the Origami project, I just used a simple Capsule game object in my test app. I added a TextMesh called "My3DTextObject".
    I then adjusted the position of the textmesh object in the inspector so that it would appear in front of me (Z = 1) and adjusted my X and Y so that it would show above my capsule and sort of centered in view with the default "Hello World" text.

    Note, I also adjusted the Character Size and the Font Size so that they played nice in the real world mixed reality scale.

    Once you are sure that the textmesh and your 3D scene objects are displaying correctly to start with, if you then change the Update routine from the above post to be the following:

        void Update () {
    
            if (keyboard != null && keyboard.active == false)
            {
                if (keyboard.done == true)
                {
                    keyboardText = keyboard.text;
                    keyboard = null;
    
                    TextMesh txt3DObject;
                    GameObject temp = GameObject.Find("My3DTextObject");
                    txt3DObject = temp.GetComponent<TextMesh>();
                    txt3DObject.text = keyboardText;
                }
            }   
        }
    

    You should then automatically return from the 2D system keyboard view to the 3D view with the updated text that you submitted on the keyboard.

    Let me know if that works or if you have further questions.

    Windows Holographic User Group Redmond

    WinHUGR.org - - - - - - - - - - - - - - - - - - @WinHUGR
    WinHUGR YouTube Channel -- live streamed meetings

  • @HoloSheep said:
    Hi @Main,

    So first thing I would suggest is to make sure that your TextMesh object and your 3D objects from your scene are showing up in your camera view in the Unity editor.

    Since I started from scratch instead of the Origami project, I just used a simple Capsule game object in my test app. I added a TextMesh called "My3DTextObject".
    I then adjusted the position of the textmesh object in the inspector so that it would appear in front of me (Z = 1) and adjusted my X and Y so that it would show above my capsule and sort of centered in view with the default "Hello World" text.

    Note, I also adjusted the Character Size and the Font Size so that they played nice in the real world mixed reality scale.

    Once you are sure that the textmesh and your 3D scene objects are displaying correctly to start with, if you then change the Update routine from the above post to be the following:

      void Update () {
            
            if (keyboard != null && keyboard.active == false)
            {
                if (keyboard.done == true)
                {
                    keyboardText = keyboard.text;
                    keyboard = null;
    
                    TextMesh txt3DObject;
                    GameObject temp = GameObject.Find("My3DTextObject");
                    txt3DObject = temp.GetComponent<TextMesh>();
                    txt3DObject.text = keyboardText;
                }
            } 
      }
    

    You should then automatically return from the 2D system keyboard view to the 3D view with the updated text that you submitted on the keyboard.

    Let me know if that works or if you have further questions.

    Thank you so much @HoloSheep . For some reason, my unity build doesn't work consistently on the Hololens emulator. Sometimes it works, sometimes it shows the input window without a keyboard. However, after 5 tries of rebuilding, it did work perfectly.

    FYI: I did something very similar, by adding a TextMesh with a box collider. I tried to make the keyboard appear upon a Select gesture when the gaze cursor is upon the TextMesh.
    Here was my code, in case anyone else wants to try the same:

    using UnityEngine;
    using UnityEngine.UI;
    using System.Collections;

    public class HoloTextController : MonoBehaviour
    {
        private TextMesh holoText;
        TouchScreenKeyboard keyboard;
    
        public static string keyboardText = "";
    
        void Start()
        {
            holoText = GetComponent<TextMesh>();
        }    
        void OnSelect()
        {
            keyboard = new TouchScreenKeyboard("", TouchScreenKeyboardType.Default, false, false, false, false, "Keyboard:");
        }
    
            void Update()
        {
    
            if (keyboard !=null && keyboard.active == false)
            {
                if (keyboard.done == true)
                {
                    keyboardText = keyboard.text;
                    keyboard = null;
                    holoText.text = keyboardText;
                }
            }
        }
    }
    
  • @neerajwadhwa said:
    @_Kyle we merged your question with a similar question asked in the past. If you think this is not the answer you were looking for please respond appropriately.

    Similar to the original question posted by @_Kyle, I'm also interested in understanding how to switch between a 2D XAML view (e.g. File Picker) and holographic 3D view using Unity (not DirectX). The example of the TouchScreenKeyboard provided in this thread was helpful in understanding 2D/3D switching capabilities; however, that particular 2D component is already built into Unity itself. Is anyone aware of examples or might be able to provide guidance on switching between a non-Unity component, such as a XAML/UWP-based File Picker, and a holographic view, all within a single app?

  • @ptoliver said:

    @neerajwadhwa said:
    @_Kyle we merged your question with a similar question asked in the past. If you think this is not the answer you were looking for please respond appropriately.

    Similar to the original question posted by @_Kyle, I'm also interested in understanding how to switch between a 2D XAML view (e.g. File Picker) and holographic 3D view using Unity (not DirectX). The example of the TouchScreenKeyboard provided in this thread was helpful in understanding 2D/3D switching capabilities; however, that particular 2D component is already built into Unity itself. Is anyone aware of examples or might be able to provide guidance on switching between a non-Unity component, such as a XAML/UWP-based File Picker, and a holographic view, all within a single app?

    I was up in Redmond two weeks ago and got to work with a member from the HoloLens team on this problem (switching between 3D and custom 2D views). I haven't had a chance to write up the blog post yet, but I have a way to do this now that's very easy. Please see the last post in the thread linked below and let me know if you have any questions.

    http://forums.hololens.com/discussion/comment/7750/#Comment_7750

    Our Holographic world is here

    RoadToHolo.com      WikiHolo.net      @jbienz
    I work in Developer Experiences at Microsoft. My posts are based on my own experience and don't represent Microsoft or HoloLens.

  • @jbienzms said:

    @ptoliver said:

    @neerajwadhwa said:
    @_Kyle we merged your question with a similar question asked in the past. If you think this is not the answer you were looking for please respond appropriately.

    Similar to the original question posted by @_Kyle, I'm also interested in understanding how to switch between a 2D XAML view (e.g. File Picker) and holographic 3D view using Unity (not DirectX). The example of the TouchScreenKeyboard provided in this thread was helpful in understanding 2D/3D switching capabilities; however, that particular 2D component is already built into Unity itself. Is anyone aware of examples or might be able to provide guidance on switching between a non-Unity component, such as a XAML/UWP-based File Picker, and a holographic view, all within a single app?

    I was up in Redmond two weeks ago and got to work with a member from the HoloLens team on this problem (switching between 3D and custom 2D views). I haven't had a chance to write up the blog post yet, but I have a way to do this now that's very easy. Please see the last post in the thread linked below and let me know if you have any questions.

    http://forums.hololens.com/discussion/comment/7750/#Comment_7750

    thanks so much ,you saved my life , i've tried so many times ,but all failed until i got you code :)

  • So glad it helped you @cindywen. If anyone else needs a working reference sample of how to switch from 3D to 2D and back again, you can check out my FitBox and IPD application here:

    https://github.com/jbienzms/HoloTools/tree/master/FitBoxIPD

    That sample still leverages my Adept library, but you really only need the AppViewManager class.

    Some day when I have more time I plan to do a pull request for this class and a sample into the main toolkit. So many things, so little time!

    Our Holographic world is here

    RoadToHolo.com      WikiHolo.net      @jbienz
    I work in Developer Experiences at Microsoft. My posts are based on my own experience and don't represent Microsoft or HoloLens.

  • @_Kyle @HoloSheep @cindywen @ptltorres
    Hi all,

    Our apps needs air tap to open Edge Browser and switch back to App.

    I achieve to Open browser with Application.OpenUri("http://www.introprogramming.info");

    but I cant able to switch back to app.
    Please Help thanks in advance.

Sign In or Register to comment.