Hello everyone.

We have decided to phase out the Mixed Reality Forums over the next few months in favor of other ways to connect with us.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

The plan between now and the beginning of May is to clean up old, unanswered questions that are no longer relevant. The forums will remain open and usable.

On May 1st we will be locking the forums to new posts and replies. They will remain available for another three months for the purposes of searching them, and then they will be closed altogether on August 1st.

So, where does that leave our awesome community to ask questions? Well, there are a few places we want to engage with you. For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality. If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers. And always feel free to hit us up on Twitter @MxdRealityDev.

Rendering HTML in 3D View

Hi, I'm working on an app based off the C# DirectX 11 template and want to be able to display web content in the 3D holographic view.

Rendering a XAML WebView to a texture seemed like a good way to do it and I was able to shoehorn the DirectX and XAML templates together and use the XAML view's dispatcher to successfully call methods and properties (such as Navigate and DocumentTitle) on a WebView control from the holographic view (which I switch to at the start):

XAMLTest2Window.App.dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, new Windows.UI.Core.DispatchedHandler(async () =>
{
    XAMLTest2Window.App.webView.Navigate(new Uri("http://www.google.com"));
    RenderTargetBitmap renderTargetBitmap = new RenderTargetBitmap();
    await renderTargetBitmap.RenderAsync(XAMLTest2Window.App.webView);
    Debug.WriteLine(renderTargetBitmap.ToString());
}));

However, the call to RenderAsync blocks until I 'bloom' back to the 2D view. I can programmatically switch to 2D and then back to 3D a second later, but this is obviously not a practical solution.

I've tried adding the DISABLE_XAML_GENERATED_MAIN constant and using the [MTAThread] Main() method from the DX11 template as the entry point, but that didn't seem to change anything.

I've also tried ApplicationViewSwitcher.TryShowAsStandaloneAsync instead of ApplicationViewSwitcher.SwitchAsync, using both XAML and programmatically-instantiated WebViews as well as both possible values for the WebView's WebViewExecutionMode constructor parameter, all to no avail.

I've also investigated 3rd-party embedded browser solutions like Awesomium, but I haven't had any luck getting anything to run under UWP.

So my question is: am I missing something obvious (I'm new to a lot of this) or does anybody have any idea if there's a workaround to this problem or another solution that I could use?

Thanks,
- Simon

Tagged:

Best Answer

Answers

  • @Simon Hey, I've been trying to do almost the exact same thing: getting a webview rendered onto a 3D surface as texture. The part I'm struggling with is the D3D/XAML shoehorning part. It seems you were able to get this done; do you have any tips or references or maybe even a GitHub repo that might help me?

  • @TimKim said:
    @Simon Hey, I've been trying to do almost the exact same thing: getting a webview rendered onto a 3D surface as texture. The part I'm struggling with is the D3D/XAML shoehorning part. It seems you were able to get this done; do you have any tips or references or maybe even a GitHub repo that might help me?

    Hi @TimKim, Sorry for not replying earlier; I don't check this forum very often. Unfortunately I'm not in a position to share a repo with you, but if I recall correctly, I tried unsuccessfully to follow a few different online tutorials on converting the DirectX project to XAML and in the end got it working by creating two projects, copying the classes from the DX one into the XAML one and starting the DX one from App.xaml.cs like so:

    var newView = CoreApplication.CreateNewView(new NamespaceOfDXCode.AppViewSource());
    
    await newView.Dispatcher.RunAsync(
        CoreDispatcherPriority.Normal, new DispatchedHandler(async () =>
        {
            var viewId = ApplicationView.GetForCurrentView().Id;
            CoreWindow.GetForCurrentThread().Activate();
    
            await CoreApplication.MainView.Dispatcher.RunAsync(
                CoreDispatcherPriority.Normal, new DispatchedHandler(() =>
                {
                    //var asyncAction = ApplicationViewSwitcher.SwitchAsync(viewId, ApplicationView.GetForCurrentView().Id);
                    var asyncAction = ApplicationViewSwitcher.TryShowAsStandaloneAsync(viewId, ViewSizePreference.Default);
                })
            );
        })
    );
    

    I'm not sure if that is the right way, but it worked for me. If you haven't got it working already, I hope that helps.

  • When I run your code, it only displays one window and closes the other one. Did you ever run into this problem?

    I want to verify that you are able to display a 2D and 3D view together. So you are in the Unity 3D view and you are seeing a XAML page together.

  • @ems316 said:
    When I run your code, it only displays one window and closes the other one. Did you ever run into this problem?

    I want to verify that you are able to display a 2D and 3D view together. So you are in the Unity 3D view and you are seeing a XAML page together.

    I wasn't using Unity; I was working with code adapted from the DirectX 11 C# templates. I'm not sure what your goal is, but mine was to access pixels from the XAML WebView control so I could write them into a texture that I could render on a quad inside my 3D DirectX application. In that regard, I do have a 2D web window floating in 3D space, but it's not a XAML window (although it does grab pixels from a XAML control in a window that is no longer visible). The CapturePreviewToStreamAsync(...) method that solved the problem for me seems to be specific to WebView, so if you're trying to similarly mirror a whole XAML page, I'm not sure if there is a solution.

    I did briefly try to create a hybrid XAML/3D project in Unity as well to try the same trick, but I ran into some issue (can't remember what it was, but I didn't spend a lot of time trying to get around it). However, I do know that it is possible to create textures from your own bytes in Unity, so if you can create a similar hybrid project in Unity, you should at least be able to create a web window in 3D.

  • @Simon said:
    See my reply to your comment in the post I made. This might not answer your question, but I don't think it's strictly true to say there are no solutions without switching out of 3D.

    If you're already able to switch between 3D and your own XAML window, you should be able to call CapturePreviewToStreamAsync(...) while in 3D using the XAML window's dispatcher and put the result into a texture using Texture2D.LoadRawTextureData.

    With a little bit of maths (which Unity probably makes easier), you should also be able to find the intersection between the gaze ray and the quad you display the texture on. You can then normalise that into mouse cursor coordinates and pass Javascript to the WebView to fire mouse clicks when gestures are triggered. With a little bit more JS and CSS, you can also send back the state of the cursor to have the proper pointer icon for links, etc.

    I hope that helps.

    The idea you are getting at is to dynamically wrap a XAML around a 2D Unity Gameobject using CapturePreviewToStreamAsync(...) to grab an instance of our texture/XAML and then wrapping it with Texture2D.LoadRawTextureData?

  • How do you communicate between Unity and the UWP, so that you can pass the bytes?

  • Hello,
    I return to this subject because it corresponds to my own question. The Pluggin "Embedded Browser" by Zen Fulcrum (https://zenfulcrum.com/browser#demo) seems perfect for introducing an html page into a 3D view. It works on Unity and it will allow to mix sites HTML in view 2d with holograms in 3d ... Has anyone tried it for hololens, does it work well?

  • @Liliaze I have used it and found it to work well on Windows PC. It is not yet adapted to run on UWP, I think because of UWP restrictions on file moving. It sets up a separate chrome browser to get textures. Overall approach works well. I have initially looked into doing the UWP port and may return to this after I publish my product in the next month or so. It may also be possible to do offscreen camera rendering with a XAML browser. I am still learning the oddities of UWP restrictions and its intersection with Unity, so I am guessing on the XAML browser screen textures.

  • Hey, I am facing a similar problem and I need to connect Unity with the XAML view. Did you find a way to pass the bytes and display the content in 3D view through Unity? Any hints or examples how to do this?

  • @Simon said:

    @ems316 said:
    When I run your code, it only displays one window and closes the other one. Did you ever run into this problem?

    I want to verify that you are able to display a 2D and 3D view together. So you are in the Unity 3D view and you are seeing a XAML page together.

    I wasn't using Unity; I was working with code adapted from the DirectX 11 C# templates. I'm not sure what your goal is, but mine was to access pixels from the XAML WebView control so I could write them into a texture that I could render on a quad inside my 3D DirectX application. In that regard, I do have a 2D web window floating in 3D space, but it's not a XAML window (although it does grab pixels from a XAML control in a window that is no longer visible). The CapturePreviewToStreamAsync(...) method that solved the problem for me seems to be specific to WebView, so if you're trying to similarly mirror a whole XAML page, I'm not sure if there is a solution.

    I did briefly try to create a hybrid XAML/3D project in Unity as well to try the same trick, but I ran into some issue (can't remember what it was, but I didn't spend a lot of time trying to get around it). However, I do know that it is possible to create textures from your own bytes in Unity, so if you can create a similar hybrid project in Unity, you should at least be able to create a web window in 3D.

    It is an old topic, but i have the same situation. I need to render a webpage to a 3d object using directx.
    @Simon i tried to do that following your hint, without luck.
    Can you explain how you manage to do it?

    Thanks a lot!

  • @opi133 I didn't go very far with Unity; I didn't get to a point where I was accessing XAML controls with it, so I don't know if it's possible or not.

    @csonaglioni How far have you got? The first steps should be to:

    1. Create a XAML project based on the template in Visual Studio (by the way, IIRC, you need to install the emulator to get those templates).
    2. Add a WebView control to the XAML page.
    3. Copy your classes from your DirectX project (I also used one based on a Visual Studio template) into the XAML project.
    4. Add code to launch the DirectX view from the XAML one (using something like the code I showed here). In my case, I fired that code from a button on my XAML page.
    5. Refer to this example on how to access the pixels from a WebView using WebView.CaptureToPreviewAsync(...).
    6. Display those pixel bytes somehow (I used D3DDeviceContext.UpdateSubresourceSafe(...) to update a DirectX texture with them).

    I hope that helps. Sorry for the very late reply.

  • @opi133 Have you made any progress with displaying webpage inside your Hololens app? Please. let me know - I'm struggling to figure out how it is possible to implement it. Many thanks!

  • Hi,

    Do you have any updates on this? How to show a webView inside a holographic app session?

Sign In or Register to comment.