Hello everyone.

We have decided to phase out the Mixed Reality Forums over the next few months in favor of other ways to connect with us.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

The plan between now and the beginning of May is to clean up old, unanswered questions that are no longer relevant. The forums will remain open and usable.

On May 1st we will be locking the forums to new posts and replies. They will remain available for another three months for the purposes of searching them, and then they will be closed altogether on August 1st.

So, where does that leave our awesome community to ask questions? Well, there are a few places we want to engage with you. For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality. If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers. And always feel free to hit us up on Twitter @MxdRealityDev.

WebView.CapturePreviewToStreamAsync() fails, but only on HoloLens with Unity3D

Hello everybody,

Currently we're trying to combine the use of Unity3D holograms and HTML rendering in one app. The idea works as follows:
1) start a XAML app whose main window contains a WebView component (using the XAML app template)
2) Switch to Holographic (DirectX) mode (essentially copy over the code from the Holographic app template)
3) The WebView still runs in background
4) We can call the WebView.Navigate() method
5) Render the contents of the WebView via the CapturePreviewToStreamAsync() API to a bitmap
6) Convert the bitmap to a texture and display it on the DirectX side- thus having HTML rendering with holograms

Our experiments so far work in:
1) C++/CX
2) C#

However we have problems if we try to do the same thing with Unity3D. Here is what we do:
1) We create a Unity3D project with mixed reality support
2) We export the Unity project with "UWP Build type" set to "XAML" (instead of D3D).
That app builds within MSVS 2017 and starts as a XAML app which automatically switches to holographic space, showing correctly the scenes as defined in Unity.
3) We further edit the main XAML page to contain a WebView element (pointing to a webpage).

Now if we call the CapturePreviewToStreamAsync() API of the WebView component, a
System.Exception is thrown with message:
Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))

The error is 100% reproducible and occurs if we use both the Hololens glasses and the emulator.
But if we run the app on the local (desktop) machine*, the call works fine.

Any suggestions what might be the cause of the behavior? Seems that Unity creates the main window of the XAML part of the app in a different way, but we don't have access to that since the window creation happens mainly within the re-compiled Unity modules.

Best Regards
Martin

*It seems the local UWP windows 10 machine with MSVS 2017 can "emulate" the holographic space by just drawing a 3D scene

Best Answer

  • MrVachMrVach
    Accepted Answer

    Although we couldn't find the reason for the exception, we found a workaround, by following the instructions in this blog entry by Jonathan Antoine :
    jonathanantoine.com/2016/06/29/hololens-launch-a-3d-holograms-app-from-a-2d-xaml-app-and-going-back-to-it-uwp/

    The way to make things work was to:
    1. Create a new XAML page for the Unity3D generated app from above (let's name it WebViewPage, which contains our WebView component)
    2. Modify the Unity generated XAML app to navigate to WebViewPage on startup instead of to the MainPage
    3. When needed to switch to holographic space, just navigate to the MainPage the usual way:

    var frame = new Frame();
    Window.Current.Content = frame;
    frame.Navigate(typeof(MainPage));  
    

    4. You might also need to move the appCallbacks object from the generated app from the App class to the MainPage class

    That solved the problem. Now the CapturePreviewToStreamAsync() call works and returns the data.

    NOTE: We noticed that the app leaves the HoloLens emulator (10.0.14393.1358) in some state which doesn't allow the app to be launched a second time without restarting the emulator. But that problem doesn't show up with on a real device

    Hope that helps someone
    Martin

Answers

  • MrVachMrVach
    Accepted Answer

    Although we couldn't find the reason for the exception, we found a workaround, by following the instructions in this blog entry by Jonathan Antoine :
    jonathanantoine.com/2016/06/29/hololens-launch-a-3d-holograms-app-from-a-2d-xaml-app-and-going-back-to-it-uwp/

    The way to make things work was to:
    1. Create a new XAML page for the Unity3D generated app from above (let's name it WebViewPage, which contains our WebView component)
    2. Modify the Unity generated XAML app to navigate to WebViewPage on startup instead of to the MainPage
    3. When needed to switch to holographic space, just navigate to the MainPage the usual way:

    var frame = new Frame();
    Window.Current.Content = frame;
    frame.Navigate(typeof(MainPage));  
    

    4. You might also need to move the appCallbacks object from the generated app from the App class to the MainPage class

    That solved the problem. Now the CapturePreviewToStreamAsync() call works and returns the data.

    NOTE: We noticed that the app leaves the HoloLens emulator (10.0.14393.1358) in some state which doesn't allow the app to be launched a second time without restarting the emulator. But that problem doesn't show up with on a real device

    Hope that helps someone
    Martin

  • Hi! Your project seems to be exactly what I'm trying to do. Were you able to have the WebView in the same space as the holograms?

    Thanks in advance for the time,
    Teresa

Sign In or Register to comment.