WebView.CapturePreviewToStreamAsync() fails, but only on HoloLens with Unity3D

MrVach

Hello everybody,

Currently we're trying to combine the use of Unity3D holograms and HTML rendering in one app. The idea works as follows:
1) start a XAML app whose main window contains a WebView component (using the XAML app template)
2) Switch to Holographic (DirectX) mode (essentially copy over the code from the Holographic app template)
3) The WebView still runs in background
4) We can call the WebView.Navigate() method
5) Render the contents of the WebView via the CapturePreviewToStreamAsync() API to a bitmap
6) Convert the bitmap to a texture and display it on the DirectX side- thus having HTML rendering with holograms

Our experiments so far work in:
1) C++/CX
2) C#

However we have problems if we try to do the same thing with Unity3D. Here is what we do:
1) We create a Unity3D project with mixed reality support
2) We export the Unity project with "UWP Build type" set to "XAML" (instead of D3D).
That app builds within MSVS 2017 and starts as a XAML app which automatically switches to holographic space, showing correctly the scenes as defined in Unity.
3) We further edit the main XAML page to contain a WebView element (pointing to a webpage).

Now if we call the CapturePreviewToStreamAsync() API of the WebView component, a
System.Exception is thrown with message:
Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))

The error is 100% reproducible and occurs if we use both the Hololens glasses and the emulator.
But if we run the app on the local (desktop) machine*, the call works fine.

Any suggestions what might be the cause of the behavior? Seems that Unity creates the main window of the XAML part of the app in a different way, but we don't have access to that since the window creation happens mainly within the re-compiled Unity modules.

Best Regards
Martin

*It seems the local UWP windows 10 machine with MSVS 2017 can "emulate" the holographic space by just drawing a 3D scene

MrVach

Although we couldn't find the reason for the exception, we found a workaround, by following the instructions in this blog entry by Jonathan Antoine :
jonathanantoine.com/2016/06/29/hololens-launch-a-3d-holograms-app-from-a-2d-xaml-app-and-going-back-to-it-uwp/

The way to make things work was to:
1. Create a new XAML page for the Unity3D generated app from above (let's name it WebViewPage, which contains our WebView component)
2. Modify the Unity generated XAML app to navigate to WebViewPage on startup instead of to the MainPage
3. When needed to switch to holographic space, just navigate to the MainPage the usual way:

var frame = new Frame();
Window.Current.Content = frame;
frame.Navigate(typeof(MainPage));  

4. You might also need to move the appCallbacks object from the generated app from the App class to the MainPage class

That solved the problem. Now the CapturePreviewToStreamAsync() call works and returns the data.

NOTE: We noticed that the app leaves the HoloLens emulator (10.0.14393.1358) in some state which doesn't allow the app to be launched a second time without restarting the emulator. But that problem doesn't show up with on a real device

Hope that helps someone
Martin

MrVach

Although we couldn't find the reason for the exception, we found a workaround, by following the instructions in this blog entry by Jonathan Antoine :
jonathanantoine.com/2016/06/29/hololens-launch-a-3d-holograms-app-from-a-2d-xaml-app-and-going-back-to-it-uwp/

The way to make things work was to:
1. Create a new XAML page for the Unity3D generated app from above (let's name it WebViewPage, which contains our WebView component)
2. Modify the Unity generated XAML app to navigate to WebViewPage on startup instead of to the MainPage
3. When needed to switch to holographic space, just navigate to the MainPage the usual way:

var frame = new Frame();
Window.Current.Content = frame;
frame.Navigate(typeof(MainPage));  

4. You might also need to move the appCallbacks object from the generated app from the App class to the MainPage class

That solved the problem. Now the CapturePreviewToStreamAsync() call works and returns the data.

NOTE: We noticed that the app leaves the HoloLens emulator (10.0.14393.1358) in some state which doesn't allow the app to be launched a second time without restarting the emulator. But that problem doesn't show up with on a real device

Hope that helps someone
Martin

TeresaRC

Hi! Your project seems to be exactly what I'm trying to do. Were you able to have the WebView in the same space as the holograms?

Thanks in advance for the time,
Teresa