Understanding Coexistence of Apps and Their Holograms (+Unity)

stepan_stulov

Hi guys.

Finally got my hands on the real device and deployed the very first example from the 101 course. I launched, pinned onto the wall and entered the standard Holograms app and placed a few pretty animals around me. Then I also launched and pinned the Origami (Unity) app. I immediately noticed some app life-cycle inconsistencies when entering the Origami app:

1. When inside of the Holograms app placing the holograms I could see both the holograms and the two pinned windows of Holograms and Origami apps on the wall.
2. When inside of the Origami app I could see no pinned co-existing windows of both apps, only the origami scene.

This makes me wonder: are these different modes of running HoloLens apps? Is it like you can mark some apps to have internal content being able to co-exist with other pinned apps' windows while some other apps are owning the world exclusively? Also is the behaviour I've experienced simply special to Unity apps only?

Another thing I'm having trouble to grasp is the overall nesting / lifecycle of apps. Is it like there is the Start Menu mode, further down there are pinned app windows, and even deeper down within each of the pinned windows there are the apps' content objects? Can they co-exist? May they not? I can't understand clear separation of contexts as from the interaction of Holograms app and Origami (Unity) app all these context seem to bleed into each other.

It's not specifically a question but a request. Could someone please give me a definitive explanation of when, where and what (co-)exists?

I highly appreciate any help! Thank you!

Jesse_McCulloch

Hey Stepan,

So what you are running up against is a preinstalled app (Holograms) not behaving like the rest of the apps you are using. The Holograms app is the only one that behaves that way, that I am aware of. You should not expect that the behavior of it is standard at this point in development.

To further answer your question:

The base level is the shell. This is where the start menu lives. It is also responsible for keeping track of all the 2-D windows that are pinned in your world, such as pinned pictures, and any UWP apps that have been placed in world space, and the 2-D representative windows for any 3-D apps that have been placed in world space.

You can have any number of 2-D windows(instances) if apps visible at all times, but only 3 of them can be considered "active". Inactive ones will appear with a different color gradient, and a sleep cursor when you gaze at them.

You can also have any number of 2-D windows for 3-D applications in world space. However, once you launch into one by clicking it, all other holographic representations in world space disappear, and you only get the content from the 3-D application you are currently inside. While inside, you can use the bloom gesture to exit to the shell, where the 2-D representation for that app should remain, and all other 2-D apps should become visible again.

The hope is that in the future we will be able to have multiple 3-D apps running at the same time, occupying a shared world space as it makes sense. For instance, having a weather widget in 3-D, as well as a To-Do list app and maybe a calendar all in 3-D at the same time. Other instances, say, when playing Minecraft, it doesn't make sense for there to be "shared" world space with other holograms.

Hopefully this clears things up a little bit. If you have any other questions, please let us know.

Jesse_McCulloch

Hey Stepan,

So what you are running up against is a preinstalled app (Holograms) not behaving like the rest of the apps you are using. The Holograms app is the only one that behaves that way, that I am aware of. You should not expect that the behavior of it is standard at this point in development.

To further answer your question:

The base level is the shell. This is where the start menu lives. It is also responsible for keeping track of all the 2-D windows that are pinned in your world, such as pinned pictures, and any UWP apps that have been placed in world space, and the 2-D representative windows for any 3-D apps that have been placed in world space.

You can have any number of 2-D windows(instances) if apps visible at all times, but only 3 of them can be considered "active". Inactive ones will appear with a different color gradient, and a sleep cursor when you gaze at them.

You can also have any number of 2-D windows for 3-D applications in world space. However, once you launch into one by clicking it, all other holographic representations in world space disappear, and you only get the content from the 3-D application you are currently inside. While inside, you can use the bloom gesture to exit to the shell, where the 2-D representation for that app should remain, and all other 2-D apps should become visible again.

The hope is that in the future we will be able to have multiple 3-D apps running at the same time, occupying a shared world space as it makes sense. For instance, having a weather widget in 3-D, as well as a To-Do list app and maybe a calendar all in 3-D at the same time. Other instances, say, when playing Minecraft, it doesn't make sense for there to be "shared" world space with other holograms.

Hopefully this clears things up a little bit. If you have any other questions, please let us know.

stepan_stulov

Outstanding answer, Jesse_McCulloch! Thank you so much!

DiegoV

Hi! I am so interested in this. I have a 3D App and I need to get out of it to interact with other Apps. Do you know if there is any news on this? Thanks!

dionysis

Is there any movement on getting this enabled for 3rd party applications? This seems like a particularly huge issue after all of the announcements around Windows Holographic at Build 2017.

It's going to be really underwhelming for a powerful PC to only have 2-D windows visible in the 3-D virtual shell. The first thing people are going to want to make happen is to recreate the sort of experiences from the 2015 marketing content on their new affordable mixed reality device only to find it's vapor still.

I've only been working with HoloLens for the last 2 weeks or so and it really feels like a slice of the future at times. But you can see the seams where it's not quite living up to it's amazing potential in things like the inability for apps to live in shared volumes. I understand that this will require some deeper sophistication in the application life cycle to allow for a well behaved, performant shared simulation. But the time to address this is now.

Developers need to be learning how to make this happen soon to enable the sorts of experiences users are going to want this fall when the headsets from Asus and HP launch. It will almost be more limiting in that sort of VR space for there to be no ability to create interactive volumes.

I could see one of the first things people want to do would be to create artwork in a tool like TiltBrush or Medium and then place that art from the app in their virtual space. Ideally they could do that with HoloLens right now... only it isn't possible with the limitations in the API.

A 3-D desk calendar, weather app, wall clock, or other sorts of widgets with non-immersive holographic content seems like it would be expected. A calendar with a unique funny cat animation/visual, event/person in history animation/visual, or a dozen other options would be apps filling the store that customers would want to customize their space. The weather app conceptualized in the 2015 marketing content would be something some people would want. Utilities like file management or messaging shouldn't require dropping into an immersive app, but should offer 3-D visualizations. An incoming text message could be read by a custom avatar of the sender or by a Cortana like personal assistant. Files could be visualize and managed in dozens of ways from skeuomorphic cabinets or shelves to purely abstract tree structures where the user could physically grab and manipulate trees of cubes representing different directories and work with them on their virtual desk space.

A waterfall sculpture or zen garden work environment could be hugely popular if they were available. Something like what Facebook is doing with spaces where you can drop any 360 video or image as your skybox for the space (the way people do with wallpaper on their desktop) and then create their virtual space inside of that background should be feasible, but it will require some ability for apps to interact with volumes rather than be immersive or flat. Ideally, 3-D objects with interactive behavior and physics should be "Apps" I can purchase from the windows store to use in an optionally multi-user shell space.

"Monitors" on dozens of surfaces or floating in mid air is great and will serve for many use cases, but realistically some mashup between what is there now, some of the ideas put forward by Mike Alger in this video from last year (https://www.youtube.com/watch?v=id86HeV-Vb8&t=10s), some of the ideas from Facebook's VR spaces... should be the short term goals for Windows Holographic. Is there upcoming announcements for this summer or should I only be thinking about apps being immersive or flat with no mixed for the foreseeable future?

Jesse_McCulloch

@dionysis - I completely agree with you, and I keep passing this feedback on to my contacts with the HoloLens team. There have really been no announcements yet about an update to the HoloLens OS or any really information about the upcoming Mixed Reality Platform API changes.

We really don't even know when those are going to be coming at this point. It could be as late as August when the developer edition pre-order headsets are said to be shipping.

Essentially, there is a void of information right now. I wish I could provide you with more info....

dionysis

@Jesse_McCulloch - Thanks for your response and your efforts to message this back to the development teams. This seems like something very worthwhile for Microsoft to push for. Android "O" is rolling in better VR UI / Window management to support their integrated Daydream hardware, Steam VR is showing a solid trend of improved integration with web/messages/etc, many independent developers are attempting to build desktop integration suites (easily one of the most common "utility" category apps), and Oculus is showing some promising bits around Facebook Spaces, Samsung Internet, etc. Now is the time to have the strongest influence in what a well managed mixed reality work space looks like. The nouns and verbs of the space will be getting defined over the next 6-18mo and 18 may be generous.

The need for a smooth interaction between standard compute "tasks" (web browsing, messaging, etc) and VR environments is clear and the lack of a good solution is a big negative motivator for VR use/adoption simply due to the overhead of putting on/removing the headset and the cognitive dissonance from the immersion shift... I know as an avid MR/VR fan, I avoid using it at times when I don't think I have time or if I'm trying to do any kind of multitasking. This needs to change because more than even a standard desktop, streaming video while referring to multiple written references, physically manipulating a spatial design diagram, and writing code should be able to share the same space... on an iceberg floating in a frozen ocean in under the aurora borealis... or where ever you are feeling most productive today.

I think the flip up design on the new Microsoft headsets are an excellent improvement and the few weeks I've spent with HoloLens have been mind bleedingly creative and surprisingly productive. I'm really looking forward to working with these tools and would like them to be proactive in creating a fully functional mixed reality work space.