Spatial Mapping Renderer component in Unity not drawing wireframe

trzy

Hi,

I'd like to use spatial mapping in my app (primarily for occlusion culling initially but eventually collision as well) and thought I'd start with the "Spatial Mapping Renderer" AR component. The offline documentation indicates this can be attached to any object. I went ahead and added it to my main camera, set "Custom Render Setting" to "Custom Material" and the "Custom Material" to "SpatialMappingWireframe", which should draw a wireframe color coded by distance.

Unfortunately, I get nothing. I've also tried Microsoft's "Origami" tutorial. They have a "Spatial Mapping" prefab that can render a wireframe and it includes the "Spatial Mapping Renderer" component, but as far as I can tell, it is their own script that ends up rendering a monochrome wireframe. The component does not seem to do anything at all.

What am I missing? What is necessary to use this component? I feel as though I may be misunderstanding its purpose and that additional support code is required in a script, but that is not what the offline docs seem to suggest.

Thank you,

Bart

ahillier

Hi @trzy,
Double-check that you have the 'SpatialPerception' capability checked. You won't get spatial mapping at all, if this option not checked.

The SpatialMappingRenderer can be attached to the MainCamera, but I would recommend only doing that for occlusion (not physics). Preferably, you can create an empty game object and attach the Spatial Mapping components to that object. In Holograms 101, we use a manger class (SpatialMapping.cs) to swap between render states for the mesh. It updates the SpatialMappingRenderer component which is part of the prefab. Having some additional code to help manage the renderer can be helpful if you want to enable/disable rendering during different states of your application.

ahillier

Hi @trzy,
The Unity components: SpatialMappingRenderer and SpatialMappingCollider are new, and I haven't done any performance evaluations of them. However, I could hypothesize that attaching the spatial renderer to the camera will be more expensive than placing it in a static position in the world (like what we do in the spatial mapping 230 course), since the camera is constantly moving and causing you to regenerate the mesh. I would suggest disabling the spatial renderer on your camera, and placing one in the world to see if this improves your performance. You should also try increasing 'Time Between Updates' and 'Removal Update Count' and decreasing 'Level Of Detail' to get better framerate. However, if mapping dynamic objects (humans, pets, etc) is important to your app, this will have an impact on the quality of your occlusion mesh.

I believe you're correct, that having more than one spatial renderer in your scene is going to be more expensive, since they will generate/update their own version of the spatial mapping meshes.

Based on the design of your app, you might consider having a 'room scanning' period at the beginning, which will allow you to capture the play area and then freeze the spatial mapping updates, so you can work with static mesh data from then on. This won't work for all situations (when you need to detect dynamic objects, or allow movement between multiple rooms), but it can be a major advantage to applications that will do expensive processing of the mesh (like plane finding or object detection), such as RoboRaid, Fragments, and Conker.

Finally, no matter how quickly you update the mesh, or how finely you scan the area, there will be holes or ghosts which can make occlusion appear chunky. Personally, I prefer to use a secondary shader for rendering my holograms when they are occluded (like the rim or grid shaders used in Holograms 230). This will help the user understand that the object is occluded, and will also makes holes or chunks in the mesh less noticeable.

If you still can't get decent perf with the Unity components, I would suggest trying our Spatial Mapping prefab from the HoloToolkit. We've been able to get decent framerate when using a medium mesh (500 triangles per cubic meter) with the occlusion material we provide.

ContextVR

@trzy Can you post a reference to the documentation where you found "Spatial Mapping Renderer" can be attached to any object? I don't think you're using the SpatialMappingRenderer component correctly by attaching it to the camera, you'll need to attach it to an object actually being rendered.

You probably don't need to use the SpatialMappingRenderer component at all.
If you follow Chapter 6 in Holograms 101, you'll need to check "Draw Visual Meshes" and set "Draw Material" to "Wireframe" so see the spatial mesh as you scan the room.

It would also help if you can describe your use case.

ahillier

Hi @trzy,
Double-check that you have the 'SpatialPerception' capability checked. You won't get spatial mapping at all, if this option not checked.

The SpatialMappingRenderer can be attached to the MainCamera, but I would recommend only doing that for occlusion (not physics). Preferably, you can create an empty game object and attach the Spatial Mapping components to that object. In Holograms 101, we use a manger class (SpatialMapping.cs) to swap between render states for the mesh. It updates the SpatialMappingRenderer component which is part of the prefab. Having some additional code to help manage the renderer can be helpful if you want to enable/disable rendering during different states of your application.

mattsich

yeah that was the solution. Follow the steps here: https://developer.microsoft.com/en-us/windows/holographic/holograms_230

trzy

@ContextVR said:
@trzy Can you post a reference to the documentation where you found "Spatial Mapping Renderer" can be attached to any object? I don't think you're using the SpatialMappingRenderer component correctly by attaching it to the camera, you'll need to attach it to an object actually being rendered.

You probably don't need to use the SpatialMappingRenderer component at all.
If you follow Chapter 6 in Holograms 101, you'll need to check "Draw Visual Meshes" and set "Draw Material" to "Wireframe" so see the spatial mesh as you scan the room.

It would also help if you can describe your use case.

Sorry for the belated reply. The Spatial Mapping Renderer can indeed be attached to the camera. It definitely works in my demo (https://github.com/trzy/hololens/).

I use it for occlusion but initially, to debug, I wanted to see it render the mesh because it was clear something wasn't working.

My understanding of the component is as follows: it periodically requests a spatial mesh at a given update frequency, complexity, and within a given volume. It then renders it. Therefore, it makes sense that only a single instance of this component should be present in the program for occlusion culling. Otherwise, you would be requesting multiple copies of the same mesh for each object the renderer is attached to, and rendering multiple passes of the spatial mesh (unless the component is intelligent enough to understand it has been instantiated more than once).

AFAIK, occlusion works simply by rendering the spatial mesh to the stencil buffer and then enabling a stencil test for all subsequent rendering. This seems to imply that you would want only a single of these components.

trzy

@ahillier said:
Hi @trzy,
Double-check that you have the 'SpatialPerception' capability checked. You won't get spatial mapping at all, if this option not checked.

The SpatialMappingRenderer can be attached to the MainCamera, but I would recommend only doing that for occlusion (not physics). Preferably, you can create an empty game object and attach the Spatial Mapping components to that object. In Holograms 101, we use a manger class (SpatialMapping.cs) to swap between render states for the mesh. It updates the SpatialMappingRenderer component which is part of the prefab. Having some additional code to help manage the renderer can be helpful if you want to enable/disable rendering during different states of your application.

Hi ahillier,

Can you comment further on the performance of the Spatial Map Renderer component (for occlusion only)? As you know, I've attached it to my main camera. I never disable it and always want occlusion active.

However, I've noticed that the performance of my application degrades when using it. Strangely, the degradation is not immediate, but seems to be linked either with a coroutine that I use to move a game object (the spatial map component is not attached to this moving game object, only to the camera) or possibly other meshes placed near the main game object.

Rather difficult to explain but you can see my thread on the topic here: http://forums.hololens.com/discussion/1968/30fps-performance-with-simple-meshes-in-unity#latest

Performance also degrades when the distance between object and camera is very low. When the Spatial Map Renderer component is used, once the performance drops to 30FPS, it seems to stay there indefinitely (except when gazing away).

I'm having a very hard time figuring out what the culprits are but it looks like there are multiple conditions that cause it.

What are the best practices for occlusion in Unity at a reasonable frame rate?

Thanks,

Bart

ahillier

Hi @trzy,
The Unity components: SpatialMappingRenderer and SpatialMappingCollider are new, and I haven't done any performance evaluations of them. However, I could hypothesize that attaching the spatial renderer to the camera will be more expensive than placing it in a static position in the world (like what we do in the spatial mapping 230 course), since the camera is constantly moving and causing you to regenerate the mesh. I would suggest disabling the spatial renderer on your camera, and placing one in the world to see if this improves your performance. You should also try increasing 'Time Between Updates' and 'Removal Update Count' and decreasing 'Level Of Detail' to get better framerate. However, if mapping dynamic objects (humans, pets, etc) is important to your app, this will have an impact on the quality of your occlusion mesh.

I believe you're correct, that having more than one spatial renderer in your scene is going to be more expensive, since they will generate/update their own version of the spatial mapping meshes.

Based on the design of your app, you might consider having a 'room scanning' period at the beginning, which will allow you to capture the play area and then freeze the spatial mapping updates, so you can work with static mesh data from then on. This won't work for all situations (when you need to detect dynamic objects, or allow movement between multiple rooms), but it can be a major advantage to applications that will do expensive processing of the mesh (like plane finding or object detection), such as RoboRaid, Fragments, and Conker.

Finally, no matter how quickly you update the mesh, or how finely you scan the area, there will be holes or ghosts which can make occlusion appear chunky. Personally, I prefer to use a secondary shader for rendering my holograms when they are occluded (like the rim or grid shaders used in Holograms 230). This will help the user understand that the object is occluded, and will also makes holes or chunks in the mesh less noticeable.

If you still can't get decent perf with the Unity components, I would suggest trying our Spatial Mapping prefab from the HoloToolkit. We've been able to get decent framerate when using a medium mesh (500 triangles per cubic meter) with the occlusion material we provide.

trzy

Thanks @ahillier. I will try some of your suggestions. A room scanning period is something many have suggested and it makes sense. I haven't experimented with using shaders for occlusion (not yet sure how a second shader is activated in Unity but will check Holograms 230).

However, re: the spatial mapping component and camera, my understanding is that the HoloLens is continually mapping the room and returning (presumably world-locked) meshes. So in that sense, it really should be irrelevant whether or not the component is attached to a moving object or not.

In other words, it doesn't make sense to transform the meshes by the parent game object position and as far as I can tell, the component doesn't even attempt to. Nevertheless, for cleanness of code, it makes sense to refactor this into its own scriptable object.

daviddenker

Sorry guys for picking up this old topic.

I'm having a new problem with the SpatialMapping.cs:

NullReferenceException: Object reference not set to an instance of an object
SpatialMapping.Start () (at Assets/Holograms/Support/Spatial Mapping/SpatialMapping.cs:81)

I opened the unity-project from the CatchUpChapters folder as well, having the same issue.

I'm currently using Unity 2017.3.0b4.
Do you have any idea how to solve the problem?

Thanks!
David

james_ashley

@daviddenker,

It would help to know what object isn't instantiated inside the Start method -- have you tried putting a break point in there? My guess would be that some scripts are missing from the scene itself.