Holograms in low light/darkness

runamuck

Hello - I'm a "Wave 2" developer still awaiting my device. I've been planning and preparing my app in the emulator. One question I have is how HoloLens handles dim lighting/dark rooms? I was planning to have my app prompt the user to dim their lights before starting the game - and of course this works fine in the emulator - but how does this work on the actual device? Can Holograms function as expected in low-light/darkness?

Jesse_McCulloch

The actual device uses visible light cameras for spatial tracking, and it will go into limited mode and not let you do anything in darkness. I'm not sure just how much light it needs, but I know watching movies and such, I cannot turn off the lights for a "theater" experience without losing the HoloLens.

Jarrod1937

@runamuck said:

@Jesse_McCulloch said:
The actual device uses visible light cameras for spatial tracking, and it will go into limited mode and not let you do anything in darkness. I'm not sure just how much light it needs, but I know watching movies and such, I cannot turn off the lights for a "theater" experience without losing the HoloLens.

Hi Jesse - thank you - that's somewhat disappointing but understandable. I was under the assumption that the device had more Kinect-like cameras to actually perform spatial mapping under little to no lighting. My hope is that it may still work correctly with limited lighting.

I believe that is correct, to an extent. The depth is achieved via infrared, so lighting doesn't matter (except really bright sunlight). However, it would make sense that for the actual environment tracking, of the users movement, as well as higher frequency data for better holographic stabilization (so they don't jitter), that visible light is needed.

Pentose

Note that while you do lose positional tracking in darkness, you do still have rotation information. In this mode you cannot render holograms that is fixed to a location in a world (world locked), but you can render holograms relative to the user (body locked). This is what happens when you tap on the "Use limited mode" button when the shell loses tracking.

For more information, see https://developer.microsoft.com/en-us/windows/holographic/coordinate_systems#Handling_tracking_errors and https://developer.microsoft.com/en-us/windows/holographic/tracking_loss_in_unity

Jesse_McCulloch

The actual device uses visible light cameras for spatial tracking, and it will go into limited mode and not let you do anything in darkness. I'm not sure just how much light it needs, but I know watching movies and such, I cannot turn off the lights for a "theater" experience without losing the HoloLens.

Jarrod1937

What would be interesting to see if you can illuminate the room with a non-visible wavelength. We know the Hololens uses infrared for depth, and probably visible more for stabilization/placement tracking (it provides more granular data). However, what portion of the infrared (near, far) it uses, and if the camera system is sensitive to the range that the depth sensors are not, who knows. I assume it's the common CCD sensors for the cameras, which are capable of a broad range of light outside the visible area as well. However, the cameras may already be filtering infrared to not get interference from the depth sensors... So who knows, but something worth investigating, a possible non/near-non-visible light source that can give the cameras their needed input to allow functioning in low light.

runamuck

@Jesse_McCulloch said:
The actual device uses visible light cameras for spatial tracking, and it will go into limited mode and not let you do anything in darkness. I'm not sure just how much light it needs, but I know watching movies and such, I cannot turn off the lights for a "theater" experience without losing the HoloLens.

Hi Jesse - thank you - that's somewhat disappointing but understandable. I was under the assumption that the device had more Kinect-like cameras to actually perform spatial mapping under little to no lighting. My hope is that it may still work correctly with limited lighting.

Jarrod1937

@runamuck said:

@Jesse_McCulloch said:
The actual device uses visible light cameras for spatial tracking, and it will go into limited mode and not let you do anything in darkness. I'm not sure just how much light it needs, but I know watching movies and such, I cannot turn off the lights for a "theater" experience without losing the HoloLens.

Hi Jesse - thank you - that's somewhat disappointing but understandable. I was under the assumption that the device had more Kinect-like cameras to actually perform spatial mapping under little to no lighting. My hope is that it may still work correctly with limited lighting.

I believe that is correct, to an extent. The depth is achieved via infrared, so lighting doesn't matter (except really bright sunlight). However, it would make sense that for the actual environment tracking, of the users movement, as well as higher frequency data for better holographic stabilization (so they don't jitter), that visible light is needed.

runamuck

@Jarrod1937 said:

@runamuck said:

@Jesse_McCulloch said:
The actual device uses visible light cameras for spatial tracking, and it will go into limited mode and not let you do anything in darkness. I'm not sure just how much light it needs, but I know watching movies and such, I cannot turn off the lights for a "theater" experience without losing the HoloLens.

Hi Jesse - thank you - that's somewhat disappointing but understandable. I was under the assumption that the device had more Kinect-like cameras to actually perform spatial mapping under little to no lighting. My hope is that it may still work correctly with limited lighting.

I believe that is correct, to an extent. The depth is achieved via infrared, so lighting doesn't matter (except really bright sunlight). However, it would make sense that for the actual environment tracking, of the users movement, as well as higher frequency data for better holographic stabilization (so they don't jitter), that visible light is needed.

It makes complete sense. I guess my next question would be is whether or not there is an ambient light sensor on the unit that we can read whether sufficient lighting exists for the holograms to appear? Perhaps in my app I can ask the user to "dim" their lights but if the light level is detected to be too low I issue a warning?

wintermoot

@runamuck said:

@Jarrod1937 said:

@runamuck said:

@Jesse_McCulloch said:
The actual device uses visible light cameras for spatial tracking, and it will go into limited mode and not let you do anything in darkness. I'm not sure just how much light it needs, but I know watching movies and such, I cannot turn off the lights for a "theater" experience without losing the HoloLens.

Hi Jesse - thank you - that's somewhat disappointing but understandable. I was under the assumption that the device had more Kinect-like cameras to actually perform spatial mapping under little to no lighting. My hope is that it may still work correctly with limited lighting.

I believe that is correct, to an extent. The depth is achieved via infrared, so lighting doesn't matter (except really bright sunlight). However, it would make sense that for the actual environment tracking, of the users movement, as well as higher frequency data for better holographic stabilization (so they don't jitter), that visible light is needed.

It makes complete sense. I guess my next question would be is whether or not their is an ambient light sensor on the unit that we can read whether sufficient lighting exists for the holograms to appear? Perhaps in my app I can ask the user to "dim" their lights but if the light level is detected to be too low I issue a warning?

There is an ambient light sensor in the specs, but I'm not sure it's exposed to use for use. I would absolutely love if we could be as smart about too bright/dark as we are about spatial perception. Right now we just get "limited mode / tracking lost". Specific messaging based on the sensor would be elite.

Pentose

Note that while you do lose positional tracking in darkness, you do still have rotation information. In this mode you cannot render holograms that is fixed to a location in a world (world locked), but you can render holograms relative to the user (body locked). This is what happens when you tap on the "Use limited mode" button when the shell loses tracking.

For more information, see https://developer.microsoft.com/en-us/windows/holographic/coordinate_systems#Handling_tracking_errors and https://developer.microsoft.com/en-us/windows/holographic/tracking_loss_in_unity

runamuck

@Pentose said:
Note that while you do lose positional tracking in darkness, you do still have rotation information. In this mode you cannot render holograms that is fixed to a location in a world (world locked), but you can render holograms relative to the user (body locked). This is what happens when you tap on the "Use limited mode" button when the shell loses tracking.

For more information, see https://developer.microsoft.com/en-us/windows/holographic/coordinate_systems#Handling_tracking_errors and https://developer.microsoft.com/en-us/windows/holographic/tracking_loss_in_unity

Ah - fantastic! This could be the workaround I need when the user enters a much darker area. Thank you