The Mixed Reality Forums here are no longer being used or maintained.
There are a few other places we would like to direct you to for support, both from Microsoft and from the community.
The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.
For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.
If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.
And always feel free to hit us up on Twitter @MxdRealityDev.
Can Hololens detect more than one finger?
Is there any documentation on looking/processing more than one finger on HoloLens?
Best Answers
-
Options
Patrick
mod
Aside from the documented gestures, there are no additional gestures available. In the gestures, the concept of a 'finger' doesn't even exist, at least not at the layers we have access to.
===
This post provided as-is with no warranties and confers no rights. Using information provided is done at own risk.(Daddy, what does 'now formatting drive C:' mean?)
5 -
OptionsPatrick
mod
Here's documentation to get access to the RGB camera in Unity.
I've not done this personally, but I believe that you can use OpenCV in a UWP application, and it looks like finger tracking is in their samples. No idea how reliable it is. I can imagine the contrast between piano keys and hands to be large enough to get started and the hands are in the field of view. I'm dubious about other instruments. A saxophone player cannot usually see their fingers while they are playing. The finger positions for guitar can be ambiguous unless you can detect where the pressure is being applied on the frets and which strings are being plucked.
I don't think that this is an impossible tool to create, but the HoloLens may not be the device that is doing the 'finger tracking'. Rather, you could use a camera attached to a PC, let the PC do the processing, and send data to the HoloLens about what the external camera sees. You could give the Saxophone player 'XRay vision' to help them place their fingers. That'd be pretty awesome.
===
This post provided as-is with no warranties and confers no rights. Using information provided is done at own risk.(Daddy, what does 'now formatting drive C:' mean?)
6
Answers
Aside from the documented gestures, there are no additional gestures available. In the gestures, the concept of a 'finger' doesn't even exist, at least not at the layers we have access to.
===
This post provided as-is with no warranties and confers no rights. Using information provided is done at own risk.
(Daddy, what does 'now formatting drive C:' mean?)
Do you think it is possible for us developers to implement 'fingers' currently? Any suggestions where to start?
Perhaps you could do image processing on the RGB feed. I would advise against doing so. Even if you get perfect hand tracking from RGB, the RGB camera sees a different frame than the gesture frame. Users would have to hold their arms a bit higher than they otherwise would. This leads to fatigue.
I wonder if there is a way to implement the experience you are imagining without requiring multiple fingers. There are advantages to keeping familiar interactions across applications. You know how to use practically every mouse to interact with practically every application. Now if some application requires a 'squid', you have to learn how to use a 'squid' before you can use the application. If you can find another application that solves the same problem but uses a mouse, you'll probably stick with the application that uses a mouse.
===
This post provided as-is with no warranties and confers no rights. Using information provided is done at own risk.
(Daddy, what does 'now formatting drive C:' mean?)
Thanks Patrick,
One of the most requested features through "Share your Idea" was teaching instruments. To play chords on most instruments, a person would need to use more than one finger. I think something like this does not disadvantage people because it is a familiar interaction.
Could you link me to documentation on RGB processing with HoloLens when you get a chance?
Here's documentation to get access to the RGB camera in Unity.
I've not done this personally, but I believe that you can use OpenCV in a UWP application, and it looks like finger tracking is in their samples. No idea how reliable it is. I can imagine the contrast between piano keys and hands to be large enough to get started and the hands are in the field of view. I'm dubious about other instruments. A saxophone player cannot usually see their fingers while they are playing. The finger positions for guitar can be ambiguous unless you can detect where the pressure is being applied on the frets and which strings are being plucked.
I don't think that this is an impossible tool to create, but the HoloLens may not be the device that is doing the 'finger tracking'. Rather, you could use a camera attached to a PC, let the PC do the processing, and send data to the HoloLens about what the external camera sees. You could give the Saxophone player 'XRay vision' to help them place their fingers. That'd be pretty awesome.
===
This post provided as-is with no warranties and confers no rights. Using information provided is done at own risk.
(Daddy, what does 'now formatting drive C:' mean?)
I have done tests and the SpatialGestureRecognizer class does indeed see more than one hand. However, I have not found ways to distinguish between different fingers. So you could basically implement something that would work for two hands simultaneously but that's about it.
I have tried some OpenCV but that's very hard to do: the CPU gets overloaded pretty quickly.
Dennis Vroegop
Emerging Experiences MVP
Is there a way to access the depth camera? Looks like there is a lot of finger tracking for Kinect depth sensor. ie http://makematics.com/code/FingerTracker/
The camera information in holographic locatable_camera
contains all the information about HoloLens Camera