Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

Different object recognition methods for HoloLens

In short: I'm trying to add the possibility to find out if the HoloLens user is pointing somewhere with a finger. I haven't been able to find a good solution yet. Does anyone know a good solution for this?

In Detail:

  • I tried OpenCV for Hololens. Details: https://forums.hololens.com/discussion/10636/hololens-opencv#latest

  • I tried HoloLensForCV. https://forums.hololens.com/discussion/10679/issues-with-hololensforcv#latest

  • I tried OpenCVForUnity. The HoloLens example with facial recognition works. But the Code is confusing and I don't get much help from the support.

  • I looked into cognitive services. It's cloud based and seems to have a big database for objects to find. I'm not sure if it's the right approach if I only want to have one object to detect. Custom Vision allowed me to define objects with pictures, but I don't know how to run it on the HoloLens. There is also the Project Gesture which I also don't know if it works for HoloLens.

So I don't know where to continue any more. If anyone has any advice, I would be very thankful.

Best Answers

  • pstuevenpstueven ✭✭
    Answer ✓

    @fyzzo said:
    So do I understand correctly that I can't find the hand when it is pointing somewhere with this?

    As far as I know, this is correct.

    You could try tracking a hand in ready state and the using WinML image classification to recognize a part of the HoloLenses camera view. In my tests image classification worked quite well with a well trained model. But if you use custom vision to train it, be sure about privacy concerns as there are no "Terms of use" pointing out what happens to the images uploaded.

Answers

  • Hey @fyzzo,
    pointing is a difficult thing as HoloLens does not track your hand unless it is in ready or hold state. If this is enough, you can track the position of the user's hand and make a RayCast from the HoloLens' position toward the detected hand position. I had pretty good results with this.

  • @pstueven said:
    Hey @fyzzo,
    pointing is a difficult thing as HoloLens does not track your hand unless it is in ready or hold state. If this is enough, you can track the position of the user's hand and make a RayCast from the HoloLens' position toward the detected hand position. I had pretty good results with this.

    Hey @pstueven

    Thanks for your answer. So do I understand correctly that I can't find the hand when it is pointing somewhere with this? I guess that's why I try another approach of "object recognition". But I had no success with this yet.

  • pstuevenpstueven ✭✭
    Answer ✓

    @fyzzo said:
    So do I understand correctly that I can't find the hand when it is pointing somewhere with this?

    As far as I know, this is correct.

    You could try tracking a hand in ready state and the using WinML image classification to recognize a part of the HoloLenses camera view. In my tests image classification worked quite well with a well trained model. But if you use custom vision to train it, be sure about privacy concerns as there are no "Terms of use" pointing out what happens to the images uploaded.

  • @pstueven said:

    @fyzzo said:
    So do I understand correctly that I can't find the hand when it is pointing somewhere with this?

    As far as I know, this is correct.

    You could try tracking a hand in ready state and the using WinML image classification to recognize a part of the HoloLenses camera view. In my tests image classification worked quite well with a well trained model. But if you use custom vision to train it, be sure about privacy concerns as there are no "Terms of use" pointing out what happens to the images uploaded.

    Hey again! I googled for WinML examples on the HoloLens and found this https://github.com/meulta/mixedreality-azure-samples/tree/master/Standalone-Samples/WindowsML-CustomVision-Hololens which actually sounds very promising. It's the cognitive service in which you can define your own objects to find, but its supposed to work offline on the HoloLens.

    I downloaded it and opened it in Unity, but it doesn't look like there is a Scene file where it uses the Scripts as an example. Do you have an idea how to run it? I added the script to a game object, I think it also wants me to define a Label Game Object.

    I really don't know why someone would put an example online without much explanation to it. :/

    Thank you for all your help so far!

  • fyzzofyzzo
    edited August 2018

    @pstueven said:
    Hey @fyzzo ,
    I remember having trouble with this repo, too. And I did not want to mess with it.

    I used Mike Taulty's blog to get CustomVision working on our HoloLens
    https://mtaulty.com/2018/03/29/third-experiment-with-image-classification-on-windows-ml-from-uwp-on-hololens-in-unity/
    Also check out the first and second experiment by Mike on this. His explainations are really good :)

    Regards
    pstueven

    That looks amazing! I'm trying to get through it right now. At the moment I have issues installing the converter tools (winmltools). My python throws some errors while getting the dependencies, especially onnx ( https://pastebin.com/arYVmWvD "Failed building wheel for onnx" ). I'm not very experienced in installing python packages and google doesn't really find something that solves this.

    You already helped me so much, maybe you know how to do this?

  • pstuevenpstueven ✭✭
    edited August 2018

    You don't need a converter if you use custom vision, you can export it directly as *.onnx

    Maybe this is also interesting for @mtaulty to update his blog post. By the way: Thank you for your blog, it already helped me out twice :)

  • @pstueven said:
    You don't need a converter if you use custom vision, you can export it directly as *.onnx

    Maybe this is also interesting for @mtaulty to update his blog post. By the way: Thank you for your blog, it already helped me out twice :)

    Hi, I'm back from my vacation ;)

    Okay, now I'm skipping that step, because I have the onnx file already. Now I'm trying to generate some code, but I can't find the mlgen.exe and the Links that are linked in the blog only say I need the Windows SDK, which I should already have ... since I develop for HoloLens.

    I don't understand why I run against these walls all the time ...

  • Mike Taulty uploaded his project on GitHub. Have you tried running it? It helped me a lot. When it comes to your own model, you can just add your onnx model to the project and code will be generated (or you can edit the labels in Mike Taulty's code).

  • Hi - one thought around not being able to find the tools. Perhaps you don't have the right SDK? Are you following the guide here and using an SDK version that's later than 17723?

  • @mtaulty said:
    Hi - one thought around not being able to find the tools. Perhaps you don't have the right SDK? Are you following the guide here and using an SDK version that's later than 17723?

    Oh right. I have 17134. It looked kind of similar so I thought it's the correct one- But I also tried to get 17723, but it says I need to be a member of the Windows Insider program and when I click on the button that I want to join it ( here https://insider.windows.com ) I get a "403 Forbidden" error.

    But maybe I can just avoid that by using your code. Thank you btw. for the work and the blogposts. I have seen Custom Vision before and it could help to implement my idea, but I was missing the puzzle piece to run it on the HoloLens.

    The next part might be interesting for you as well:

    @pstueven said:
    Mike Taulty uploaded his project on GitHub. Have you tried running it? It helped me a lot. When it comes to your own model, you can just add your onnx model to the project and code will be generated (or you can edit the labels in Mike Taulty's code).

    Ah right. I can't test it on a HoloLens right now, but I prepared the Unity Project to test it as soon as I can. Doing so I got some issues/questions:

    1. A minor one: The project doesn't load the main scene automatically and is missing the standard HoloLens Settings. Just for convenience I think a published project should have those delivered. right?
    2. The Scriptholder was not finding the script for some reason. For this and Point 1 I read the blog post to manually set it up correctly.
    3. I don't know why the model is suddenly a *.bytes file. The blog only says that the onnx file was imported. Is the C# Script searching for *.bytes? Was it converted or just renamed from *.onnx?

    Thank you again for your time. You two are helping me a lot!

  • @fyzzo said:
    Oh right. I have 17134. It looked kind of similar so I thought it's the correct one- But I also tried to get 17723, but it says I need to be a member of the Windows Insider program and when I click on the button that I want to join it ( here https://insider.windows.com ) I get a "403 Forbidden" error.

    That's weird! As RS4 is already rolled out to everyone, you should be able to use the SDK...

    @fyzzo said:
    Ah right. I can't test it on a HoloLens right now, but I prepared the Unity Project to test it as soon as I can. Doing so I got some issues/questions:

    1. A minor one: The project doesn't load the main scene automatically and is missing the standard HoloLens Settings. Just for convenience I think a published project should have those delivered. right?
    2. The Scriptholder was not finding the script for some reason. For this and Point 1 I read the blog post to manually set it up correctly.
    3. I don't know why the model is suddenly a *.bytes file. The blog only says that the onnx file was imported. Is the C# Script searching for *.bytes? Was it converted or just renamed from *.onnx?
    1. I guess it is a problem of missing *.meta files. It is recommended not to put the these into the git repository. But I had the same issue. Could have told you, sorry.
      1. Had the same. See 1.
      2. I just renamed it and it worked.

    Regards
    @pstueven

  • fyzzofyzzo
    edited September 2018

    @pstueven said:

    @fyzzo said:
    Oh right. I have 17134. It looked kind of similar so I thought it's the correct one- But I also tried to get 17723, but it says I need to be a member of the Windows Insider program and when I click on the button that I want to join it ( here https://insider.windows.com ) I get a "403 Forbidden" error.

    That's weird! As RS4 is already rolled out to everyone, you should be able to use the SDK...

    At least that is showing when I click on the "Windows 10 SDK" link in the docs here: https://docs.microsoft.com/en-us/windows/ai/mlgen (Microsoft isn't great in using their own correct links sometimes...)

    I also tried googling the sdk directly and landed here https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk it wants me to uninstall the 17134.12 first.

    I don't think I should do that, because other people using this computer might need it for their HoloLens app. And we had already problems with updating windows systems breaking things. I first thought I only need it to get the mlgen (that I don't need anymore), but now I try to open the built Unity project in Visual Studio to deploy it on the hololens but it tells me I need a newer SDK version. So I can't open it with only 17134?

    I guess I need to ask some people here and if there is a possibility to update or I can't continue.

    Thanks for all the help anyway!

  • That's right, I was asked to do that, too. Any possibility of setting up a Virtual Machine?

  • fyzzofyzzo
    edited September 2018

    I think I made a mistake by updating Visual Studio or something, because I can't deploy any Unity Projects any more. VS always tells me I need the new SDK now.

    Edit: ah I just can't open it with VS 2015 any more. I'm still trying things out.

  • Hey @pstueven ,

    I feel like I should know this, but I'm running the app now on my HoloLens but it isn't showing anything when I'm looking at dogs or anything else. What am I missing here?

Sign In or Register to comment.