Hello everyone.

We have decided to phase out the Mixed Reality Forums over the next few months in favor of other ways to connect with us.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

The plan between now and the beginning of May is to clean up old, unanswered questions that are no longer relevant. The forums will remain open and usable.

On May 1st we will be locking the forums to new posts and replies. They will remain available for another three months for the purposes of searching them, and then they will be closed altogether on August 1st.

So, where does that leave our awesome community to ask questions? Well, there are a few places we want to engage with you. For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality. If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers. And always feel free to hit us up on Twitter @MxdRealityDev.

Computer Vision for Unity/UWP

DanglingNeuronDanglingNeuron ✭✭✭
edited August 2016 in Questions And Answers

Object Tracking and Marker/Shape Detection will require computer vision library that could be used in Unity/C# scripts OR a plugin for UWP - since the documentation states that these are not part of the HoloLens/UWP SDK.

Anyone knows of a computer vision library that could be used in Unity3D/C#/UWP?

Most of the readily available Computer Vision stuff is in C++ and not UWP compatible :sweat:

Any thoughts? @mavasher @MarkMMullin

Healthcare IT professional by day - Indie GameDev for UWP and mobile platforms by night

Tagged:

Best Answers

Answers

  • DanglingNeuronDanglingNeuron ✭✭✭
    edited March 2016

    @james_ashley said:
    There are OpenCV wrappers in the Unity Asset store. Lucian Wischik is currently working on an OpenCV build that works with UWP and the first iteration is available on NuGet.

    Ye asked teh Enox publisher about the "beta" status for UWP..

    Playing around with Lucian solution to use it from C#/UWP as a first step.

    The only caveat is.. too many round trips across boundaries - dont know how it effects perf... or maybe I dont know what I am talkin about :)

    e.g. Hololens camera feed captured in Unity scripts via LocatableWebCam (of course the frame goes up from the native side) , then my unity script code sending it back down using a custom unity plugin to C# and then sending it to OpenCV WRT component and then to OpenCV - and results go all the way up.

    All the good stuff is in C/C++ :)

    Healthcare IT professional by day - Indie GameDev for UWP and mobile platforms by night

  • DanglingNeuronDanglingNeuron ✭✭✭
    edited March 2016

    @holovr said:
    vuforia is working on a windows 10 version of their AR SDK. that will allow you to do markbased AR very well. Let hololens do the actual tracking and have Vuforia do the marker recognition and object instantiation

    That would be great .. used vuforia for Android, pretty cool. But not too hopeful about the port, they dint do it for WP8/8.1 as well. Now they announces Windows 10, but not sure if that really means Windows 10 UWP or Windows 10 old style :)

    Healthcare IT professional by day - Indie GameDev for UWP and mobile platforms by night

  • Any updates on this? Has anyone successfully used the Unity OpenCV wrappers (Enox) to run computer vision on HoloLens?

  • @adityas said:
    Any updates on this? Has anyone successfully used the Unity OpenCV wrappers (Enox) to run computer vision on HoloLens?

    The only viable option may be enox unity plugin (even though he says Windows 10 UWP is in beta) . I havent yet bought the plugin yet.

    But good news is Vuforia was demoed a week ago in BUILD 2016 so they have ported it to Unity/UWP. Developer use is free of course but there is licensing cost when app goes live. Vuforia is still in beta as well but when its delivered, it will be awesome. So either wait for Vuforia.. or purchase Enox plugin.

    The OpenCV wrappers wont work because the wrappers are regular .Net wrappers.. they are not UWP compatible.

    Healthcare IT professional by day - Indie GameDev for UWP and mobile platforms by night

  • You might want to check out the theater presentation "Lessons from the Lowe's Kitchen 2: Obtaining Precision Alignment" from Build on Channel 9 when/if they post the recording. I recall Kevin spoke about some of their challenges with using markers and mentioned something about aligning regular images like posters in the near future.

    Windows Holographic User Group Redmond

    WinHUGR.org - - - - - - - - - - - - - - - - - - @WinHUGR
    WinHUGR YouTube Channel -- live streamed meetings

  • The OpenCV wrapper on Unity Store by Enox works on hololens. According to their website, some module is not ported yet, but most of the basic function can be called without problem.

  • Hi ! Just tried Vuforia on Hololens, and tried stereoAr demo. If you are interested I can upload a video so you can judge for yourself.

  • JasonJason ✭✭
    edited November 2016

    @XaviGuardia Hi, if you don't mind, I'd appreciate viewing a video. I ran the 'HelloWorld' sample a few months ago (images of the stones with the teapots overlaid on top) so I'm curious to see any new features for Vuforia with HoloLens. Thanks.

  • ksamranksamran
    edited November 2016

    Hello Everyone,

    Pretty Useful discussion going on. I have read in above discussion that OpenCV wrapper on Unity Store by Enox works on hololens @ YuJen,

    Can you guide me a bit. I am able to ran openCv lib on unity using the same wrapper and also compile it successfully. But when I upload the app on hololens I can not see Opencv lib working in the app.

    Any idea? Can anyone share gud viseo using the same OpenCV wrapper

    Thank You

  • If you are comfortable with native C++. You should try using a wrapper of native image processing and call it in C#.
    I built a repository to interface ARToolKit from HoloLens.
    Check here: https://github.com/qian256/HoloLensARToolKit

  • I tried opencvsharp, found at https://github.com/shimat/opencvsharp.
    I made it working in the unity.
    However, when compiling in the VS, it failed with Severity Code Description Project File Line Suppression State Error The command " "D:\Hololens Projects\OpenCVSharpLearning\App\Unity\Tools\AssemblyConverter.exe -platform=uap -lock="D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\project.lock.json" -bits=32 -configuration=Release -removeDebuggableAttribute=False -path="." -path="C:\Program Files\Unity\Editor\Data\PlaybackEngines\MetroSupport\Players\UAP\dotnet\x86\Release" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\Assembly-CSharp.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\Assembly-CSharp-firstpass.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.UI.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\OpenCvSharp.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.HoloLens.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.Networking.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.VR.dll"" exited with code 1. EmguLearning D:\Hololens Projects\EmguLearning\App\EmguLearning\EmguLearning.csproj 278
    And here is stack trace.
    D:\Hololens Projects\OpenCVSharpLearning\App\Unity\Tools>AssemblyConverter -platform=uap -lock="D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\project.lock.json" -bits=32 -configuration=Release -removeDebuggableAttribute=False -path="." "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\OpenCvSharp.dll" Failed to fix references for attribute System.Security.SuppressUnmanagedCodeSecurityAttribute Failed to fix references for type OpenCvSharp.NativeMethods System.NullReferenceException: Object reference not set to an instance of an object. at Unity.FixReferencesStep.Visit(ICustomAttributeProvider provider, GenericContext context) at Unity.FixReferencesStep.Visit(TypeDefinition type) at Unity.TypeDefinitionDispatcher.DispatchType(TypeDefinition type) at Unity.TypeDefinitionDispatcher..ctor(ModuleDefinition module, ITypeDefinitionVisitor visitor) at Unity.FixReferencesStep.ProcessModule() at Unity.ModuleStep.Execute() at Unity.FixReferencesStep.Execute() at Unity.Step.Execute(OperationContext operationContext, IStepContext previousStepContext) at Unity.Operation.Execute() at Unity.Program.Main(String[] args)

    Basically Hololense AssemblyConverter does not like SuppressUnmanagedCodeSecurityAttribute in the wrapper.

    I guess it would be same if any openCV wrapper lib uses this attribute, it won't compile into hololens.

  • For people who still working on this, you can get the package provided from unity store. However, when I watched the demo video long time back the performance was not that good.

    Here is the steps I took couple of months ago to consume opencv on HoloLens.

    1. Create a simple Unity native plugin wrapper to pass frame from webcam to the native plugin to do opencv process.
    2. Create Unity plugin reference opencv to heavy processing. (Passing mat crossing native and managed binaries could get a lot perf hit)
    3. Copy UWP opencv build to hololens.

    The simple app I did was tracking a red/blue ball via HoloLens, it did work but it was hard to estimate the correct distance due to the 3D environment mapping accuracy and lighting change when head moves. The object recognition on every frame is taking a lot of resources, 5 fps seems reasonable for processing video frames.

    Reference:
    Unity Native Plugin
    Build UWP compatible opencv binaries

    FYI, opencvsharp is not windows 10 uwp compatible so it can't be used on HoloLens, more importantly, also all API consumed need to be store compliant for listing on Windows store.

    Hope this will help.

Sign In or Register to comment.