Computer Vision for Unity/UWP

DanglingNeuron

Object Tracking and Marker/Shape Detection will require computer vision library that could be used in Unity/C# scripts OR a plugin for UWP - since the documentation states that these are not part of the HoloLens/UWP SDK.

Anyone knows of a computer vision library that could be used in Unity3D/C#/UWP?

Most of the readily available Computer Vision stuff is in C++ and not UWP compatible

Any thoughts? @mavasher @MarkMMullin

james_ashley

There are OpenCV wrappers in the Unity Asset store. Lucian Wischik is currently working on an OpenCV build that works with UWP and the first iteration is available on NuGet.

holovr

vuforia is working on a windows 10 version of their AR SDK. that will allow you to do markbased AR very well. Let hololens do the actual tracking and have Vuforia do the marker recognition and object instantiation

mavasher

My team is actively looking at the various possibilities. We've used Vuforia on Android before, and before that, we created our own object tracking solution using a combination RGB and TOF camera. Right now, the question is really how to get the right packages into Unity with the appropriate speed:

• Enox's Unity Asset Store Entry: $95 is worth every penny if this works well, but my team is worried about the fact that it seems to be the JAVA build of OpenCV made into a plugin. We could be wrong about that. I'm not sure what kind of performance hit we'll take on under this kind of arrangement.
• EMGU CV - It's a GPL license which makes things difficult. You can purchase a commercial license but they are pretty pricy.
• OpenCvSharp - don't know too much about this one.

It's kind of frustrating because it feels like we are re-inventing the wheel here. I would have hoped that robust object tracking would be included natively. It's also frustrating that we're limited to just the RGB camera given all the sensors on the hardware.

It's still ambiguous to me whether Vuforia will play nice with the Hololens even if a Windows 10 version becomes available. In the early days of the Epson Moverio, we basically hacked the Vuforia tracking system to play nice on a transparent, binocular view. That took some doing. Later they came out with the Vuforia for digital eyewear which basically got the same results we had.

Our biggest issue at this point is mostly that we are tired of waiting around. We want to get moving on the project. We've been developing for several years now and the Hololens represents the first time we've had hardware that can handle our application.

I'd love to know what your plan is for implementing object tracking. We're still investigating.

james_ashley

There are OpenCV wrappers in the Unity Asset store. Lucian Wischik is currently working on an OpenCV build that works with UWP and the first iteration is available on NuGet.

holovr

vuforia is working on a windows 10 version of their AR SDK. that will allow you to do markbased AR very well. Let hololens do the actual tracking and have Vuforia do the marker recognition and object instantiation

DanglingNeuron

@james_ashley said:
There are OpenCV wrappers in the Unity Asset store. Lucian Wischik is currently working on an OpenCV build that works with UWP and the first iteration is available on NuGet.

Ye asked teh Enox publisher about the "beta" status for UWP..

Playing around with Lucian solution to use it from C#/UWP as a first step.

The only caveat is.. too many round trips across boundaries - dont know how it effects perf... or maybe I dont know what I am talkin about

e.g. Hololens camera feed captured in Unity scripts via LocatableWebCam (of course the frame goes up from the native side) , then my unity script code sending it back down using a custom unity plugin to C# and then sending it to OpenCV WRT component and then to OpenCV - and results go all the way up.

All the good stuff is in C/C++

DanglingNeuron

@holovr said:
vuforia is working on a windows 10 version of their AR SDK. that will allow you to do markbased AR very well. Let hololens do the actual tracking and have Vuforia do the marker recognition and object instantiation

That would be great .. used vuforia for Android, pretty cool. But not too hopeful about the port, they dint do it for WP8/8.1 as well. Now they announces Windows 10, but not sure if that really means Windows 10 UWP or Windows 10 old style

mavasher

My team is actively looking at the various possibilities. We've used Vuforia on Android before, and before that, we created our own object tracking solution using a combination RGB and TOF camera. Right now, the question is really how to get the right packages into Unity with the appropriate speed:

• Enox's Unity Asset Store Entry: $95 is worth every penny if this works well, but my team is worried about the fact that it seems to be the JAVA build of OpenCV made into a plugin. We could be wrong about that. I'm not sure what kind of performance hit we'll take on under this kind of arrangement.
• EMGU CV - It's a GPL license which makes things difficult. You can purchase a commercial license but they are pretty pricy.
• OpenCvSharp - don't know too much about this one.

It's kind of frustrating because it feels like we are re-inventing the wheel here. I would have hoped that robust object tracking would be included natively. It's also frustrating that we're limited to just the RGB camera given all the sensors on the hardware.

It's still ambiguous to me whether Vuforia will play nice with the Hololens even if a Windows 10 version becomes available. In the early days of the Epson Moverio, we basically hacked the Vuforia tracking system to play nice on a transparent, binocular view. That took some doing. Later they came out with the Vuforia for digital eyewear which basically got the same results we had.

Our biggest issue at this point is mostly that we are tired of waiting around. We want to get moving on the project. We've been developing for several years now and the Hololens represents the first time we've had hardware that can handle our application.

I'd love to know what your plan is for implementing object tracking. We're still investigating.

adityas

Any updates on this? Has anyone successfully used the Unity OpenCV wrappers (Enox) to run computer vision on HoloLens?

DanglingNeuron

@adityas said:
Any updates on this? Has anyone successfully used the Unity OpenCV wrappers (Enox) to run computer vision on HoloLens?

The only viable option may be enox unity plugin (even though he says Windows 10 UWP is in beta) . I havent yet bought the plugin yet.

But good news is Vuforia was demoed a week ago in BUILD 2016 so they have ported it to Unity/UWP. Developer use is free of course but there is licensing cost when app goes live. Vuforia is still in beta as well but when its delivered, it will be awesome. So either wait for Vuforia.. or purchase Enox plugin.

The OpenCV wrappers wont work because the wrappers are regular .Net wrappers.. they are not UWP compatible.

HoloSheep

You might want to check out the theater presentation "Lessons from the Lowe's Kitchen 2: Obtaining Precision Alignment" from Build on Channel 9 when/if they post the recording. I recall Kevin spoke about some of their challenges with using markers and mentioned something about aligning regular images like posters in the near future.

YuJen

The OpenCV wrapper on Unity Store by Enox works on hololens. According to their website, some module is not ported yet, but most of the basic function can be called without problem.

XaviGuardia

Hi ! Just tried Vuforia on Hololens, and tried stereoAr demo. If you are interested I can upload a video so you can judge for yourself.

Jason

@XaviGuardia Hi, if you don't mind, I'd appreciate viewing a video. I ran the 'HelloWorld' sample a few months ago (images of the stones with the teapots overlaid on top) so I'm curious to see any new features for Vuforia with HoloLens. Thanks.

ksamran

Hello Everyone,

Pretty Useful discussion going on. I have read in above discussion that OpenCV wrapper on Unity Store by Enox works on hololens @ YuJen,

Can you guide me a bit. I am able to ran openCv lib on unity using the same wrapper and also compile it successfully. But when I upload the app on hololens I can not see Opencv lib working in the app.

Any idea? Can anyone share gud viseo using the same OpenCV wrapper

Thank You

longqian

If you are comfortable with native C++. You should try using a wrapper of native image processing and call it in C#.
I built a repository to interface ARToolKit from HoloLens.
Check here: https://github.com/qian256/HoloLensARToolKit

changgyu

I tried opencvsharp, found at https://github.com/shimat/opencvsharp.
I made it working in the unity.
However, when compiling in the VS, it failed with Severity Code Description Project File Line Suppression State Error The command " "D:\Hololens Projects\OpenCVSharpLearning\App\Unity\Tools\AssemblyConverter.exe -platform=uap -lock="D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\project.lock.json" -bits=32 -configuration=Release -removeDebuggableAttribute=False -path="." -path="C:\Program Files\Unity\Editor\Data\PlaybackEngines\MetroSupport\Players\UAP\dotnet\x86\Release" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\Assembly-CSharp.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\Assembly-CSharp-firstpass.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.UI.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\OpenCvSharp.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.HoloLens.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.Networking.dll" "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\UnityEngine.VR.dll"" exited with code 1. EmguLearning D:\Hololens Projects\EmguLearning\App\EmguLearning\EmguLearning.csproj 278
And here is stack trace.
D:\Hololens Projects\OpenCVSharpLearning\App\Unity\Tools>AssemblyConverter -platform=uap -lock="D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\project.lock.json" -bits=32 -configuration=Release -removeDebuggableAttribute=False -path="." "D:\Hololens Projects\OpenCVSharpLearning\App\OpenCVSharpLearning\OpenCvSharp.dll" Failed to fix references for attribute System.Security.SuppressUnmanagedCodeSecurityAttribute Failed to fix references for type OpenCvSharp.NativeMethods System.NullReferenceException: Object reference not set to an instance of an object. at Unity.FixReferencesStep.Visit(ICustomAttributeProvider provider, GenericContext context) at Unity.FixReferencesStep.Visit(TypeDefinition type) at Unity.TypeDefinitionDispatcher.DispatchType(TypeDefinition type) at Unity.TypeDefinitionDispatcher..ctor(ModuleDefinition module, ITypeDefinitionVisitor visitor) at Unity.FixReferencesStep.ProcessModule() at Unity.ModuleStep.Execute() at Unity.FixReferencesStep.Execute() at Unity.Step.Execute(OperationContext operationContext, IStepContext previousStepContext) at Unity.Operation.Execute() at Unity.Program.Main(String[] args)

Basically Hololense AssemblyConverter does not like SuppressUnmanagedCodeSecurityAttribute in the wrapper.

I guess it would be same if any openCV wrapper lib uses this attribute, it won't compile into hololens.

KennethZhang

For people who still working on this, you can get the package provided from unity store. However, when I watched the demo video long time back the performance was not that good.

Here is the steps I took couple of months ago to consume opencv on HoloLens.

1. Create a simple Unity native plugin wrapper to pass frame from webcam to the native plugin to do opencv process.
2. Create Unity plugin reference opencv to heavy processing. (Passing mat crossing native and managed binaries could get a lot perf hit)
3. Copy UWP opencv build to hololens.

The simple app I did was tracking a red/blue ball via HoloLens, it did work but it was hard to estimate the correct distance due to the 3D environment mapping accuracy and lighting change when head moves. The object recognition on every frame is taking a lot of resources, 5 fps seems reasonable for processing video frames.

Reference:
Unity Native Plugin
Build UWP compatible opencv binaries

FYI, opencvsharp is not windows 10 uwp compatible so it can't be used on HoloLens, more importantly, also all API consumed need to be store compliant for listing on Windows store.

Hope this will help.