Running OpenCV 3 on HoloLens (C#)

holodoctor

Hi,

There were a few related threads on this forum, but none seemed to be particularly helpful (in terms of details). I'm interested in using plain vanilla OpenCV 3.1 in my Unity/C# project. There seem to be a few wrapper libraries that expose C# interface, but my preference would be to be able to use the OpenCV distribution directly (for various reasons). Also, at this point, the prudent thing to do for a person like me (who have no or little experience with OpenCV) is to invest on OpenCV3 rather than on the older API. (I hear that OpenCV3 is much better in terms of performance, for one thing.) I can imagine that I'll end up spending quite a bit of time coding with OpenCV if I work on HoloLens apps moving forward.

I have a few questions to the community.

(1) Has anybody successfully run OpenCV 3 on HoloLens (for Unity/C# apps)? If so, would you mind sharing some details? The current binary distribution for OpenCV3 (C/C++) seems to include x64 but not x86. Did you have to build an x86 binary from scratch? Is there some kind of additional binary repo (outside the sourceforge download site)?

(2) I'm somewhat disappointed by a lack of support for C# language in general, and for C#/UWP in particular. The OpenCV official effort includes support for Java/Android and Python, among other things. But, there is no C# version. It appears that the C++ version can be run on UWP, and hence in theory it can be made accessible from C# (or, from any UWP-supported language). Has anybody tried? Any problems?

(3) I did some research on the web, and there is really no useful information on running OpenCV3 on UWP (e.g., on 32bit OS). I can see what is generally needed to make it work (and it seems straightforward enough), but the fact that I cannot find any concrete information makes me a bit suspicious. Would it be a waste of time for me even to try? Why hasn't anybody (including Microsoft) built a nuget package or something like that so that we do not all have to go through the same tedious steps of building a binary for OpenCV?

Thanks!
~h

keymaster

@Colton_TSBD It's important to note that the standard AprilTag library does not work on the Hololens. I used a version that was in the HoloTookit, however that appears to have been removed, in favour of the poster recognition sample that @jbienzms posted. I have also moved to a similar approach now, but using the OpenCV dlls by @sylvain and the aruco module in OpenCV. I should be able to release that soon. For now, the best approach is probably to use the Companion Kit code or make your own plugin to suit your needs.

ahillier

@holodoctor,
I have not tried OpenCV on the HoloLens yet, but I recently purchased this from the Unity Asset store (for experimenting with during my free time):
https://www.assetstore.unity3d.com/en/#!/content/21088

The current version lists beta-level support for UWP apps, so I'm not sure how well it will work on the HoloLens. It's a bit pricey, so I'd suggest reading the comments and deciding if the investment is worth it, or if you really want to take on the challenge of wrapping it for Unity yourself.

HoloSheep

@ahillier having an OpenCV Windows 10 managed wrapper available as part of the platform (or as a HoloToolkit extra) would be an awesome move by Microsoft to support creative next generation development on the Windows Holographic Platform.

I think such an idea might get a lot of support as a feature request from members of the forum.

holodoctor

@ahillier Thanks for the tip. I'll give it a try. @HoloSheep I wonder if there are people on this forum who want to work on a project like this together. Microsoft seems to have been briefly involved with the OpenCV project, but their effort was stopped almost a year ago (right around the time Windows 10/UWP was released). I am not sure if/when they will do this again.

Jimbohalo10

Well I built the kit from the https://sourceforge.net/projects/opencvlibrary
version 3.1, changed the Visual Studio Solution generated by CMake.
For VS 2015 with Universal Windows and x86, built with error.
On trying to the Auto Install supplied with OpensCV, the thing fails
even though ALL_BUILD(Universal Windows) and the library is built in x86. Now we find that VC++ is not supported in Windows Store build.

---

Error MSB8016 This project does not support the current Configuration Type (Utility).
ALL_BUILD C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppBuild.targets line 358

Line 358 is !-- Error out if Windows Store app projects has an unsupported Configuration Type --!

---

So no you wont have OpenCV on HoloLens basically because it VC++ and because Windows.Target= Windows.Holographic is ONLY supported in Visual C# Sharp. The code is UWP Windows 10 target because CMake creates the templates so just tweaking in the project files make this work in Visual Studio

Well here is the library built as it should be built in x86

Hopefully this will save someone some work.
The instructions to build with Visual Studio 2015 are at
http://homepages.ed.ac.uk/cblair2/opencv/opencv_tutorial.pdf

Well this saved me some dollars and OpenCV is a really good package.

holodoctor

@Jimbohalo10 So, were you able to use the x86 binary in Unity apps for HoloLens?

I spent some time yesterday, and as it turned out, building a binary wasn't a big huddle for me. (I just cloned opencv/opencv. I ran cmake, and I built the binaries using the generated VS sln file. No errors whatsoever.) I tried to tweak some generated VS settings, but it didn't work. Essentially, these are unmanaged ("native") VC++ binaries. Based on the information I got, these binaries are "compatible" with WinRT (Windows 8.1), which I think is good enough for UWP.

So, my presumption at this point is that this native C++ OpenCV binary (x86/x64) should be usable in any UWP apps (including holographic apps).

The next step for me was to figure out how to use unmanaged C++ library in a Unity app (C#). In general, this is straightforward enough. I tested a simple scenario, and it worked just fine. There seemed to be multiple ways to do this, but if you have access to the source code (which we do, for opencv), all you have to do is to export symbols and use the C binding (extern "C"). (Search for "plugins" in Unity documentation if you are not familiar with steps of using "Unmanaged C++" plugins.)

I think there are a few ways we can "wrap" OpenCV binary.
(1) We can just use exported methods in Unity scripts (with appropriate extern declarations). No additional work needed assuming that we do all the necessary work on the opencv codebase.
(2) We can create a managed C# wrapper library (e.g., "Universal Class Library" VS template), which exposes a C# interface to Unity apps.
(3) Or, we can create a managed C++ wrapper library (e.g., "Windows Runtime Component" or winmd), which can be consumed in C# code.
(4) One other option is to wrap a "functionality/module" in a managed C++ library. I think it is, in some sense, the best option. Suppose that you have a certain task to perform using OpenCV (say, detecting spoons on a table). Then, write that function in C++, and package it as a winmd.

I was going to explore these options yesterday. But, then I realized that this might not work.

I spent quite a bit of time going through, and trying to understand, these horrible mess of .Net versioning and different runtimes (WinRT, UWP, Mono, ..), etc., and what I learned is that the "wrapping" may not be as straightforward.

The biggest problem I see is that, the Mono used by Unity is compatible with .Net 3.5. On the other hand, the current .Net version which UWP uses is 4.5 or 4.6. (On top of this, there is a new .Net (.Net.Core?) coming out, which is not even compatible with 4.x as far as I can tell.) I cannot be entirely sure but based on VS settings I think UWP is compatible with down to .Net 4.0, but not with .Net 3.5. So, I feel like I'm in a no-man's land. It seem like I cannot produce a managed library that is compatible with both Unity and UWP.

I could be wrong, but that's what I figured so far. I'll try to pick up where I left off next weekend, if possible. But, for now, I'm pretty discouraged in pursuing this further.

Any input will be greatly appreciated.

Thanks,
~h

Jimbohalo10

@holodoctor Managed Plugin may be the way forward
http://docs.unity3d.com/Manual/Plugins.html
And Managed Plugin page http://docs.unity3d.com/Manual/UsingDLL.html
or maybe get the old Unity version for free
http://enoxsoftware.github.io/OpenCVForUnity/
See the video on how to install https://www.youtube.com/watch?v=HnXGIvHvU9I&feature=youtu.be

holodoctor

@Jimbohalo10 Yes, that's what I'm thinking of doing, creating a managed plugin from an unmanaged/native C++ dll (which doesn't seem as straightforward at this point). @ahillier also points out OpenCVforUnity, which seems to be one of the most popular opencv lib assets on the Asset store, but I am not sure if that's the way to go. The product page claims that the C# version is a port of OpenCV for Java. That means, the C# version does not share any codebase with the classic OpenCV (C/C++). It simply uses the same (or similar) API as OpenCV Java, and the implementation is a backport of Java implementation (which may or may not be optimal for C#). At least, that's what I think it is. I would much rather use an open source version of OpenCV (in some way) than a third party software. It's not just about saving a few bucks. (BTW, thanks for the pointers. The GitHub page you posted, however, contains only sample codes not the actual library source.) Regards, ~h

Jimbohalo10

@holodoctor

The product page claims that the C# version is a port of OpenCV for Java. That means, the C# version does not share any codebase with the classic OpenCV (C/C++). It simply uses the same (or similar) API as OpenCV Java, and the implementation is a backport of Java

Well then there is a problem with Java on Edge/HoloLens Emulator/Device Java is not supported. See screen below

Oh dear, no Java No OpenCV for Unity?

holodoctor

@Jimbohalo10 I don't think Java is required to be able to use OpenCV4Unity. As far as I can tell, it is written in C#. The publisher says that it is a "clone" (or, port) of OpenCV Java. (My guess is that they probably used some kind of Java-to-C# translator tool to get started.) In any case, that's an interesting observation. We cannot run Java on HoloLens? (In hindsight, that's obvious. You cannot run Java apps on UWP.) ~h

Jimbohalo10

@holodoctor
The answer could be OpenCvSharp

https://nuget.org/packages/OpenCvSharp-WithoutDll/
and
Unity How-to-integrate-opencv-emgu-or-opencvsharp

Sources to OpenCvSharp 3.1 at https://github.com/shimat/opencvsharp/releases

Oh there is so much on this. I am down to what seems like 3 errors. No more time to play fixing this today.
Much more easy than exporting headers
I have got the some VC# samples to run in VS2015

holodoctor

@Jimbohalo10 yes, as a matter of fact, I was just looking at opencvsharp GitHub repo. The nuget package opencvsharp-anycpu (OpenCV 3.1) does not install into my UWP test app project. But, since the source code is available, I can take a look and see if it's something we can leverage on. (There was an issue open requesting UWP support, but there were no replies from the developers.) I'm now testing Lucian's UWP builds, https://www.nuget.org/packages/OpenCV.UWP.native.all/

Edit Just FYI, I also found an interesting project, OpenCV.Net: https://bitbucket.org/horizongir/opencv.net. The nuget package does not install on UWP, however. I'm not sure how easy/difficult it will be to add UWP support....

Jimbohalo10

@holodoctor Its exciting that there are so many OpenCv options available. I need to get a good builds of OpenCvSharp, then convert to get it into Unity and convert to UWP, build the libraries for HoloLens and run test units on HoloLens emulator, my only worry is how the Locate Camera for HoloLens will work

holodoctor

@Jimbohalo10 yes, it's good there are so many options. obviously, the task for us is to make (at least) ONE work on UWP, and in HoloLens Unity/C# apps. I'm pretty optimistic, but I still feel like it's a crap shoot at this point. I'll keep you posted if I make any progress on this front. ~h

Fireman

@holodoctor I am also looking for using OpenCV on hololens. I found there is a OpenCV C# wrapper. http://www.emgu.com/wiki/index.php/Main_Page. Can you use that?

holodoctor

@Fireman I think Emgu CV is probably one of the most widely used OpenCV clones/wrappers on .net. As far as I can tell, however, it does not support UWP. In fact, they have a Unity asset on Asset Store, https://www.assetstore.unity3d.com/en/#!/content/24681, but the product page only mentions windows 8.1 not UWP.

holodoctor

OK, if anybody is still interested at this point , I made it! (It's been only a few days, but it feels like it's been much much longer to me for some reason.)

I suggested earlier a number of different ways C++ OpenCV can be used in UWP apps (including HoloLens). I succeeded in using option (4) (and, by generalization, (3)). Ideally, we should have a C# wrapper, but maybe later. For now, we can use OpenCV in C# UWP apps (and, hence in HoloLens apps), and that's good enough for me.

Here's what I did.

(a) Create a Visual C++ project, Windows -> Universal -> DLL. (My preference would be using Windows Runtime Component, but I couldn't make it work due to the way Unity generates VS projects, etc.)
(b) Add a nuget package OpenCV.UWP.native.all (by Lucian).
(c) Create a method that encapsulates a desired function (e.g., counting spoons on a table) using C++ OpenCV API. Use the C binding (extern "C") and "dllexport" the method.
(d) Add the generated dlls (and opencv dlls) into a Unity app project's "Plugins" directory. (You can use "x86" and "x64" subdirectories if you target both x86 and x64 platforms.)
(e) In a C# script, "dllimport" the target method.

You cannot run the app in Unity Editor if you have external plugins like this. You will need to generate VS projects first and run the app from Visual Studio.

To be honest, I'm still confused as to what is managed and what is unmanaged, etc. I am not sure why I had to "dllimport" (known as "explicit pinvoke") when I created a "universal" DLL. (Shouldn't this be a managed C++/CLI dll?) Don't know. I might be able to "simplify" this process somewhat, but maybe later. For now, the most important thing is "it works" (which provides an incentive to invest more time and effort along this line).

Remaining tasks at this point:

(A) Reproduce the OpenCV build process for UWP so that we can build the current OpenCV code base. (Lucian's builds, in nuget repo, are several months old.)
(B) We can potentially "wrap" all OpenCV APIs into universal C++ dlls or winmds. (Option (3) above.) Obviously, this will be a major undertaking, and it will require some kind of "team effort".
(C) Create a set of OpenCV test cases, and test them in Unity apps.
(D) Finally, most importantly, test this method on HoloLens.

Also, working on some kind of a "C# wrapper" is still an option.

Anyways, if anybody needs more information, I can provide some more details.

Best,
~h
"HoloLens + OpenCV = Better HoloLens"

Jimbohalo10

@holodoctor Really pleased you have managed to build your OpenCV application.
I have since found some OpenCV debug tools for Visual Studio
2 Tools results for OpenCv

The might help when looking at images OpenCV apps.
Image Watch: viewing in-memory images in the Visual Studio debugger

holodoctor

@Jimbohalo10 Thanks for the links. Those tools look pretty promising/interesting, but unfortunately they seem somewhat outdated. (e.g., Ceemple only supports VS2013.) I'll do some more digging, and if I find anything notable, I'll keep you informed. Thanks, ~h

Fireman

@holodoctor Did you actually run it on hololens?

holodoctor

@Fireman Not yet. I'm still working on the "Remaining tasks (A)".

Fireman

@holodoctor Also found AForge.Net. It has many computer vision routines in C#

holodoctor

@Fireman thanks for the link. the project seems to be non-active at this point, but I'll take a look. At this point, I'm still struggling with "Task (A)". Creating a nuget package for native binaries seems to be 100 times more difficult than creating a nuget for .Net libraries. If anybody's interested, I'll post a followup if/when I make any progress. ~h

Fireman

@holodoctor If you can post your experience, a lot of people can benefit from it. Your effort is deeply appreciated.

holodoctor

@Fireman Thanks for the kind words.

I made some progress. I can now build nuget packages for OpenCV C++ native binaries on Windows. It is only a small part of the whole process I'm trying to create, but still I view this as an important step (albeit optional). Using nuget packages can help speed up general development process.

Here's what I achieved so far.
(1) I created nuget packages of OpenCV C++ builds, based on the current codebase (as of 1~2 days ago). You can find it on NuGet Gallery. The package id is opencv.win.native.
(2) More importantly, if anybody wants to build a nuget package based on their customized OpenCV binaries, it can be easily done now. As it turns out, building nuget packages for native DLLs is not a trivial task. I created a GitHub repo for the benefit of others:

• OpenCV for Windows

Please let me know if you need any further information.
~h

Jimbohalo10

@holodoctor You have made great progress with this project. Many people will benefit from your new NuGet Package. Thank you sharing the source packages with everyone. To have got this far in such a short time is very impressive of you skills and commitment

holodoctor

@Jimbohalo10 thanks for the compliment. I do hope it'll be useful for people who are trying to use OpenCV on Windows (not just on HoloLens), but at the end of the day it's just a first step. There are a lot more work to be done. ~h

Jimbohalo10

@holodoctor I created a VC# Holographic UWP solution, which is for testing all the OpenCV package. EmguCV.221.x86 from NuGet loads successfully into UWP, into the VS 2015 Update 3,VC# Holographic UWP template. I have built the template see below

I know its Open CV version 2.21, but the 3.1 version fails win 10 UWP conflict.

My app is called HoloVCsharpEmguCV221x86_1.0.0.0_x86_
and it runs inside the HoloLens Emulator

What I need now is some code to activate/test the library? If anyone who can write OpenCV C++ code which may convert has any Visual C# sharp code they could give.

Small tests at first to see if or how it errors.

In the meanwhile I will look into the Examples in OpenCvSharp.

holodoctor

@Jimbohalo10 Glad to hear that you are making a progress. The best test case will be, to load an image (from a file or from a camera) and do simple manipulations. You can easily find a sample code on the Web. When I tried OpenCV last week, I didn't even bother with that. The primary goal for me was just to make sure that things compile and link. My test case was like this (probably the simplest code snippet you can imagine using OpenCV):

cv::Mat img1(320, 260, CV_8UC3, cv::Scalar(125, 0, 0));
std::cout << "img1 = " << img1 << std::endl;

Good luck!
~h

glopes

@holodoctor just stumbled upon this discussion on the interwebs. Great to hear about your progress! I'm actually the developer of one of the wrapper solutions you posted earlier (OpenCV.NET) and for a while I've been trying to come up with the best way to bring OpenCV 3 into the managed world.

C++/CLI definitely works for UWP, but the problem is it will not work for non-windows platforms since C++/CLI is Windows only. Another way to work around this is to first make a C-wrapper around the C++ classes and then P/Invoke into that. Once we have this C-wrapper library I believe we could start upgrading OpenCV.Net to take advantage of it.

For a long time I've been proposing to the OpenCV community that they actually automatically generate this C-wrapper, since C is still the best language for interop (not just for C# but for many other languages): https://github.com/opencv/opencv/issues/4705

Still on hold unfortunately, but it could in theory be created automatically the same way Python, Java and other bindings are generated.

Just wanted to provide my feedback on this and say I would be very interested to work on a wrapper or simply upgrade OpenCV.NET once this C-binding issue has been resolved.

Jimbohalo10

@holodoctor Update so far
OpenCvSharp documentation

has been great help to me in building the plugins libraries for Unity 5.4f1 HTL, 32 bit.

OpenCvSharp is a wrapper of OpenCv C++

The OpenCvSharp plugins works best inside Mono Developer for Unity and teaches you to write OpenCvSharp code by just using the dot syntax.

So far 27 lines of code converted from both C++ and OpenCV for Java samples have been built using Unity Builder and tested on Windows x86 and run standalone in Windows 10.

The real problem is that when this is sent to via Unity to Visual Studio 2015 UWP, there is some sort of library conflict resulting in VS/Unity C# Net 3.5 and UWP Net 5.1 freezing

Windows Store Application builds, but refuses to start, complaining the is no executable code or startup project.

The template Unity is creating is not UWP solution and is not attaching to Unity, rather creating an Windows 10 executable.

The important point is OpenCvSharp runs under Unity as an plugin extension to any project.

My next idea is to try running OpenCvSharp plugins inside HoloLens tutorial, which will generate hopefully the correct templates for Visual Studio.

Summary

Lots of progress for many hours work , but lot of fun as well