Hello everyone.
The Mixed Reality Forums here are no longer being used or maintained.
There are a few other places we would like to direct you to for support, both from Microsoft and from the community.
The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.
For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.
If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.
And always feel free to hit us up on Twitter @MxdRealityDev.
The Mixed Reality Forums here are no longer being used or maintained.
There are a few other places we would like to direct you to for support, both from Microsoft and from the community.
The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.
For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.
If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.
And always feel free to hit us up on Twitter @MxdRealityDev.
Options
Answers
Fantastic thread. Thanks for the ground work!
I've tried to make a UWP DLL that uses the OpenCV nuget packages, following the instructions @holodoctor gave above. I made a simple function to test using OpenCV (it just makes a matrix and returns its size), which gets DllImported in a C# script on the unity side. The DLL compiles just fine, however I'm running into an error while running on the HoloLens emulator:
Loading opencv_features2d310.dllFailed to load 'opencv_features2d310.dll', expected x86 architecture, but was Unknown architecture. You must recompile your plugin for x86 architecture.(Filename: C:\buildslave\unity\build\Runtime/Misc/Plugins.cpp Line: 139)This error gets thrown on startup of the unity app, for each of the OpenCV dlls and for my interface dll. I'm definitely building these dlls for x86, and every tool I've used on the dlls confirms this.
I haven't been able to test these on an actual HoloLens, but I tested an unrelated pre-built dll on the emulator and it ran, so I don't suspect it's the emulator causing problems.
I've spent several days trying to find the issue, but I can't find a way to fix this. If someone has any idea what may resolve this, I'd hugely appreciate it!
Hi,
This might help you (binaries are included).
https://github.com/sylvain-prevost
and now with the proper nuget install / might save you some time.
https://www.nuget.org/packages/OpenCV-Hololens/
Thanks! I have actually moved away from OpenCV in the meantime, since I just want to do marker tracking, so I'm now using AprilTag, of which HoloToolkit happens to include a port for the HoloLens. However this will come in handy if I want to do any other image processing!
@keymaster
Do you have any tips on how to get started with AprilTag for Hololens? I'm trying to integrate it in a Unity app but I can't seem to get anywhere.
@aerr So I'm using the version of AprilTag that's included in the HoloToolkit repository here.
Probably the easiest way to use it in a unity project is to just use the HoloToolkit-Unity package, since that includes C# wrapping files for everything. Specifically, you want this file.
The core method here is FindTags, which takes the image to track as a byte array. You can get by taking whatever texture you have, and calling Colors32() to get it's data. That's just rgba data, so you can copy it to a byte array and pass it to the tracking library.
The rest of it is pretty self-explanatory, except the callbacks parameter, for which I had to first create a class so I could set my own functions. I've uploaded that one here. Just make an instance of it and assign your callback functions before you pass it to the FindTags method.
In case you don't want to use the whole of the HoloToolkit in your project, or need more direct access to AprilTag, you can also use the BuildLibs.bat file in the directory I linked at the start to create static build files for AprilTag. Using those you can then make a native plugin for unity (i.e. a c++ dll) and call that from your C# code.
Hope this helps, let me know if you have more questions!
@keymaster
tried the same as you, but: LocatedEvent get never fired (altough the CompletedEvent does
). I'm not sure, waths going wrong: perhaps the conversion from my Texture2D to byte-array. Any advice ?
And: how's he performance ?
@keymaster
Got it (my Texture2d has to be mirrored ...) Now it works !
@keymaster
Many thanks, I got it working! your answer helped me a lot!
@keymaster I've tried getting AprilTag working as you described but I am getting a fun "The thread tried to read or write to a virtual address for which it does not have the appropriate access" crash. I'm pretty sure it has to do with my tenuous understanding of how C memory management and pointers work. Is there an example scene of FindTags working on Hololens?
I don't know the answer to this but I'm super interested in it too.
One thing I wanted to mention is that the HoloLens Companion Kit has a simple marker tracker in it and it looks like it uses components from OpenCV. Maybe this will help you in your research.
The sample is here.
Our Holographic world is here
RoadToHolo.com WikiHolo.net @jbienz
I work in Developer Experiences at Microsoft. My posts are based on my own experience and don't represent Microsoft or HoloLens.
@Colton_TSBD It's important to note that the standard AprilTag library does not work on the Hololens. I used a version that was in the HoloTookit, however that appears to have been removed, in favour of the poster recognition sample that @jbienzms posted. I have also moved to a similar approach now, but using the OpenCV dlls by @sylvain and the aruco module in OpenCV. I should be able to release that soon. For now, the best approach is probably to use the Companion Kit code or make your own plugin to suit your needs.
@holodoctor,
I use Cmake compile opencv x86 Library and dll.When i use the dll in Unity to release a hololens App.Debug: DllNotFoundException.But it can operate in Windows well.Do you know ?How is your problem solved?
@Libra_YaN This may be an issue with DLL dependencies - even if you have one OpenCV dll, it will require other modules, and if those are missing Unity just reports the original dll as missing! I'd recommend copying over all OpenCV modules to the unity plugins folder and test with that. If the issue is gone, you can then remove dlls one at a time to see which ones are required in your case and which ones aren't.
@keymaster I use Cmake compile opencv x86 Library and it produced two dll:opencv_ffmpeg310.dll ,opencv_world310.dll. I copy the two dll to the unity plugins folder.But it still report the issue ,DllNotFoundException.
@Libra_YaN You should definitely have way more dlls. A proper OpenCV build should produce one dll for each of these folders: https://github.com/opencv/opencv/tree/master/modules
I don't know which OpenCV module you're trying to use yourself, but depending on what you're importing you'll need some subset of all those dlls (not just the one you're importing in C#).
If you really want to proceed with your own builds, you should make sure it builds all these modules, but I'd recommend just using these builds provided here - they've worked well for me: https://github.com/sylvain-prevost
@keymaster Thank you for your help! My project is Built by Unity so I can not use Nugut. And the opencv_world310.dll included all opencv modules.So my operation should not have problems.Do you have other advice or questions?
@Libra_YaN I didn't know the world module existed, but yes that should cover any possibly dependency problems. My suggestion would be to compare your own build settings with those in the nuget project above (it includes solution files to build from source) to make sure you're not missing any settings. Other than that I can't say much more without seeing your specific setup.
Hello,
I wanted to ask if anyone succeeded in using OpenCV with HoloLens.
I did as @holodoctor suggested:
1) Created a Visual C++ project, Windows -> Universal -> DLL.
2) Added the nuget package "OpenCV for Windows 310.3.0" using "Install-Package opencv.win.native -Version 310.3.0"
3) Created a method that just created a OpenCV Mat and fills it. Used the extern "C" and dllexport
4) Added the generated dlls and the opencv dlls to unity in the "Plugins" folder
5) Created a C# script and used the dllimport to call my method.
Now a dll that does not reference and call OpenCV is working fine. I tried returning a string and a number and everything is working fine. But when I reference OpenCV and try to call the method, the Unity VS project crashes with DllNotFoundException.
After a lot of investigation and after using the "Dependency Walker", I found that OpenCV uses x64 dlls, in which I think is where the problem is, as HoloLens is x86 and all modules/dlls should be the same CPU type.
Any one can help?
Thank you.
@alia_adly4 I managed to get things working, pretty much through the same process as you described. However, I used the nuget package provided by @sylvain, OpenCV-Hololens
If that doesn't fix it, I have my plugin that uses OpenCV in this repo. It includes the VS projects to build the plugin both for the HoloLens and for the Unity Editor, maybe it would be useful to look at an example library that works.
@keymaster
Thank you for your quick reply and thank you for sharing your repo. However, it worked fine as you said with @sylvain nuget package!
So...Has anybody successfully run OpenCV on HoloLens in Unity project?
@keymaster @alia_adly4 Please help!!! I have done exactly what you guys mentioned. I compiled the dll for x86 Universal Windows in Release mode, copied all the dlls (including @sylvain opencv dlls) to Plugin/WSA folder of Unity and deployed to the HoloLens, but the code failed exactly at where the functions from my plugins were called and the app just crashed without further error logs. If I remove the plugin calls, the code runs just fine so the problem must be from there. But I can't think of what can possibly be the cause.
@vertex Yes I have and I'm sure there are others.
@ethanttbui
The reason your app is "crashing" is probably because your FPS is actually just super low which is why it is crashing without an error log. You need to optimize your code so that it doesn't take up so much memory. If you just take it as it is the app runs at like 1 to 2 fps. An idea would be to display your video frames on a Canvas with a smaller FOV and do your calculations from there. Also only run the OpenCV camera when you need to.
OpenCV is a very taxing resource and needs to be seriously optimized for running it on a mobile device, otherwise it would be on every phone app. Some big money workstations have trouble running OpenCV functions. The HoloLens is not a workstation just a mobile device. It has to be toned down a bit.
AR Developer
@ethanttbui I suggest trying the official examples from the HoloLens for CV repo: https://github.com/Microsoft/HoloLensForCV/