The Mixed Reality Forums here are no longer being used or maintained.
There are a few other places we would like to direct you to for support, both from Microsoft and from the community.
The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.
For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.
If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.
And always feel free to hit us up on Twitter @MxdRealityDev.
Using Windows 10 Face detection for Hololens application
Would it be possible to use the windows 10 face detection and camera API to detect faces along with being able to place the face rectangles generated onto people in a Unity 3D application for hololens? Any ideas as to how this would work or what I would need to do to make this functional?
Best Answers
-
OptionsTimK mod
All Windows 10 Universal APIs, including face detection/tracking, can be used with HoloLens. The Windows.Media.Capture APIs allows streaming of preview frames from which face detection can be performed.
I recommend starting with the BasicFaceTracking sample located here: https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/BasicFaceTracking
The sample only builds a 2D app, but it demonstrates how to stream preview frames from MediaCapture and run the FaceTracking on them. Please note there are two different "face detector" classes:
- Windows.Media.FaceAnalysis.FaceDetector is intended for finding faces in a single image or frame
- Windows.Media.FaceAnalysis.FaceTracker is optimized to track faces in a video stream, which is what you want to use in this case
However, to use face detection in 3D space is a little tricky since all you'll get from these APIs is a 2D bounding box in pixel space. One possibility is to simply estimate the distance of the face based on the size of the bounding box in the frame image. If you're simply "tagging" faces with a simple graphic or lable, this might be good enough.
Another option is to use the Sharing experience from Holograms 240 class. You can map the faces returned by FaceTracker with the 3D positions of participants in the session to render some effect on or around their faces.
Be aware the output of FaceTracker is pretty noisy, and the size of the bounding box will quickly change depending on the orientation of the person's face, e.g. a profile view of the face will give you a tall, slender box while a frontal view produces a more square box. You'll probably need to perform some filtering of the face tracking output in your app.
8 -
Options
I've built something like this and have made it open source at https://github.com/UoA-eResearch/hololens_facial_recognition
When you air tap (or click with the clicker) it takes a photo and posts the image data to the Microsoft Face API, then displays the response as floating text next to any of the faces detected. Looks like this -
Enjoy!
6
Answers
Does the Hololens even have a camera preview to apply effects to such as a face rectangle?
All Windows 10 Universal APIs, including face detection/tracking, can be used with HoloLens. The Windows.Media.Capture APIs allows streaming of preview frames from which face detection can be performed.
I recommend starting with the BasicFaceTracking sample located here: https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/BasicFaceTracking
The sample only builds a 2D app, but it demonstrates how to stream preview frames from MediaCapture and run the FaceTracking on them. Please note there are two different "face detector" classes:
However, to use face detection in 3D space is a little tricky since all you'll get from these APIs is a 2D bounding box in pixel space. One possibility is to simply estimate the distance of the face based on the size of the bounding box in the frame image. If you're simply "tagging" faces with a simple graphic or lable, this might be good enough.
Another option is to use the Sharing experience from Holograms 240 class. You can map the faces returned by FaceTracker with the 3D positions of participants in the session to render some effect on or around their faces.
Be aware the output of FaceTracker is pretty noisy, and the size of the bounding box will quickly change depending on the orientation of the person's face, e.g. a profile view of the face will give you a tall, slender box while a frontal view produces a more square box. You'll probably need to perform some filtering of the face tracking output in your app.
I've built something like this and have made it open source at https://github.com/UoA-eResearch/hololens_facial_recognition
When you air tap (or click with the clicker) it takes a photo and posts the image data to the Microsoft Face API, then displays the response as floating text next to any of the faces detected. Looks like this -
Enjoy!
@neonninja
Good attempt.
The marker wont show in the correct position if there are multiple faces detected in different locations.
Hi @TimK,
I tried building the sample code in holographic face tracking and got errors as follows:
1) constexpr function return is non-constant error in functions in DirectXMath.h:
inline XM_CONSTEXPR float XMConvertToRadians(float fDegrees) { return fDegrees * (XM_PI / 180.0f); }
inline XM_CONSTEXPR float XMConvertToDegrees(float fRadians) { return fRadians * (180.0f / XM_PI); }
2) static assertion failed with "Model constant buffer size must be..." in ShaderStructures:
static_assert((sizeof(CubeModelConstantBuffer) % (sizeof(float) * 4)) == 0, "Model constant buffer size must be 16-byte aligned (16 bytes is the length of four floats).");
static_assert((sizeof(QuadModelConstantBuffer) % (sizeof(float) * 4)) == 0, "Model constant buffer size must be 16-byte aligned (16 bytes is the length of four floats).");
I am new to microsoft development and would like to ask some help in resolving these errors.
Thank you and I am hoping for your kind response.
Regards,
Trish
I can build, deploy and run this project on Emulator. But get error when I deploy it to HoloLens.
The error is:
Can anybody help me to fix this?
Thanks a lot.
YL