How to use fiducials to place holograms/spatial anchors?
I am trying to use a fiducial marker tracked by the Hololen's front facing camera to initialize a spatial anchor/place a hologram in space. The idea is to be able to set a scene up for the user without a manual intervention (i.e. no hand-placing of holograms).
Unfortunately, there seem to be some issues - we need very precise real world object overlays, so correct hologram placement is essential. However, we are getting some unexplained offsets between the marker/real world object and the hologram.
Here is a summary of what our Hololens application is doing and where the problems are.
The application is derived from the default C# DirectX sample for Hololens (i.e. not using Unity). The idea is to track a fiducial marker using the front-facing camera and use this to place a world anchor in order to initialize the application for the user.
The fiducials are detected and tracked with OpenCV, using the Aruco module - fairly standard piece of software, works on pretty much anything, even on Hololens the marker detection and pose estimation work fine.
In order to place an object on the marker the code extracts the Coordinate System, CameraViewTransform and CameraProjectionTransform from the video frame metadata, as described in the Locatable Camera documentation.
// AppCoordinateSystem is the current spatial coordinate system established by the application // FrameCoordinateSystem is the the coordinate system recovered from the video frame Matrix4x4 World2Cam = AppCoordinateSystem.TryGetTransformTo(FrameCoordinateSystem); // The MFSampleExtension_Spatial_CameraViewTransform obtained from the video frame Matrix4x4 Cam2View; // The "world" position of the marker is then calculated as: Matrix4x4 View2Cam = new Matrix4x4(); Matrix4x4 Cam2World = new Matrix4x4(); Matrix4x4.Invert(Cam2View, out View2Cam); Matrix4x4.Invert(World2Cam, out Cam2World); Matrix4x4 View2World = View2Cam * Cam2World; // matrix bringing my marker centres into world space var position = Vector3.Transform(marker_pos, View2World); var rotation = Quaternion.CreateFromRotationMatrix(Matrix4x4.CreateFromAxisAngle(axis, angle) * View2World);
And this is the pose where I put my hologram. I am not anchoring it for the time being because the idea is to verify that the calculated pose is correct before "freezing it" using an anchor. The pose is recalculated continuously, the expected result being that the hologram will be perfectly overlapping the square marker (e.g. 100mm square marker will be perfectly covered by a 100mm large cube).
The problem - there is always an offset between the hologram and the marker, i.e. the hologram is never properly centered on the marker. The offset is about ~1-2cm at the bottom and right edge of the marker, i.e. the hologram always appears shifted up and left. This offset stays on the same side even when the marker rotates - there is always an offset at the bottom and right edge, thus it is not coming from the marker detection itself (the offset would rotate with the marker/hologram otherwise). I have also verified that the marker is being tracked correctly drawing an overlay into the video from the front camera - no problems there.
Unity using Vuforia shows the same problem. There is even a report on the Vuforia forums with a video showing the same symptom as I have:
Setting the focus point has no effect on this - I am setting it at the marker center.
The second problem relates to how to obtain the correct camera intrinsics - focal length, principal point and distortion parameters (if any - I understand that the Hololens hw performs some undistortion processing already). The VideoMediaFrame.CameraIntrinsics property is always null on Hololens - this is apparently a known issue? (see https://forums.hololens.com/discussion/10344/cameraintrinsics-always-null)
I have got a set of intrinsics from the internet and also attempted my own calibration, but they are not very good.
The C++ examples (https://github.com/Microsoft/HoloLensForCV) get the camera intrinsics correctly, using those eliminates the left-right part of the offset above (the principal point seems to have been shifted) but the top-bottom offset between the marker and the projected hologram still remains.
I am also aware of ARToolkit for Hololens - but that shows even worse errors than my Aruco-based code.
Did anyone manage to get a stable hologram overlaid on a fiducial marker with Hololens? Is there some example how this should be correctly done? (mainly how to deal with the Locatable Camera transformations - the documentation is not very clear there).