Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.
Options

Pixel Coordinates to World Coordinates

jonathanhuang13jonathanhuang13 ✭
in Questions And Answers

Currently, I'm trying to take a picture with the Hololens, find some object in the picture, and then place an icon in the world at where that object is. I've been able to take the picture and find the object in the picture, but how do you transform pixel coordinates from an image taken by the Hololens to world coordinates so I can place an icon there?

I've looked at https://developer.microsoft.com/en-us/windows/holographic/locatable_camera and https://forum.unity3d.com/threads/holographic-photo-blending-with-photocapture.416023/ but I'm still confused as to how the projectionMatrix work. Is there a way to find the depth of an object in an image?

Answers

  • Options
    CurvSurfCurvSurf ✭✭
    edited November 2016

    @jonathanhuang13

    1. Extract an object in the image captured by the locatable camera (You have done. You know the pixel image coordinates of the object).
    2. Draw the ray from the camera COP through the image object by using the pixel coordinates, focal length of the camera, and projectionMatrix.
    3. Determine the colliding point of the ray onto the spatial mapping.
    4. Place the icon onto the colliding point, facing towards the device (you).
    5. Track the image object in the video stream and repeat 2~4.

    projectionMatrix:
    The projectionMatrix capsulates together the focal length, position, view and head-up directions of the device.

    HoloLens and the object must move slowly. Otherwise, the tracking might be going lost and the colliding points inaccurate.

  • Options
    jonathanhuang13jonathanhuang13 ✭

    Thanks @CurvSurf! One question, though. How exactly do I use the pixel coordinates, focal length and projectionMatrix?

    Currently, I position and rotate the camera using the cameraToWorldMatrix:
    position = cameraToWorldMatrix.MultiplyPoint(Vector3.zero);
    rotation = Quaternion.LookRotation(-cameraToWorldMatrix.GetColumn(2), cameraToWorldMatrix.GetColumn(1));

    Then, I scale the image coordinates by the Screen.width/cameraResolution.width. How would you do it using the projectionMatrix?

    Thanks again!

  • Options
    CurvSurfCurvSurf ✭✭

    @jonathanhuang13
    I describe in general manner.

    A ray (3-D line) has a starting point and a direction.

    In the camera coordinate system (CCS), the camera COP with (0.0, 0.0, 0.0) is the starting point, and the image point with dirRay = (imgCrdX/focalLength, imgCrdY/focalLength, 1.0) is the direction of the observation ray. dirRay may have to be normalized to dirRay / ||dirRay||.

    The image coordinates (imgCrdX, imgCrdY) are referenced to the image center. If necessary, they must have been corrected by considering the distortions, offsets, scaling parameters.

    In the world coordinate system (WCS), the camera has the position of (camX, camY, camZ) and poseMatrix of R. The rows of R represent the directions of the camera axes seen from WCS (The columns of R represent the directions of the WCS axes seen from CCS).

    Then, the observation ray in WCS has the starting point of (CamX, CamY, CamZ), and the directions of (<dirRay, 1stColumnR>, <dirRay, 2ndColumnR>, <dirRay, 3rdColumnR>). (<a, b>: dot product of two vectors a and b).

  • Options
    Matthias1991Matthias1991 ✭

    I know it's an older question, but I wanted to do the same. Did it work for you? I'm new to hololens and all this transformation stuff, would you mind to post your result?

    Thanks

Sign In or Register to comment.