Converting 2D coordinates into 3D

Khronossos

Hello fellow hololens developers,

I want to convert 2D coordinates from a picture send from the hololens to another PC. The user of the PC picks a spot from the picture, sends the coordinates back to the hololens and the hololens should place an object at that spot.
My problem is to figure out how the 3D coordinates are measured. Currently I am using the holographicApp demo content with the spinning cube. I added 0.25f on each x and y coordinates of it's position. My intention was to figure out if it positions one quarter of the viewport, for the Y coordinate it seems to be right, but not for the X.

Let the picture speek for it. I assumed the cube at the blue circle.

I hope that makes any sense to you guys.
Thank you very much in advance for your help.

Cheers

stepan_stulov

Assuming you're using Unity.

Do you want to transform normalized viewport coordinated to points in the world where your user looks at? Then you need to run a ray from the camera (1), through the near clipping plane's point corresponding to that viewport coordinate (2), and down to an intersection point with the spatial mapping (3).

1. Simply take the camera.transform.position
2. You can use a variety of methods here from built-in to doing your own frustum math. Amongst others is https://docs.unity3d.com/ScriptReference/Camera.ViewportToWorldPoint.html Note how exactly the viewport coordinates are normalized by this link. Bottom left is 0,0 and top right is 1,1.
3. You can now make a ray from (1) through (2) and find (3) via Physics.Raycast().

Additionally make sure that your cube's visual content's "centre of masses" is in it's 0,0,0 so that placing the cube to the right place also looks right.

Did I understand you correctly?

stepan_stulov

Assuming you're using Unity.

Do you want to transform normalized viewport coordinated to points in the world where your user looks at? Then you need to run a ray from the camera (1), through the near clipping plane's point corresponding to that viewport coordinate (2), and down to an intersection point with the spatial mapping (3).

1. Simply take the camera.transform.position
2. You can use a variety of methods here from built-in to doing your own frustum math. Amongst others is https://docs.unity3d.com/ScriptReference/Camera.ViewportToWorldPoint.html Note how exactly the viewport coordinates are normalized by this link. Bottom left is 0,0 and top right is 1,1.
3. You can now make a ray from (1) through (2) and find (3) via Physics.Raycast().

Additionally make sure that your cube's visual content's "centre of masses" is in it's 0,0,0 so that placing the cube to the right place also looks right.

Did I understand you correctly?

stepan_stulov

Ok, additionally to my previous post, which was only one direction. It's very similar. You just use one of the inverse methods like:
https://docs.unity3d.com/ScriptReference/Camera.WorldToViewportPoint.html

stepan_stulov

Also now I'm thinking you're trying to map the HoloLens's viewport onto a picture taken by HoloLens's camera? I believe this is where the discrepancy can come from. HoloLens's camera "sees" wider than what you see on its rendering displays, I'm pretty sure. These are two separate "visions". Normalized coordinate on the entire picture is not the same as normalized coordinate on the part of that picture that represents the edges of the viewport. So when the user selects the point, you need to calculate that point's coordinates not relativeto the entire picture but to the red rectangle / sub-picture. To calculate that you need to know the coordinates of the corners. Which you will need to either ask the user to specify or find somehow. But there is a problem with this as taking a photo doesn't include holograms as seen by the HoloLens wearer as far as I know. So you can't really place some markers to be found be either a human or image processing. Or is this not a picture taken by the HoloLens's camera but a render / snapshot of the mixed reality capture?

stepan_stulov

Ok, there seems to be a very good read with examples on how to transform between LocatableCamera's taken pictures and in-app 3D coordinates:

https://developer.microsoft.com/en-us/windows/holographic/locatable_camera

Khronossos

@stepan_stulov said:
Ok, there seems to be a very good read with examples on how to transform between LocatableCamera's taken pictures and in-app 3D coordinates:

https://developer.microsoft.com/en-us/windows/holographic/locatable_camera

Hi Stephan,
thank you very much for your answers. I am using directX and the simplecommunication sample + mixedreality sample from Microsoft github page. The hololens streams a live video to an app. The user selects spots on the received video. The 2D coords should be sent back, and a marker should display at the excact position.

Edit:

I think it is not that simple than I thought. I came up with these steps, please correct me if I am wrong:

1. Raycast through the spot, find collision with the mesh. There I can get the distance
2. Create some kind of projection matrix with the distance and calculate the position of the marker

But how?