Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

How can I access the Hololens Locatable Camera view and projection matrices with the WinRT API?

The documentation for the Hololens gives an example of how to extract the Locatable Camera view and projection matrices using the Media Foundation API here: https://developer.microsoft.com/en-us/windows/mixed-reality/locatable_camera#locating_the_device_camera_in_the_world.

It claims that this is also possible with the WinRT API, and links to this documentation reference: https://docs.microsoft.com/en-us/uwp/api/Windows.Media.Devices.Core.CameraIntrinsics.

However, this class does not seem to have any API to retrieve the extended Hololens attributes, only the default Windows Phone ones like the distortion matrix and pinhole properties.

Is the Hololens documentation incorrect and is it simply not possible to retrieve the Locatable Camera metadata in the WinRT API? Or am I missing something?

The Spatial Coordinate System (3rd and last extended sample metadata attribute) does seem to be available as MediaFrameReference.CoordinateSystem (https://docs.microsoft.com/en-us/uwp/api/windows.media.capture.frames.mediaframereference), which makes this even more confusing...

Answers

  • I'm currently struggling with similar issues. I've attempted to get the data directly by using "MFSampleExtension_Spatial_CameraViewTransform" and "MFSampleExtension_Spatial_CameraProjectionTransform", but to no avail.

    I've been trying to modify the HolographicFaceTracking sample, which provides a MediaFrameReference object for each incoming video frame. Given a MediaFrameReference frame, I can get the SpatialCoordinateSystem from frame->CoordinateSystem.

    To attempt to get the view transform, for example, I've been doing frame->Properties->HasKey(MFSampleExtension_Spatial_CameraViewTransform) which returns true (suggesting that the properties does in fact have the data), but if I try to get the data (props->Lookup(MFSampleExtension_Spatial_CameraViewTransform)), I just get a Platform::Object^ file that I don't know what to do with. Checking the type of that Object reveals it's an ArrayReference<UINT8>, but attempting to cast it to a UINT8* yields data that is nonsensical.

    Can anyone provide insight on what to do here? The info in https://developer.microsoft.com/en-us/windows/mixed-reality/locatable_camera_in_directx about IMFStreamSinks is extremely confusing and I've been unable to find resources in the mountain of obscure documentation that is MSDN about how to actually get something like this set up. A test application would be greatly appreciated.

  • Any solution here :neutral: ?

  • Found a solution:

    `

    MediaFrameReference^ frame = ...;
    auto projectionTransformProperty= (Windows::Foundation::IPropertyValue^) frame->Properties->Lookup(MFSampleExtension_Spatial_CameraProjectionTransform);
    Platform::Array<unsigned char>^ projectionMatrixByteArray = ref new Platform::Array<unsigned char>(4*4*4);
    projectionTransformProperty->GetUInt8Array(&projectionMatrixByteArray);
    float* projectionMatrixValues= reinterpret_cast<float*>(projectionMatrixByteArray->Data);
    

    `

Sign In or Register to comment.