A background app in Hololens that continuously streams Hololens position data to a laptop

pumpkin

I want to continuously send the Hololens position data (Camera.main.transform.position) to my laptop while another app (like https://developer.microsoft.com/en-us/windows/holographic/holograms_101e) is running on the Hololens. I will be sending the data over a UDP connection to the laptop. Is it possible to enable such a background process which sends position data every 30 milliseconds? Is there sample code available to do the same?

thebanjomatic

Background processes aren't supported at the OS level. You might be able to accomplish this using the DevicePortal API:

https://github.com/Microsoft/WindowsDevicePortalWrapper
https://developer.microsoft.com/en-us/windows/holographic/device_portal_api_reference#holographic_perception

Depending on the realtime/offline needs, you could also use the "Perception Simulation Recording" capabilities provided by the DevicePortal.

HoloSheep

@pumpkin

You may also want to keep in mind that HoloLens position data (Camera.main.transform.position) is relative to the origin of the device at the moment when the Unity app initializes. So even when/if background apps become supported the Camera origins between apps and between devices may need some sort of sharing logic and support similar to the gameRootAnchor concept used in Holograms 240 to give that position any sort of meaning.

Just food for thought.

pumpkin

I want to enable a background task that sends position data from Hololens to my laptop in real-time. However, I do not need this background task to be an app. Also, it is fine if the Camera origins change.
Real-time needs makes "Perception Simulation Recording" not helpful as it records the data sends it as a file later. Is there a hack to get real-time position data streaming background process/task work?

thebanjomatic

@pumpkin Answering your edited message, I haven't tried this myself, but the documentation says that it works over WebSocket, so you would need to do something like this:

const webSocket = new WebSocket('wss://IPADDRESS/api/holographic/perception/client');
webSocket.onMessage = function (e) { 
  console.log(JSON.stringify(e.data, null, 2));
};

pumpkin

Please point me to documentation to understand the output of /api/holographic/perception/client. I want to obtain the data corresponding to headset position, i.e., Camera.main.transform.position.

On running following code:

CODE

var connection = new WebSocket('wss://IPADDRESS/api/holographic/perception/client');
// Log messages from the server
connection.onmessage = function (e) {
console.log('message from server', e.data);

};

I get following output

OUTPUT

message from server {"TrackingState" : 3, "Camera" : {"Displays" : [{"Frustum" : {"Bottom" : [0.006326,-0.997893,0.064567,0.090161], "Far" : [0.049452,0.079039,-0.995644,-1000.228943], "Left" : [-0.979840,-0.025178,0.198192,-0.011032], "Near" : [-0.049453,-0.079030,0.995645,0.380177], "Right" : [0.949619,-0.015447,0.313026,0.052084], "Top" : [-0.020988,0.972734,0.230970,-0.066238]}, "Projection" : [3.762149,0.000000,0.000000,0.000000,-0.014389,6.659828,0.000000,0.000000,0.037827,0.038883,-1.000300,-1.000000,0.000000,0.000000,-0.300090,0.000000], "View" : [0.998688,-0.013528,-0.049452,0.000000,0.009590,0.996825,-0.079029,0.000000,0.050363,0.078451,0.995641,0.000000,0.031609,-0.079524,0.080175,1.000000]},{"Frustum" : {"Bottom" : [0.002245,-0.997937,0.064166,0.091165], "Far" : [0.044227,0.078993,-0.995894,-1000.279358], "Left" : [-0.975307,-0.027971,0.219073,0.051666], "Near" : [-0.044227,-0.078994,0.995894,0.379664], "Right" : [0.956465,-0.012491,0.291579,-0.010945], "Top" : [-0.015338,0.972869,0.230848,-0.067444]}, "Projection" : [3.771098,0.000000,0.000000,0.000000,0.008667,6.673673,0.000000,0.000000,-0.030171,0.038280,-1.000300,-1.000000,0.000000,0.000000,-0.300090,0.000000], "View" : [0.998987,-0.008642,-0.044227,0.000000,0.005126,0.996860,-0.078993,0.000000,0.044770,0.078685,0.995889,0.000000,-0.031603,-0.080627,0.079662,1.000000]},{"Frustum" : {"Bottom" : [0.006326,-0.997893,0.064567,0.090161], "Far" : [0.049452,0.079039,-0.995644,-1000.228943], "Left" : [-0.979840,-0.025178,0.198192,-0.011032], "Near" : [-0.049453,-0.079030,0.995645,0.380177], "Right" : [0.949619,-0.015447,0.313026,0.052084], "Top" : [-0.020988,0.972734,0.230970,-0.066238]}, "Projection" : [3.762149,0.000000,0.000000,0.000000,-0.014389,6.659828,0.000000,0.000000,0.037827,0.038883,-1.000300,-1.000000,0.000000,0.000000,-0.300090,0.000000], "View" : [0.998688,-0.013528,-0.049452,0.000000,0.009590,0.996825,-0.079029,0.000000,0.050363,0.078451,0.995641,0.000000,0.031609,-0.079524,0.080175,1.000000]}]}, "LSRPlane" : {"distance" : 2.000000, "x" : 0.000000, "y" : 0.000000, "z" : -1.000000}, "HeadToAttachedFor" : [0.998836,0.007357,0.047557,0.000000,-0.011083,0.996847,0.078554,0.000000,-0.046828,-0.078989,0.995779,0.000000,0.002860,0.086067,-0.073378,1.000000], "OriginToAttachedFor" : [0.997681,0.000012,-0.068029,0.000000,-0.000010,1.000000,0.000041,0.000000,0.068029,-0.000040,0.997686,0.000000,0.029821,-0.018559,0.003618,1.000000]}

sbagchi

This is something I'm interested in doing, though it doesn't need to be a background app. Does anyone have other suggestions for how to do this?

dirk15

I have the same problem, does anyone find a solution now?

dirk15

I have the same problem, does anyone find a solution now?