Object Detection - Hololens

Has anyone gotten anything working with object detection and the Hololens?

I'm hoping to automatically detect a physical object (or a marker on that object) and overlay a hologram on top of it.

Answers

  • I've just begun my search, but I am interested in this as well.
  • I suppose we could use a library like OpenCV based on raw images. But, HoloLens is a powerful computer vision device. Is there a possibility Microsoft might expose some high level object detection API?

  • ContextVRContextVR ✭✭✭
    edited July 2016

    @Maggie You can try and make ARToolkit work on HoloLens, but Vuforia is a device-neutral SDK that now supports HoloLens: http://www.ptc.com/news/2016/ptc-brings-vuforia-to-hololens

  • @ContextVR I've looked into Vuforia. Turns out their "spring" release date for the Hololens actually means this fall. I applied for the Beta program, but am still looking for an alternative while I wait for the results for that.

    Any advice on how to make ARToolkit work for the Hololens? I was also looking into that before, but was not quite sure how to get it to work.

  • ContextVRContextVR ✭✭✭
    edited July 2016

    @Maggie I am also waiting for access to Vuforia for HoloLens. They say they will provide access, but emails fall into a black hole. Checking out AR Tooolkit is on my todo list, are you seeing any compilation errors?

    Another alternative might be Wikitude, but even though they talk a lot about HoloLens and the future of AR, but they don't list HoloLens as officially supported.

    @ahillier I remeber we had a conversation about markers, have you tried AR Toolkit?

  • I've been looking into AR Toolkit for HoloLens, it seems that the current version supports only Windows 8.1 and Windows 8.1 Universal. I'm currently investigating how to retarget the toolkit for Windows 10 Universal/Holographic.

    Owner of the startup Information Experience Sweden AB, http://informationexperience.se
    Information Management, SharePoint, CAD, Visualization for AEC. Architecting and developing solutions for HoloLens, combined with AR Toolkit, Kinect, RealSense, Leap Motion.

  • @Maggie I would like to do the same thing. Currently I achieved the following:

    1. Capture image with camera
    2. Convert the camera image to raw data (2D array etc)
    3. Convert raw data back to texture2D and display in hololens view.

    My plan is to do recognition work between 2 and 3. I am also searching for some libraries. It turned out there is only limited options for C#. Even I found something in C#, i am not sure if I can run it on hololens. Also, It seemed to me that image processing on hololens with C# was very slow.

  • ContextVRContextVR ✭✭✭

    @Fireman I can also attest to CPU image processing with C# being very slow, even for simple things like re-sampling. My next point of investigation is to try something with HW acceleration, e.g. OpenCV. I also found this but it supports .NET 4.5+.

  • Vuforia made their beta public: https://developer.vuforia.com/downloads/beta

    Taqtile

  • mavashermavasher ✭✭
    edited August 2016

    Downloaded the EAP Beta of Vuforia6 for Hololens: I have to say - I'm disappointed.

    The object tracking does a pretty intelligent thing - it ties the image markers to the spacial mapping so that it can leverage the Hololen's natural environment tracking. The problem becomes when you move the marker against the natural environment, which is the whole point of object tracking: the tracking is incredibly slow and imprecise.

    I found this video which discusses this issue in detail.

    It's disappointing because the Caterpillar video is pretty deceptive- they had the marker stay still through the Hololens segment of the video so that once the object was instantiated Vuforia was just leveraging the Hololens' environment tracking, and there was no real object tracking occurring.

    You could have gotten basically the exact same functionality in the video from an intelligently designed voice command to pull up the appropriate model.
    The barcode functionality (which I haven't tested) may be useful but the object tracking ability of Vuforia at this stage is inadequate for even basic object tracking.

  • Hi all,

    HoloLens does not support any of detection, recognition, or measurement of objects.

    It would be the Holy Grail of Computer Vision: Detect, recognize, and measure objects.

    HoloLens' spatial mapping generates asynchronously meshes from the raw depth data stream that is not available to developers. We developers like to access the raw depth data stream as with Kinect, Intel RealSense, and Google Tango. 'Microsoft' HoloLens team has never been seriously responding to this issue of raw depth data stream that is available with 'Microsoft' Kinect. We developers are hearng...

    The current available AR tools are based on planes, in other words, recognize and detect images (photos, videos) or planes (floor, wall, ceiling). But, there are many other shapes than planes, e.g. boxes, spheres, cylinders, cones, rings, etc.

    Most man-made objects are composed of planes, boxes, spheres, cylinders, cones, and tori. Do you have ever thought about it?

    An object has shape, size, position, and pose (i.e. rotation/orientation).

    The next generation of AR is the 3D-AR: Detect, recognize, and measure 3D objects in real-time.

    Microsoft HoloLense with spatial mapping points:

    Google Tango with depth data:

    Intel RealSense with depth data:

    CurvSurf's FindSurface SDK for 3D-AR will be prepared for public distribution in 1Q 2017.

    You can find more on:
    Web page CurvSurf
    YouTube Channel CurvSurf
    Google Plus CurvSurf

  • @mavasher said:
    Downloaded the EAP Beta of Vuforia6 for Hololens: I have to say - I'm disappointed.

    I agree.
    Wikitude 3D is much better than EAP Beta of Vuforia6 for Hololens IMHO, simply because it supports Unity 5 properly.

    EAP Beta of Vuforia6 for HoloLens documentation is "sparse" in the PDF it states

    Quote: from "Working with the HoloLens Sample in Unity"
    "Note that the Unity HoloLens Technical Preview does not currently support Vuforia’s Play Mode simulator."

    Basically this means developing on Android, or iOS. There is a Universal Windows Programming version as well, so lets try that. Well hours later I cannot find the missing files, simple ones for the include library, for the examples so UWP appears broken as well.

    So develop on Android and IMPORT to Unity 5.4 HTP says the PDF

    There is no official install video for Vuforia6.

    Wikitude 3D has the wikitude.com videos/ Tutorial set and in particular Wikitude Tutorial for Unity setup for Android
    I was up and running for Android in about17 minutes.

    Contrast this with the two days or 19 hours Vuforia6. If this was commercial I would have been sacked!

    Well I can see the need to finish the OpenCvSharp for Unity, using UWP. We need something on HoloLens we can support and control the development!!!.

    There as so many implementations of OpenCv that one must be right for HoloLens

  • Not exactly an answer to your question, but I have used a marker similar to QR codes to overlay holograms onto real objects. I setup each object, such as size, etc and then store the associated marker code with that data then when the hololens detects a marker, it reads it and overlays the associated hologram on top of the marker's object.

  • I think Wolfram is also working on something similar...

    https://www.imageidentify.com/

    Stephen Hodgson
    Microsoft HoloLens Agency Readiness Program
    Virtual Solutions Developer at Saab
    HoloToolkit-Unity Moderator

  • xantaxanta
    edited November 2016

    @LemuelBotha said:
    Not exactly an answer to your question, but I have used a marker similar to QR codes to overlay holograms onto real objects. I setup each object, such as size, etc and then store the associated marker code with that data then when the hololens detects a marker, it reads it and overlays the associated hologram on top of the marker's object.

    may i ask what tool you use to make this?
    i am using the Vuforia but it has a problem that even if i lay my model
    right over the tracking image in Unity, when i wrap it into Hololens and after recognition the model is far away off the image, in random direction, in fact not random but i just cant tell why.

  • Hi,
    I had some success using OpenCV in order to process front camera frames.

    Since I'm doing it outside of Unity, I adjusted the sources to generate appropriate UWP binaries which can then be referenced directly in a C++ holo-app.

    Depending on your needs, and acceptable conditions like ambient light & relative motion of hololens camera vs object tracked, you can still achieve decent results.

    It does not perform as fast as what you'd get on a recent desktop or phone because there is no compatible GPU acceleration available on the Hololens; the CPU is good but not as fast as one would image initially (it seems that the HPU does most of heavy lifting).

    If it can help, here is the git repo for OpenCv adjusted for Hololens UWP:
    https://github.com/sylvain-prevost/opencv-hololens

  • Has anyone reached anything new?

  • @CurvSurf said:

    When to release the FindSurface SDK

  • @elmowuming said:

    @CurvSurf said:

    When to release the FindSurface SDK

    https://forums.hololens.com/discussion/1290/augmented-reality-with-3-d-object-measurement#latest
    https://developers.curvsurf.com/docu.jsp

    The next release of FindSurface SDK is planned end Oct. 2017 including:
    1. FindSurface SDK for Windows, not yet UWP
    2. C# source code for Autodesk Revit plugin for automatic feature extraction from point cloud.

    • The C# source code is free for any usage including modification and distribution
    • FindSurface SDK is free for one month trial.

    Joon
    -
    -
    -
    FindSurface SDK for Windows C# source code for Autodesk Revit plugin.
    Functions:
    • Import point cloud (off-line)
    • Automatic feature extraction of plane, sphere, cylinder, cone, torus. Or, any of them if the measurement accuracy is good enough
    • Show/hide individual features, inliers, outliers
    • Set control parameter values.
    Usage:
    1. Set program control parameter values:

    • Point measurement accuracy
    • Touch radius: Size of point cloud patch
    • Mean distance: Point cloud density
    • Threshold of cone’s vertex angle
    • Region growing: Radial expansion, lateral extension.
    1. Select feature shape to be extracted: Plane, sphere, cylinder, cone, torus, or any
    2. Click a point among the point cloud
    3. Repeat again from 1, 2, or 3.


    Program control parameters.

    • Point measurement accuracy
    • Touch radius: Size of point cloud patch
    • Mean distance: Point cloud density
    • Threshold of cone’s vertex angle
    • Region growing: Radial expansion, lateral extension.


    Extracted features


    Hide outliers. Show inliers and features


    Point cloud


    Hide inliers. Show outliers and features


    Features, inliers, and outliers.

Sign In or Register to comment.