Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

Augmented Reality with 3-D Object Measurement

We have extracted the vertex points from the spatial mapping data of our office floor and exported them to our own software (middleware) for point cloud processing:

We have found that the structure of point cloud is based on an 8x8x8-tree.

  • 1 block = 8' x 8' x 8' = 8 brick x 8 brick x 8 brick
  • 1 brick = 1' x 1' x 1' = 8 voxel x 8 voxel x 8 voxel
  • 1 voxel = 1/8' x 1/8' x 1/8'.

What we will be doing is to display a pin-up poster onto a virtual screen in a cylindrical room corner as below:

For this purpose, we have to know accurately the radius, length, position, and orientation of the real object cylinder. The results of the cylinder fitting are shown below:

The next steps are:
1. Impelementation of the procedure to HoloLens
2. Mapping & displaying a pin-up poster onto the virtual cylindrical screen
3. Rendering the wireframe cylinder, if neccessary.

A gazing point of HoloLens will the pin-up position.
We use DirectX.

Joon

Tagged:
«1

Comments

  • CurvSurfCurvSurf ✭✭
    edited June 2016

    CurvSurf develops and distributes real-time 3D object/geometry recognition, measurement, and extraction SDK for 3D imaging/measurement applications for tablets, mobile and 3D scanning devices.

    Applications include Reverse Engineering, AR (Augmented Reality), Robot Vision, Logistic Managements, Deep Learning, Collision Avoidance, UAV, Autonomous Car, and any 3D object recognition applications which needs “Human-Like” robust perception.

    CurvSurf will not develop the end-user applications, but, will help clients as software developer do the job of application development specialized in their field.

    What CurvSurf will be doing with HoloLens is to demonstrate the functionality of the CurvSurf middleware:

    • Recognize, extract, measure, and annotate the primitives gazed by HoloLens.

    As a concrete application demonstration, CurvSurf will be showing a pin-up poster (Ad-poster) on a real object cylinder with a minimum diameter of 1'.

    URL's:

    Google +CurvSurf

    YouTube Channel CurvSurf

    curvsurf.com

    PS: Beginning 2017, the CurvSurf SDK will be prepared for public distribution.

  • JeromeJerome ✭✭

    Right, HoloLens has to offer a better spatial mapping for what we gaze.
    We should be able to precisely scan what is in front of the user on demand.

  • Advantage of Google Project Tango providing the access to raw depth data.

    https://www.youtube.com/watch?v=8ruiudETvuY

    In next days, CurvSurf will post a comparable video by using HoloLens on the YouTube Channel 'CurvSurf'. You can compare between with/without access to raw depth data.

  • HoloLens Spatial Mapping Points
    https://youtube.com/watch?v=y9v0qEIB7V4

    We have extracted vertex points from the spatial mapping of our office.

    The point density seems to be too low for retrieving the geometric information of small objects. We expect we could recognize & measure the gym ball.

  • CurvSurfCurvSurf ✭✭
    edited July 2016

    Shared from my #HoloLens.

    Sphere fitting to HoloLens spatial mapping points.

    We notice:
    1. Motion tracking is highly solid.
    2. Spatial mapping truncates convex/concave details.
    3. The minimum size of extractable object is about 0.5 m.
    4. MS 'must' provide developers with the access to raw depth data.
    5. The interpupillary distance (IPD) must be measured automatically.

    Photos & videos can be mapped (displayed) onto the real object surface:
    1.The world earth map can be displayed onto the real gym ball surface and we can simulate the worldwide airplane flights.
    2. A virtual ad-poster can be displayed onto real cylidrical wall.

    https://www.youtube.com/watch?v=C7mLH_5QzvU

    We use DirectX not Unity.

  • HoloLens Spatial Mapping Points with Cylinder Extraction

    Shared from my #HoloLens.

    Cylinder fitting to the HoloLens spatial mapping points of the cylindrical wall-corner of our office.

    Because the radius, postion & pose of the real cylindrical wall are known, photos & videos can be mapped (displayed) onto the real object surface: A virtual ad-poster or video can be displayed onto the real cylidrical wall.

    https://www.youtube.com/watch?v=bhp-uZASd_k

    The next work steps:
    1. Complete the object surface list: Plane, box, sphere, cylinder, cone, and torus
    2. Automate the recognition & determination of the surface type.

  • Is there a way to see the measurements (values in metric and imperial) effectively projected onto any of the graphics? I don't see any measurements in these videos or am I missing something? Thanks!

  • @Sealeo
    The measurements are displayed implicitly per overlaying the virtual objects onto the real objects.

    In order to ovelay a virtual object onto the target real object, we need the minimal set of accurate geometric parameter values of the real object:

    • Plane: 4 corner points
    • Sphere: center, radius
    • Cylinder: top & bottom points, radius
    • Cone: top & bottom points, top & bottom radii
    • Torus: center, axis unit vector, mean & tube radii.

    Our middleware FindSurface do the job of real-time detection, recognition, and measurement of the size, position, and orientation of real objects.

    If the input point data are dense and accurate enough, also the type of real objects of plane, box, sphere, cylinder, cone, and torus can be determined automatically.

    1Q2017, the FindSurface SDK will be prepared for public distribution and open the new age of '3D Augmented Reality' (Current augmented reality technologies are '2D Augmented Reality' based on image processing and plane extraction). The applications of FindSurface include Augmented Reality, Robotics, Building Information Modeling, Logistics, etc. CurvSurf will not develop end-user applications, but help clients do their job of application development.

    CurvSurf also will be preparing a 3D data processing service. Our clients including robots send CurvSurf via wireless communication ways 3D point clouds and images captured by laser scanner, handheld scanner, smartphone, smartglass, smartwatch, or tablet, etc. Then the data processing system of CurvSurf processes the point clouds and images, and, sends the geometric information extracted from the data back to the clients.

  • CurvSurf's middleware (DLL) for real-time 3D object/geometry recognition, measurement, and extraction has a size of ca. 200kB binary. If the density and the accuracy of 3D measurements are good enough, the middleware works much well.
    CurvSurf's middleware is single thread, non multi-processing yet.

    It will be distributed publically 1Q2017.

  • CurvSurfCurvSurf ✭✭
    edited September 2016

    An application field of CurvSurf's middleware FindSurface: Logistics.
    CurvSurf's Logistics Application 2016, 3D object recognition, FindSurface
    https://www.youtube.com/watch?v=cMmln8SujMg

    HoloLens can enter the hall of logistics only after it provides not only spatial mapping but also raw depth data to developers.

    There are many other potential application fields: AR, BIM, Robotics, Game, ...

    And, there are many other 3-D cameras providing RGBD data in real-time.

  • CurvSurf is wondering if Microsoft is willing to offer developers the access to raw depth data stream.

  • This looks very promising. We are actively looking for a solution/partner to create 3D maps of walls, doors, ceiling and floor of a large space like an office building or airport. Please message me.

  • CurvSurfCurvSurf ✭✭
    edited September 2016

    @claforte
    I recommand you to take first a look at the 3-D industry:
    Laser Scanning Forum
    http://www.laserscanningforum.com/forum
    SPAR3D
    http://www.spar3d.com/
    InterGEO
    http://www.intergeo.de
    Google +CurvSurf
    https://plus.google.com/+CurvSurf
    YouTube Channel CurvSurf
    https://www.youtube.com/user/finger6207
    curvsurf.com
    http://www.curvsurf.com/

  • Object detection, recognition, and measurement by voice command.
    https://youtube.com/watch?v=4RYMR8i8suU

    Real-Time Geometric Feature Extraction with 3-D Measuring Device
    https://plus.google.com/+CurvSurf/posts/6SkiWaUNeii

  • CurvSurfCurvSurf ✭✭
    edited November 2016

    Seeing is not yet understanding.

    A camera captures a scene, without understanding the contents.
    A 3-D printer replicates an object, without understanding the structure.

    Microsoft HoloLens, Google Tango, Intel RealSense are all 3-D camera capturing 3-D space, knowing own position and pose in 3-D space, withot understanding the 3-D space captured yet.

    How understanding and interpreting the 3-D space captured is the next question.

  • "Industrial 3-D Augmented Reality"
    Smart Manufacturing Technology Conference 2016, COEX, Seoul, Korea, 12/15/2016.
    https://youtube.com/watch?v=ubRWmkN5CnQ

  • Very impressive and solid technology. I personally believe your technology also serves the critical component when doing the object extraction application. Combining with the image recognition technology powered by ML, it could have tremendous value on consumer facing scenarios.

    The beginning of wisdom is the admission of one's ignorance.

  • CurvSurfCurvSurf ✭✭
    edited March 2017
  • CurvSurf's FindSurface SDK incl. documentation is available:
    https://developers.curvsurf.com/

    Developer registration, SDK download, and license activation are possible per online.

    Supported OS: Windows Classic.

  • Another article on CurvSurf's FindSurface SDK.

    Speedy, Efficient 3D Data Processing for any 3D Capture Solution
    ...
    That means companies like CurvSurf will be crucial to the future of 3D. The South Korean company has developed an algorithm that can recognize objects in 3D point clouds. What’s more, it’s so fast and efficient that it works real-time with a live sensor feed, and can run on an AR device.

  • UPCOMING EVENT:

    Augmented World Expo 2017
    The largest AR & VR event in the world
    Santa Clara Convention Center, Califonia
    May 31 - June 2, 2017
    http://www.augmentedworldexpo.com

    Booth No. 543
    Showcase product: FindSurface SDK for Windows classic.

    CurvSurf is planning to provide live demonstrations of 3-D augmented reality with Google Tango, Intel RealSense, and Microsoft HoloLens.

  • CurvSurfCurvSurf ✭✭
    edited June 2017

    3-D Augmented Reality

    Real-time 3-D Augmented Reality demonstrated at AWE USA 2017, Booth #543 CurvSurf.
    https://www.youtube.com/watch?v=_Cao7bVZAMg

    This video is taken by Microsoft HoloLens and CurvSurf FindSurface, and edited in 3 cuts:

    Cut #1: Finding and measuring sphere, cylinder, and plane in vertex points of spatial mapping with HoloLens.
    - HoloLens provides spatial mapping and motion tracking information.
    - CurvSurf FindSurface detects, recognizes, and measures the object size, position, and orientation from 3-D point cloud.
    https://developers.curvsurf.com/docu.jsp

    With the geometric information of real objects, we can make the following simple 3-D AR examples as in Cut #2 & #3.

    Cut #2:
    - Virtual flying ball around real gym ball with occlusion
    - Virtual drawing on real doll house
    - Virtual tiling on real floor.

    Cut #3: Virtual rotating poster on real column.

    We differentiate 3-D AR from current 2-D AR:
    - 2-D AR: based on image processing and plane detection
    - 3-D AR: based on 3-D object measurement of surface type, size, position, and orientation of real objects.

    Joon

  • 3-D Augmented Reality - Google Tango

    Sphere fitting determines the position and size of real gym ball with occlusion.

    • Virtual airplane and cars rotate around the real gym ball
    • Virtual objects can be put onto.

    https://www.youtube.com/watch?v=QHp5grEi1NE

  • @CurvSurf I noticed your software is mainly extracting simple objects at the moment like balls, pipes, crates etc. Currently at its current stage of development can it do more complex objects (example toaster) or can custom shapes be recognized/created?

  • @AvionicHologram
    Ya.
    CurvSurf FindSurface is a middleware for detecting, recognizing, and measuring the shape, size, position, and pose of objects from point cloud. Object shapes include plane, box, sphere, cylinder, cone, and torus.

    https://developers.curvsurf.com/

    Most of man-made objects, e.g. room, house, building, table, factory, plant, etc. are geometrically composed of simple primitives such as plane, sphere, cylinder, cone, and torus.

    A typical table is composed of a plane and 3~4 rods. If the size of plane is about the human hip, it's a stool.

  • We have demonstrated how CurvSurf FindSurface works with Google Tango, Intel RealSense, Microsoft HoloLens for Real-time 3-D data streaming.
    https://www.youtube.com/c/CurvSurf
    https://developers.curvsurf.com/docu.jsp

    In next months, CurvSurf will reveal:

    1. A video of demonstrative AR application to Apple ARKit. As far as we have found, the point cloud from Apple ARKit using the latest iPhone/iPad without 3-D camera option is highly noisy. Anyway, CurvSurf FindSurface may extract geometric information from that such bad point cloud.

    2. An Autodesk Revit Plugin source code in C# for automatic feature extraction from point cloud by using CurvSurf FindSurface.

    3. A real-time tracking & occlusion C/C++ code for Intel RealSense ZR300.

    Joon

Sign In or Register to comment.