Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

HolographicPuzzle

lukaswerzlukaswerz
edited February 2017 in Projects

HolographicPuzzle

Holographic Puzzle is a jigsaw game for the HoloLens. Like a "normal" jigsaw puzzle, the goal is to connect the jigsaw parts correctly and to see the whole image at the end.

Short Intro Video

Hello everybody

My name is Lukas Werz and I'm a master-degree student in Information and Communication Technologies at the Interstate University of Applied Sciences of Technology Buchs, Switzerland. I have been developing Android applications for almost three years.

As an introduction in my master-thesis, I was advised to build a simple Jigsaw-Puzzle for the HoloLens, which I've finished now. You can find and download the application in the store.

Now I want to share my experiences, my problems and part of my solutions. I'm happy if someone reads this and gives some feedback, suggesting what I could change in my next project, my master-thesis.

I assume that the most people reading this are developers, so I will not comment components of the Holographic Academy. If you need more information, just let me know.

Because text size is limited, I split up my description in several parts:

  • Unity Hierarchy
  • Jigsaw Part
  • Manipulations of the jigsaw parts
  • Used Sources

Answers

  • lukaswerzlukaswerz
    edited February 2017

    Unity Hierarchy

    The game is built with Unity. The Hierarchy is shown in the image below.

    The TabToPlace GameObject appears after the SpatialMapping Process has finished. It's transformation parent is the main camera and it has a fixed distance to the cameras origin. If the user clicks on it, the ProperitiesSelection GameObject appears at this position. The reason for this additional TabToPlace is, that during a development build out of Visual Studio, it is not clear to me when the origin of the coordinate system is set. When I set the position of the ProperitiesSelection in unity manually to 0/0/2, it's located behind a wall. This is because my development monitor is in front of a wall.

    The PropertiesSelection GameObject shows the available images and resolutions of the puzzle. By resolution I mean the number of Jigsaw Parts, for example 4*3.

    The Placable Box is visible when the user should set the puzzle on a surface. The Placeable Script attached to it is from the Spatial Mapping Tutorial. The size of the Box depends on the resolution of the puzzle and the initial distance between the jigsaw parts.

    The Floor GameObject is just a Plane. The size of the Plane is the same like the ground plot of the Placable Box. Its position is set when the Placable Box is set. The reason for this is, that sometimes the meshes out of the spatial processing have holes. The jigsaw parts flying trough some of these holes are lost and the game can't be won.

    The PlayBox GameObject is a workaround for the following problem. I don't know what I have done wrong, but with the spatial mesh and the Floor GameObject it is possible to push a jigsaw part through the colliders of these objects and the jigsaw part is lost. This behavior is not possible when I simply push a jigsaw part on the floor. Instead I have to place one part on another and then push the upper one down.

    The PlayBox Bottom is just a trigger Object that lies a small distance under the floor. If a jigsaw part touches this, it means that it has fallen through the floor. In this case, the ResetPosition-Script attached to the jigsaw part resets the position of the object to the last valid position.

    The AppContext provides members for setting and retrieving data about an application's context. The only reason it is inside the Unity hierarchy, is to set and exchange a shader reference easily.

    The GameController handles the instantiation, the adjacency and the destruction of the jigsaw parts. Again here, it's only in the hierarchy to set and change references and default values.

  • lukaswerzlukaswerz
    edited February 2017

    Jigsaw Part

    One of my greatest problems was the generation of the mesh and the colliders of the jigsaw part.
    My target was to randomly create the geometry of the connectors so that each puzzle connection is different.

    Mesh
    An ellipse with parameters a and b out of a random range is created for each connection couple (once convex and once concave at the adjacent jigsaw part). The ellipse is connected to the main body with two parallel lines. The distance between these two lines is random too.

    The four corner vertices of the main body and the center of the ellipsis are at a fixed position.

    A list with all vertices of the jigsaw front elevation is created. For that, the ellipsis are interpolated with an interpolation angle (there should be a better way!). An visualization of the list is shown in the figure below.

    Then this list is passed to the Triangulator which calculates triangles.

    For the back elevation, the vertices of the front elevation are cloned and moved to the back. The triangles are cloned and reverted. The border reference vertices from the front and the back. The triangles are calculated like (pointFront[i] , pointFront[i+1], pointBack[i+1]) and (pointFront[i] , pointBack[i+1], pointBack[i]).

    Colliders
    As far as I know, Unity does not support concave colliders. But it's compulsory that a connector of a jigsaw part fits into its adjacent opposite. A convex connector should fit into a concave if the geometry allows it, and it must not fit if the geometry does not allow it.

    For the geometry of the collider, I interpolated the ellipsis with a bigger interpolation angle, to minimize the number of points.
    So I had to split up my geometry in multiple convex colliders. The first approach was to use the existing triangulator to split up the front elevation in triangles. With the triangle and the thickness of the jigsaw part I created a collider. This approach works, but is really inefficient.
    The second approach was to make a convex decomposition. After some research I recognized, that this is not a trivial job and I can't implement this in the given time. There are existing implementations, for example in CGAL (C++). Finally I found a solution adapted for unity. Originally this code is from the faseer unity asset, but user Leonardo_Temperanza modified it in such a way that only one C# class is needed.

    This results the output visualized in the following image.

    But attention! This code works perfectly in a debug or a release build. But in my case, it fails in a master build. I can't retrace why, but in a master release the exact same code throws an exception.
    I spent many days to find out what's going on, but I was unable find the reason.

    What I found was a workaround: if i exclude the convex connectors from the geometry the convex decomposition works in a master build too. I can manually split the convex connector into two convex objects.

  • lukaswerzlukaswerz
    edited February 2017

    Manipulations of the puzzleparts

    In a real jigsaw, the parts are rotated and translated in any direction. This is the behavior that I wanted to reproduce in my app. So I looked for a way to rotate and translate the jigsaw parts in an intuitive way.

    I came to the following possible solutions:

    • rotation through gravity and forces
    • two different manipulation modes, translation and rotation

    Rotation through gravity:
    If a user gazes at a specific point of the jigsaw part and makes a manipulation gesture. A force responsive to the manipulation is applied at this point. If the user lifts the piece in the air, the piece rotate until the center of gravity is under the point of force. If the user then makes a quick movement and immediately releases the hold gesture, it creates a angular moment and the piece rotates.
    But this approach does not work. It feels like try and error, the piece never rotates the way the user wants.

    Two different manipulation modes for translation and rotation:
    The user can change the manipulation mode between rotate and translate. This can be done by speech command or by airtab gesture on the jigsaw part.
    In the rotation mode, the manipulation mode rotates the puzzle part around the axis that the user manipulates. In the translation mode, the puzzle part translated.
    This approach worked fine, but it is not intuitive.

    So I decided to block the rotation for the moment.

    Dear Mircosoft Hololens Developer Team, are you planing to introduce multiple hands manipulations or finger sensitive gestures, like Zoom and Rotate on mobile phones?

    I'm happy to answer question and read your inputs!

  • Used Sources

  • Just out of curiosity, is this your master thesis in University?

    Dear Mircosoft Hololens Developer Team, are you planing to introduce multiple hands manipulations or finger sensitive gestures, like Zoom and Rotate on mobile phones?

    You can implement them yourself by using the InteractionSourceState class

  • lukaswerzlukaswerz
    edited February 2017

    Hi @Barmyard
    This is a part of my master thesis. This is just to get in touch with the HoloLens as training.

    Thank you for your answer. I will have a look at the InteractionSourceState class.

Sign In or Register to comment.