Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

Got my game idea! - Stealth

Jarrod1937Jarrod1937 ✭✭✭
edited April 2016 in Projects

I've developing an application for work using Unity. I figured while I'm learning Unity for work I may as well develop my own game along with it. At first I was thinking a miniature scale game that can be largely kept within the viewport, for comfort. However, Fragments got me thinking larger.
I think one of the best ways to build on the augmented part of the Hololens would be a real-life stealth game. I'm thinking more along the lines of of the old Metal Gear Solid VR missions, where you're simply trying to reach a waypoint to progress on.
Although I used to do video game artwork in my past life, I'll probably do a pre-made asset for the enemy that already has animations to save me time, maybe with a re-texturing for uniqueness.
Anyone see any pitfalls with this idea? Any features you'd like to see (can't make any promises)?

Things I have planned:

  • A collider extending from the characters head with a fixed distance for vision. If the camera enters the collider, the character can see the user. This may need reworking or be changed, as I want the detection to not be triggered if the user is occluded by the enviroment (wall, chair...etc).
  • Voice commands for distraction. Similar to, 'over here,' which will entice the character to investigate.
  • If I understand correctly, the built-in nav mesh will not work, as that is a baked nav setup. However, there seems to be a few nice generic realtime A* methods already implemented that I can use.

One concern is that skinned meshes will have a bit more of a performance hit, so I may be limited to one or two enemies.


  • Sounds like fun!

  • Nice idea. Young Conker should also provide some inspiration.

    I wonder if user height and pose will be a variable you'll need to consider. IE, the HoloLens acts as the camera, but the actual user is much larger than the HoloLens, and the HoloLens doesn't know if you are standing or sitting or crouching. That could add another wrinkle to the logic of if the virtual character can 'see' the player.

    This post provided as-is with no warranties and confers no rights. Using information provided is done at own risk.

    (Daddy, what does 'now formatting drive C:' mean?)

  • Jarrod1937Jarrod1937 ✭✭✭
    edited April 2016

    @Patrick, yeah, I'm considering having a box collider, that roughly gives the area of a user, attached to the camera. The collider can be made to resize it's height to fit the camera's height. It's an estimation, but it should be relatively effective I think. I want to avoid a mesh collider for performance reasons.
    After researching more, it looks like I'll use a collider attached to the enemies head that projects out to a distance of 15 meters. The collider will be set to trigger mode, so if it's area intersects with the player collider at all, the ontrigger event will be called. From here, I think doing some ray casting to 15 meters at various angles horiztontal and vertical to try and see if the enemy can "see" the player. Considering there will be no other colliders in the scene and the player is just a box collider, I would think the ray casting should be fairly cheap. I've looked at sphere casting, which may lessen the number of rays needed, but I am not sure if it would be computationally cheaper than more rays. The trigger is simply used as an optimization, so that the enemy isn't ray casting if the rays won't hit the player anyways.

    Quick question, does ray casting get blocked by the spatial mesh by default or do I have to do something special? If the player is behind a chair, I don't want the ray cast to return true. Since this is a personal project I don't have my work computer, and can't use the emulator at home... So it's a bit of a flying blind until I get the device itself (I'm allowed to take that home in the evenings).

  • @Jarrod1937 Yes ray casting can get blocked by spatial mapping, I think you can look at CursorManager in the toolkit, or possibly GazeManager, one of those ray casts and hits spatial mapping. I know this because of a project i did for work, we used gaze and cursor to interact with the environment that was spatial mapped and didn't have to do anything special.

    BTW this is a wonderful idea, looking forward to seeing what you can do, if you need help let me know, sounds like fun ;)

  • Jarrod1937Jarrod1937 ✭✭✭
    edited April 2016

    I dub thee, Box Man! Starting out with basic geometry getting the basics of the code fleshed out.
    The vision object I created by:

    • There are no pyramid primitives in unity oddly. I imported a pyramid into Unity.
    • I rotated and sized the vision pyramid to the size I wanted.
    • Attached it to the head, so when the head rotates, so does the pyramid.
    • I unchecked "Mesh Renderer" on the pyramid, so that it isn't rendered.
    • I added a Mesh Collider, and used the same mesh as a mesh collider, with Convex checked. Normally mesh colliders are poor for performance, however, this is a very basic mesh, so the performance should still be comparable to primitives.

    Now here is where I was able to do a trick to speed up the vision when doing ray casting. Triggers just let you know something triggered... That's about it, it's up to you to find where that item is that triggered it. So I did not use a trigger in this case. Instead, the pyramid has a rigidbody attached to it with all constraints checked, so that it isn't actually affected by physics. Then, onCollisionEnter, similar to onTriggerEnter, you can actually find the first contact point via:

    // Only care if it is the player/testobject, // ignore other collisions if(col.gameObject.name == "testobject") { Debug.Log("Contact at: " + col.contacts[0].point); }

    This should allow for a lot of optimization so that I only need to vaguely do a few ray casts in the direction and area of the contact point.

  • Jarrod1937Jarrod1937 ✭✭✭
    edited April 2016

    I went back to the trigger method. Turns out if you use the above method you'll have to call an ignore collider on everything you want to ignore the vision object... no fun.
    However, I have made progress. Here is the system I have now:

    • Same pyramid as before.
    • The pyramid is set to trigger.
    • On trigger I get the position of the collider box attached to the camera.
    • In the onTriggerStay (so it's not just a minor blip) I call a function that sends out 12 ray casts toward the collider box by getting the boxes bounds/extents and ray casting in that area.
    • If one of the rays hits, the "playerSeen" function returns true.

    This works pretty well, and the geometry has been kept simple so the ray casting is pretty computationally cheap. The basically functionality is there, now time to develop a state machine interface for the rough enemy AI states.
    The soldier I plan on using is:


    Now, while that asset looks great there are quite a few things I'll need to do to it. First, I'll be using one of the lower LOD's, as 10K is a bit much. Secondly, I'll be baking the ambient occlusion and normal maps into the diffuse so that it's only a diffuse textured model... This is a bit of an art and the results won't look perfect but it should run better then. The textures will be lowered in resolution as well. Lastly, because the Hololens renders black as transparent, he'll be getting a brighter retexture of the darker areas.

  • Jarrod1937Jarrod1937 ✭✭✭
    edited April 2016

    Added the soldier model. The LOD system is pretty nice, though I have yet to optimize the textures. The screenshot below shows the vision pyramid attached and moving with the animated head of the soldier. Since me, the player, is within the trigger range, you can see the ray casts, though since I am behind the wall there, the debug is registering 'NOT', as in not visible to the enemy. It seems like the newer, non-legacy animation system is a bit more complicated than the legacy system I saw in videos, so I'll need some time to learn that. However, as I said, the next stage is the AI state machine. I must say though, that as a first time user of Unity, it sure does make prototyping easy.

  • @Jarrod1937 Ya Unity is really good at just getting up and going, very little ground work to do. Excellent progress on this project, very cool!

  • Jarrod1937Jarrod1937 ✭✭✭
    I have the state machine done. I followed an example tutorial closely as it fit my game pretty well already. However, the next stage is to provide actions and movement for each state. For this I am going to implement pathfinding first. For performance sake I'm going to have the user scan the environment to play in, then freeze updates, then generate a navmesh at that time. This should help with computation a lot since I doubt the environment will change that drastically for a game like this.
  • Jarrod1937Jarrod1937 ✭✭✭

    Just to give an update. I have basic pathfinding implemented (via Apex Pathfinding for Unity), and have it generating a path grid graph at runtime. I have yet to test if the pathfinding will work with the spatial mesh, however you can use Apex to generate the grid at runtime based on a particular mesh layer, and the Spatial Collider script seems to generate the mesh in a particular layer of your choice... Logically you think it would work fine.
    For some non-coding I got around to simplifying the material and textures of the main enemy solider. I switched the material to use the Hololens standard fast shader, turned off highlights and reflections, used albedo alpha, and removed all normal and ambient occlusion maps. I then baked these into the diffuse, used the green channel of the normal for the baked lighting and did a multi-layer bake with the ambient occlusion (just a plain overlay didn't look as nice). The balaclava (ski mask) I lightened to a light gray to have it not be too transparent in the Hololens.
    The results look pretty good considering it's purely a diffuse texture at this point. I will probably have to adjust it a bit more when I get the Hololens due to having a hard time judging black/gray as transparent and semi-transparent respectively.
    Next is to use the path finding to start connecting movement and other behavior to my state machine AI setup.

  • Jarrod1937Jarrod1937 ✭✭✭
    edited May 2016

    I now have the enemy soldier animated. He is idle, then when he sees you he goes into attack mode, runs after you (via path finding), once he is 3 meters away he stops and fires. I don't think I'll actually spawn any bullets, I think the test will be if he is firing and one of his vision rays is hitting the player, then he will hit the player and the player is dead.
    Next is to do a patrolling state, the tricky part is figuring out how to procedurally place the patrol way points.
    While the soldier has vision, I want to simulate hearing as well. I'm thinking another collider, a sphere, that is centered at the guys position. If the player enters this sphere, the closer they get the slower they have to move to avoid detection. The issue with the game is that I need it to be difficult, even though the player will have precision never before seen in a game. The Hololens allows the player to easily control their height, position, speed, acceleration...etc, makes for interesting game design challenges.

  • It's so fun to read your project updates! You'll have to post a video link soon :)

  • Jarrod1937Jarrod1937 ✭✭✭

    @ahillier, glad you like the updates :smile:
    I may not do a video just yet... It's one of those things where things are a bit too rough to show in video form just yet. I've mostly been fleshing out the functionality, the display portion is incomplete. It's getting there though.
    The latest things I've done is to have a threshold for attack versus just an alert state. If the enemy sees you for less than 0.7 seconds, he'll go to that point to investigate, anything over and he knows exactly where you are and will attack. Less exciting I programmed a main singleton application class for storing global state info and for storing user prefs and data.
    I think I figured out the patrol routes. Obviously in a game where the environment is dynamic you can't set the patrol routes manually ahead of time. So, I'm thinking to procedurally place the goal hologram, which has a spherical collider attached to it. This collider then gives the bounds for the patrol routes of the soldiers. After all, they are protecting the goal anyways, so it makes sense to define their patrols around it. Then do some sort of periodic ray in multiple directions and go with the longest one for moving to, go there, stay for a few seconds, rinse and repeat, maybe with a bit of randomness to prevent locking into the same two points. Seems good in theory, I'll let you know how well it goes.

  • Jarrod1937Jarrod1937 ✭✭✭

    Just posting to let everyone know I'm still making progress. I was disappointed to find that the Apex path finding code I was using doesn't work with UWP. Since then, thanks to a forum member here, I think I have a path finding solution that'll work, but I'll have to migrate my code over to it. In the meantime, I've been working on using and augmenting the plane finding code that's part of Holotoolkit. I've managed to get relatively stable plane finding, and code to place prefabs at wall planes. I am cycling through all floor planes and finding the one with the largest bounds and using that as the main floor reference, same with the ceiling planes. This doesn't seem to work perfectly however, so I may experiment with picking the one that's the highest Y coordinate for floors and the lowest for ceiling. Next is to check the bounds of the prefab for the best placement on the planes.
    I also experimented with world space triplanar shaders. This has the interesting property in that I can apply this shader to all found planes and they will in essence UV map themselves to a predefined tile amount, no matter their orientation or placement. The result is that the wall planes texture themselves to a defined wall texture, same with floors and ceilings. This worked okay, but the imprecise nature of the plane finding and placement made it look a bit hacky. I think the best approach is to use prefabs onto the planes for augmenting them, rather than straight texture them.

  • Jarrod1937Jarrod1937 ✭✭✭

    An update, making progress. I now have path finding functioning, the soldier spawning and walking toward me. He is slightly (about an inch or so) into the floor, but that can be fixed easily enough. The plane finding closes up the doors with walls, so each room is segmented off, but I am working on messing with the planefinding DLL to prevent this from happening.
    I'll try and post a video if I have the time. It's somewhat intimidating to have a big beefy soldier walking towards you in your living room, haha, but that's the feeling I'm going for.

  • Would have helped if we had access to the plane finding dll code. for me the plane finding is very wonky.. especially for table tops and floors. walls and ceilings work well. How are you doing with the floor and table tops.. are you able to patch up holes and such?

    Healthcare IT professional by day - Indie GameDev for UWP and mobile platforms by night

  • Jarrod1937Jarrod1937 ✭✭✭

    @DanglingNeuron said:
    Would have helped if we had access to the plane finding dll code. for me the plane finding is very wonky.. especially for table tops and floors. walls and ceilings work well. How are you doing with the floor and table tops.. are you able to patch up holes and such?

    Actually, they do give us access, though it wasn't initially intuitive that the .dll used by the Unity toolkit is part of the native toolkit:

    I didn't know until a mod here mentioned it. I've downloaded the code and have been messing around with the planefinding code.
    In regards to tables... it's hit or miss, but floors and walls for the most part do pretty well. Thankfully for my project, tables aren't of any importance beyond being obstacles to avoid, so the results can be rough in my case. I just need to get the individual rooms to not be segmented, unless I want to limit game play to a single room.

  • Jarrod1937Jarrod1937 ✭✭✭

    Here is a video update for you guys. Still rough, but it's slowly getting there:

  • Wow, nice work!


  • Jarrod1937Jarrod1937 ✭✭✭

    Thanks! Next stage is to develop the patrolling portion of the enemy AI. The AI is pretty similar:

    • Player places goal in room.
    • Enemy spawns around goal, generates patrol points, moves between them, with random intervals.
    • If enemy 'sees' you, meaning the ray cast hits you, it will:
      a. Cause them to investigate, if ray casting hits the players body and not their head.
      b. Straight into alert mode if the enemy's ray cast hits the payers head.

    • For the most part, it's game over if the enemy spots you, as this is pure stealth, but I'll have the enemy shoot the player.

  • Really doing well, such a great amount of progress. Keep going nice video

  • JeromeJerome ✭✭

    did you check the RAIN AI tool for unity? there is stuff to manage the field of view of the enemy as well as sound detection, the GUI is pretty good to create complex AI patterns.

  • Jarrod1937Jarrod1937 ✭✭✭
    edited June 2016

    @Jerome said:
    did you check the RAIN AI tool for unity? there is stuff to manage the field of view of the enemy as well as sound detection, the GUI is pretty good to create complex AI patterns.

    I had not, but I'll certainly check it out. I've just became cautious of assets that relate to code, since they sometimes reference old .NET items that don't function in the Hololens's UWP architecture (like Apex mentioned above). If it looks to not rely on anything, then I think such a thing will be a time saver, thanks for the suggestion!

  • Jarrod1937Jarrod1937 ✭✭✭

    An update:
    I have programmed a random waypoint system. Since the environment can be anything, the waypoints for the patrol route can't be set ahead of time. So the new system is:

    • Soldier spawns.
    • Do a 360 degree raycasting in 10 degree increments with 3 distance rings (15, 10, 5 feet). Each raycast that doesn't hit anything is recorded in an array for each distance.
    • The longest distance raycast is selected randomly for the first move, thereafter it picks the longest ray with the least amount of angle change from the previous to the new vector.
    • This is repeated until the max number of way points is reached (currently set to 4).

    The waypoints chosen are then fed to the path finding code, just in case the raycasting missed small details, the pathfinding will route the soldier around any obstacles.
    For the most part this works. The waypoints are chosen based on the environment, the least angle change keeps the waypoints in a natural fashion (before then it was zigzagging). This code is separated from the main code so it can be isolated into a patrol state.
    The next portion is to figure out how to interpolate the motion. The pathing code makes it a bit difficult to ramp up and slow down the AI entities motion. After this, it's putting the pieces together so that I have the three basic states:
    1.) Patrol
    2.) Investigate
    3.) Attack

  • Hey Jarrod,

    We're solving similar issues! For my game, I'm also coming up with a custom navigation system based on the spatial mapping and having an autonomous agent in game. I'll be making an update video of my progress soon so you can see how I accomplish the navmesh and getting around the doorway problem you mentioned in your video.

  • Jarrod1937Jarrod1937 ✭✭✭

    Great, look forward to your progress as well :)

  • This is great. Seeing how far you have come it gives me hope that I can get my concept put together.

  • Thanks for sharing. Great work!

  • Just to update everyone, this project isn't dead or anything. Since I am working on this in my spare time, I've had to make slow progress on it when I'm busy. I am working on a new software system for my work, after I'm done with this I'll start working on my Hololens work application (holographic warehouse fulfillment software). If I'm allowed I'll post details on that project too.

Sign In or Register to comment.