How to Share and Update GameObject Transforms over multiple Hololens on a Sharing Service?

Davon92 · August 2016

Hello, I am a Unity Programmer and I am having a very difficult time with Updating transforms for game objects between multiple Hololens over the sharing service/network.

I have managed to get the generic sharing prefab and network service working on my own project and I have also applied the remote head manager successfully which creates a primitive cube and updates its transform across multiple Hololens. I even have SpatialMapping and tap to place working as well.

I now want to take my own custom game objects from within the app, place them on a wall or floor and have them update their location on other headsets.

I've Searched through the Question and Answers as well as the forums on both this and Unity's site. I have re-watched Hololens academy multiple times including 240 sharing. But the issue of updating transforms of game objects within an app hasn't been brought up but once back in July.

I am not trying to use Photon or Unet unless i absolutely have to as i would like to keep this as simple to follow as possible.

and As for trying to read how the custom message can be altered, i'm not sure I have enough networking background to figure it out on my own. would anyone be able to help me figure this out?

Patrick · August 2016

I can help you, but you're going to have to believe in yourself.

I don't expect that with the information we gave in Holograms 240 that you would be able to spawn arbitrary objects in a shared world. A super confident programmer could look through the code and piece together the patterns, but let's face it how many of us are really super confident? I'll try to lay it out in a way that you hopefully can achieve this.

You don't need to know any networking, per se, to edit the custom messages. You do need to keep in mind that you aren't maintaining one world, but you are instead maintaining a world for each user in the experience, and in general you want all of those worlds to look the same.

The player avatar store in combination with the projectile launcher in the project is probably super close to what you need, if we can just squeeze them together somehow.

Let's think about what you get from the player avatar store.

You want to let people put objects in the world, and it sounds like you'd like having a list of objects available. The PlayerAvatarStore has a list of game objects that you can set from the Unity Editor and will spawn them in a long line. It's not the best UI, but it works in a pinch. You could take the time to make the interface nicer-like put the objects inside some sort of panel with a way to scroll so that you can support 100's of objects, but we'll start simple, and could use the UI this script affords;

The way PlayerAvatarStore is put together isn't exactly what you want though. You don't want to change the digital persona of the user, but instead want to change what the user places in the world. To do this, you'll want to make a new script called something like 'SpawnedItemSelector' that keeps track of the index into the PlayerAvatars[] array. And PlayerAvatars should probably be renamed to something like 'SpawnableItems'. So it's probably cut and paste time. Just make a new script called 'SpawnedItemStore: Singleton' and rename the things that need to be renamed.

So the SpawnedItemSelector should be used in this block instead of the AvatarSelector:

// Add AvatarSelector component to handle avatar selection by user.
            AvatarSelector avatarSelector = nextAvatar.GetComponent<AvatarSelector>();
            if (avatarSelector == null)
            {
                avatarSelector = nextAvatar.AddComponent<AvatarSelector>();
            }

            avatarSelector.AvatarIndex = index;

Now let's look and see what AvatarSelector does, and see how we can morph AvatarSelector into 'SpawnedItemSelector'.

Fortunately AvatarSelector is a very small script. It does a couple things:
1. It dismisses the picker. (This might not be what you want. In your case maybe you want to have the picker always open, I don't know).
2. It tells the LocalPlayerManager what AvatarIndex is used. In our case we can have it tell the LocalPlayerManager what SpawnedItemIndex is used.
3. And in the local player manager, setting the AvatarIndex causes the local player manager to send one of those custom messages to everyone else in the scene so they know what to render for your avatar. In the case of the SpawnedItem we don't actually need to do anything over the network just yet. It's just important that we keep track of the User's most recently selected SpawnedItemIndex

Okay, so at this point we have a SpawnedItemStore and a SpawnedItemSelector and our Local Player Manager can know which SpawnedItem to spawn. Now we need to spawn the item.

Here we are going to follow the projectile launcher behavior. Maybe we'd even make a new script called "ItemSpawner". This script will need to do three things:
1. Spawn the users selected Spawned item in their world view at the position of their choosing
2. Send a custom message with enough information for other users in the scene to spawn the same item in the same location.
3. Listen for other users in the scene to spawn their items at their requested location in the local user's scene.

So for the item spawner we are going to need to create a custom message that lets us send information about what to spawn where. What information do we need to spawn an item?
1. Probably the user that owns the item, though not necessarily.
2. The index in to the SpawnedItemStore's list of items.
3. A Vector3 for position and a Quaternion for rotation.

In the ProjectileLauncher start function we see that it requests to handle the 'ShootProjectile' custom message. We'll do the same thing, but we need to setup a 'SpawnItem' custom message. In CustomMessages.cs close to the top is the TestMessageID enum. Add an item BEFORE the 'Max' entry called SpawnItem.

Then if you search through customMessages for 'shootprojectile' you'll find a SendShootProjectile function that takes a Vector3 and a direction. Copy and paste that function and rename the copy to SendSpawnedItem. Remember that we need to send a UserID, the SpawnedItemIndex, a position and a rotation. So SendSpawnedItem should look like:

public void SendSpawnedItem(int SpawnedItemIndex, Vector3 position, Quaternion rotation)
    {
        // If we are connected to a session, broadcast our head info
        if (this.serverConnection != null && this.serverConnection.IsConnected())
        {
            // Create an outgoing network message to contain all the info we want to send
            NetworkOutMessage msg = CreateMessage((byte)TestMessageID.SendSpawnedItem);
            msg.Write(SpawnedItemIndex);
            AppendVector3(msg, position);
            AppendQuaternion(msg, rotation);

            // Send the message as a broadcast, which will cause the server to forward it to all other users in the session.  
            this.serverConnection.Broadcast(
                msg,
                MessagePriority.Immediate,
                MessageReliability.Reliable,
                MessageChannel.Avatar);
        }
    }

if you are paying super close attention you'll notice that this function isn't sending the user ID. That's because CreateMessage() always adds the user ID.

Patrick · August 2016

Meanwhile back at projectile launcher...

The SpawnProjectile function handles spawning locally and calling the custom messages class with all the information needed. ShootProjectile actually creates the GameObject in the world. There's a lot of noise in here that won't apply to us. We just need to Instantiate a copy of the GameObject at the LocalPlayers 'SpawnedItemIndex' at whatever position you like, then send all that information to CustomMessages.Instance.SendSpawnedItem. If you can, it would be best if these items could be spawned as children of the anchor object. In this case you can send the transform.localPosition and transform.localRotation as the Vector3 and Quaternion. Otherwise you'll need to do the anchor.inversetransform... stuff on the way out and the anchor.transform on the way in to keep the objects in the same coordinate system.

Finally we have ProcessRemotePRojectile, which we can call ProcessRemoteSpawnedItem. Here we need to read the network message in the same order as it was written. The userID is always part of the message and you always need to read it, even if you aren't using it. Then you need to read the remote item index, the position, and the rotation. Then you need to call the SpawnItem function you based off of the SpawnProjectile function.

Something like:

void ProcessRemoteSpawnedItem(NetworkInMessage msg)
    {
        // Parse the message
        long userID = msg.ReadInt64();
        int RemoteItemIndex = msg.ReadInt32();
        Vector3 remoteItemPosition = CustomMessages.Instance.ReadVector3(msg);
        Quaternion remoteItemDirection = CustomMessages.Instance.ReadQuaternion(msg);


        SpawnItem(RemoteItemIndex, remoteItemPosition, remoteItemDirection);
    }

/whew.

Okay, so I'm still leaving some gaps like wiring up the ItemSpawner as opposed to the ProjectileLauncher. That's handled in the AppStateManager.cs as the 'overridefocussedobject'. I'm happy to answer questions about that as well, but the networking bit is what you seemed most interested in.

Davon92 · August 2016

Ok so still trying to process what i need to make it a simpler test.

I do not need a store as of yet as i'm using objects already in the scene.

I do need a Send Object Transform function in custom messages that acts like send Head Transform function but do i have to add an item index for specific items?

Can I instead have the script on a game object and that object will handle its own call to send its local position and rotation to custom messages?

Next i need the process remote Spawned item function this seems pretty straight forward as it reads the information then calls the spawn item function.

For my instance of the spawn projectile function which calls the custom message send shoot projectile. I am misunderstanding the need or the transform anchor. How is the anchor transform variable sending information to the custom message that the vector 3 position and quaternion variables could not send instead?

Also I am using the tap to place script the tell my object where to eventually end up.

So is it correct to assume that i should create a callback when the user has finished placing that will call to the "spawn projectile" function or even a keyword update that will act as the point of reference to call the script with the position and rotation?

Patrick · August 2016