UDP-Communication [SOLVED]

DrNeurosurg

After a little bit searching I found a solution for UDP-Communication (sending & receiving).
I found a very simple solution at

https://github.com/rondagdag/hololens-littlebits/blob/master/UnityScripts/UDPCommunication.cs

and modified it a "little bit" for more flexibility ..

Some remarks:

• the code works ONLY in an UWP-App (Unity) - on Hololens and PC - not in Unity-Editor !
• for testing the communication I used https://packetsender.com/download

UDPCommunication.cs

using UnityEngine;
using System;
using System.IO;
using System.Text;
using System.Linq;
using HoloToolkit.Unity;
using System.Collections.Generic;
using UnityEngine.Events;

#if !UNITY_EDITOR
using Windows.Networking.Sockets;
using Windows.Networking.Connectivity;
using Windows.Networking;
#endif

[System.Serializable]
public class UDPMessageEvent : UnityEvent<string, string, byte[]>
{

}

public class UDPCommunication : Singleton<UDPCommunication>
{
    [Tooltip ("port to listen for incoming data")]
    public string internalPort = "12345";

    [Tooltip("IP-Address for sending")]
    public string externalIP = "192.168.17.110";

    [Tooltip("Port for sending")]
    public string externalPort = "12346";

    [Tooltip("Send a message at Startup")]
    public bool sendPingAtStart = true;

    [Tooltip("Conten of Ping")]
    public string PingMessage = "hello";

    [Tooltip("Function to invoke at incoming packet")]
    public UDPMessageEvent udpEvent = null;

    private readonly  Queue<Action> ExecuteOnMainThread = new Queue<Action>();


#if !UNITY_EDITOR

    //we've got a message (data[]) from (host) in case of not assigned an event
    void UDPMessageReceived(string host, string port, byte[] data)
    {
        Debug.Log("GOT MESSAGE FROM: " + host + " on port " + port + " " + data.Length.ToString() + " bytes ");
    }

    //Send an UDP-Packet
    public async void SendUDPMessage(string HostIP, string HostPort, byte[] data)
    {
        await _SendUDPMessage(HostIP, HostPort, data);
    }



    DatagramSocket socket;

    async void Start()
    {
        if (udpEvent == null)
        {
            udpEvent = new UDPMessageEvent();
            udpEvent.AddListener(UDPMessageReceived);
        }


        Debug.Log("Waiting for a connection...");

        socket = new DatagramSocket();
        socket.MessageReceived += Socket_MessageReceived;

        HostName IP = null;
        try
        {
            var icp = NetworkInformation.GetInternetConnectionProfile();

            IP = Windows.Networking.Connectivity.NetworkInformation.GetHostNames()
            .SingleOrDefault(
                hn =>
                    hn.IPInformation?.NetworkAdapter != null && hn.IPInformation.NetworkAdapter.NetworkAdapterId
                    == icp.NetworkAdapter.NetworkAdapterId);

            await socket.BindEndpointAsync(IP, internalPort);
        }
        catch (Exception e)
        {
            Debug.Log(e.ToString());
            Debug.Log(SocketError.GetStatus(e.HResult).ToString());
            return;
        }

        if(sendPingAtStart)
            SendUDPMessage(externalIP, externalPort, Encoding.UTF8.GetBytes(PingMessage));

    }




    private async System.Threading.Tasks.Task _SendUDPMessage(string externalIP, string externalPort, byte[] data)
    {
        using (var stream = await socket.GetOutputStreamAsync(new Windows.Networking.HostName(externalIP), externalPort))
        {
            using (var writer = new Windows.Storage.Streams.DataWriter(stream))
            {
                writer.WriteBytes(data);
                await writer.StoreAsync();

            }
        }
    }


#else
    // to make Unity-Editor happy :-)
    void Start()
    {

    }

    public void SendUDPMessage(string HostIP, string HostPort, byte[] data)
    {

    }

#endif


    static MemoryStream ToMemoryStream(Stream input)
    {
        try
        {                                         // Read and write in
            byte[] block = new byte[0x1000];       // blocks of 4K.
            MemoryStream ms = new MemoryStream();
            while (true)
            {
                int bytesRead = input.Read(block, 0, block.Length);
                if (bytesRead == 0) return ms;
                ms.Write(block, 0, bytesRead);
            }
        }
        finally { }
    }

    // Update is called once per frame
    void Update()
    {
        while (ExecuteOnMainThread.Count > 0)
        {
            ExecuteOnMainThread.Dequeue().Invoke();

        }
    }

#if !UNITY_EDITOR
    private void Socket_MessageReceived(Windows.Networking.Sockets.DatagramSocket sender,
        Windows.Networking.Sockets.DatagramSocketMessageReceivedEventArgs args)
    {
         Debug.Log("GOT MESSAGE FROM: " + args.RemoteAddress.DisplayName);
        //Read the message that was received from the UDP  client.
        Stream streamIn = args.GetDataStream().AsStreamForRead();
        MemoryStream ms = ToMemoryStream(streamIn);
        byte[] msgData = ms.ToArray();


        if (ExecuteOnMainThread.Count == 0)
        {
            ExecuteOnMainThread.Enqueue(() =>
            {
                Debug.Log("ENQEUED ");
                if (udpEvent != null)
                    udpEvent.Invoke(args.RemoteAddress.DisplayName, internalPort, msgData);
            });
        }
    }


#endif
}

Assign this Script to an (empty) GameObject, set the Properties and assign a UnityEvent to UdpEvent ... like this:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class UDPResponse : MonoBehaviour {


    public TextMesh tm = null;

    public void ResponseToUDPPacket(string incomingIP, string incomingPort, byte[] data)
    {

        if (tm != null)
            tm.text = System.Text.Encoding.UTF8.GetString(data);

#if !UNITY_EDITOR

        //ECHO 
        UDPCommunication comm = UDPCommunication.Instance;
        comm.SendUDPMessage(incomingIP, comm.externalPort, data);

#endif
    }
}

Any comments and/or modifications are welcome !

araujo

Hi! I was wondering what latency you get with this implementation? Thanks!

JKukulsky

Hey DrNeurosurg! Great work.
I have prior exposure to serial comms, but not over IP.
Just for clarification, UDPCommunication.cs gets assigned to an empty game object to be run in a UWP app from the computer, and UDPResponse.cs gets assigned to an empty game object to be run in a UWP on the Hololens (just to echo back)?
Please correct me if I'm wrong.

DrNeurosurg

@JKukulsky:

No - On Hololens-UWP-App Assign the UDPCommunication.cs to an empty Gameobject and also UDPResponse.cs to the same or diffrent Gameobject ...
It's both for receiving.
If you want send from an Unity-UWP-Desktop-PC-App you need only to assign UDPCommunication.cs to an Gameobject...

I'll post a Github-link to both examples in a short time ..

summerIntern

@DrNeurosurg
How would you use this to control a game object using the message received from a udp server?

marklee

This does not work properly.

Weirdly, Hololens will only be able to receive UDP messages from externalIP after it does SendUDPMessage(externalIP, externalPort,Encoding.UTF8.GetBytes(PingMessage));

Somehow it does not listen like a UDPServer. It needs to send to a Server (externalIP) 1st then wait to receive messages.

I tried out the scripts here on Hololens and does not work too for a typical UDP Server https://docs.microsoft.com/en-us/windows/uwp/networking/sockets

Anyone has insights?

THanks !

Huachen

@DrNeurosurg have you post your example to the GitHub-link? Thank you.

seb

Can we get some official word from Microsoft on this? It is so weird that the Hololens doesn't listen to UDP packets. This should not have to do with implementation on the application layer. It should listen to UDP period. What is going on?

marklee

@seb said:
Can we get some official word from Microsoft on this? It is so weird that the Hololens doesn't listen to UDP packets. This should not have to do with implementation on the application layer. It should listen to UDP period. What is going on?

Got to be patient .... or wait for Hololens 2 ?

沐冉楠

nice

seb

Doesn't work for me. Microsoft this is on you. I have client projects currently stalled because of this thing. How can a 2016 device NOT work with UDP? I'm not sure if I should cry or laugh at this.

I have done complete testing and it works fine in standalone UWP-app, it works fine in other settings like mobile, just not the Hololens.

So much for UWP.

seb

@marklee said:

@seb said:
Can we get some official word from Microsoft on this? It is so weird that the Hololens doesn't listen to UDP packets. This should not have to do with implementation on the application layer. It should listen to UDP period. What is going on?

Got to be patient .... or wait for Hololens 2 ?

Yeah that doesn't fly with clients though. And the rest of the internet. UDP is not something new.

summerIntern

Hi everyone, this code works very well. Sincere thanks to Dr Neurosurg . I was able to use it to receive information from a udp server (that I programmed in ubuntu) to overlay body pose estimation on people in near real-time, check out the video here (> 1 min 22 s):
https://www.youtube.com/watch?v=lflbEWC5VsA&t=24s

DaveArendash

@summerIntern said:
Hi everyone, this code works very well. Sincere thanks to Dr Neurosurg . I was able to use it to receive information from a udp server (that I programmed in ubuntu) to overlay body pose estimation on people in near real-time, check out the video here (> 1 min 22 s):
https://www.youtube.com/watch?v=lflbEWC5VsA&t=24s

Hi summerIntern: by 'this code' do you mean the code pasted in this thread above? Or some other code you worked on? Any tips?

Jimbohalo10

@Huachen said:
@DrNeurosurg have you post your example to the GitHub-link? Thank you.

Yes @DrNeurosurg did see DrNeuroSurg -GitHub repositories

JeffGullicksen

I have this code working on a PC. I use the sharing service to communicate with the HoloLenses and this code to communicate with other stuff. Thanks for sharing this code.

mtaulty

I'm unsure of whether this might help this thread but as part of the set of experimental blog posts that I wrote which started here;
https://mtaulty.com/2017/12/29/experiments-with-shared-holograms-and-azure-blob-storage-udp-multicasting-part-1/

I wrote a small UDP multicasting library to send/receive small messages between HoloLens devices.

I also wrote a console-based test application and a 2D UWP XAML based test application for that library.

That library and those apps seem to work ok for me when running from HoloLens<->PC although I will flag that sometimes I find that the HoloLens doesn't seem to pick up messages from the PC and I find that I have to reboot the device to clear that problem although I'm not 100% sure that it isn't my code not doing the right things with respect to timeouts on sockets.

Regardless...I thought I'd add this info to this thread in case any of that helps with some of the UDP explorations that people are making.

As an aside, the blog posts go on to build on top of this library but the first post only points to the library itself so that might be relevant here around UDP.

toriop

because I failed to use this program, is there anybody know how to set "externalIP"? it is the ip of my computer?

mumblefish

@DrNeurosurg @JeffGullicksen I got a version of this solution working one-way (hololens -> pc) but I can't seem to send anything from pc -> hololens, it's driving me nuts. would anyone be kind enough to share a working example of this code?

Gabrio

Hi, thanks for sharing.
I'm able to receive a webcam stream to my hololens, but I've a problem of memory leak. Did you have it too?
If nothing pass over the UDP, the memory is stable, when I turn on the UDP sender (a smartphone), the memory increase and I have only the UDPCommunication in my scene.
Where could I add something to free the memory?
Thanks

UPDATE: I saw that the line that increase the memory process od the Hololens when I receive data is this one:

Stream streamIn = args.GetDataStream().AsStreamForRead();

Why should it increase constantly? I'm using GC.Collect() every 10 sec, but nothing happens to the memory unfortunatelly.
Please help.

Holomoon

@DrNeurosurg Thanks for your post. In my application, I need to send a picture fro HoloLens to pc for processing and then send back the result to HoloLens. I am really new to this, could you please let me know how I should modify the code?

ChiaraPalu

Dear All,
I m using the code kindly shared by @DrNeurosurg to send info from a server. The script is able to send a ping to the server and gets connected, but then when it has to receive a message from the server it crashes and I have the following message on output repeatedly: Dropping touch event part of canceled gesture.Dropping touch event part of canceled gesture.........
Does anyone have an idea on why this is happening? It seems hololens is not allowed to receiveMessage from the server. It always stops before entering in the Socket_MessageReceived. Do you know if I have to do something on the device itself to allow this? I ve also tried to deploy my app on local machine (the pc) and everything works fine, but when I deploy the same app on the hololens device it doesn t work! Thanks a lot!