Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

Issues with socket sending/receiving data with Python code

yawenlyawenl
edited October 2017 in Questions And Answers

I'm currently using Unity 5.6 to create a Holographic app on Hololens.
The app involves face detection and we decide to put the face detection code (written in Python) on a laptop and use socket to connect it with Hololens, since we want to create a better version of face detection and Python is the best language to do it.

I set Python code as a "server" and uses C# in Hololens as "client" to connect to Python code. Currently I'm having issue to keep the socket connected/open throughout the process. I have to open and close the socket on client side in every Update(). This approach takes much longer time in connecting/disconnecting. I would like to seek a way to keep the socket connected throughout the whole app process.

Now my code looks like this:
Server side (Python): It first takes the data size as data_length and then receive bytes with this length in order to receive any length of data.

host = socket.gethostbyname('IP_ADDRESS')
port = 80
s.bind((host, port))

print('Waiting for a connection...')
while (True):
    s.listen(1)
    c, addr = s.accept()
    print('Connection from: ' + str(addr))

    data_length = 0
    while True:
        data = c.recv(1024)
        if not data:
            break
        buffer += data

        if data_length == 0 and b':' in buffer:
            # retrieve data_length
            data_length, ignored, buffer = buffer.partition(b':')
            data_length = int(data_length.decode())

        if data_length == 0:
            continue

        if len(buffer) < data_length:
            print('message not long enough')
            continue
        message = buffer[:data_length]
        buffer = buffer[data_length:]
        data_length = 0

        # DO SOMETHING WITH MESSAGE

        # returned_message is the message I want to send back to client
        c.send(returned_message.encode())  # echo
        # close connection 
        c.close()
        break
# close socket
s.close()

Client Side (C# using UWP): Update() first connects to Python. It sends the data size, and then send the data to Python. Next, it waits for returned result. Finally it closes the connection.

    public class PythonConnector : MonoBehaviour
    {
        private Task exchangeTask;
        private Windows.Networking.Sockets.StreamSocket socket;
        private void connectPython()
        {
            string PORT_NO = "80";
            string SERVER_IP = "SERVER_IP";
            ConnectUWP(SERVER_IP, PORT_NO).Wait();
        }

        private async Task ConnectUWP(string host, string port)
        {

                Debug.Log("ConnectUWP");
                if (exchangeTask != null) StopExchange();
                socket = new Windows.Networking.Sockets.StreamSocket();
                Windows.Networking.HostName serverHost = new Windows.Networking.HostName(host);
                try {
                    await socket.ConnectAsync(serverHost, port);
                    byteWriter = socket.OutputStream.AsStreamForWrite();
                    writer = new StreamWriter(byteWriter) { AutoFlush = true };
                } catch (Exception e) {
                    errorStatus = e.ToString();
                    Debug.Log("connection error:" +errorStatus);
                }

        }

        public void StopExchange()
        {
            if (exchangeTask != null)
            {
                exchangeTask.Wait();
                socket.Dispose();
                writer.Dispose();
                socket = null;
                exchangeTask = null;
                writer = null;
            }

        }

        void Update()
        {

            connectPython();

            byte[] data_to_send;
            // send data size to Python
            writer.Write(data_to_send.Length + ":");

            // send data over
            try {
                byteWriter.Write(data_to_send, 0, data_to_send.Length);
                byteWriter.Flush();
            }
            catch (Exception e)
            {
                Debug.Log("write error:" + e.ToString());
            }

            // read result from Python
            Stream readStream = socket.InputStream.AsStreamForRead();
            int bytesReceived = 0;

            byte[] readBuffer = new byte[1000];
            int numBytesRead;
            try
            {
                using (MemoryStream ms = new MemoryStream())
                while ((numBytesRead = readStream.Read(readBuffer, 0, readBuffer.Length)) > 0)
                {
                    ms.Write(readBuffer, 0, numBytesRead);
                    string res = bytesReceived.ToString();
                }
                received = Encoding.ASCII.GetString(ms.ToArray(), 0, (int)ms.Length);
            }
            catch (Exception ex)
            {
                Debug.Log(ex.Message);
            }

            // stop socket connection
            StopExchange();

        }
    }

I tried removing StopExchange() and connectPython() from Update() and connect the socket at the beginning of the app, but the code would stuck at reading session after finishing sending the data over.
Does anyone has an idea why it is stuck?

Tagged:

Answers

  • Hey yawenl,
    I'm not here to sole your problem, but for asking you help on this kind of communication.

    My problem is very similar to the your:
    I have a C# app on the Hololens that works as Client and a Python application on my local machine that work as Server.

    I want that:
    the client (hololens) sends some pictures to the server, then the server handles them and sends some information to the hololens again.

    But the problem is that hololens and local machine can't estabilish a connection.
    In particular, the client App stuck at the istruction:
    await socket.ConnectAsync(serverHost, port);

    Why?

    I'm not using Unity for the code, I wrote all in Visual Studio 2017 editor.

    The server code is exactly the your (I know there is the part of the lenght etc. etc. but I don't mind because, first of all, I want that the two parts estabilish the communication).

    The client code is this:

    namespace IndustrialTerminator
    {
    internal class Program
    {
    [MTAThread]
    static void Main() => CoreApplication.Run(new UrhoAppViewSource());
    }

    public class CognitiveServicesApp : StereoApplication
    {
    
        Node busyIndicatorNode; 
        MediaCapture mediaCapture;  
        bool inited;
        bool busy;
        bool withPreview;
    
        public CognitiveServicesApp(ApplicationOptions opts) : base(opts) { }
    
        protected override async void Start()
        {
            ResourceCache.AutoReloadResources = true;
            base.Start();
    
            EnableGestureTapped = true;
    
            SpatialCursor cursor = new SpatialCursor();
            cursor.CursorEnabled = true;
    
            busyIndicatorNode = Scene.CreateChild();    
            busyIndicatorNode.SetScale(0.06f);  
            busyIndicatorNode.CreateComponent<BusyIndicator>();
    
            mediaCapture = new MediaCapture();  
            await mediaCapture.InitializeAsync(); 
            await mediaCapture.AddVideoEffectAsync(new MrcVideoEffectDefinition(), MediaStreamType.Photo); 
            await RegisterCortanaCommands(new Dictionary<string, Action> {
                    {"Describe", () => CaptureAndShowResult(false)},
                    {"Enable preview", () => EnablePreview(true) },
                    {"Disable preview", () => EnablePreview(false) },
                    {"Help", Help }
                }); 
    
            ShowBusyIndicator(true);
            await TextToSpeech("Welcome to the Tensorflow application for HoloLens and UrhoSharp.");
            ShowBusyIndicator(false);
    
            inited = true; 
        }
    
        async void Help()
        {
            await TextToSpeech("Available commands are:");
            foreach (var cortanaCommand in CortanaCommands.Keys)
                await TextToSpeech(cortanaCommand); 
        }
    
        async void EnablePreview(bool enable)
        {
            withPreview = enable; 
            await TextToSpeech("Preview mode is " + (enable ? "enabled" : "disabled"));
        }
    
        public override void OnGestureDoubleTapped() 
        {
            CaptureAndShowResult(false); 
        }
    
        async void CaptureAndShowResult(bool readText)  
        {
            if (!inited || busy) 
                return; 
    
            ShowBusyIndicator(true);    
            var desc = await CaptureAndAnalyze(readText); 
            InvokeOnMain(() => ShowBusyIndicator(false)); 
    
        }
    
        void ShowBusyIndicator(bool show)
        {
            busy = show; 
            busyIndicatorNode.Position = LeftCamera.Node.WorldPosition + LeftCamera.Node.WorldDirection * 1f;
            busyIndicatorNode.GetComponent<BusyIndicator>().IsBusy = show;
        }
    
        async Task<string> CaptureAndAnalyze(bool readText = false)
        {
    
            var imgFormat = ImageEncodingProperties.CreateJpeg(); 
    
            //NOTE: this is how you can save a frame to the CameraRoll folder:
            //var file = await KnownFolders.CameraRoll.CreateFileAsync($"MCS_Photo{DateTime.Now:HH-mm-ss}.jpg", CreationCollisionOption.GenerateUniqueName);
            //await mediaCapture.CapturePhotoToStorageFileAsync(imgFormat, file);
            //var stream = await file.OpenStreamForReadAsync();
    
            // Capture a frame and put it to MemoryStream
            var memoryStream = new MemoryStream();
            using (var ras = new InMemoryRandomAccessStream()) 
            {
                await mediaCapture.CapturePhotoToStreamAsync(imgFormat, ras); 
                ras.Seek(0); //Resetta la posizione corrente del flusso 
                using (var stream = ras.AsStreamForRead())
                    stream.CopyTo(memoryStream); 
            }
    
            var imageBytes = memoryStream.ToArray(); 
            memoryStream.Position = 0; 
    
            ***Some operations*
            //await TextToSpeech("Try to use Python connector");
            //var pythonConnector = new PythonConnector();
            //pythonConnector.connectPython();
    
            return "END";
        }
    
    }
    
    public class PythonConnector : MonoBehaviour
    {
        private Task exchangeTask;
        private Windows.Networking.Sockets.StreamSocket socket;
        private Stream byteWriter;
        private StreamWriter writer;
    
        public void connectPython()
        {
            string PORT_NO = "80";
            string SERVER_IP = "192.168.1.24";
            ConnectUWP(SERVER_IP, PORT_NO).Wait();
        }
    
        private async Task ConnectUWP(string host, string port)
        {
    
            UnityEngine.Debug.Log("ConnectUWP");
            if (exchangeTask != null) StopExchange();
            socket = new Windows.Networking.Sockets.StreamSocket();
            Windows.Networking.HostName serverHost = new Windows.Networking.HostName(host);
            try
            {
                await socket.ConnectAsync(serverHost, port);
                byteWriter = socket.OutputStream.AsStreamForWrite();
                writer = new StreamWriter(byteWriter) { AutoFlush = true };
            }
            catch (Exception e)
            {
                string errorStatus = e.ToString();
                UnityEngine.Debug.Log("connection error:" + errorStatus);
            }
    
        }
    
        public void StopExchange()
        {
            if (exchangeTask != null)
            {
                exchangeTask.Wait();
                socket.Dispose();
                writer.Dispose();
                socket = null;
                exchangeTask = null;
                writer = null;
            }
    
        }
    
        void Update()
        {
    
            connectPython();
    
            /*
            byte[] data_to_send;
            // send data size to Python
            writer.Write(data_to_send.Length + ":");
    
            // send data over
            try
            {
                byteWriter.Write(data_to_send, 0, data_to_send.Length);
                byteWriter.Flush();
            }
            catch (Exception e)
            {
                UnityEngine.Debug.Log("write error:" + e.ToString());
            }
    
            // read result from Python
            Stream readStream = socket.InputStream.AsStreamForRead();
            int bytesReceived = 0;
    
            byte[] readBuffer = new byte[1000];
            int numBytesRead;
            try
            {
                using (MemoryStream ms = new MemoryStream())
                    while ((numBytesRead = readStream.Read(readBuffer, 0, readBuffer.Length)) > 0)
                    {
                        ms.Write(readBuffer, 0, numBytesRead);
                        string res = bytesReceived.ToString();
                    }
                received = Encoding.ASCII.GetString(ms.ToArray(), 0, (int)ms.Length);
            }
            catch (Exception ex)
            {
                Debug.Log(ex.Message);
            }
            */
    
            // stop socket connection
            StopExchange();
    
        }
    }
    

    }

Sign In or Register to comment.