Communication between Python and C#

Question

I have a Python backend running machine learning algorithms. I want to use the same backend for both an Excel plugin (C#) and a website. I want both interfaces to send my training data (thousands of lines of numbers in arrays) to the same Python application and retrieve the results in the form of another array up to a few thousand lines.

The website would fetch data from a SQL database and send that data to Python, while the Excel plugin would take the data that is in the current worksheet and send that data to Python. I need to be able to create numpy arrays in Python before continuing to process the data. Note that the website would be running on the same machine where the Python application resides. I still haven't decided what I will use to code the website, but I was leaning towards Node.js.

I have done some research and found a few options:

1- Named pipes
2- Sockets
3- RPC server such as gRPC or XML-RPC.
4- Writing the data to a file and reading it back in Python
5- Web Service

Note: I would need the Python "server" to be stateful and keep the session running between calls. So I would need to have a kind of daemon running, waiting for calls.

Which one would you experts recommend and why? I need flexibility to handle several parameters and also large arrays of numbers. Using IronPython is not an option because I am running Keras on Python, which apparently does not support IronPython.

Answer 1

I had the same problem recently. I used a named pipe to transport data from python to my c# server, hope it helps you.

Python:

import win32pipe, win32file

class PipeServer():
    def __init__(self, pipeName):
        self.pipe = win32pipe.CreateNamedPipe(
        r'\\.\pipe\\'+pipeName,
        win32pipe.PIPE_ACCESS_OUTBOUND,
        win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT,
        1, 65536, 65536,
        0,
        None)

    #Carefull, this blocks until a connection is established
    def connect(self):
        win32pipe.ConnectNamedPipe(self.pipe, None)

    #Message without tailing '\n'
    def write(self, message):
        win32file.WriteFile(self.pipe, message.encode()+b'\n')

    def close(self):
        win32file.CloseHandle(self.pipe)


t = PipeServer("CSServer")
t.connect()
t.write("Hello from Python :)")
t.write("Closing now...")
t.close()

For this code to work you need to install pywin32 (best choice is from binarys): https://github.com/mhammond/pywin32

C#-Server:

using System;
using System.IO;
using System.IO.Pipes;

class PipeClient
{
    static void Main(string[] args)
    {
        using (NamedPipeClientStream pipeClient =
            new NamedPipeClientStream(".", "DMXServer", PipeDirection.In))
        {

            // Connect to the pipe or wait until the pipe is available.
            Console.Write("Attempting to connect to pipe...");
            pipeClient.Connect();

            Console.WriteLine("Connected to pipe.");
            Console.WriteLine("There are currently {0} pipe server instances open.",
               pipeClient.NumberOfServerInstances);
            using (StreamReader sr = new StreamReader(pipeClient))
            {
                // Display the read text to the console
                string temp;
                while ((temp = sr.ReadLine()) != null)
                {
                    Console.WriteLine("Received from server: {0}", temp);
                }
            }
        }
        Console.Write("Press Enter to continue...");
        Console.ReadLine();
    }
}