Is it thread-safe when using tf.Session in inference service?

python multithreading tensorflow tensorflow-serving
tobe picture tobe · Aug 1, 2016 · Viewed 7.8k times · Source

Now we have used TensorFlow to train and export an model. We can implement the inference service with this model just like how tensorflow/serving does.

I have a question about whether the tf.Session object is thread-safe or not. If it's true, we may initialize the object after starting and use the singleton object to process the concurrent requests.

Answer

mrry picture mrry · Aug 1, 2016

The tf.Session object is thread-safe for Session.run() calls from multiple threads.

Before TensorFlow 0.10 graph modification was not thread-safe. This was fixed in the 0.10 release, so you can add nodes to the graph concurrently with Session.run() calls, although this is not advised for performance reasons; instead, it is recommended to call sess.graph.finalize() before using the session from multiple threads, to prevent accidental memory leaks.

Related questions

How-to run TensorFlow on multiple core and threads

I should start saying that I am completely new to any kind of parallelism/multithreading/multiprocessing programming. Now, I have the chance to run my TensorFlow CNN on 32 cores (each with 2 hyperthreads). I've spent a lot of time trying to …

Read More
How to prefetch data using a custom python function in tensorflow

I am trying to prefetch training data to hide I/O latency. I would like to write custom Python code that loads data from disk and preprocesses the data (e.g. by adding a context window). In other words, one …

Read More
How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More