TensorFlow in production for real time predictions in high traffic app - how to use?

python machine-learning tensorflow tensorflow-serving
Nir picture Nir · Feb 15, 2016 · Viewed 8.1k times · Source

What is the right way to use TensorFlow for real time predictions in a high traffic application.

Ideally I would have a server/cluster running tensorflow listening on a port(s) where I can connect from app servers and get predictions similar to the way databases are used. Training should be done by cron jobs feeding the training data through the network to the same server/cluster.

How does one actually use tensorflow in production? Should I build a setup where the python is running as a server and use the python scripts to get predictions? I'm still new to this but I feel that such script will need to open sessions etc.. which is not scalable. (I'm talking about 100s of predictions/sec).

Any pointer to relevant information will be highly appreciated. I could not find any.

Answer

mrry picture mrry · Feb 16, 2016

This morning, our colleagues released TensorFlow Serving on GitHub, which addresses some of the use cases that you mentioned. It is a distributed wrapper for TensorFlow that is designed to support high-performance serving of multiple models. It supports both bulk processing and interactive requests from app servers.

For more information, see the basic and advanced tutorials.

Related questions

TensorFlow: How to predict from a SavedModel?

I have exported a SavedModel and now I with to load it back in and make a prediction. It was trained with the following features and labels: F1 : FLOAT32 F2 : FLOAT32 F3 : FLOAT32 L1 : FLOAT32 So say I want to …

Read More
Tensorflow: how to save/restore a model?

After you train a model in Tensorflow: How do you save the trained model? How do you later restore this saved model?

Read More
What is logits, softmax and softmax_cross_entropy_with_logits?

I was going through the tensorflow API docs here. In the tensorflow documentation, they used a keyword called logits. What is it? In a lot of methods in the API docs it is written like tf.nn.softmax(logits, name=…

Read More