Limit number of cores used in Keras

python tensorflow keras cpu-cores
Mohamed AL ANI picture Mohamed AL ANI · Sep 26, 2017 · Viewed 18.4k times · Source

I have a shared machine with 64 cores on which I have a big pipeline of Keras functions that I want to run. The thing is that it seems that Keras automatically uses all the cores available and I can't do that.

I use Python and I want to run 67 neural networks in a for loop. I would like to use half of the available cores.

I can't find any way of limiting the number of cores in Keras... Do you have any clue?

Answer

Mohamed AL ANI picture Mohamed AL ANI · Sep 26, 2017

As @Yu-Yang suggested, I used this line before each fit I do : 

from keras import backend as K
K.set_session(K.tf.Session(config=K.tf.ConfigProto(intra_op_‌​parallelism_threads=‌​32, inter_op_parallelism_threads=32)))

Check the CPU usage (htop) : enter image description here

Related questions

Can I run Keras model on gpu?

I'm running a Keras model, with a submission deadline of 36 hours, if I train my model on the cpu it will take approx 50 hours, is there a way to run Keras on gpu? I'm using Tensorflow backend and running it …

Read More
Keras, How to get the output of each layer?

I have trained a binary classification model with CNN, and here is my code model = Sequential() model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1], border_mode='valid', input_shape=input_shape)) model.add(Activation('relu')) model.add(Convolution2…

Read More
Tensorflow 2.0 - AttributeError: module 'tensorflow' has no attribute 'Session'

When I am executing the command sess = tf.Session() in Tensorflow 2.0 environment, I am getting an error message as below: Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'tensorflow' has no attribute …

Read More