How to "reset" tensorboard data after killing tensorflow instance

mathetes picture mathetes · Dec 24, 2015 · Viewed 35.7k times · Source

I'm testing different hyperparameters for a cnn model I built, but I'm having a small annoyance when viewing the summaries in Tensorboard. The problem seems to be that the data is just "added" in consecutive runs, so the functions result in a weird superposition unless I see the information as "relative" instead of "by step". See here:

X Type: Step

X Type: Relative

I've tried killing tensorboard's process and erasing the log files, but it seems it is not enough.

So the question is, how do I reset this information?

Thanks!!

Answer

dandelion picture dandelion · May 24, 2016

Note: The solution you've posted (erase TensorBoard's log files and kill the process) will work, but it isn't preferred, because it destroys historical information about your training.

Instead, you can have each new training job write to a new subdirectory (of your top-level log directory). Then, TensorBoard will consider each job a new "run" and will create a nice comparison view so you can see how the training differed between iterations of your model.

In the following an example from https://www.tensorflow.org/tensorboard/get_started:

model = create_model()
...
model.compile(...)

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.fit(..., callbacks=[tensorboard_callback])