tensorflow Found more than one graph event per run

Question

I am loading a tensorboard for my ml engine experiment that is running in local mode and got the following warning:

"Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W0825 19:26:12.435613 Reloader event_accumulator.py:311] Found more than one metagraph event per run. Overwriting the metagraph with the newest event."

Originally, I suspected that this was because I had not cleared my --logdir=$OUTPUT_PATH (as other posts suggested -- however, even if I performed rm -rf $OUTPUT_PATH/* I am still getting this error for a local train. Could this error be indicative of a larger issue in my graph?

Answer 1

It looks like you may have already come across this post, but without more information, it's the best information I can provide:

This is a known issue, TensorBoard doesn't like it when you write multiple event files from separate runs in the same directory. It will be fixed if you use a new subdirectory for every run (new hyperparameters = new subdirectory).

You may be inadvertently writing multiple event files in the same directory (e.g. training and eval?).

Also, be sure you are returning an appropriate tf.estimator.EstimatorSpec when in modes.EVAL. From the census sample:

if mode == Modes.EVAL:
  labels_one_hot = tf.one_hot(
      label_indices_vector,
      depth=label_values.shape[0],
      on_value=True,
      off_value=False,
      dtype=tf.bool
  )
  eval_metric_ops = {
      'accuracy': tf.metrics.accuracy(label_indices, predicted_indices),
      'auroc': tf.metrics.auc(labels_one_hot, probabilities)
  }
  return tf.estimator.EstimatorSpec(
      mode, loss=loss, eval_metric_ops=eval_metric_ops)