Obtaining total number of records from .tfrecords file in Tensorflow

tensorflow tfrecord
HuckleberryFinn picture HuckleberryFinn · Nov 7, 2016 · Viewed 18k times · Source

Is it possible for obtain the total number of records from a .tfrecords file ? Related to this, how does one generally keep track of the number of epochs that have elapsed while training models? While it is possible for us to specify the batch_size and num_of_epochs, I am not sure if it is straightforward to obtain values such as current epoch, number of batches per epoch etc - just so that I could have more control of how the training is progressing. Currently, I'm just using a dirty hack to compute this as I know before hand how many records there are in my .tfrecords file and the size of my minibatches. Appreciate any help..

Answer

drpng picture drpng · Nov 7, 2016

To count the number of records, you should be able to use tf.python_io.tf_record_iterator.

c = 0
for fn in tf_records_filenames:
  for record in tf.python_io.tf_record_iterator(fn):
     c += 1

To just keep track of the model training, tensorboard comes in handy.

Related questions

How do I convert a directory of jpeg images to TFRecords file in tensorflow?

I have training data that is a directory of jpeg images and a corresponding text file containing the file name and the associated category label. I am trying to convert this training data into a tfrecords file as described in …

Read More
How to inspect a Tensorflow .tfrecord file?

I have a .tfrecord but I don't know how it is structured. How can I inspect the schema to understand what the .tfrecord file contains? All Stackoverflow answers or documentation seem to assume I know the structure of the file. …

Read More
TensorFlow - Read all examples from a TFRecords at once?

How do you read all examples from a TFRecords at once? I've been using tf.parse_single_example to read out individual examples using code similar to that given in the method read_and_decode in the example of the …

Read More