which is faster for load: pickle or hdf5 in python

python pandas numpy dataframe hdf5
denvar picture denvar · Jun 20, 2016 · Viewed 36.4k times · Source

Given a 1.5 Gb list of pandas dataframes, which format is fastest for loading compressed data: pickle (via cPickle), hdf5, or something else in Python?

  • I only care about fastest speed to load the data into memory
  • I don't care about dumping the data, it's slow but I only do this once.
  • I don't care about file size on disk

Answer

MaxU picture MaxU · Jun 20, 2016

I would consider only two storage formats: HDF5 (PyTables) and Feather

Here are results of my read and write comparison for the DF (shape: 4000000 x 6, size in memory 183.1 MB, size of uncompressed CSV - 492 MB).

Comparison for the following storage formats: (CSV, CSV.gzip, Pickle, HDF5 [various compression]):

                  read_s  write_s  size_ratio_to_CSV
storage
CSV               17.900    69.00              1.000
CSV.gzip          18.900   186.00              0.047
Pickle             0.173     1.77              0.374
HDF_fixed          0.196     2.03              0.435
HDF_tab            0.230     2.60              0.437
HDF_tab_zlib_c5    0.845     5.44              0.035
HDF_tab_zlib_c9    0.860     5.95              0.035
HDF_tab_bzip2_c5   2.500    36.50              0.011
HDF_tab_bzip2_c9   2.500    36.50              0.011

But it might be different for you, because all my data was of the datetime dtype, so it's always better to make such a comparison with your real data or at least with the similar data...

Related questions

Convert pandas dataframe to NumPy array

I am interested in knowing how to convert a pandas dataframe into a NumPy array. dataframe: import numpy as np import pandas as pd index = [1, 2, 3, 4, 5, 6, 7] a = [np.nan, np.nan, np.nan, 0.1, 0.1, 0.1, 0.1] b = [0.2, np.nan, 0.2, 0.2, 0.2, np.nan, np.nan] c = […

Read More
How to take column-slices of dataframe in pandas

I load some machine learning data from a CSV file. The first 2 columns are observations and the remaining columns are features. Currently, I do the following: data = pandas.read_csv('mydata.csv') which gives something like: data = pandas.DataFrame(np.…

Read More
Pandas conditional creation of a series/dataframe column

I have a dataframe along the lines of the below: Type Set 1 A Z 2 B Z 3 B X 4 C Y I want to add another column to the dataframe (or generate a series) of the same length as the dataframe (…

Read More