hdf5 file to pandas dataframe

python pandas hdf5
Graham Slick picture Graham Slick · Nov 7, 2016 · Viewed 24.1k times · Source

I downloaded a dataset which is stored in .h5 files. I need to keep only certain columns and to be able to manipulate the data in it.

To do this, I tried to load it in a pandas dataframe. I've tried to use:

pd.read_hdf(path)

But I get: No dataset in HDF5 file.

I've found answers on SO (read HDF5 file to pandas DataFrame with conditions) but I don't need conditions, and the answer adds conditions about how the file was written but I'm not the creator of the file so I can't do anything about that.

I've also tried using h5py: 

df = h5py.File(path)

But this is not easily manipulable and I can't seem to get the columns out of it (only the names of the columns using df.keys()) Any idea on how to do this ?

Answer

drj picture drj · Jan 11, 2017

Pandas HDF support needs the HDF file to be formated very specifically. You can see https://stackoverflow.com/a/33644128/4128030 for more info.

Related questions

"Large data" work flows using pandas

I have tried to puzzle out an answer to this question for many months while learning pandas. I use SAS for my day-to-day work and it is great for it's out-of-core support. However, SAS is horrible as a piece of …

Read More
ImportError HDFStore requires PyTables No module named tables

import pandas as pd dfs = pd.HDFStore('xxxxx.h5') throws this error: "ImportError: HDFStore requires PyTables, "No module named tables" problem importing" I tried to install PyTables, which Requires Cython. I have Cython 0.21 installed, but it is throwing an …

Read More
Pandas ParserError EOF character when reading multiple csv files to HDF5

Using Python3, Pandas 0.12 I'm trying to write multiple csv files (total size is 7.9 GB) to a HDF5 store to process later onwards. The csv files contain around a million of rows each, 15 columns and data types are mostly strings, but …

Read More