save numpy array in append mode

python numpy save
user3820991 picture user3820991 · May 21, 2015 · Viewed 42.7k times · Source

Is it possible to save a numpy array appending it to an already existing npy-file --- something like np.save(filename,arr,mode='a')?

I have several functions that have to iterate over the rows of a large array. I cannot create the array at once because of memory constrains. To avoid to create the rows over and over again, I wanted to create each row once and save it to file appending it to the previous row in the file. Later I could load the npy-file in mmap_mode, accessing the slices when needed.

Answer

rth picture rth · May 21, 2015

The build-in .npy file format is perfectly fine for working with small datasets, without relying on external modules other then numpy.

However, when you start having large amounts of data, the use of a file format, such as HDF5, designed to handle such datasets, is to be preferred [1].

For instance, below is a solution to save numpy arrays in HDF5 with PyTables,

Step 1: Create an extendable EArray storage 

import tables
import numpy as np

filename = 'outarray.h5'
ROW_SIZE = 100
NUM_COLUMNS = 200

f = tables.open_file(filename, mode='w')
atom = tables.Float64Atom()

array_c = f.create_earray(f.root, 'data', atom, (0, ROW_SIZE))

for idx in range(NUM_COLUMNS):
    x = np.random.rand(1, ROW_SIZE)
    array_c.append(x)
f.close()

Step 2: Append rows to an existing dataset (if needed)

f = tables.open_file(filename, mode='a')
f.root.data.append(x)

Step 3: Read back a subset of the data

f = tables.open_file(filename, mode='r')
print(f.root.data[1:10,2:20]) # e.g. read from disk only this part of the dataset

Related questions

Saving numpy array to txt file row wise

I have an numpy array of form a = [1,2,3] which I want to save to a .txt file such that the file looks like: 1 2 3 If I use numpy.savetxt then I get a file like: 1 2 3 There should be a easy solution …

Read More
Setting the fmt option in numpy.savetxt

I am looking at the numpy.savetxt, and am stuck at the fmt option. I tried looking at here and also the reference in the link below all the letters that can be used for the fmt option sort give …

Read More
TypeError: Mismatch between array dtype ('object') and format specifier ('%.18e')

I have the following array: X = np.array([image_array_to_vector1,image_array_to_vector2,image_array_to_vector3,image_array_to_vector4]) A print out of X looks as follows: [array([167, 167, 169, ..., 1, 1, 1], dtype=uint8) array([42, 43, 43, ..., 41, 36, 34], dtype=uint8) array([0, 0, 0, ..., 0, 0, 0], dtype=…

Read More