Pandas: Sampling a DataFrame

python partitioning pandas
Blender picture Blender · Aug 30, 2012 · Viewed 79k times · Source

I'm trying to read a fairly large CSV file with Pandas and split it up into two random chunks, one of which being 10% of the data and the other being 90%.

Here's my current attempt:

rows = data.index
row_count = len(rows)
random.shuffle(list(rows))

data.reindex(rows)

training_data = data[row_count // 10:]
testing_data = data[:row_count // 10]

For some reason, sklearn throws this error when I try to use one of these resulting DataFrame objects inside of a SVM classifier:

IndexError: each subindex must be either a slice, an integer, Ellipsis, or newaxis

I think I'm doing it wrong. Is there a better way to do this?

Answer

Wouter Overmeire picture Wouter Overmeire · Aug 30, 2012

What version of pandas are you using? For me your code works fine (i`m on git master).

Another approach could be:

In [117]: import pandas

In [118]: import random

In [119]: df = pandas.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))

In [120]: rows = random.sample(df.index, 10)

In [121]: df_10 = df.ix[rows]

In [122]: df_90 = df.drop(rows)

Newer version (from 0.16.1 on) supports this directly: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html

Related questions

pyspark partitioning data using partitionby

I understand that partitionBy function partitions my data. If I use rdd.partitionBy(100) it will partition my data by key into 100 parts. i.e. data associated with similar keys will be grouped together Is my understanding correct? Is it advisable …

Read More
How do I list all files of a directory?

How can I list all files of a directory in Python and add them to a list?

Read More
Iterating over dictionaries using 'for' loops

I am a bit puzzled by the following code: d = {'x': 1, 'y': 2, 'z': 3} for key in d: print key, 'corresponds to', d[key] What I don't understand is the key portion. How does Python recognize that it needs only to …

Read More