Convert Pandas column containing NaNs to dtype `int`

python pandas na
Zhubarb picture Zhubarb · Jan 22, 2014 · Viewed 197k times · Source

I read data from a .csv file to a Pandas dataframe as below. For one of the columns, namely id, I want to specify the column type as int. The problem is the id series has missing/empty values.

When I try to cast the id column to integer while reading the .csv, I get:

df= pd.read_csv("data.csv", dtype={'id': int}) 
error: Integer column has NA values

Alternatively, I tried to convert the column type after reading as below, but this time I get:

df= pd.read_csv("data.csv") 
df[['id']] = df[['id']].astype(int)
error: Cannot convert NA to integer

How can I tackle this?

Answer

Andy Hayden picture Andy Hayden · Jan 22, 2014

The lack of NaN rep in integer columns is a pandas "gotcha".

The usual workaround is to simply use floats.

Related questions

specifying "skip NA" when calculating mean of the column in a data frame created by Pandas

I am learning Pandas package by replicating the outing from some of the R vignettes. Now I am using the dplyr package from R as an example: http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html R script planes &…

Read More
Selecting pandas cells with None value

I have a column of a pandas dataframe that I got from a database query with blank cells. The blank cells become "None" and I want to check if each of the rows is None: In [325]: yes_records_sample['name'] …

Read More
Fill empty cells in column with value of other columns

I have a HC list in which every entry should have an ID, but some entries do not have an ID. I would like to fill those empty cells by combining the the first name column and the last name …

Read More