NumPy or Pandas: Keeping array type as integer while having a NaN value

python numpy int pandas type-conversion
ely picture ely · Jul 18, 2012 · Viewed 66.3k times · Source

Is there a preferred way to keep the data type of a numpy array fixed as int (or int64 or whatever), while still having an element inside listed as numpy.NaN?

In particular, I am converting an in-house data structure to a Pandas DataFrame. In our structure, we have integer-type columns that still have NaN's (but the dtype of the column is int). It seems to recast everything as a float if we make this a DataFrame, but we'd really like to be int.

Thoughts?

Things tried:

I tried using the from_records() function under pandas.DataFrame, with coerce_float=False and this did not help. I also tried using NumPy masked arrays, with NaN fill_value, which also did not work. All of these caused the column data type to become a float.

Answer

Wes McKinney picture Wes McKinney · Jul 18, 2012

NaN can't be stored in an integer array. This is a known limitation of pandas at the moment; I have been waiting for progress to be made with NA values in NumPy (similar to NAs in R), but it will be at least 6 months to a year before NumPy gets these features, it seems:

http://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na

(This feature has been added beginning with version 0.24 of pandas, but note it requires the use of extension dtype Int64 (capitalized), rather than the default dtype int64 (lower case): https://pandas.pydata.org/pandas-docs/version/0.24/whatsnew/v0.24.0.html#optional-integer-na-support )

Related questions

Convert pandas dataframe to NumPy array

I am interested in knowing how to convert a pandas dataframe into a NumPy array. dataframe: import numpy as np import pandas as pd index = [1, 2, 3, 4, 5, 6, 7] a = [np.nan, np.nan, np.nan, 0.1, 0.1, 0.1, 0.1] b = [0.2, np.nan, 0.2, 0.2, 0.2, np.nan, np.nan] c = […

Read More
How do I create an empty array/matrix in NumPy?

I can't figure out how to use an array or matrix in the way that I would normally use a list. I want to create an empty array (or matrix) and then add one column (or row) to it at …

Read More
ValueError: setting an array element with a sequence

This Python code: import numpy as p def firstfunction(): UnFilteredDuringExSummaryOfMeansArray = [] MeanOutputHeader=['TestID','ConditionName','FilterType','RRMean','HRMean', 'dZdtMaxVoltageMean','BZMean','ZXMean','LVETMean','Z0Mean', 'StrokeVolumeMean','CardiacOutputMean','VelocityIndexMean'] dataMatrix = BeatByBeatMatrixOfMatrices[column] roughTrimmedMatrix = p.array(dataMatrix[1:,1:17]) trimmedMatrix = p.array(roughTrimmedMatrix,dtype=p.float64) #ERROR …

Read More