Importing data and variable names from a text file in Python

Question

I have a text file containing simulation data (60 columns, 100k rows):

a  b   c  
1  11 111
2  22 222
3  33 333
4  44 444

... where in the first row are variable names, and beneath (in columns) is the corresponding data (float type).

I need to use all these variables with their data in Python for further calculations. For example, when I insert:

print(b)

I need to receive the values from the second column.

I know how to import data:

data=np.genfromtxt("1.txt", unpack=True, skiprows = 1)

Assign variables "manually":

a,b,c=np.genfromtxt("1.txt", unpack=True, skiprows = 1)

But I'm having trouble with getting variable names:

reader = csv.reader(open("1.txt", "rt"))
for row in reader: 
   list.append(row)
variables=(list[0])  

How can I change this code to get all variable names from the first row and assign them to the imported arrays ?

Answer 1

Instead of trying to assign names, you might think about using an associative array, which is known in Python as a dict, to store your variables and their values. The code could then look something like this (borrowing liberally from the csv docs):

import csv
with open('1.txt', 'rt') as f:
  reader = csv.reader(f, delimiter=' ', skipinitialspace=True)

  lineData = list()

  cols = next(reader)
  print(cols)

  for col in cols:
    # Create a list in lineData for each column of data.
    lineData.append(list())


  for line in reader:
    for i in xrange(0, len(lineData)):
      # Copy the data from the line into the correct columns.
      lineData[i].append(line[i])

  data = dict()

  for i in xrange(0, len(cols)):
    # Create each key in the dict with the data in its column.
    data[cols[i]] = lineData[i]

print(data)

data then contains each of your variables, which can be accessed via data['varname'].

So, for example, you could do data['a'] to get the list ['1', '2', '3', '4'] given the input provided in your question.

I think trying to create names based on data in your document might be a rather awkward way to do this, compared to the dict-based method shown above. If you really want to do that, though, you might look into reflection in Python (a subject I don't really know anything about).