Realtime data stream to python from CSV file

Question

I have a CSV file that has data from a random sensor recorded over a few minutes time. Now i want to stream that data from CSV file to my pyhton code as if it were receiving data from the sensor itself directly. (the code is for taking readings from two different sensors/csv files and averaging them) Someone suggested to use Apache Spark to stream data, but i feel thats a bit too complex for me. Might there be a simpler solution?

Answer 1

You could also use pandas read_csv() function to read the big csv file in small chunks, the basic code is written below:

import pandas as pd
chunksize = 100
for chunk in pd.read_csv('myfile.csv', chunksize=chunksize):
    print(chunk)

This link explains how this works: http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking