Convert a simple one line string to RDD in Spark

python apache-spark pyspark distributed-computing rdd
poiuytrez picture poiuytrez · Oct 2, 2014 · Viewed 44.7k times · Source

I have a simple line:

line = "Hello, world"

I would like to convert it to an RDD with only one element. I have tried 

sc.parallelize(line)

But it get:

sc.parallelize(line).collect()
['H', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']

Any ideas?

Answer

michaeltang picture michaeltang · Oct 2, 2014

try using List as parameter: 

sc.parallelize(List(line)).collect()

it returns 

res1: Array[String] = Array(hello,world)

Related questions

How to change dataframe column names in pyspark?

I come from pandas background and am used to reading data from CSV files into a dataframe and then simply changing the column names to something useful using the simple command: df.columns = new_column_name_list However, the same …

Read More
Load CSV file with Spark

I'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing : sc.textFile('file.csv') .map(lambda line: (line.split(',')[0], line.split(',')[1])) .collect() I would expect …

Read More
How do I add a new column to a Spark DataFrame (using PySpark)?

I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column. I've tried the following without any success: type(randomed_hours) # => list # Create in Python and transform to RDD new_col = pd.DataFrame(randomed_hours, …

Read More