Scala: Replacing double quotes with single quotes

scala dataframe spark-dataframe double-quotes single-quotes
SFatima picture SFatima · Jan 3, 2017 · Viewed 11.1k times · Source

How do you replace single quotes with double quotes in Scala? I have a data file that has some records with "abc" (double quotes). I need to replace these quotes with single quotes and convert it to a data frame. 

val customSchema_1 =        
  StructType(Array(
  StructField("ID", StringType, true),
  StructField("KEY", StringType, true),
  StructField("CODE", StringType, true))

val df_1 = sqlContext.read
  .format("com.databricks.spark.csv")
  .option("delimiter", "¦")
  .schema(customSchema_1)
  .load("example")

Answer

Alex Fruzenshtein picture Alex Fruzenshtein · Jan 3, 2017

Read line by line your file and apply following example to each of them:

val text: String = """Here is a lot of text and "quotes" so you may think that everything is ok until you see something "special" or "weird"
"""

text.replaceAll("\"", "'")

This will give you a new String value with quotes instead of double quotes.

Related questions

Renaming column names of a DataFrame in Spark Scala

I am trying to convert all the headers / column names of a DataFrame in Spark-Scala. as of now I come up with following code which only replaces a single column name. for( i <- 0 to origCols.length - 1) { df.…

Read More
How to define partitioning of DataFrame?

I've started using Spark SQL and DataFrames in Spark 1.4.0. I'm wanting to define a custom partitioner on DataFrames, in Scala, but not seeing how to do this. One of the data tables I'm working with contains a list of transactions, …

Read More
Joining Spark dataframes on the key

I have constructed two dataframes. How can we join multiple Spark dataframes ? For Example : PersonDf, ProfileDf with a common column as personId as (key). Now how can we have one Dataframe combining PersonDf and ProfileDf?

Read More