How to make shark/spark clear the cache?

hadoop hive apache-spark shark-sql
venkat picture venkat · Dec 11, 2013 · Viewed 45.4k times · Source

when i run my shark queries, the memory gets hoarded in the main memory This is my top command result.

Mem: 74237344k total, 70080492k used, 4156852k free, 399544k buffers Swap: 4194288k total, 480k used, 4193808k free, 65965904k cached

this doesn't change even if i kill/stop shark,spark, hadoop processes. Right now, the only way to clear the cache is to reboot the machine.

has anyone faced this issue before? is it some configuration problem or a known issue in spark/shark?

Answer

Henrique Florencio picture Henrique Florencio · May 19, 2017

To remove all cached data:

sqlContext.clearCache()

Source: https://spark.apache.org/docs/2.0.1/api/java/org/apache/spark/sql/SQLContext.html

If you want to remove an specific Dataframe from cache:

df.unpersist()

Related questions

Save Spark dataframe as dynamic partitioned table in Hive

I have a sample application working to read from csv files into a dataframe. The dataframe can be stored to a Hive table in parquet format using the method df.saveAsTable(tablename,mode). The above code works fine, but I …

Read More
SparkSQL vs Hive on Spark - Difference and pros and cons?

SparkSQL CLI internally uses HiveQL and in case Hive on spark(HIVE-7292) , hive uses spark as backend engine. Can somebody throw some more light, how exactly these two scenarios are different and pros and cons of both approaches?

Read More
Spark SQL "Limit"

Env : spark 1.6 using Hadoop. Hortonworks Data Platform 2.5 I have a table with 10 billion records and I would like to get 300 million records and move them to a temporary table. sqlContext.sql("select ....from my_table limit 300000000").repartition(50) .write.saveAsTable("temporary_…

Read More