What is the fastest way to load data into Cassandra column-family

csv cassandra bulk-load loaddata
Pedro Cunha picture Pedro Cunha · Oct 28, 2015 · Viewed 9.2k times · Source

I created a Cassandra column-family and I need to load data from a CSV file for this column family. The csv file has a 15 Gb volume.

I am using the CQL 'COPY FROM' command but this takes a long time to make loading the data. What is the best/simplest way to load large amounts of data to Cassandra from csv files?

Answer

BrianC picture BrianC · Oct 28, 2015

The CQLSH built-in copy to/from CSV files is pretty simple and is intended for small to moderate sized data sets. You didn't mention which Cassandra version you're using, but there were a lot of performance improvements made in 2.1.5 (CASSANDRA-8225).

An alternative tool that has had good results for larger data is cassandra-loader. You could try that with a subset of your file (like 1000 rows) to confirm it works, then try with your whole file to see the performance.

Related questions

COPY cassandra table from csv file

I'm setting up a demo landscape for Cassandra, Apache Spark and Flume on my Mac (Mac OS X Yosemite with Oracle jdk1.7.0_55). The landscape shall work as a proof of concept for a new analytics platform and therefore I need …

Read More
Export large amount of data from Cassandra to CSV

I'm using Cassandra 2.0.9 for store quite big amounts of data, let's say 100Gb, in one column family. I would like to export this data to CSV in fast way. I tried: sstable2json - it produces quite big json files …

Read More
How to concatenate text from multiple rows into a single text string in SQL server?

Consider a database table holding names, with three rows: Peter Paul Mary Is there an easy way to turn this into a single string of Peter, Paul, Mary?

Read More