What's the difference between Flume and Sqoop?

hadoop sqoop flume
Cacheing picture Cacheing · Oct 22, 2013 · Viewed 31.7k times · Source

Both Flume and Sqoop are meant for data movement, then what is the difference between them? Under what condition should I use Flume or Sqoop?

Answer

techuser soma picture techuser soma · Oct 22, 2013

From http://flume.apache.org/

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

Flume helps to collect data from a variety of sources, like logs, jms, Directory etc.
Multiple flume agents can be configured to collect high volume of data.
It scales horizontally.

From http://sqoop.apache.org/

Apache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.

Sqoop helps to move data between hadoop and other databases and it can transfer data in parallel for performance.

Related questions

How to copy data from one HDFS to another HDFS?

I have two HDFS setup and want to copy (not migrate or move) some tables from HDFS1 to HDFS2. How to copy data from one HDFS to another HDFS? Is it possible via Sqoop or other command line?

Read More
Using Sqoop to import data from MySQL to Hive

I am using Sqoop (version 1.4.4) to import data from MySQL to Hive. The data will be a subset of one of tables, i.e. few columns from a table. Is it necessary to create table in Hive before hand. Or …

Read More
Multiple table join in hive

I have migrated Teradata tables' data into hive . Now I have to build summary tables on top of imported data. Summary table needs to be built from five source tables If I go with joins I'll need to join five …

Read More