STORE output to a single CSV?

apache-pig
JasonA picture JasonA · Mar 28, 2012 · Viewed 14.6k times · Source

Currently, when I STORE into HDFS, it creates many part files.

Is there any way to store out to a single CSV file?

Answer

Chris White picture Chris White · Mar 29, 2012

You can do this in a few ways:

  • To set the number of reducers for all Pig opeations, you can use the default_parallel property - but this means every single step will use a single reducer, decreasing throughput:

    set default_parallel 1;

  • Prior to calling STORE, if one of the operations execute is (COGROUP, CROSS, DISTINCT, GROUP, JOIN (inner), JOIN (outer), and ORDER BY), then you can use the PARALLEL 1 keyword to denote the use of a single reducer to complete that command:

    GROUP a BY grp PARALLEL 1;

See Pig Cookbook - Parallel Features for more information

Related questions

Difference between Pig and Hive? Why have both?

My background - 4 weeks old in the Hadoop world. Dabbled a bit in Hive, Pig and Hadoop using Cloudera's Hadoop VM. Have read Google's paper on Map-Reduce and GFS (PDF link). I understand that- Pig's language Pig Latin is a …

Read More
When to use Hadoop, HBase, Hive and Pig?

What are the benefits of using either Hadoop or HBase or Hive ? From my understanding, HBase avoids using map-reduce and has a column oriented storage on top of HDFS. Hive is a sql-like interface for Hadoop and HBase. I would …

Read More
PIG how to count a number of rows in alias

I did something like this to count the number of rows in an alias in PIG: logs = LOAD 'log' logs_w_one = foreach logs generate 1 as one; logs_group = group logs_w_one all; logs_count = foreach logs_group generate …

Read More