Pig Batch mode: how to set logging level to hide INFO log messages?

apache-pig
Polymerase picture Polymerase · May 18, 2013 · Viewed 10.9k times · Source

Using Apache Pig version 0.10.1.21 (rexported). When I execute a pig script, there are a lots of INFO logging lines which looks like that: 

2013-05-18 14:30:12,810 [Thread-28] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local_0005_r_000000_0' done.
2013-05-18 14:30:18,064 [main] WARN  org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0005
2013-05-18 14:30:18,094 [Thread-31] WARN  org.apache.hadoop.mapred.JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
2013-05-18 14:30:18,114 [Thread-31] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-05-18 14:30:18,254 [Thread-32] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3fcb2dd1
2013-05-18 14:30:18,265 [Thread-32] INFO  org.apache.hadoop.mapred.MapTask - io.sort.mb = 10

Is there a SET command within the pig script or a command line flag to allow the logging level? Basically I would like to hide the [Thread-xx] INFO messages. Only showing WARNING and ERROR. I have tried the command line debug flag. Unfortunately, the INFO messages still show up:

pig -x local -d WARN MyScript.pig

Hope there is a solution. Thanks in advance for any help.

SOLVED: Answer by Loran Bendig, set the log4j.properties. Summarized here for convenience

Step1: copy the log4j config file to the folder where my pig scripts are located.

cp /etc/pig/conf.dist/log4j.properties log4j_WARN

Step2: Edit log4j_WARN file and make sure these two lines are present

log4j.logger.org.apache.pig=WARN, A
log4j.logger.org.apache.hadoop = WARN, A

Step3: Run pig script and instruct it to use the custom log4j

pig -x local -4 log4j_WARN MyScript.pig

Answer

Vahid Mirjalili picture Vahid Mirjalili · Mar 1, 2016

Another setting could be also like this:

Create a file named nolog.conf, with the following content

log4j.rootLogger=fatal

and then run pig as follows

pig -x local -4 nolog.conf

Related questions

Difference between Pig and Hive? Why have both?

My background - 4 weeks old in the Hadoop world. Dabbled a bit in Hive, Pig and Hadoop using Cloudera's Hadoop VM. Have read Google's paper on Map-Reduce and GFS (PDF link). I understand that- Pig's language Pig Latin is a …

Read More
When to use Hadoop, HBase, Hive and Pig?

What are the benefits of using either Hadoop or HBase or Hive ? From my understanding, HBase avoids using map-reduce and has a column oriented storage on top of HDFS. Hive is a sql-like interface for Hadoop and HBase. I would …

Read More
PIG how to count a number of rows in alias

I did something like this to count the number of rows in an alias in PIG: logs = LOAD 'log' logs_w_one = foreach logs generate 1 as one; logs_group = group logs_w_one all; logs_count = foreach logs_group generate …

Read More