Concat Avro files using avro-tools

hadoop hdfs avro avro-tools
54l3d picture 54l3d · Jan 18, 2016 · Viewed 8.5k times · Source

Im trying to merge avro files into one big file, the problem is concat command does not accept the wildcard

hadoop jar avro-tools.jar concat /input/part* /output/bigfile.avro

I get:

Exception in thread "main" java.io.FileNotFoundException: File does not exist: /input/part*

I tried to use "" and '' but no chance.

Answer

Clément MATHIEU picture Clément MATHIEU · Jan 20, 2016

I quickly checked Avro's source code (1.7.7) and it seems that concat does not support glob patterns (basically, they call FileSystem.open() on each argument except the last one).

It means that you have to explicitly provide all the filenames as argument. It is cumbersome, but following command should do what you want:

IN=$(hadoop fs -ls /input/part* | awk '{printf "%s ", $NF}')
hadoop jar avro-tools.jar concat ${IN} /output/bigfile.avro

It would be a nice addition to add support of glob pattern to this command.

Related questions

What are the pros and cons of parquet format compared to other formats?

Characteristics of Apache Parquet are : Self-describing Columnar format Language-independent In comparison to Avro, Sequence Files, RC File etc. I want an overview of the formats. I have already read : How Impala Works with Hadoop File Formats , it gives some insights …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to copy file from HDFS to the local file system

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through …

Read More