Repair HBase table (unassigned region in transition)

hadoop hbase cloudera
Mario picture Mario · Jun 13, 2012 · Viewed 27.8k times · Source

I'm a bit stuck repairing a faulty table (on Hbase 0.92.1-cdh4.0.0, Hadoop 2.0.0-cdh4.0.0)

There is a region in transition that doesn't finish:

Region    State
bf2025f4bc154914b5942af4e72ea063 counter_traces,1329773878.35_766a0b4df75e4381a686fbc07db9e333,1339425291230.bf2025f4bc154914b5942af4e72ea063. state=OFFLINE, ts=Tue Jun 12 11:43:53 CEST 2012 (0s ago), server=null

When I run sudo -u hbase hbase hbck -repair, I get this:

Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
ERROR: Region { meta => counter_traces,1329773878.35_766a0b4df75e4381a686fbc07db9e333,1339425291230.bf2025f4bc154914b5942af4e72ea063., hdfs => hdfs://hbase001:8020/hbase/counter_traces/bf2025f4bc154914b5942af4e72ea063, deployed =>  } not deployed on any region server.
Trying to fix unassigned region...
12/06/12 11:44:40 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {NAME => 'counter_traces,1329773878.35_766a0b4df75e4381a686fbc07db9e333,1339425291230.bf2025f4bc154914b5942af4e72ea063.', STARTKEY => '1329773878.35_766a0b4df75e4381a686fbc07db9e333', ENDKEY => '1329793347.58_163865765c0a11e184ab003048f0e77e', ENCODED => bf2025f4bc154914b5942af4e72ea063,}

and it just loops.

If I don't do the -repair, I get this:

Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
ERROR: Region { meta => counter_traces,1329773878.35_766a0b4df75e4381a686fbc07db9e333,1339425291230.bf2025f4bc154914b5942af4e72ea063., hdfs => hdfs://hbase001:8020/hbase/counter_traces/bf2025f4bc154914b5942af4e72ea063, deployed =>  } not deployed on any region server.
ERROR: Region { meta => counter_traces,1329816776.95_95b5561f3c1e496598421359a19ac665,1339425297099.ee1fd808d954c9adc9ed95ec2f29d119., hdfs => hdfs://hbase001:8020/hbase/counter_traces/ee1fd808d954c9adc9ed95ec2f29d119, deployed =>  } not deployed on any region server.
12/06/12 11:45:59 DEBUG util.HBaseFsck: There are 134 region info entries
ERROR: There is a hole in the region chain between 1329773878.35_766a0b4df75e4381a686fbc07db9e333 and 1329793347.58_163865765c0a11e184ab003048f0e77e.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: There is a hole in the region chain between 1329816776.95_95b5561f3c1e496598421359a19ac665 and 1329847231.75_b3c50776778b43e088dd7ed865e11331.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table counter_traces

I've run -repair a couple of times before, and it helped. But this time, not anymore.

Ok, so this states to do a manual intervention to fix this. Could someone point me in the right direction on how to do this? A recipe, webpage, example, anything will help.

Thanks, Mario

Answer

sulabhc picture sulabhc · Jul 3, 2012

Mario,

So one of the reasons why a region gets stuck in transition is because, when it is being moved across regionservers, it is unassigned from the source regionserver but is never assigned to another regionserver. One fix that always works for me is by forcibly ASSIGNing it from the hbase shell by :- 

assign regionName

Related questions

Is there a way to add nodes to a running Hadoop cluster?

I have been playing with Cloudera and I define the number of clusters before I start my job then use the cloudera manager to make sure everything is running. I’m working on a new project that instead of using …

Read More
How to run HBase shell against a remote cluster

I'm running HBase in pseudo-distributed mode on my workstation. We also have HBase running on a cluster. Using the HBase shell, I'd like to access the HBase instance that's running on the cluster from my workstation. I would like to …

Read More
Import data from HDFS to HBase (cdh3u2)

I have Installed hadoop and hbase cdh3u2. In hadoop i have a file at the path /home/file.txt. it has the data like one,1 two,2 three,3 I want to import this file into hbase. in that, the first …

Read More