Change IP address of a Hadoop HDFS data node server and avoid Block pool errors

Question

I'm using the cloudera distribution of Hadoop and recently had to change the IP addresses of a few nodes in the cluster. After the change, on one of the nodes (Old IP:10.88.76.223, New IP: 10.88.69.31) the following error comes up when I try to start the data node service.

Initialization failed for block pool Block pool BP-77624948-10.88.65.174-13492342342 (storage id DS-820323624-10.88.76.223-50010-142302323234) service to hadoop-name-node-01/10.88.65.174:6666
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(10.88.69.31, storageID=DS-820323624-10.88.76.223-50010-142302323234, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster25;nsid=1486084428;c=0)
    at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:656)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3593)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:899)
    at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:91), I was unable to start the datanode service due to the following error:

Has anyone had success with changing the IP address of a hadoop data node and join it back to the cluster without data loss?

Answer 1

CHANGE HOST IP IN CLOUDERA MANAGER

1. Change Host IP on all node

   sudo nano /etc/hosts
   

2. Edit the ip cloudera config.ini on all node if the master node ip changes

   sudo nano /etc/cloudera-scm-agent/config.ini
   

3. Change IP in PostgreSQL Database

4. For the password Open PostgreSQL password

   cat /etc/cloudera-scm-server/db.properties
   

5. Find the password lines

   Example. com.cloudera.cmf.db.password=gUHHwvJdoE
   

6. Open PostgreSQL

   psql -h localhost -p 7432 -U scm
   

7. Select table in PostgreSQL

   select name,host_id,ip_address from hosts;
   

8. Update table IP

   update hosts set ip_address = 'xxx.xxx.xxx.xxx' where host_id=x;
   

9. Exit the tool

   \q
   

10. Restart the service on all node

    service cloudera-scm-agent restart
    

11. Restart the service on master node

    service cloudera-scm-server restart