HDFS Home Directory

hadoop cluster-computing hdfs user-permissions
Tapan Avasthi picture Tapan Avasthi · Apr 9, 2012 · Viewed 26k times · Source

I have setup a single node multi-user hadoop cluster. In my cluster, there is an admin user that is responsible for running the cluster (superuser). All other users are allocated a hdfs directory like /home/xyz where xyz is a username.

In unix, we can change the default home directory for a user in /etc/passwd. And by default, landing directory for a user is the home directory.

How do I do it in hadoop for hdfs file system. I want for example, if user types: $hadoop dfs -ls on the unix prompt. It shall list the contents of the home directory allocated by me.

Further, hdfs directories are created by the superuser who runs the cluster(hadoop superuser and not unix root) and then transfers the ownership to a particular user.

Answer

Chris White picture Chris White · Apr 9, 2012

I'm not sure this is something that can be configured - the source for DistributedFileSystem(line 150) has a call for getHomeDirectory that seems to be hard-coded:

@Override
public Path getHomeDirectory() {
  return makeQualified(new Path("/user/" + dfs.ugi.getShortUserName()));
}

You do have two possible choices if you want to be able to change this:

  • Submit a ticket to hadoop asking for a new feature - See this link
  • Amend the source yourself and re-build + re-distribute the hadoop-core jar across your cluster (simple in your single node pseudo cluster)

Related questions

Is there a way to add nodes to a running Hadoop cluster?

I have been playing with Cloudera and I define the number of clusters before I start my job then use the cloudera manager to make sure everything is running. I’m working on a new project that instead of using …

Read More
Set hadoop system user for client embedded in Java webapp

I would like to submit MapReduce jobs from a java web application to a remote Hadoop cluster but am unable to specify which user the job should be submitted for. I would like to configure and use a system user …

Read More
How to submit a spark job on a remote master node in yarn client mode?

I need to submit spark apps/jobs onto a remote spark cluster. I have currently spark on my machine and the IP address of the master node as yarn-client. Btw my machine is not in the cluster. I submit my …

Read More