How to import a custom module in a MapReduce job?

Question

I have a MapReduce job defined in main.py, which imports the lib module from lib.py. I use Hadoop Streaming to submit this job to the Hadoop cluster as follows:

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar -files lib.py,main.py 
    -mapper "./main.py map" -reducer "./main.py reduce" 
    -input input -output output

In my understanding, this should put both main.py and lib.py into the distributed cache folder on each computing machine and thus make module lib available to main. But it doesn't happen: from the log I see that files are really copied to the same directory, but main can't import lib, throwing ImportError.

Why does this happen and how can I fix it?

UPD. Adding the current directory to the path didn't work:

import sys    
sys.path.append(os.path.realpath(__file__))
import lib
# ImportError

though, loading the module manually did the trick:

import imp
lib = imp.load_source('lib', 'lib.py')

But that's not what I want. So why does the Python interpreter see other .py files in the same directory, but can't import them? Note that I have already tried adding an empty __init__.py file to the same directory without effect.

Answer 1

I posted the question to Hadoop user list and finally found the answer. It turns out that Hadoop doesn't really copy files to the location where the command runs, but instead creates symlinks for them. Python, in its turn, can't work with symlinks and thus doesn't recognize lib.py as Python module.

Simple workaround here is to put both main.py and lib.py into the same directory, so that symlink to the directory is placed into MR job working directory, while both files are physically in the same directory. So I did the following:

1. Put main.py and lib.py into app directory.

2. In main.py I used lib.py directly, that is, import string is just

   import lib

3. Uploaded app directory with -files option.

So, final command looks like this:

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar -files app 
       -mapper "app/main.py map" -reducer "app/main.py reduce" 
       -input input -output output