How to activate a specific Python environment as part of my submission to Slurm?

python slurm sbatch
bib picture bib · Nov 29, 2018 · Viewed 8.7k times · Source

I want to run a script on cluster (SBATCH file). How can active my virtual environment (path/to/env_name/bin/activate). Does i need only to add:

 module load python/2.7.14
source "/pathto/Python_directory/ENV2.7_new/bin/activate"

in my_script.sh file?

Answer

Trond Kristiansen picture Trond Kristiansen · Nov 29, 2018

You mean to activate a specific Python environment as part of your submission to Slurm? This is what I add to my job script and it works well. Note that I use Anaconda, which by default adds the required paths to my .bashrc script after installation. Hope this helps. 

....
# define and create a unique scratch directory
SCRATCH_DIRECTORY=/global/work/${USER}/kelp/${SLURM_JOBID}
mkdir -p ${SCRATCH_DIRECTORY}
cd ${SCRATCH_DIRECTORY}

# Activate Anaconda work environment for OpenDrift
source /home/${USER}/.bashrc
source activate MyEnvironment 

# we execute the job and time it
time mpirun python slurmscript.py

Related questions

Using python's multiprocessing on slurm

I am trying to run some parallel code on slurm, where the different processes do not need to communicate. Naively I used python's slurm package. However, it seems that I am only using the cpu's on one node. For example, …

Read More
How do I list all files of a directory?

How can I list all files of a directory in Python and add them to a list?

Read More
Iterating over dictionaries using 'for' loops

I am a bit puzzled by the following code: d = {'x': 1, 'y': 2, 'z': 3} for key in d: print key, 'corresponds to', d[key] What I don't understand is the key portion. How does Python recognize that it needs only to …

Read More