Test MPI on a cluster
I am learning OpenMPI on a cluster. Here is my first example. I expect the output would show response from different nodes, but they all respond from the same node node062. I just wonder why and how I can actually get report from different nodes to show MPI actually is distributing processes to different nodes? Thanks and regards!
ex1.c
/* test of MPI */
#include "mpi.h"
#include <stdio.h>
#include <string.h>
int main(int argc, char **argv)
{
char idstr[2232]; char buff[22128];
char processor_name[MPI_MAX_PROCESSOR_NAME];
int numprocs; int myid; int i; int namelen;
MPI_Status stat;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
MPI_Get_processor_name(processor_name, &namelen);
if(myid == 0)
{
printf("WE have %d processors\n", numprocs);
for(i=1;i<numprocs;i++)
{
sprintf(buff, "Hello %d", i);
MPI_Send(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD); }
for(i=1;i<numprocs;i++)
{
MPI_Recv(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD, &stat);
printf("%s\n", buff);
}
}
else
{
MPI_Recv(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &stat);
sprintf(idstr, " Processor %d at node %s ", myid, processor_name);
strcat(buff, idstr);
strcat(buff, "reporting for duty\n");
MPI_Send(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
}
MPI_Finalize();
}
ex1.pbs
#!/bin/sh
#
#This is an example script example.sh
#
#These commands set up the Grid Environment for your job:
#PBS -N ex1
#PBS -l nodes=10:ppn=1,walltime=1:10:00
#PBS -q dque
# export OMP_NUM_THREADS=4
mpirun -np 10 /home/tim/courses/MPI/examples/ex1
compile and run:
[tim@user1 examples]$ mpicc ./ex1.c -o ex1
[tim@user1 examples]$ qsub ex1.pbs
35540.mgt
[tim@user1 examples]$ nano ex1.o35540
----------------------------------------
Begin PBS Prologue Sat Jan 30 21:28:03 EST 2010 1264904883
Job ID: 35540.mgt
Username: tim
Group: Brown
Nodes: node062 node063 node169 node170 node171 node172 node174 node175
node176 node177
End PBS Prologue Sat Jan 30 21:28:03 EST 2010 1264904883
----------------------------------------
WE have 10 processors
Hello 1 Processor 1 at node node062 reporting for duty
Hello 2 Processor 2 at node node062 reporting for duty
Hello 3 Processor 3 at node node062 reporting for duty
Hello 4 Processor 4 at node node062 reporting for duty
Hello 5 Processor 5 at node node062 reporting for duty
Hello 6 Processor 6 at node node062 reporting for duty
Hello 7 Processor 7 at node node062 reporting for duty
Hello 8 Processor 8 at node node062 reporting for duty
Hello 9 Processor 9 at node node062 reporting for duty
----------------------------------------
Begin PBS Epilogue Sat Jan 30 21:28:11 EST 2010 1264904891
Job ID: 35540.mgt
Username: tim
Group: Brown
Job Name: ex1
Session: 15533
Limits: neednodes=10:ppn=1,nodes=10:ppn=1,walltime=01:10:00
Resources: cput=00:00:00,mem=420kb,vmem=8216kb,walltime=00:00:03
Queue: dque
Account:
Nodes: node062 node063 node169 node170 node171 node172 node174 node175 node176
node177
Killing leftovers...
End PBS Epilogue Sat Jan 30 21:28:11 EST 2010 1264904891
----------------------------------------
UPDATE:
I would like to run several background jobs in a single PBS script, so that the jobs can run at the same time. e.g. in the above example, I added another call to run ex1 and change both runs to be background in ex1.pbs
#!/bin/sh
#
#This is an example script example.sh
#
#These commands set up the Grid Environment for your job:
#PBS -N ex1
#PBS -l nodes=10:ppn=1,walltime=1:10:00
#PBS -q dque
echo "The first job starts!"
mpirun -np 5 --machinefile /home/tim/courses/MPI/examples/machinefile /home/tim/courses/MPI/examples/ex1 &
echo "The first job ends!"
echo "The second job starts!"
mpirun -np 5 --machinefile /home/tim/courses/MPI/examples/machinefile /home/tim/courses/MPI/examples/ex1 &
echo "The second job ends!"
(1) The result is fine after qsub this script with previous compiled executable ex1.
The first job starts!
The first job ends!
The second job starts!
The second job ends!
WE have 5 processors
WE have 5 processors
Hello 1 Processor 1 at node node063 reporting for duty
Hello 2 Processor 2 at node node169 reporting for duty
Hello 3 Processor 3 at node node170 reporting for duty
Hello 1 Processor 1 at node node063 reporting for duty
Hello 4 Processor 4 at node node171 reporting for duty
Hello 2 Processor 2 at node node169 reporting for duty
Hello 3 Processor 3 at node node170 reporting for duty
Hello 4 Processor 4 at node node171 reporting for duty
(2) However, I think the running time of ex1 is too quick and probably the two background jobs do not have much running time overlapping, which is not the case when I apply the same way to my real project. So I added sleep(30) to ex1.c to extend the running time of ex1 so that two jobs running ex1 in background will be running simultaneously almost all the time.
/* test of MPI */
#include "mpi.h"
#include <stdio.h>
#include <string.h>
#include <unistd.h>
int main(int argc, char **argv)
{
char idstr[2232]; char buff[22128];
char processor_name[MPI_MAX_PROCESSOR_NAME];
int numprocs; int myid; int i; int namelen;
MPI_Status stat;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
MPI_Get_processor_name(processor_name, &namelen);
if(myid == 0)
{
printf("WE have %d processors\n", numprocs);
for(i=1;i<numprocs;i++)
{
sprintf(buff, "Hello %d", i);
MPI_Send(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD); }
for(i=1;i<numprocs;i++)
{
MPI_Recv(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD, &stat);
printf("%s\n", buff);
}
}
else
{
MPI_Recv(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &stat);
sprintf(idstr, " Processor %d at node %s ", myid, processor_name);
strcat(buff, idstr);
strcat(buff, "reporting for duty\n");
MPI_Send(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
}
sleep(30); // new added to extend the running time
MPI_Finalize();
}
But after recompilation and qsub again, the results seems not all right. There are processes aborted. in ex1.o35571:
The first job starts!
The first job ends!
The second job starts!
The second job ends!
WE have 5 processors
WE have 5 processors
Hello 1 Processor 1 at node node063 reporting for duty
Hello 2 Processor 2 at node node169 reporting for duty
Hello 3 Processor 3 at node node170 reporting for duty
Hello 4 Processor 4 at node node171 reporting for duty
Hello 1 Processor 1 at node node063 reporting for duty
Hello 2 Processor 2 at node node169 reporting for duty
Hello 3 Processor 3 at node node170 reporting for duty
Hello 4 Processor 4 at node node171 reporting for duty
4 additional processes aborted (not shown)
4 additional processes aborted (not shown)
in ex1.e35571:
mpirun: killing job...
mpirun noticed that job rank 0 with PID 25376 on node node062 exited on signal 15 (Terminated).
mpirun: killing job...
mpirun noticed that job rank 0 with PID 25377 on node node062 exited on signal 15 (Terminated).
I wonder why there are processes aborted? How can I qsub background jobs correctly in a PBS script?
Answer
couple things: you need to tell mpi where to launch processes, assuming you are using mpich, look at mpiexec help section and find machine file or equivalent description. Unless machine file is provided, it will run on one host
PBS automatically creates nodes file. Its name is stored in PBS_NODEFILE environment variable available in PBS command file. Try the following:
mpiexec -machinefile $PBS_NODEFILE ...
if you are using mpich2, you have two boot your mpi runtime using mpdboot. I do not remember the details of command, you will have to read man page. Remember to create secret file otherwise mpdboot will fail.
I read your post again, you will use open mpi, you still have to supply machines file to mpiexec command, but you do not have to mess with mpdboot