OpenMPI MPI_Barrier problems

Question

I having some synchronization issues using the OpenMPI implementation of MPI_Barrier:

int rank;
int nprocs;

int rc = MPI_Init(&argc, &argv);

if(rc != MPI_SUCCESS) {
    fprintf(stderr, "Unable to set up MPI");
    MPI_Abort(MPI_COMM_WORLD, rc);
}

MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);


printf("P%d\n", rank);
fflush(stdout);

MPI_Barrier(MPI_COMM_WORLD);

printf("P%d again\n", rank);

MPI_Finalize();

for mpirun -n 2 ./a.out

output should be: P0 P1 ...

output is sometimes: P0 P0 again P1 P1 again

what's going on?

Answer 1

The order in which your print out lines appear on your terminal is not necessarily the order in which things are printed. You are using a shared resource (stdout) for that so there always must be an ordering problem. (And fflush doesn't help here, stdout is line buffered anyhow.)

You could try to prefix your output with a timestamp and save all of this to different files, one per MPI process.

Then to inspect your log you could merge the two files together and sort according to the timestamp.

Your problem should disappear, then.