I have an MPI program which compiles and runs, but I would like to step through it to make sure nothing bizarre is happening. Ideally, I would like a simple way to attach GDB to any particular process, but I'm not really sure whether that's possible or how to do it. An alternative would be having each process write debug output to a separate log file, but this doesn't really give the same freedom as a debugger.
Are there better approaches? How do you debug MPI programs?
I have found gdb quite useful. I use it as
mpirun -np <NP> xterm -e gdb ./program
This the launches xterm windows in which I can do
run <arg1> <arg2> ... <argN>
usually works fine
You can also package these commands together using:
mpirun -n <NP> xterm -hold -e gdb -ex run --args ./program [arg1] [arg2] [...]