Creating an MPI_Datatype for a structure containing pointers

Question

I have the following structure.

typedef struct
{
 int *Ai;
 double *Ax;
 int nz;
}column;

I want to transfer this structure using MPI_Send and MPI_Receive. How do I create an MPI_Datatype for this structure?

Answer 1

MPI is designed to work with arrays of structures rather that with structures of arrays.

The MPI_Hindexed that @suszterpatt proposed is a terrible hack. It will only allow you to send one element of the structure type and only the element that was used to define the MPI data type. For other variables of the same structure type it is mostly guaranteed that the computed offsets will be wrong. Besides Hindexed types use one and the same MPI data type for all elements and thus does not allow you to send both ints and doubles.

The wise thing to do is to transform your program to use arrays of structures:

 typedef struct
 {
    int i;
    double z;
 } point;

 typedef struct
 {
    point *A;
    int nz;
 } column;

Now you can create an MPI structured type point_type and use it to send nz elements of that type giving column.A as the buffer address:

 int lens[3];
 MPI_Aint base, disps[2];
 MPI_Datatype oldtypes[2], point_struct, point_type;

 MPI_Get_address(&point, disps);
 MPI_Get_address(&point.z, disps+1);
 base = disps[0];

 lens[0] = 1; disps[0] = MPI_Aint_diff(disps[0], base); oldtypes[0] = MPI_INT;
 lens[1] = 1; disps[1] = MPI_Aint_diff(disps[1], base); oldtypes[1] = MPI_DOUBLE;
 MPI_Type_create_struct(2, lens, disps, oldtypes, &point_struct);
 MPI_Type_create_resized(point_struct, 0, sizeof(point), &point_type);
 MPI_Type_commit(&point_type);

 MPI_Send(column.A, column.nz, point_type, ...);

This first creates an MPI datatype point_struct that describes the layout of the structure members, but does not account for any padding at the end and therefore cannot be used to reliably send an array of such structures. Therefore, a second datatype point_type with the correct extent is created using MPI_Type_create_resized.

On the receiver side you would peek the message with MPI_Probe, extract the number of elements with MPI_Get_count with a type of point_type (that goes straight to the nz field), allocate the A field and use it in MPI_Recv to receive the nz elements:

 MPI_Status status;
 MPI_Probe(source, tag, comm, &status);
 MPI_Get_count(&status, point_type, &column.nz);
 if (nz == MPI_UNDEFINED)
   ... non-integral message was received, do something
 column.A = (point *)malloc(column.nz*sizeof(point));
 MPI_Recv(column.A, column.nz, point_type, source, tag, comm, MPI_STATUS_IGNORE);

If that code change is impossible you can still go through the intermediate step of transforming your structure before sending it, a process usually called (un-)marshaling. In your case do something like this (I assume that you store the number of array elements in both Ai and Ax in the nz field):

 point *temp = (point *)malloc(nz*sizeof(point));
 for (int i = 0; i < column.nz; i++)
 {
   temp[i].i = column.Ai[i];
   temp[i].z = column.Az[i];
 }
 MPI_Send(temp, nz, point_type, ...);
 free(temp);

On the receiver side you must do the opposite: allocate a large enough buffer that can hold the structure, receive the message in it and then do the opposite transformation.

Once again, you do not need to transmit the actual value of nz since it can be easily extracted from the length of the message using MPI_Get_count.