appending to a memory-mapped file

Question

I'm constantly appending to a file of stock quotes (ints, longs, doubles, etc.). I have this file mapped into memory with mmap.

What's the most efficient way to make newly appended data available as part of the memory mapping?

I understand that I can open the file again (new file descriptor) and then mmap it to get the new data but that seems to be inefficient. Another approach that has been suggested to me is to pre-allocate the file in 1mb chunks, write to a specific position until reaching the end then ftruncate the file to +1mb.

Are there other approaches?

Doest Boost help with this?

Answer 1

Boost.IOStreams has fixed-size only memory mapped files, so it won't help with your specific problem. Linux has an interface mremap which works as follows:

void *new_mapping = mremap(mapping, size, size + GROWTH, MREMAP_MAYMOVE);
if (new_mapping == MAP_FAILED)
    // handle error
mapping = new_mapping;

This is non-portable, however (and poorly documented). Mac OS X seems not to have mremap.

In any case, you don't need to reopen the file, just munmap it and mmap it again:

void *append(int fd, char const *data, size_t nbytes, void *map, size_t &len)
{
    // TODO: check for errors here!
    ssize_t written = write(fd, data, nbytes);
    munmap(map, len);
    len += written;
    return mmap(NULL, len, PROT_READ, 0, fd, 0);
}

A pre-allocation scheme may be very useful here. Be sure to keep track of the file's actual length and truncate it once more before closing.