read() from a HDD with O_DIRECT fails with 22 (EINVAL, Invalid Argument)

Question

I want to make a basic read() from a SATA HDD /dev/sdd. A write() seems to work. Also read() and write() works without the O_DIRECT Flag. I've read, that it has to be aligned to the blocksize. So I used this to get the blocksize:

root$ blockdev --getsize /dev/sdd
488397168

root$ blockdev --getsize64 /dev/sdd
250059350016

root$ python -c "print 250059350016.0/488397168"
512.0

As you can see I have root. The HDD is connected via a PCIe SATA Card and lspci -vv shows me, that it uses the basic ahci (drivers/ata/ahci.c) driver. I work with the 3.2.0 Linux Kernel on a 64 bit Power Architecture.

Here is my code:

#define _GNU_SOURCE

#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdlib.h> 
#include <unistd.h> 
#include <errno.h>

int main() {

    int r;
    char *buffer, *tmp_buffer;
    // alloc more than necesarry to align the real buffer
    tmp_buffer = malloc(2*512*sizeof(char)); 
    long align = (unsigned long)tmp_buffer%512;
    printf("tmp_buffer is at: %x \% 512 = %d\n",tmp_buffer,align);

    buffer = tmp_buffer+(512-align);
    printf("buffer is at: %x \% 512 = %d\n",buffer,(unsigned long)buffer%512);

    memset(buffer,0,sizeof(512));

    // OPEN
    int fd = open("/dev/sdd",O_DIRECT | O_RDWR | O_SYNC);
    if(fd!=3) printf("fd = %d\n",fd);

    // READ
    printf("try to read and then dump buffer:\n");

    r = read(fd,buffer,sizeof(512));
    if(r == -1) printf("Error: %s\n",strerror(errno));
    else {
        // DUMP BUFFER
        int i;
        for(i=0; i<sizeof(512); i++)
            printf("%c",buffer[i]);
    }
    printf("\n");
    return 0;
}

The output is:

tmp_buffer is at: 1cc80010 % 512 = 16
buffer is at: 1cc80200 % 512 = 0
try to read and then dump buffer:
Error: Invalid argument

edit: I have updated my source as suggested by Brett Hale's answer. Unfortunately I still get the error. Is my way to find out the blocksize ok? Have I done the aligning right?

Thank you very much for reading,
Fabian

Answer 1

Direct DMA transfer typically requires the buffer to be aligned. From man:

The O_DIRECT flag may impose alignment restrictions on the length and address of userspace buffers and the file offset of I/Os. ... Under Linux 2.6, alignment to 512-byte boundaries suffices.

So char buffer[512]; might need to be aligned to a 512-byte address.

It may not be possible to achieve this alignment on the stack, so something like:

static char buffer[512] __attribute__ ((__aligned__ (512)));

may work. Or maybe this alignment will work on the stack. Alternatively, if you're using x86, you could use the <mm_malloc.h> intrinsic support functions: _mm_malloc and _mm_free.