What is the max size of AF_UNIX datagram message in Linux?
Currently I'm hitting a hard limit of 130688 bytes. If I try and send anything larger in one message I get a ENOBUFS error.
I have checked the net.core.rmem_default, net.core.wmem_default, net.core.rmem_max, net.core.wmem_max, and net.unix.max_dgram_qlen sysctl options and increased them all but they have no effect because these deal with the total buffer size not the message size.
I have also set the SO_SNDBUF and SO_RCVBUF socket options, but this has the same issue as above. The default socket buffer size are set based on the _default socket options anyways.
I've looked at the kernel source where ENOBUFS is returned in the socket stack, but it wasn't clear to me where it was coming from. The only places that seem to return this error have to do with not being able to allocate memory.
Is the max size actually 130688? If not can this be changed without recompiling the kernel?
Answer
AF_UNIX SOCK_DATAGRAM/SOCK_SEQPACKET datagrams need contiguous memory. Contiguous physical memory is hard to find, and the allocation fails, logging something similar to this on the kernel log:
udgc: page allocation failure. order:7, mode:0x44d0
[...snip...]
DMA: 185*4kB 69*8kB 34*16kB 27*32kB 11*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3788kB
Normal: 13*4kB 6*8kB 100*16kB 62*32kB 24*64kB 10*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 7012kB
[...snip...]
unix_dgram_sendmsg() calls sock_alloc_send_skb() lxr1, which calls sock_alloc_send_pskb() with data_len = 0 and header_len = size of datagram lxr2. sock_alloc_send_pskb() allocates header_len from "normal" skbuff buffer space, and data_len from scatter/gather pages lxr3. So, it looks like AF_UNIX sockets don't support scatter/gather on current Linux.