I have a Java service that currently runs with a 14GB heap. I am keen to try out the -XX:+UseLargePages option to see how this might affect the performance of the system. I have configured the OS as described by Oracle using appropriate shared memory and page values (these can also be calculated with an online tool).
Once the OS is configured, I can see that it allocates the expected amount of memory as huge-pages. However, starting the VM with the -XX:+UseLargePages
option set always results in one of the following errors:
When -Xms
/ -Xmx
is almost equal to the huge page allocation:
Failed to reserve shared memory (errno = 28). // 'No space left on device'
When -Xms
/ -Xmx
is less than the huge page allocation:
Failed to reserve shared memory (errno = 12). // 'Out of memory'
I did try introducing some leeway - so on a 32GB system I allocated 24GB of shared memory and hugepages to use with a JVM configured with a 20GB heap, of which only 14GB is currently utilized. I also verified that the user executing the JVM did have group rights consistent with /proc/sys/vm/hugetlb_shm_group
.
Can anyone give me some pointers on where I might be going wrong and what I could try next?
-Xms
/ -Xmx
- 20GB/proc/sys/kernel/shmmax
- 25769803776 (24GB)/proc/sys/vm/nr_hugepages
- 12288Thanks to @jfgagne for providing an answer that lead to a solution. In addition to the /proc/sys/kernel/shmall
setting (specified as 4KB pages), I had to add entries to /etc/security/limits.conf
as described on Thomas' blog. However, as my application is started using jsvc
I also had to duplicate the settings for the root user (note that the limits are specified in KB):
root soft memlock 25165824
root hard memlock 25165824
pellegrino soft memlock 25165824
pellegrino hard memlock 25165824
It's also worth mentioning that settings could be tested quickly by starting the JVM with the -version
argument:
java -XX:+UseLargePages -Xmx20g -version
When you use huge pages with Java, there is not only the heap using huge pages, but there is also the PermGen: do not forget to allocate space for it. It seems this is why you have a different errno message when you set Xmx
near the amount of huge pages.
There is also the shmall
kernel parameter that needs to be set which you did not mention, maybe it is what is blocking you. In your case, you should set it to 6291456.
The last thing to say: when using huge pages, the Xms
parameter is not used anymore: Java reserves all Xmx
in shared memory using huge pages.