Is there a way to set heap size from a running Java program?
No.
What you can do with an app that has very variable heap requirements is to set your max heap size very high with -Xmx
and tune -XX:MaxHeapFreeRatio
and -XX:MinHeapFreeRatio
so that the app will not hang on to a lot of memory when the heap shrinks (it does that with default settings).
But note that this may cause performance problems when the memory actually used by the app varies both strongly and quickly - in that case you're better off having it hang on to all the memory rather than give it back to the OS only to claim it again a second later. You might also want to fiddle with the GC options to ensure that the GC doesn't leave too much unclaimed objects lying around, which it tends to do when there's a lot of room for the heap to grow, and which would defeat the goal of wanting the heap size to adjust to the app's needs.