I would like to create a custom theme. It is possible to apply other people's custom themes (http://www.gethemes.com/free-themes) but I cannot find documentation on how to create Google Sites themes.
I don't know how to create a entirely custom theme from scratch, but you can create a theme that is extremely "customized". In other words, you can create a new site, apply an existing theme, then customize colors, fonts, and background images (or textures) for all the various parts of the page. You will not be able to change the size of the various elements (sidebar, header area, etc), but as long as you can find a base theme with the general layout that you'd like, you're set. You are then free to publish your new theme as a template. If this provides the flexibility that you hoped for, here are the required steps:
UPDATE: You can find similar instructions in the official documentation.