AWS cloudformation: One big template file or many small ones?

Question

I'm about to rewrite a lot of my aws deployment code to launch everything with cloudformation controlled by boto, instead of bringing up each element on its own with boto. Does anyone know if its "best practice" to use one giant template file, which kicks everything off together, or a lot of smaller ones?

The advantage of one giant one seems to be that AWS handles all the dependancies for you so will bring things up slightly faster. The clear disadvantage is that it seems like a nightmare to maintain.

Has anyone tried combining their template files at run time so that they are treated as one large one, or does that get difficult to maintain?

Answer 1

There's no easy answer, but several important points to keep in mind:

• when you write several small templates, write a master template which will call the small ones (nested stacks). When you want to update a small one, do the change in the file, and update the master one. Only the resources that have changed will be updated, and the result of the stack update will be atomic (all clear or rollback everything). CloudFormation will still run the nested stacks in parallel, so it's not that slower.

• there's a limitation in CloudFormation regarding the number of resources (200 resources per stack). It's very easy to reach, if you've got SecurityGroupIngress/Egress rules for example. Not sure this limit can be updated by the support.

• on the other hand, feeding parameters to nested stacks can result in big files with not that many information... consider this: you have to feed all the parameters to the call, inside the nested stack you have to declare all the parameters again. Two levels of nesting is a real pain, believe me!

The best solution I've found is using a CloudFormation template frontend (I use troposphere - in python), so that you really describe infrastructure as code, with all the advantages of code (loops, conditionals, external file, functions) and in the end, you've got a genuine CloudFormation template.

I've been able to write huge CloudFormation templates with this system, without any maintenance nightmare...