Using Cloudwatch you can monitor your EC2 instances by several criteria, such as network usage, CPU usage, and so on…
Unfortunately, there is no metric for memory consumption. First of all, just out of curiosity, I would like to know, why? Can anybody explain why it is possible to, e.g., monitor CPU usage, but not memory usage? At least to me, that's not obvious.
And then, my actual question: Okay, given that Cloudwatch doesn't allow monitoring the EC2 instances' memory usage - what is the alternative? How should I setup an alarm if, e.g. > 80% of the memory of an instance is being used?
Memory and Disk specific statistics require AWS to monitor at the OS level rather than the host level, so that is why they leave it out by default. It will probably be added at some point but since it has been on the wish list for about 7 years, we can assume it's a very low priority item.
The recommended way to monitor memory usage is to create a custom Cloudwatch metric by using your own monitoring scripts on your instances. AWS have published documentation on how to achieve this on Linux instances with a set of (unsupported) scripts.
Once your instances are publishing the custom metrics, you will be able to attach alarms to them in CloudWatch.