How to build a movie recommender system?
- What is the best approach?
- What are the algorithms used? What are their strengths and weaknesses?
- Why do current movie recommender systems fail at providing good recommendations?
Answer
This is a very open-ended question that involves a lot of different concepts.
As an initial discussion point, consider the k-nearest neighbor algorithm. It's widely used in problems similar to your movie picker. One big problem with this algorithm is the human input in deciding how many dimensions you use to segment your feature-space and choosing the properties of each of those dimensions so that each adds value, rather than duplicating the value of another dimension.
Directly related to the k-NN algorithm is the field of cluster analysis. When you plot data points for information that has clumps within more scattered outliers you can intuitively see that there is some nature of similarity in the clumped points. You may be able to easily group some of the scattered outliers with one or another clump, but there will be many points that lie between clumps that could fit with two or more competing clumps. The only way to remedy this dilemma is to add more dimensional parameters to your data points so that those uncommitted outliers are drawn to one clump. (Follow the link to see a nice picture of clumped data.)
This brief introduction leads to the next concept: Pattern Recognition. This subject is math-heavy and the subject of a lot of research in the fields of Theoretical Computing Science, Statistics, Artificial Intelligence, Machine Learning, and Clairvoyance. That last one is a semi-joke, but it points to the crux of your problem: How can a computer predict what you will do in the future? The short answer is that it can't. The longer answer tries to explain why your tastes and moods change in seemingly random directions at seemingly random times. A good pattern recognition system might pick 20 movies that you really enjoy and then recommend another one from the same clump that the other 20 did that you thoroughly hate. Where did the system fail? Was it in the algorithm implementation, the initial selection of parameters for the dimensions of your feature-space, or did your profile get messed up because someone else used your Netflix account to order 'Howard the Duck', 'Cruising', and 'Beaches'?
The wikipedia page for 'Pattern Recognition' lists a lot of different algorithms and methods. You can start reading there to get a better handle on individual strengths and weaknesses. You could also try asking this question in the Theoretical Computer Science stack to get the long-haired answers.