Django-way for building a "News Feed" / "Status update" / "Activity Stream"

Question

I'd like to create a reusable Django app that handles status updates of the Users. Much like facebook's "news feed".

Use cases includes, for example:

• A Professor can create an Assignment due to an specific date and every student can see on the news feed that the assignment was created, with a short description, the date that it's due and a link to see the full description.
• He also can upload a new PDF that he finds interesting for his students. On the news feed, the info regarding this should be displayed, eg, the description of the pdf, an link to download and a link to preview it.
• A link to a YouTube video can be posted and on the News Feed is displayed an small thumbnail and, with a click, the video is embbeded using javascript and the user can watch it right away.

One concern is how to handle different kinds of Updates and display the correct "html snippet" for it. The other, which is more important, is how to design the Models of this "Django way".

About the former, I could think of two ways of doing it:

1. Using Model inheritance;
2. Using Generic relations.

I searched before posting here, but I didn't find anything. I checked Pinax to see if they had it implemented, but they don't. So, I'm here looking for more suggestions on how to handle this in a nice and non-hacky way.

Thanks in advance,

Answer 1

Python is actually a great language for building Activity Streams and Newsfeeds. Tommaso and I have written the Stream Framework package. https://github.com/tschellenbach/stream-framework It is currently the most used Python solution for building newsfeeds. We're also offering a hosted solution at https://getstream.io. The Django client is by far the easiest to get started with: https://github.com/GetStream/stream-django and python can be found here (https://github.com/getstream/stream-python)

The templating part works like this

{% load stream_django %}

{% for activity in activities %}
    {% render_activity activity %}
{% endfor %}

This will render a template located in activity/tweet.html with the activity as context. For instance

{{ activity.actor.username }} said "{{ activity.object.body }} {{ activity.created_at|timesince }} ago"

The full docs are here: https://github.com/GetStream/stream-django#templating

The Stream Framework allows you to build any type of newsfeed using either Redis or Cassandra. It's build to scale and creates the individual newsfeeds using a fanout process.

Besides the Stream Framework (which I obviously prefer) there are many other solutions out there. A full list is available on django packages: https://www.djangopackages.com/grids/g/activities/

Note that with newsfeeds there are a few scaling issues to keep in mind. In general there are 3 common approaches:

Denormalization strategies

Pull Most users start out this way. When you open the feed page you simply query the feeds from all the users you follow. If the users feeds are stored in memory this will keep on working for quite some time. Eventually it is quite hard to keep on using such as strategy though as you often have to query most of the nodes storing your user's feeds.

Push The push approach writes your activity to all of your followers feeds. Of course this means you're wasting a ton of resources, but the end result is a pre computed feed per user. This approach (though initially not very efficient) scales nicely.

Combination Some optimized systems use a combination of these two approaches. Also see the Yahoo paper on this topic.

Storage options

In terms of storing all this data the most common options are Redis, Cassandra and MongoDB. Let's quickly compare these:

Redis Redis is Extremely easy to setup and maintain. It however only stores data in memory. This means you'll have to optimize how you serialize data and maybe fallback to the database for less frequently queried data. Another problem is that it's not trivial to add machines to your Redis cluster.

MongoDB Mongo DB is used primarily by a few ruby projects and it's also available as a backend for pump.io by e14n. I've personally never ran it in production so I can't properly evaluate this option. There are however a lot of blogposts covering issues with the performance, scalability and maintainability of mongo.

Cassandra Fashiolista, Instagram and Spotify are all using Cassandra. Our hosted solution also uses Cassandra as a backend. It's extremely cost effective to operate and you can add more nodes with ease. The only problem is that it's hard to setup and maintain.

Articles

In addition have a look at this high scalability post were we explain some of the design decisions involved: http://highscalability.com/blog/2013/10/28/design-decisions-for-scaling-your-high-traffic-feeds.html

To learn more about feed design I highly recommend reading some of the articles which we based Feedly on:

• Yahoo Research Paper
• Twitter 2013 Redis based, with fallback
• Cassandra at Instagram
• Etsy feed scaling
• Facebook history
• Django project, with good naming conventions. (But database only)
• http://activitystrea.ms/specs/atom/1.0/ (actor, verb, object, target)
• Quora post on best practises
• Quora scaling a social network feed
• Redis ruby example
• FriendFeed approach
• Thoonk setup
• Twitter's Approach