Ideas for scaling chat in AWS?

Question

I'm trying to come up with the best solution for scaling a chat service in AWS. I've come up with a couple potential solutions:

1. Redis Pub/Sub - When a user establishes a connection to a server that server subscribes to that user's ID. When someone sends a message to that user, a server will perform a publish to the channel with the user's id. The server the user is connected to will receive the message and push it down to the appropriate client.

2. SQS - I've thought of creating a queue for each user. The server the user is connected to will poll (or use SQS long-polling) that queue. When a new message is discovered, it will be pushed to the user from the server.

3. SNS - I really liked this solution until I discovered the 100 topic limit. I would need to create a topic for each user, which would only support 100 users.

Are their any other ways chat could be scaled using AWS? Is the SQS approach viable? How long does it take AWS to add a message to a queue?

Answer 1

Building a chat service isn't as easy as you would think.

I've built full XMPP servers, clients, and SDK's and can attest to some of the subtle and difficult problems that arise. A prototype where users see each other and chat is easy. A full features system with account creation, security, discovery, presence, offline delivery, and friend lists is much more of a challenge. To then scale that across an arbitrary number of servers is especially difficult.

PubSub is a feature offered by Chat Services (see XEP-60) rather than a traditional means of building a chat service. I can see the allure, but PubSub can have drawbacks.

Some questions for you: 1. Are you doing this over the Web? Are users going to be connecting and long-poling or do you have a Web Sockets solution?

2. How many users? How many connections per user? Ratio of writes to reads?

3. Your idea for using SQS that way is interesting, but probably won't scale. It's not unusual to have 50k or more users on a chat server. If you're polling each SQS Queue for each user you're not going to get anywhere near that. You would be better off having a queue for each server, and the server polls only that queue. Then it's on you to figure out what server a user is on and put the message into the right queue.

I suspect you'll want to go something like:

1. A big RDS database on the backend.
2. A bunch of front-end servers handling the client connections.
3. Some middle tier Java / C# code tracking everything and routing messages to the right place.

To get an idea of the complexity of building a chat server read the XMPP RFC's: RFC 3920 RFC 3921