Process a continuous stream of JSON

Question

The (now defunct) page http://stream.twitter.com/1/statuses/sample.json used to return a continuous and endless stream of JSON data.

I'd like to process it using jQuery (or JavaScript, but preferably jQuery) inside my own web page to be able to display visual effects based on the live feed of tweets.

Since as far as I know the jQuery parseJSON function will only execute the callback function after all the data has been sent by the server, but this is actually a continuous stream of data. How can I process the data "as it happens" but still keep the connection running?

Answer 1

This sort of thing is best done using WebSockets now, which according to CanIUse.Com is available in all major browsers except Opera Mini (see that link for more details about older or all browsers, and click the Resources tab to see even more links). As an overview, websockets are supported in IE 10+, Firefox 11+ (38+ if within a WebWorker context), Chrome 16+, Opera 12.1+, Safari 7+, Android 4.4+, Opera Mobile 12.1+.

Note: you will likely want to learn about Service Workers and Web Workers as well, though those have different use cases and different abilities.

It looks like this:

var connection = new WebSocket(
   'ws://html5rocks.websocket.org/echo',
   ['soap', 'xmpp']
);

Attaching some event handlers immediately to the connection allows you to know when the connection is opened, when you've received incoming messages, or if an error has occurred.

Sending messages becomes as easy as this:

connection.send('your message');
connection.send(binaryData);

See Introducing WebSockets: Bringing Sockets to the Web for a full explanation on how to do this.

ASP.Net developers: if for some reason you need to support older browsers and don't want to figure out for yourself how to deal with those that don't support WebSockets, consider using a library such as SignalR.

The Old EventSource API Answer For Older Browsers

Most browsers now implement the EventSource API, which makes long polling really easy, as long as the stream can be delivered with content-type text/event-stream. Older browsers or those developers who for any reason can't engineer the stream to have that content-type can use some helper script to do the same thing.

Here's an example:

var jsonStream = new EventSource('https://example.com/yourstreamingservice')
jsonStream.onmessage = function (e) {
   var message = JSON.parse(e.data);
   // handle message
};

This is basically a full-fledged version of the exact thing that I outline below.

The Even Older Service Streaming Answer For REALLY OLD Browsers

What you want is called long polling. You'll need a custom AJAX onreadystatechange handling function. Instead of waiting until the entire stream has completed (since it never will), you'll need to examine the contents periodically. Note that you'll need to do some heavy lifting for this to work in IE 9 and lower, using an iframe.

Roughly:

• Respond to each onreadystatechange event and examine the stream you've been given up to the current character to see if there is enough data to consume one or more discrete events. You'll need to parse the stream yourself with javascript string-handling functions. A combination of split, indexOf, regular expressions, looping, and so on can be used to accomplish this task.
• If there's not enough content yet, then exit and wait for the next event.
• I am pretty sure that each time the onreadystatechange handler fires, the responseText will be all the data that has been received so far. Define a persistent variable that will hold the position of the first character that hasn't been properly processed yet.
• Once there is enough content for one or more discrete events to appear in the stream, take them out one at a time and pass them to your JSON parser to actually render the text as objects. Use them normally.

Check out this HTTP Streaming gist for one resource, or Streaming as an alternative to polling the server at SoftwareAs. If you must support IE 9 or older, then you'll need to use the iframe method for that.

Here is a quote from the book Ajax Design Patterns: Creating Web 2.0 Sites with Programming and Usability Patterns:

In summary, Service Streaming makes the HTTP Streaming approach more flexible, because you can stream arbitrary content rather than Javascript commands, and because you can control the connection's lifecycle. However, it combines two technologies that aren't consistent across browsers, with predictable portability issues. Experiments suggest that the Page Streaming technique does work on both IE [9 and older] and Firefox, but Service Streaming only works on Firefox, whether XMLHTTPRequest or IFrame is used. In the first case IE [9 and older] suppresses the response until its complete, with the IFrame it works if a workaround is used: The IE [9 and older] accepts a message from the server after the first 256 bytes so the only thing to do is to send 256 dummy Bytes before sending the messages. After this all messages will arrive as expected. So a full Service Streaming is possible in IE [9 and older], too!

Mind you that it is from 2006, so it is definitely out of date, but if you have to support older browsers, it's still relevant.

Security Issues

Normal AJAX cannot go cross-domain, meaning (now that I pay attention to the fact that you want to stream from twitter) that you won't be able to do what you're asking. This can be worked around with JSONP, but JSONP by nature can't be service streamed and moreover isn't offered by twitter anyway. There is also Cross-Origin Resource Sharing (CORS) but twitter's not going to set that up for you--that's the kind of thing they'd only do for domains affiliated with them. And CORS requires a modern browser.

Your only option is thus to create a proxy service on your web server that performs the requests to twitter for you and then hands out the data. This can only be done from the same domain as the main page was served from. Doing this would also allow you to create a version that will work for IE using the iframe technique. If you don't care about old IE versions, you can implement CORS yourself to defeat the domain restriction, if you know the domain that will be making the requests.

If you have full control of the client software (like if this is for a corporate intranet) there is another option: hosting the web browser inside of a compiled locally-executed application's user form. I have only done this using C# but I imagine it's possible from other languages. When you use the right browser object, because it's hosted inside a C# application, the C# application can defeat the cross-domain security restrictions, reading and writing all page content no matter what domain it comes from. I doubt your situation is this one but I wanted to put the option here for others who might appreciate it.