Recording user data for heatmap with JavaScript

Question

I was wondering how sites such as crazyegg.com store user click data during a session. Obviously there is some underlying script which is storing each clicks data, but how is that data then populated into a database? It seems to me the simple solution would be to send data via AJAX but when you consider that it's almost impossible to get a cross browser page unload function setup, I'm wondering if there is perhaps some other more advanced way of getting metric data.

I even saw a site which records each mouse movement and I am guessing they are definitely not sending that data to a database on each mouse move event.

So, in a nutshell, what kind of technology would I need in order to monitor user activity on my site and then store this information in order to create metric data? I am not looking to recreate GA, I'm just very interested to know how this sort of thing is done.

Thanks in advance

Answer 1

Heatmap analytics turns out to be WAY more complicated than just capturing the cursor coordinates. Some websites are right-aligned, some are left-aligned, some are 100%-width, some are fixed-width-"centered"... A page element can be positioned absolutely or relatively, floated etc. Oh, and there's also different screen resolutions and even multi-monitor configurations.

Here's how it works in HeatTest (I'm one of the founders, have to reveal that due to the rules):

1. JavaScript handles the onClick event: document.onclick = function(e){ } (this will not work with <a> and <input> elements, have to hack your way around)
2. Script records the XPath-address of the clicked element (since coordinates are not reliable, see above) in a form //body/div[3]/button[id=search] and the coordinates within the element.
3. Script sends a JSONP request to the server (JSONP is used because of the cross-domain limitations in browsers)
4. Server records this data into the database.

Now, the interesting part - the server.

1. To calculate the heatmap the server launches a virtual instance of a browser in-memory (we use Chromium and IE9)
2. Renders the page
3. Takes a screenshot,
4. Finds the elements' coordinates and then builds the heatmap.

It takes a lot of cpu-power and memory usage. A lot. So most of the heatmap-services including both us and CrazyEgg, have stacks of virtual machines and cloud servers for this task.