Web Scraping in a Google Chrome Extension (JavaScript + Chrome APIs)

javascript google-chrome google-chrome-extension xmlhttprequest web-scraping
Seb Nilsson picture Seb Nilsson · Jun 28, 2011 · Viewed 22.4k times · Source

What are the best options for performing Web Scraping of a not currently open tab from within a Google Chrome Extension with JavaScript and whatever more technologies are available. Other JavaScript-libraries are also accepted.

The important thing is to mask the scraping to behave like a normal web-request. No indications of AJAX or XMLHttpRequest, like X-Requested-With: XMLHttpRequest or Origin.

The scraped content must be accessible from JavaScript for further manipulation and presentation within the extension, most probably as a string.

Are there any hooks in any WebKit/Chrome-specific API:s that can be used to make a normal web-request and get the results for manipulation?

var pageContent = getPageContent(url); // TODO: Implement
var items = $(pageContent).find('.item');
// Display items with further selections

Bonus-points to make this work from a local file on disk, for initial debugging. But if that is the only point is stopping a solution, then disregard the bonus-points.

Answer

Eli Grey picture Eli Grey · Aug 25, 2011

Attempt to use XHR2 responseType = "document" and fall back on (new DOMParser).parseFromString(responseText, getResponseHeader("Content-Type")) with my text/html patch. See https://gist.github.com/1138724 for an example of how I detect responseType = "document support (synchronously checking response === null on an object URL created from a text/html blob).

Use the Chrome WebRequest API to hide X-Requested-With, etc. headers.

Related questions

How do I send an HTTP GET request from a Chrome extension?

I'm working on a chrome extension that sends an HTTP request using the method GET. How do I send at www.example.com the parameter par with value 0? www.example.com?par=0 (the server reads the parameter par and does …

Read More
Chrome sendrequest error: TypeError: Converting circular structure to JSON

I've got the following... chrome.extension.sendRequest({ req: "getDocument", docu: pagedoc, name: 'name' }, function(response){ var efjs = response.reply; }); which calls the following.. case "getBrowserForDocumentAttribute": alert("ZOMG HERE"); sendResponse({ reply: getBrowserForDocumentAttribute(request.docu,request.name) }); break; However, my code never …

Read More
Use a content script to access the page context variables and functions

I'm learning how to create Chrome extensions. I just started developing one to catch YouTube events. I want to use it with YouTube flash player (later I will try to make it compatible with HTML5). manifest.json: { "name": "MyExtension", "version": "1.0", "…

Read More