How do craigslist mashups get data?

Question

I'm doing some research work into content aggregators, and I'm curious how some of the current craigslist aggregators get data into their mashups.

For example, www.housingmaps.com and the now closed www.chicagocrime.org

If there is a URL that can be used for reference, that would be perfect!

Answer 1

For AdRavage.com I use a combination of Magpie RSS (to extract the data returned from searches) and a custom screen scraping class to properly populate the city/category information used when building searches.

For example, to extract the categories you could:

//scrape category data
$h = new http();
$h->dir = "../cache/"; 
$url = "http://craigslist.org/";

if (!$h->fetch($url, 300)) {
  echo "<h2>There is a problem with the http request!</h2>";      
  exit();
}

//we need to get all category abbreviations (data looks like: <option value="ccc">community)
preg_match_all ("/<option value=\"(.*)\">([^`]*?)\n/", $h->body, $categoryTemp);

$catNames = $categoryTemp['2']; 

//return the array of abreviations
if(sizeof($catNames) > 0)   
    return $catNames;   
else
    return $emptyArray = array();