I've already seen this question about scraping ajax, but python isn't mentioned there. I considered using scrapy, i believe they have some docs on that subject, but as you can see the website is down. So i don't know what to do. I want to do the following:
I only have one url, example.com you go from page to page by clicking submit, the url doesn't change since they're using ajax to display the content. I want to scrape the content of each page, how to do it?
Lets say that i want to scrape only the numbers, is there anything other than scrapy that would do it? If not, would you give me a snippet on how to do it, just because their website is down so i can't reach the docs.
First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.
Speaking about handling ajax while web scraping. Basically, the idea is rather simple:
XHR
request is going to the serverXHR
request in your spiderAlso see:
Hope that helps.