The Scrapy shell is an interactive shell where you can try and debug your scraping code very quickly, without having to run the spider.
I have followed How can I find an element by CSS class with XPath? which gives the selector to use …
xpath web-scraping scrapy scrapy-shellI know that you can scrapy shell -s USER_AGENT='custom user agent' 'http://www.example.com' to change the …
scrapy scrapy-shellWe've been using scrapy-splash middleware to pass the scraped HTML source through the Splash javascript engine running inside a docker …
web-scraping scrapy scrapy-splash scrapy-shell splash-js-renderI use Scrapy shell without problems with several websites, but I find problems when the robots (robots.txt) does not …
python scrapy web-crawler robots.txt scrapy-shell