Package "rvest" for web scraping https site with proxy

r web-scraping rvest
Stanislav Shlykov picture Stanislav Shlykov · Mar 16, 2016 · Viewed 8k times · Source

I want to scrap a https website, but I failed.

Here is my code:

require(rvest)
url <- "https://www.sunnyplayer.com/de/"
content <- read_html(url)

But I have error in console- "Error in open.connection(x, "rb") : Timeout was reached" How I can fix this problem?

Answer

Matt picture Matt · Jul 19, 2016

The same thing happens to me on a proxy. To get around this, use download.file and specify a download location. You can then parse the file with read_html.

download.file(url, destfile = 'C://whatever.html')
content <- read_html('C://whatever.html')

Related questions

rvest how to select a specific css node by id

I'm trying to use the rvest package to scrape data from a web page. In a simple format, the html code looks like this: <div class="style"> <input id="a" value="123"> <input id="b"> &…

Read More
Using 'rvest' to extract links

I am trying to scrape data from Yelp. One step is to extract links from each restaurant. For example, I search restaurants in NYC and get some results. Then I want to extract the links of all the 10 restaurants Yelp …

Read More
How to scrape a table with rvest and xpath?

using the following documentation i have been trying to scrape a series of tables from marketwatch.com here is the one represented by the code bellow: The link and xpath are already included in the code: url <- "http://…

Read More