Retrieve contents of URL as string

ruby open-uri
AP257 picture AP257 · Jul 7, 2010 · Viewed 44.8k times · Source

For tedious reasons to do with Hpricot, I need to write a function that is passed a URL, and returns the whole contents of the page as a single string.

I'm close. I know I need to use OpenURI, and it should look something like this:

require 'open-uri'
open(url) {
  # do something mysterious here to get page_string
}
puts page_string

Can anyone suggest what I need to add?

Answer

Carlo Pecchia picture Carlo Pecchia · Jul 7, 2010

You can do the same without OpenURI:

require 'net/http'
require 'uri'

def open(url)
  Net::HTTP.get(URI.parse(url))
end

page_content = open('http://www.google.com')
puts page_content

Or, more succinctly:

Net::HTTP.get(URI.parse('http://www.google.com'))

Related questions

In Ruby/Rails, how can I encode/escape special characters in URLs?

How do I encode or 'escape' the URL before I use OpenURI to open(url)? We're using OpenURI to open a remote url and return the xml: getresult = open(url).read The problem is the URL contains some user-input text …

Read More
Is there a workaround to open URLs containing underscores in Ruby?

I'm using open-uri to open URLs. resp = open("http://sub_domain.domain.com") If it contains underscore I get an error: URI::InvalidURIError: the scheme http does not accept registry part: sub_domain.domain.com (or bad hostname?) I understand …

Read More
Ruby's open-uri and cookies

I would like to store the cookies from one open-uri call and pass them to the next one. I can't seem to find the right docs for doing this. I'd appreciate it if you could tell me the right way …

Read More