Ruby open-uri redirect forbidden

ruby open-uri
Vikaton picture Vikaton · Dec 10, 2014 · Viewed 18.3k times · Source

I have this simple html parser(for learning purposes) that I have been working on.:

require 'open-uri'
puts "Enter URL to parse HTML: "
url = gets.chomp
puts "Enter tag to parse from: "
tag = gets.chomp
response = open(url).read
title1 = response.index(tag)
title2 = response.index(tag.insert(1,'/')) -1
result = response[(title1 + tag.length - 1)..title2]
print result

and when I input http://twitter.com, I get this error message:

ERROR: `open_loop': redirection forbidden: http://twitter.com -> https://twitter.com/ (RuntimeError)
from /usr/local/rvm/rubies/ruby-2.1.4/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
from /usr/local/rvm/rubies/ruby-2.1.4/lib/ruby/2.1.0/open-uri.rb:704:in `open'
from /usr/local/rvm/rubies/ruby-2.1.4/lib/ruby/2.1.0/open-uri.rb:34:in `open'
from /home/ubuntu/workspace/htmlparse.rb:6:in `<main>'

Any advise or help? Im new to Ruby and I am aware of other html parsing modules, but Im doing this to learn Ruby basics. Thanks.

Answer

fivedigit picture fivedigit · Dec 10, 2014

Have a look at the open_uri_redirections gem.

It patches Ruby's OpenURI to allow redirections from HTTP to HTTPS or the other way around.

Related questions

In Ruby/Rails, how can I encode/escape special characters in URLs?

How do I encode or 'escape' the URL before I use OpenURI to open(url)? We're using OpenURI to open a remote url and return the xml: getresult = open(url).read The problem is the URL contains some user-input text …

Read More
Retrieve contents of URL as string

For tedious reasons to do with Hpricot, I need to write a function that is passed a URL, and returns the whole contents of the page as a single string. I'm close. I know I need to use OpenURI, and …

Read More
Is there a workaround to open URLs containing underscores in Ruby?

I'm using open-uri to open URLs. resp = open("http://sub_domain.domain.com") If it contains underscore I get an error: URI::InvalidURIError: the scheme http does not accept registry part: sub_domain.domain.com (or bad hostname?) I understand …

Read More