Ruby `split': invalid byte sequence in UTF-8 (ArgumentError)

Question

I am trying to populate the movie object, but when parsing through the u.item file I get this error:

`split': invalid byte sequence in UTF-8 (ArgumentError)

File.open("Data/u.item", "r") do |infile|
            while line = infile.gets
                line = line.split("|")
            end
end

The error occurs only when trying to split the lines with fancy international punctuation.

Here's a sample

543|Misérables, Les (1995)|01-Jan-1995||http://us.imdb.com/M/title-exact?Mis%E9rables%2C%20Les%20%281995%29|0|0|0|0|0|0|0|0|1|0|0|0|1|0|0|0|0|0|0

Is there a work around??

Answer 1

I had to force the encoding of each line to iso-8859-1 (which is the European character set)... http://en.wikipedia.org/wiki/ISO/IEC_8859-1

a=[]
IO.foreach("u.item") {|x| a << x}
m=[]
a.each_with_index {|line,i| x=line.force_encoding("iso-8859-1").split("|"); m[i]=x}