Encoding::UndefinedConversionError

ruby encoding sinatra sequel
martriay picture martriay · Oct 22, 2012 · Viewed 43.2k times · Source

I keep getting an Encoding::UndefinedConversionError - "\xC2" from ASCII-8BIT to UTF-8 every time I try to convert a hash into a JSON string. I tried with [.encode | .force_encoding](["UTF-8" | "ASCII-8BIT" ]), chaining .encode with .force_encoding, backwards, switching parameters but nothing seemed to work so I caught the error like this:

begin
  menu.to_json
rescue Encoding::UndefinedConversionError
  puts $!.error_char.dump
  p $!.error_char.encoding
end

Where menu is a sequel's dataset.to_hash with content from a MySQL DB, utf8_general_ci encoding and returned this:

"\xC2"

<#Encoding:ASCII-8BIT>

The encoding never changes, no matter what .encode/.force_encoding I use. I've even tried to replace the string .gsub!(/\\\xC2/) without luck.

Any ideas?

Answer

martriay picture martriay · Jan 3, 2013 
menu.to_s.encode('UTF-8', invalid: :replace, undef: :replace, replace: '?')

This worked perfectly, I had to replace some extra characters but there are no more errors.

Related questions

How to convert a Net::HTTP response to a certain encoding in Ruby 1.9.1?

I have a Sinatra application (http://analyzethis.espace-technologies.com) that does the following Retrieve an HTML page (via net/http) Create a Nokogiri document from the response.body Extract some info and send it back in the response. The response …

Read More
ruby 1.9: invalid byte sequence in UTF-8

I'm writing a crawler in Ruby (1.9) that consumes lots of HTML from a lot of random sites. When trying to extract links, I decided to just use .scan(/href="(.*?)"/i) instead of nokogiri/hpricot (major speedup). The problem is that …

Read More
How to convert a string to UTF8 in Ruby

I'm writing a crawler which uses Hpricot. It downloads a list of strings from some webpage, then I try to write it to the file. Something is wrong with the encoding: "\xC3" from ASCII-8BIT to UTF-8 I have items …

Read More