Is there a way to remove the BOM from a UTF-8 encoded file?

ruby byte-order-mark
Abe picture Abe · Feb 16, 2011 · Viewed 20k times · Source

Is there a way to remove the BOM from a UTF-8 encoded file?

I know that all of my JSON files are encoded in UTF-8, but the data entry person who edited the JSON files saved it as UTF-8 with the BOM.

When I run my Ruby scripts to parse the JSON, it is failing with an error. I don't want to manually open 58+ JSON files and convert to UTF-8 without the BOM.

Answer

knut picture knut · Oct 15, 2011

With ruby >= 1.9.2 you can use the mode r:bom|utf-8

This should work (I haven't test it in combination with json):

json = nil #define the variable outside the block to keep the data
File.open('file.txt', "r:bom|utf-8"){|file|
  json = JSON.parse(file.read)
}

It doesn't matter, if the BOM is available in the file or not.

Andrew remarked, that File#rewind can't be used with BOM.

If you need a rewind-function you must remember the position and replace rewind with pos=:

#Prepare test file
File.open('file.txt', "w:utf-8"){|f|
  f << "\xEF\xBB\xBF" #add BOM
  f << 'some content'
}

#Read file and skip BOM if available
File.open('file.txt', "r:bom|utf-8"){|f|
  pos =f.pos
  p content = f.read  #read and write file content
  f.pos = pos   #f.rewind  goes to pos 0
  p content = f.read  #(re)read and write file content
}

Related questions

How to avoid tripping over UTF-8 BOM when reading files

I'm consuming a data feed that has recently added a Unicode BOM header (U+FEFF), and my rake task is now messed up by it. I can skip the first 3 bytes with file.gets[3..-1] but is there a more …

Read More
How to write BOM marker to a file in Ruby

I have some working code with a crutch to add BOM marker to a new file. #writing File.open name, 'w', 0644 do |file| file.write "\uFEFF" file.write @data end #reading File.open name, 'r:bom|utf-8' do |file| …

Read More
How to check if a value exists in an array in Ruby

I have a value 'Dog' and an array ['Cat', 'Dog', 'Bird']. How do I check if it exists in the array without looping through it? Is there a simple way of checking if the value exists, nothing more?

Read More