How to convert EBCDIC with chinese chars to UTF-8 format

utf-8 ebcdic
SNL picture SNL · Mar 11, 2011 · Viewed 9.3k times · Source

I have a requirement to convert a file with EBCDIC encoding which is encoded using the IBM937 code page to UTF-8 format for loading the file into a multi-byte enabled DB2 database.

I have tried unix recode and iconv. None of them has the ability to convert IBM 937 to UTF8. I'm looking for any utility (java, perl, unix ) in this world which can do that on a unix based system. Can someone help me here?

SL

Answer

Nicholas Carey picture Nicholas Carey · Mar 11, 2011

Take a look at ICU (International Components for Unicode): http://site.icu-project.org/

It has a converter for IBM-937: http://demo.icu-project.org/icu-bin/convexp?conv=ibm-937_P110-1999&s=ALL

CU is a mature, widely used set of C/C++ and Java libraries providing Unicode and Globalization support for software applications. ICU is widely portable and gives applications the same results on all platforms and between C/C++ and Java software. ICU is released under a nonrestrictive open source license that is suitable for use with both commercial software and with other open source or free software.

Here are a few highlights of the services provided by ICU:

  • Code Page Conversion: Convert text data to or from Unicode and nearly any other character set or encoding. ICU's conversion tables are based on charset data collected by IBM over the course of many decades, and is the most complete available anywhere.

  • Collation: Compare strings according to the conventions and standards of a particular language, region or country. ICU's collation is based on the Unicode Collation Algorithm plus locale-specific comparison rules from the Common Locale Data Repository, a comprehensive source for this type of data.

  • Formatting: Format numbers, dates, times and currency amounts according the conventions of a chosen locale. This includes translating month and day names into the selected language, choosing appropriate abbreviations, ordering fields correctly, etc. This data also comes from the Common Locale Data Repository.

  • Time Calculations: Multiple types of calendars are provided beyond the traditional Gregorian calendar. A thorough set of timezone calculation APIs are provided.

  • Unicode Support: ICU closely tracks the Unicode standard, providing easy access to all of the many Unicode character properties, Unicode Normalization, Case Folding and other fundamental operations as specified by the Unicode Standard.

  • Regular Expression: ICU's regular expressions fully support Unicode while providing very competitive performance.

  • Bidi: support for handling text containing a mixture of left to right (English) and right to left (Arabic or Hebrew) data.

  • Text Boundaries: Locate the positions of words, sentences, paragraphs within a range of text, or identify locations that would be suitable for line wrapping when displaying the text.

And much more. Refer to the ICU User Guide for details.

Related questions

UTF-8 to EBCDIC in Java

Our requirement is to send EBCDIC text to mainframe. We have some chinese characters thus UTF8 format. So, is there a way to convert the UTF-8 characters to EBCDIC? Thanks, Raj Mohan

Read More
Conversion from EBCDIC to UTF8 in Linux

I have imported with Perl a table from our database AS/400 DB2. The problem is that the string are encoded in EBCDIC Latin-1 (italian language). How can I convert the resulting file to plain utf-8 in Linux bash?

Read More
String Hex Encoding and Decoding

I am converting a String from UTF-8 to CP1047 and then performing hex encoding on it, which works great. Next what I am doing is converting back, using decoding the hex String and displaying it on console in UTF-8 format. …

Read More