Remove ^@ Characters in a Unix File

unix sed character tr invisible
Manikanta G picture Manikanta G · Aug 6, 2015 · Viewed 8k times · Source

I have a question about removing invisible characters which can be only be seen when we try to view the file using "vi" command. We have a file that's being generated by Datastage Application (Source is a DB2 Table -> Target is a .txt File). File has data with different data types. I'm having an issue with just 3 columns which have their datatypes defined as CHAR.

If you open the file in a Textpad you'd see spaces. But when you view the same file on Unix via vi command, we see ^@ characters in blue color. My file is a delimiter file with the delimiter as ^@^ (I know it's kinda sounds weird) .

I have tried:

  1. tr -d [:cntrl:] <Filename >NewFileName — Still no luck — [Delimiters are removed but the spaces remain]
  2. tr -s "^@" <Filename >NewFilename — Still no luck — I see file reduce in file size but the invisible characters still stay.
  3. Tried changing the delimiter — but still see the same invisible characters.
  4. Used sed "s/^@/g/" (and other sed commands) <Filename — still no luck.

Any suggestions are really appreciated. I have researched the posts on this website but I couldn't find one. If it's a simple please excuse me and share your thoughts.

Answer

Robby Cornelissen picture Robby Cornelissen · Aug 6, 2015

In vi, NUL characters are represented as ^@. To get rid of them:

tr

Using tr, you should be able to remove the NUL characters as follows:

tr -d '\000' < file-name > new-file-name

Related questions

Find and replace in file and overwrite file doesn't work, it empties the file

I would like to run a find and replace on an HTML file through the command line. My command looks something like this: sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html > index.html When …

Read More
How can I extract a predetermined range of lines from a text file on Unix?

I have a ~23000 line SQL dump containing several databases worth of data. I need to extract a certain section of this file (i.e. the data for a single database) and place it in a new file. I know both …

Read More
Bash tool to get nth line from a file

Is there a "canonical" way of doing that? I've been using head -n | tail -1 which does the trick, but I've been wondering if there's a Bash tool that specifically extracts a line (or a range of lines) from a …

Read More