Which characters need to be escaped when using Bash?
Is there any comprehensive list of characters that need to be escaped in Bash? Can it be checked just with sed?
In particular, I was checking whether % needs to be escaped or not. I tried
echo "h%h" | sed 's/%/i/g'
and worked fine, without escaping %. Does it mean % does not need to be escaped? Was this a good way to check the necessity?
And more general: are they the same characters to escape in shell and bash?
Answer
There are two easy and safe rules which work not only in sh but also bash.
1. Put the whole string in single quotes
This works for all chars except single quote itself. To escape the single quote, close the quoting before it, insert the single quote, and re-open the quoting.
'I'\''m a s@fe $tring which ends in newline
'
sed command: sed -e "s/'/'\\\\''/g; 1s/^/'/; \$s/\$/'/"
2. Escape every char with a backslash
This works for all characters except newline. For newline characters use single or double quotes. Empty strings must still be handled - replace with ""
\I\'\m\ \a\ \s\@\f\e\ \$\t\r\i\n\g\ \w\h\i\c\h\ \e\n\d\s\ \i\n\ \n\e\w\l\i\n\e"
"
sed command: sed -e 's/./\\&/g; 1{$s/^$/""/}; 1!s/^/"/; $!s/$/"/'.
2b. More readable version of 2
There's an easy safe set of characters, like [a-zA-Z0-9,._+:@%/-], which can be left unescaped to keep it more readable
I\'m\ a\ s@fe\ \$tring\ which\ ends\ in\ newline"
"
sed command: LC_ALL=C sed -e 's/[^a-zA-Z0-9,._+@%/-]/\\&/g; 1{$s/^$/""/}; 1!s/^/"/; $!s/$/"/'.
Note that in a sed program, one can't know whether the last line of input ends with a newline byte (except when it's empty). That's why both above sed commands assume it does not. You can add a quoted newline manually.
Note that shell variables are only defined for text in the POSIX sense. Processing binary data is not defined. For the implementations that matter, binary works with the exception of NUL bytes (because variables are implemented with C strings, and meant to be used as C strings, namely program arguments), but you should switch to a "binary" locale such as latin1.
(You can easily validate the rules by reading the POSIX spec for sh. For bash, check the reference manual linked by @AustinPhillips)