I installed Drush with sudo apt-get install drush.
How can I remove it now?
Just had the same prob, installing drush that way will get you something that doesn't work at all...
try this:
sudo apt-get remove drush
Then install it like this:
sudo wget --quiet -O - http://ftp.drupal.org/files/projects/drush-7.x-5.1.tar.gz | sudo tar -zxf - -C /usr/local/share
sudo ln -s /usr/local/share/drush/drush /usr/local/bin/drush
sudo drush