You're dealing with two different problems here:
- detecting if a face in a portrait has hair
- "removing" the hair
The first is solvable fairly easily:
- Separate the face from the background (as you've mentioned a "simple portrait image", this shouldn't be too hard).
- Convert your image to the Y'CbCr color space
- Human skin has a fairly narrow range of chrominance values, regardless of race. Check out this paper for the details.
- The approach above will help you separate skin areas of the face from non-skin areas
- Assume that the non-skin areas consist of hair. Note that facial hair will get picked up as a non-skin area, too.
As far as the second problem goes, you need to clarify exactly what you mean by "removing":
- Are you simply cutting out the part of the portrait that has hair? In this case, the solution follows directly from the detection method above.
- Are you trying to make it look like the person has no hair at all (e.g. is bald, clean-shaven?) In this case, things will be a lot harder -- there's a reason why professional photo manipulators get paid well.