I had a discussion a few weeks back with some co-workers on refactoring, and I seem to be in a minority that believes "Refactor early, refactor often" is a good approach that keeps code from getting messy and unmaintainable. A number of other people thought that it just belongs in the maintenance phases of a project.
If you have an opinion, please defend it.
Answer
Just like you said: refactor early, refactor often.
Refactoring early means the necessary changes are still fresh on my mind. Refactoring often means the changes tend to be smaller.
Delaying refactoring only ends up making a big mess which further makes it harder to refactor. Cleaning up as soon as I notice the mess prevents it from building up and becoming a problem later.