Refactoring is the process of changing the structure of code without changing its behavior. It should be used to ease the addition of features. Because the outcome is code that "smells" better, sometimes people get confused and think that refactoring is an end to itself. I disagree with that sentiment. While we should strive to write code that smells good, we shouldn't spend extra effort after we finish beautifying the code. There is always the possibility that no one will need to touch that again in which case the beautifying was wasted effort. After all, the compiler doesn't care how pretty it is.
Remember this: refactoring is not an end to itself. Programmers are paid to add features or fix bugs thus making features work better. Refactoring--by itself--accomplishes neither of these. If you are touching code just to refactor it, stop. Wait until you have a need to change something first.
Here are some rules of thumb for deciding when to refactor:
The fallout of this is that, when making the first change to a piece of code, you should resist refactoring. If the change will be difficult without refactoring, then do it. However, if there is a "hack" you can do to fix the bug or add the feature, prefer that. Most of the time, ugly and quick is better than pretty and slow. If you come back to the same code again, then refactor.