When to refactor

When you add a function
When you need to fix a bug
When you do a code review

The rule of 3 :
1st time you write some code
2nd time you do something similar, you sill do it
3rd time you do something similar, you refactor.