This advice is somewhat redundant with what I posted a few days ago but, what the heck, it's got more spicy goodness than the previous formulation.
 
Where your underlying functionality is extraordinarily fast you cannot afford to add levels of abstraction whose processing cost will dwarf the actual work that needs to be done.  This would be a case of construction with all glue and no wood. 
 
For example, if we're talking about a function that will do a few clocks worth of math, it's a bad idea to spend a few dozen clocks on assorted virtual dispatches and stack repackaging.  Even though you might get some elegant factoring of the code, it might be a bad idea to do that factoring.  The more meaty the work gets, the more compelling the factoring becomes. 
 
Said another way, just because you can factor some code into nice interfaces doesn't mean that it is a good idea to do so.
 
Know your limits, abstract responsibly.