I'm all for this idea.
It adds an extra level of strategy, for those looking for more challenge. (the randomness of dice, just isn't enough strategy sometimes)
Again to reiterate Ditocoaf's point, it would be an option to play, not a requirement, so don't worry it will slow things down too much, as ...