I really like this list of heuristics by Itamar Gilad for how much data you should have before launching product features/improvements:

• Never launch anything solely based on opinions.
• It’s ok to release minor tweaks based on assessment.
• Having supporting data (without testing) is enough to launch only very small, low-risk, easy-to-revert changes.
• Everything else should be validated through tests and experiments. However there are various levels of testing to choose from with different associated costs and confidence levels.  
• How much validation you need depends on: a) the cost of building the idea in full, b) how risky it is, and c) your risk tolerance. 

It’s also the first time I’ve come across his confidence calculator, which looks like a really useful tool (email-gated, though…).