A simple question to ask, but a decision that requires considering many factors.
In thinking about base size, it is important to remember that we are using a sample of respondents and their data patterns to project what we expect would happen in the real world among the larger population that they represent. Success isn’t about Concept A outperforming a competitor benchmark in the test, but rather it’s a question of if we expect that this would occur in the marketplace.
A small sample size, while more cost efficient, may lack a sufficient level of statistical power to detect differences, provides fewer respondents for subgroup analysis, and will have larger margins of error. There is greater risk of Type II error (missed opportunity) associated with lower base sizes.
On the other hand, a bigger sample size means less risk of drawing the wrong conclusions, but obviously is more expensive. Beyond cost, the additional caveat with large base sizes is the potential Type I error (false positive). Just because two numbers are statistically significant in our test doesn’t mean that they are meaningful different in the marketplace.
We need to balance the power to find differences with the confidence that these differences actually exist in the real world.
When selecting a sample size, we recommend considering how the findings are ultimately going to be used. If the results will be used primarily for early exploration of a topic, a smaller base size of 500 may be fine. On the other hand, if the results are going to drive a go/no go decision on building a multi-million dollar factory abroad, then you may want to consider 5,000 or more!
Political pollsters almost always use a sample size of 1,000 as a sample size that has both a reasonable margin of error (+/-3%) and is not too costly. A good strategy to determine sample size is to start at n=1,000 and adjust up or down based on how much risk exposure is involved in the final decision from the research.
Still not sure on a sample size? Toluna is always happy to discuss the specifics of your project and help you to determine the best sample size.
Back to Blog
