Published Sep 18, 2020
We’ve recently launched a new enhancement to our quota balancing sampling feature which is designed to ensure that we continue to deliver the highest quality results.
Speed and cost-efficient insights are essential for companies to quickly move ahead with their innovation lifecycles and insights agencies must ensure they can deliver the full spectrum of client needs. Despite these parameters, and the increasing focus on platforms and technology to deliver these insights, this should not come at the cost of quality. It is essential to continue applying best practice research methodologies and techniques which includes sample/quota balancing. Sample/quota balancing ensures that differences between tested concepts are real and not a facet of uneven sample composition. Speed should not always be traded against quality, and the balance and quality of sample composition is significant based on what is being measured.
Quota balancing ensures that respondents have an equal chance of being assigned to a particular concept. Companies often need to target their surveys and set quotas on specific audiences by country, region, gender, income age, and so on. For concept and pack testing especially, companies also want to put a quota on behavioural elements such as brand usage and/or consumers with different frequency of usage, which may be combined with other quotas. Quota balancing ensures a consistent proportion of the target audience is seeing and rating each concept or pack being tested.
Obtaining genuine results with a balanced sample.
In the hypothetical example below, we have two concepts being tested monadically in a survey reaching females aged 18-24 and 25-34. In the unbalanced sample example, you’ll notice that the age groups aren’t balanced, and we don’t know if results may be skewed based on the different sample compositions. This might cause us to doubt the decisions we make based on the data. In this example, Concept 2 is the winner among females 18-24 and Concept 1 is the winner among females 25-34.
If we balance the sample so each concept is seen by an even spread of the target audience, we remove potential sampling bias and we can be confident in our decision-making. Here we get a different outcome, where Concept 2 is the overall winner for everyone.
| Unbalanced sample – we don’t know if purchase intent results are real due to difference in sample balancing | Balanced sample – the sample is balanced so our results are genuine | ||||||||
| Concept/Pack 1 | Purchase intent T2B | Concept/Pack 2 | Purchase intent T2B | Concept/Pack 1 | Purchase intent T2B | Concept/pack 2 | Purchase intent T2B | ||
| Females 18-24 | 40% of completed interviews | 65% | 60% of completed interviews | 75% | 50% of completed interviews | 70% | 50% of completed interviews | 75% | |
| Females 25-34 | 60% of completed interviews | 75% | 40% of completed interviews | 65% | 50% of completed interviews | 65% | 50% of completed interviews | 75% | |
We can also go further than simply balancing the demographic composition. If we then add in custom screening and also include consumers who use two specific brands and buy the brand frequently (i.e. at least once a week), then sample balancing would enable the following to happen:
| Concept/Pack 1 – 200 interviews total | Concept/Pack 2 – 200 interviews total |
| Females 18-24 Brand A frequent users = 50 completed interviews | Females 18-24 Brand A frequent users = 50 completed interviews |
| Females 18-24 Brand B frequent users = 50 completed interviews | Females 18-24 Brand B frequent users = 50 completed interviews |
| Females 25-34 Brand A frequent users = 50 completed interviews | Females 25-34 Brand A frequent users = 50 completed interviews |
| Females 25-34 Brand B frequent users = 50 completed interviews | Females 25-34 Brand B frequent users = 50 completed interviews |
The application of quota balancing ensures a level sample playing field across all items being tested to deliver consistent analysis and insights. The only difference in results can be attributed to the stimulus, not because different audience proportions have seen different concepts or packs.
Apply quality sampling balancing to your surveys, every time.
In a traditional research world, quota balancing is typically considered a hygiene factor, something that is automatically applied as a best practice. As with all hygiene elements, it can become critical if it isn’t applied or done correctly.
The next time you ask your provider to set up and deliver a concept or pack survey for you, or you do it yourself via a self-serve platform, it’s important to verify how sample balancing works and confirm that you’re getting the quality you need to take a better and more accurate decision on which of your concepts is most likely to succeed.
*monadic is often deemed best practice as it eliminates any potential bias given each respondent only sees one concept /pack/ad and isn’t influenced by any other stimulus/options in the survey. Sequential monadic design presents at least two concepts/packs/ads to a respondent within the same survey with the order randomised across the survey.
Back to Blog