Open Nav
Logo Toluna

Quota Balancing–Delivering Best-in-Class Innovation Testing

Published Sep 18, 2020

Debbie Senior, VP, Product Automation

Share this

We’ve recently launched a new enhancement to our quota balancing sampling feature which is designed to ensure that we continue to deliver the highest quality results. 

Speed and cost-efficient insights are essential for companies to quickly move ahead with their innovation lifecycles and insights agencies must ensure they can deliver the full spectrum of client needs. Despite these parameters, and the increasing focus on platforms and technology to deliver these insights, this should not come at the cost of quality. It is essential to continue applying best practice research methodologies and techniques which includes sample/quota balancing. Sample/quota balancing ensures that differences between tested concepts are real and not a facet of uneven sample composition. Speed should not always be traded against quality, and the balance and quality of sample composition is significant based on what is being measured.

Quota balancing ensures that respondents have an equal chance of being assigned to a particular concept. Companies often need to target their surveys and set quotas on specific audiences by country, region, gender, income age, and so on. For concept and pack testing especially, companies also want to put a quota on behavioural elements such as brand usage and/or consumers with different frequency of usage, which may be combined with other quotas. Quota balancing ensures a consistent proportion of the target audience is seeing and rating each concept or pack being tested.  

Obtaining genuine results with a balanced sample.

In the hypothetical example below, we have two concepts being tested monadically in a survey reaching females aged 18-24 and 25-34. In the unbalanced sample example, you’ll notice that the age groups aren’t balanced, and we don’t know if results may be skewed based on the different sample compositions. This might cause us to doubt the decisions we make based on the data. In this example, Concept 2 is the winner among females 18-24 and Concept 1 is the winner among females 25-34. 

If we balance the sample so each concept is seen by an even spread of the target audience, we remove potential sampling bias and we can be confident in our decision-making. Here we get a different outcome, where Concept 2 is the overall winner for everyone. 

Unbalanced sample – we don’t know if purchase intent results are real due to difference in sample balancing Balanced sample – the sample is balanced so our results are genuine 
Concept/Pack 1  Purchase intent T2B Concept/Pack 2  Purchase intent T2B Concept/Pack 1  Purchase intent T2B Concept/pack 2 Purchase intent T2B
Females 18-24 40% of completed interviews  65% 60% of completed interviews 75% 50% of completed interviews  70% 50% of completed interviews 75%
Females 25-34 60% of completed interviews 75% 40% of completed interviews 65% 50% of completed interviews 65% 50% of completed interviews 75%

 

We can also go further than simply balancing the demographic composition. If we then add in custom screening and also include consumers who use two specific brands and buy the brand frequently (i.e. at least once a week), then sample balancing would enable the following to happen:

Concept/Pack 1 – 200 interviews total Concept/Pack 2 – 200 interviews total
Females 18-24 Brand A frequent users = 50 completed interviews Females 18-24 Brand A frequent users = 50 completed interviews
Females 18-24 Brand B frequent users = 50 completed interviews Females 18-24 Brand B frequent users = 50 completed interviews
Females 25-34 Brand A frequent users = 50 completed interviews Females 25-34 Brand A frequent users = 50 completed interviews
Females 25-34 Brand B frequent users = 50 completed interviews Females 25-34 Brand B frequent users = 50 completed interviews

 

The application of quota balancing ensures a level sample playing field across all items being tested to deliver consistent analysis and insights. The only difference in results can be attributed to the stimulus, not because different audience proportions have seen different concepts or packs. 

Apply quality sampling balancing to your surveys, every time.

In a traditional research world, quota balancing is typically considered a hygiene factor, something that is automatically applied as a best practice. As with all hygiene elements, it can become critical if it isn’t applied or done correctly.  

The next time you ask your provider to set up and deliver a concept or pack survey for you, or you do it yourself via a self-serve platform, it’s important to verify how sample balancing works and confirm that you’re getting the quality you need to take a better and more accurate decision on which of your concepts is most likely to succeed. 

*monadic is often deemed best practice as it eliminates any potential bias given each respondent only sees one concept /pack/ad and isn’t influenced by any other stimulus/options in the survey.  Sequential monadic design presents at least two concepts/packs/ads to a respondent within the same survey with the order randomised across the survey. 

 

 

 

 

 

Share this