Background for the Claim Substantiation Survey

In advertising for several of its cereals, the Malt-O-Meal Brands Company (MOM) claimed that in national taste tests, some of its cereals were preferred over the comparable Post cereals. Among other claims, MOM used substantiation surveys to support claims such as, “National Taste Test WINNER Fruity Dyno-Bites preferred over Post Fruity Pebbles” and “Malt-O-Meal Dyno-Bites Cereal Preferred Over Post Pebbles Cereal.” The claims in question were used on packaging, and on television, Internet and point-of-purchase advertising.

Post challenged the claims, maintaining that, (among other issues) the preference claims were unsupported, despite MOM’s substantiation survey. Post maintained that the survey did not meet industry standards for taste test superiority claim substantiation, as outlined in the American Society for Testing and Materials’ (ASTM) Guide for Sensory Claim Substantiation. Among other flaws, Post alleged that the survey did not interview the correct universe of consumers and was not conducted in the correct geography. NAD agreed with Post and recommended that MOM discontinue its national taste preference claims. Among other issues, NAD considered who was interviewed and where the survey was conducted.  

Who to Interview in Claim Substantiation Surveys?

When it comes to research in support of claims about product performance, typically, interviews focus on the user. For preference or liking claims, the ASTM Guide states in paragraph 5.1.1 that, “Hedonic (liking and preference) claims should always apply to the user population.”

Post objected to the test methodology used to support MOM’s claims, saying that results did not accurately reflect the demographics of those who consume the product. MOM’s taste preference claims involved brands such as Fruity and Cocoa Dyno-Bites, Fruity and Cocoa Pebbles, Honey Buzzers and Honey Comb. The taste test interviewed men and women between the ages of 30 and 64 years old. Post claimed that consumers under the age of 35 make up a large percentage of users, between 38% and 82%, depending on the brand. NAD agreed with Post and found that the survey sampled the population to whom the ad was targeted and not the population that uses the products.

What could MOM have done differently? They could have included younger users in their survey. Young adults are easily recruited in mall for research such as taste tests and there are companies specialized in recruiting teens and tweens. While the costs might be higher to survey a younger audience, it is usually feasible to recruit enough respondents.

Is there a scenario where MOM may not have had to survey younger users? MOM might have changed the claim to one that would say, “Shoppers prefer Cocoa Dyno-Bites over Post Cocoa Pebbles”? Perhaps in that case, the substantiation survey could have been conducted among a universe of shoppers of relevant cereals. If a large percentage of this universe is older, such as between 30 and 64 years old, this would possibly have lessened the need to interview younger consumers. While this alternative claim may have been more or less impactful with consumers, the phrasing for this new claim might have been closer to the actual survey that was conducted.

For some types of litigation surveys, shoppers instead of users are the appropriate universe. For example, for likelihood of confusion surveys, research often measures confusion between brands or packaging and involves actual and potential purchasers seeing products or advertising. In that case, shoppers may be among the target audience.

Where to Conduct Claim Substantiation Surveys?

When it comes to surveys to substantiate preference claims, it is important to consider the scope of the claim and the geographic distribution of the target sample. The ASTM Guide states in paragraphs 5.2.8.1 and 5.2.8.2 that preference may vary by region. When geography is relevant to a claim, it says, “A national claim should be based on a sample representing major geographic regions (North, East, Midwest, South, West, etc.). A minimum of two markets in each of the four regions should be included.”

Post objected to MOM’s test methodology because in the Northeast, MOM conducted the claim substantiation survey in only one market, rather than two. NAD agreed. It stated that MOM’s claims were based on a national taste test, referring to the preferences of the overall population of all sweetened breakfast cereal consumers and that the Northeast should have been represented by two markets.

Might claims research be run in fewer regions or markets? This could be the case if the scope of the claim is regional or if product usage is regional. However, when it comes to product usage, it is important to consider the users of the category, not of a specific brand. In this case, MOM explained that for their survey, using one market in the Northeast was appropriate given the lower distribution of their product in that region. However, in NAD’s opinion, the survey geography should have included two markets in the Northeast to represent the preferences of the population of category consumers, not just MOM cereal users.

What if the Who and Where of Users are Unknown?

This case illustrates that in claim substantiation research surveys, the demographics and geographical distribution of users is important. Large consumer goods companies, such as MOM and Post, often have detailed information about user demographics, which they may use for marketing.

What about the company that wants to do research to support claims in a product category where user demographics are not readily available? How would they know who and where to survey?

One way is to use an Omnibus study. This is a quantitative study where questions on many subjects are asked at the same time among a panel of respondents with a demographic profile representative of the U.S. population.

Let’s say you wanted to do claims research with users of sunscreen but don’t know their age or geographic breakdown. In the Omnibus study you could ask respondents if they use sunscreen. Results would include a profile of respondents answering yes they use sunscreen, including information such as their age, gender, income, and geography. This might indicate who and where to conduct your claims survey.

This NAD decision provides insight about the importance of selecting who and where to survey for claim substantiation research. The decision illustrates how the survey universe should reflect users along key demographics such as age, gender, and geography. There must be a good reason for leaving out a key group, or you might find yourself with the survey equivalent of crying over spilled milk!