In intellectual property matters, surveys are used to examine a variety of issues, including likelihood of confusion, secondary meaning, deceptive advertising, and consumer response to marketplace practices such as prices or packaging. Surveys are often accepted (and even expected) as evidence in venues such as Federal Court, The Trademark Trial and Appeal Board (TTAB), and the National Advertising Division (NAD) of the Better Business Bureau.
One requirement of surveys is that they replicate marketplace conditions – that is, the conditions under which a typical consumer might commonly encounter a product or service in the marketplace. The issue is important to courts and others who must decide whether a survey is reliable; as Professor McCarthy has written, “…the closer the survey methods mirror the situation in which the ordinary person would encounter the trademark, the greater the evidentiary weight of the survey results.”[1]
However, it is also widely recognized that no survey can perfectly replicate marketplace conditions, and the challenge of replicating marketplace conditions represents one of the more complex issues in designing or evaluating a survey. All surveys represent an abstraction from the marketplace, in part because surveys are designed to eliminate the effect of some factors in order to measure how consumers respond to variables of interest. The context of a survey is not the same as the actual marketplace, creating a unique environment with its own set of roles, expectations, and communication methods. As a recent article in The Trademark Reporter stated, “One critical way that surveys deviate from reality is that respondents know they are taking part in a survey rather than an actual purchase situation. Finding ways to minimize such ‘survey artifact’ effects is often the key to effective content design.”[2]
Given these considerations, the issue of marketplace conditions presents a dilemma to the survey researcher. On the one hand, a survey should provide some level of replication of marketplace conditions. On the other hand, it is not possible to match marketplace conditions exactly. This article discusses the considerations inherent in this tradeoff. We will first describe how surveys provide a unique context for communications, and then will describe how this context affects considerations regarding core survey design factors such as research mode, research method, and selection of respondents.
The Unique Context of a Survey Interview
As we consider the challenges of replicating marketplace conditions in litigation surveys, it is first worth noting the characteristics that make the survey interview a unique environment. One source has described survey responses as “…simply responses evoked in an artificial situation contrived by the researcher[3] Several characteristics make the responses provided by survey respondents different from everyday communications.
One characteristic involves the mode of communications available. In a survey interview, a live interviewer or a computer-based interviewing program provides questions that the respondent answers based on the information provided in the survey. Most surveys, including those used in intellectual property matters, do not allow two-way feedback, do not allow the respondent to ask clarifying questions, and prohibit the respondent from consulting other sources during the interview. While these rules follow generally-accepted practices for conducting interviews, they differ from a typical marketplace, where a consumer who wants more information can make inquiries of the seller or can seek additional information.
A second characteristic involves the consequences of the respondent’s actions. In the marketplace, a prospective buyer considering a purchase must decide whether to pay money in exchange for a product or service of interest, or whether to keep the money and forgo the purchase. This is true in both consumer and business contexts, even if in a business context the buyer spends money that may belong to an organization rather than to them personally.
By contrast, in a typical intellectual property survey, the respondent does not make a purchase decision, but instead answers questions about the items that are the subject of the survey. While some surveys used in intellectual property matters inquire about purchase intent or purchase likelihood, even these questions do not typically ask respondent to spend their own money. Instead, surveys typically ask the respondent to make judgments about what their attitudes and actions might involve given the situation described in the survey. Simply put, the stakes are different in a survey.
A third characteristic is the unusual nature of the task facing the respondent. During the survey interview, respondents face a series of questions for which they are asked to provide answers. No matter what instructions are provided, some survey respondents may view survey questions as a type of test in which some answers are more correct or acceptable to the interviewer. Some respondents may try to use clues from the survey interview to identify the correct or most acceptable answers. In the best case scenario, where such behavior is randomly directed and dispersed across the survey, the survey results will contain more “noise” and have lower reliability. In the worst case scenario, where the behavior systematically varies in direction or across survey stimuli or questions, the survey results can be biased and thus have lower validity.
Considered together, these and other differences mean that the survey provides an environment in which the respondent cannot ask clarifying questions, is not in an actual purchase situation, has limited information upon which to make decisions, and may try to infer clues from the survey interview. However, these are challenges that can and should be properly managed by the survey expert. When these and other issues are addressed, the survey has a number of inherent advantages that make it a highly effective way to produce probative evidence on issues critical to intellectual property matters.
I will next describe how considerations relating to marketplace conditions affect the decisions researchers make regarding specific elements of research design, using three specific examples: research mode, research stimuli, and respondent selection.
Marketplace Conditions and Research Design
Research Mode
Research mode refers to the manner in which data is gathered during a survey. Today, most litigation research is conducted either in-person or online. In-person research is often conducted by live interviewers at dedicated facilities located in or near shopping malls, while online research is typically conducted using self-administered surveys conducted over the Internet.
When considering marketplace conditions, researchers often (but not always) match the mode of data gathering to the manner in which the product is shopped for and purchased. If a product is shopped for, inspected, or purchased in person, then the research will often be conducted in person. Similarly, if a product is shopped for or purchased online, then the research is more likely to be conducted online.
While online surveys are increasingly used in litigation research,[4] litigation surveys have lagged commercial (non-litigation) surveys in the prevalence of using online surveys. Today, the majority of commercial research is conducted online due to a variety of factors, including the difficulty of reaching respondents by other modes such as telephone or in person interviews, and the availability of visuals and online research tools and methods that are difficult to replicate in alternative modes. Commercial research organizations are also more comfortable accepting the results of online research, even for products that are sold in person, because their standards and processes for evaluating and accepting research are different than litigation environments.
As a result, many surveys that would be conducted online in commercial contexts are conducted in person in litigation contexts. Many survey experts start their consideration of marketplace conditions by making sure that the research mode matches the manner in which the product is purchased.
Choice of Research Design
The consideration of marketplace conditions also affects the choice of research design. Research design refers to, among other considerations, the overall flow of questions involved in a survey and the means in which product is shown to respondents.
For example, likelihood of confusion studies are commonly conducted using either a lineup format or an Eveready format.[5] In the Eveready format, the respondent typically sees only the junior product, and is asked (among other things) who makes the product; responses that reference the senior product are taken to reflect confusion. By contrast, in a lineup study, the respondent sees a lineup of products. For example, one common implementation of a lineup study is that the respondent sees either the junior or senior product first, then sees a lineup consisting of the other product plus additional products from the category.
Marketplace conditions are important considerations in the selection between these two research designs. For example, the Eveready format is typically used only with matters involving a senior brand that is already known by respondents, while a lineup is used only “where the stimuli proximately tested in the format appear, in fact, proximately in the marketplace.”[6] When brands do not appear in immediate physical proximity in the marketplace, and are not sufficiently well-known as to support a Eveready design, researchers (under certain conditions) may choose to use a variant of a lineup study, such as a design in which products are presented sequentially rather than side-by-side in a lineup. On occasion, the same matter may involve both a lineup study and a sequential study.[7]
The decision between Eveready and lineup formats provides one example of the host of considerations relating to marketplace conditions in research design. These considerations help courts, experts, and attorneys evaluate whether a survey “…sufficiently simulated the actual marketplace conditions in which consumers encounters the parties’ products.” [8] The design of litigation surveys commonly involves a series of considerations about research design and presentation of stimuli, all of which are typically intended to create a survey that replicates marketplace conditions.
Selection of Respondents
Considerations relating to marketplace conditions also arise in selecting and qualifying respondents to ensure that the proper universe is selected for the survey.
For example, the researcher must decide who should be included in a consumer sample. For example, Professor McCarthy, in discussing secondary meaning surveys, has stressed the importance of the “reasonably informed shopper” who has “that amount of basic knowledge about the product that most people would have from news and advertising,” but is not an expert.[9] A researcher designing a survey must consider whether marketplace conditions are better reflected if the sample includes gatekeepers, such as dealers, retail buyers, or retail salespeople.
Similarly, the researcher must also decide whether marketplace conditions are better reflected if the survey sample includes past customers, prospective customers, or both. The answers depend in part on the marketplace and in part on the nature of the matter. For example, courts have indicated that including past customers in a sample for a survey on secondary meaning renders results of little value because it is meaningless to ask customers which company they bought from, as they know who they bought from. [10] As one source stated, “One’s own customers do not a fair sample make.” [11] In other matters, asking different types of questions, past customers may represent an entirely appropriate universe of respondents. For example, in a class-action matter, members of the proposed class may be the most relevant respondents if a survey is to reflect the marketplace.
Conclusions
The challenge of replicating marketplace conditions is an important one in survey design, and is too complex to completely address in a brief article. The examples provided in this article do not necessarily reflect all possible considerations for a particular case. Rather, it is our hope that these examples illustrate in a useful way the inherent trade-offs between the objective of replicating marketplace conditions and preserving the reliability and validity of the obtained survey results.
[2] M. Rappeport, “Litigation Surveys—Social ‘Science’ as Evidence,” The Trademark Reporter (2002, July-August).
[3] F.J. Fowler, Survey Research Methods (Sage, 2009).
[4] G.M. Gelb & B.D. Gelb, “Internet Surveys for Trademark Litigation: Ready or Not, Here They Come,” The Trademark Reporter (2007, September-October).
[5] See e.g., Union Carbide Corp.Houston v. Ever-Ready Inc., (Court of Appeals, 7th Cir. 1976); Exxon Corp. v. Texas Motor Exchange of, Inc., (Court of Appeals, 5th Cir. 1980); SquirtCo v. Seven-Up Co., (Court of Appeals, 8th Cir.1980).
[6] J.B. Swann, “Likelihood of Confusion Studies and the Straitened Scope of Squirt,” The Trademark Reporter (2008, May-June), p. 740.
[7] See Storck USA, LP and August Storck K.G. v. Farley Candy Co., Inc., 797, (Dist. Court, ND Illinois 1992).
[8] Thoip v. Walt Disney Co., Disney Consumer Products and Disney Destinations, (Dist. Court, SD New York 2010).
[9] McCarthy, supra note 1.
[10] See Sno-Wizard Mfg., Inc. v. Eisemann Products Co., 791 F. 2d 423 (Court of Appeals, 5th Cir. 1986).
[11] See L.E. Evans JR & D.M. Gunn, “Trademark Surveys,” The Trademark Reporter (1988).Dr. Bruce Isaacson
President
MMR Strategy Group