What Do We Call Valid?

Design research tries to extract knowledge from both the process and the outcome while building an artifact that does not yet exist. This dual character presses a set of questions specific to methodology.

However deeply a single case is examined, what can be said from it? When one work is completed, is it the work that was completed, or the knowledge? Can another researcher retrace a structure described in language? These questions differ in kind from those for which quantitative research—measuring objects and generalizing statistically—already holds answers.

The methodology of design research has not offered one correct answer to these questions. Instead, several frameworks—case study, Research through Design, constructive design research, and mixed methods—have each taken on part of the answer. Where and how the old vocabulary of validity, reliability, and generalizability applies shifts from one framework to the next. The sections below map the literature onto these three questions, together with the question of binding them into a single study, from an angle distinct from the quality-versus-quantity axis (treated in qualitative-quantitative-design-research) and the question of whether qualitative methods are mainstream (treated in design-research-methodology-mainstream).

Why Does Case Study Withstand the “Not Representative” Objection?

Design research often chooses a design that follows a small number of cases in depth. From the logic of statistical sampling, this looks like a weakness. The objection that a small n cannot represent a population and therefore cannot generalize has been raised again and again.

In Case Study Research (2014), Yin answered this objection at the level of the framework itself. A case study is an empirical mode of inquiry that investigates a contemporary phenomenon in depth and within its real-world context, where the boundary between phenomenon and context is not clearly evident¹. Here Yin formulated analytic generalization in contrast to statistical generalization. What a case study generalizes toward is not a population but a theory. Rather than estimating a parameter from a sample, it asks whether the case supports or refutes a theoretical proposition. So even n=1 can function as a sufficient refutation of a theory.

In his 1989 Academy of Management Review article, Eisenhardt presented theory building from cases as a procedure². Case selection is not random but proceeds by theoretical sampling, chosen for theoretical reasons. Deliberately selecting extreme and contrasting cases lets constructs and their relationships emerge. Eisenhardt laid out a procedure that shuttles between within-case analysis and cross-case pattern search, testing constructs against both the data and the existing literature.

The two arguments place the validity of case study on a logic distinct from statistics. For Yin, internal validity is supported by pattern matching and explanation building, and external validity by analytic generalization¹. The “not representative” objection holds only when one misreads a case study as aiming at statistical generalization.

How Does Making Become Investigating?

A case study examines an object that already exists. Design research has another lineage that turns the act of making itself into a research method.

In his 1993 Royal College of Art research paper, Frayling divided research in art and design into three³. Research into design, research for design, and research through design. The third includes the claim that the process of making is itself inquiry, and that the artifact made carries the knowledge. Frayling offered this distinction as a still-loose sketch, drawing on cases such as materials research and development work.

It was the 2007 CHI paper by Zimmerman, Forlizzi, and Evenson that gave this loose distinction the contours of an HCI research method⁴. They proposed Research through Design (RtD) as a method that generates research knowledge through artifacts made by designers, and offered four criteria for evaluating the knowledge an artifact embodies: process, invention, relevance, and extensibility. Extensibility means that the knowledge produced is documented in a form on which others can build. Here RtD moves from mere making toward a mode of cumulative knowledge production.

In their 2010 DIS paper, Zimmerman and colleagues critically examined the theory of RtD, noting as open problems that the method still lacks a theoretical model and that the research community has no shared basis for evaluating and documenting RtD outcomes⁵. For making to become investigating, there must be a path by which what is made reaches others as knowledge. The four criteria of 2007 and the self-critique of 2010 are two faces of the same question about that path.

Constructive Design Research Opens Practice onto Three Sites

Where RtD asserted “research through making,” it was constructive design research that systematized that assertion as a methodology.

In Design Research Through Practice (2011), Koskinen, Zimmerman, Binder, Redström, and Wensveen defined constructive design research as research in which design, construction, and making occupy a central part of knowledge inquiry⁶. They organize this practice onto three sites. The Lab, which tests hypothetical questions under controlled conditions; the Field, which enters the context of people’s lives; and the Showroom, which exhibits works to provoke discussion. The three sites correspond, respectively, to the traditions of psychological experiment, ethnography, and the criticism of art and design.

What makes this tripartition work as a methodology is that it makes explicit how criteria of validity change by site. In the Lab, control and reproducibility; in the Field, fidelity to context; in the Showroom, the quality of the discussion a work provokes—each supports the trustworthiness of the knowledge. Koskinen and colleagues argued that design research need not bind itself to a single epistemology; it may choose a site according to the question and take on the rigor specific to that site. Constructive design research can be read as a framework that rebundles Frayling’s loose sketch and Zimmerman et al.’s HCI method along the axis of the site of practice.

Validity and Reliability Were Forged Separately in Two Traditions

Both case study and RtD must ultimately answer the question, “is this knowledge trustworthy?” The vocabulary that guarantees trustworthiness developed separately in the quantitative and the qualitative traditions.

In Quasi-Experimentation (1979), Cook and Campbell built a framework for protecting causal inference under field conditions where experiments cannot be fully controlled⁷. They divide validity into four kinds. Statistical conclusion validity, internal validity, construct validity, and external validity. By systematizing threats to internal validity in particular (history, maturation, selection, regression, and others), they made it possible to inspect in advance which threats might intrude, and how, even in quasi-experiments without a control group. The stage of verifying a design’s effect through experiment falls within the reach of this vocabulary.

These four kinds of validity do not transfer as-is to qualitative research. In Naturalistic Inquiry (1985), Lincoln and Guba proposed criteria of trustworthiness specific to naturalistic inquiry⁸. Four criteria: credibility (corresponding to internal validity), transferability (external validity), dependability (reliability), and confirmability (objectivity). These reorganize the meaning of trustworthiness to fit the constructivist ontology, in which realities are multiple and constructed through the interaction of researcher and object. Transferability is not generalization to a population but the reader’s judgment, enabled by thick description, of whether findings transfer to their own context.

The two traditions are not in competition; they ask about trustworthiness in different places. Cook and Campbell’s framework works where variables are manipulated and effects measured. Lincoln and Guba’s framework works where context is described from within. Insofar as design research contains both exploration and evaluation in one process, both vocabularies are needed.

How Far Is Generalization from Cases Justified?

Analytic generalization (Yin) and transferability (Lincoln & Guba) both try to secure the external reach of case study by a road other than statistical generalization. Yet a persistent doubt has been directed at the very claim that “one can generalize from a case.”

In his 2006 Qualitative Inquiry article, Flyvbjerg confronted five misunderstandings about case study research⁹. The one that touches the core of methodology is his rebuttal to the misunderstanding that “one cannot generalize from a single case, so it contributes nothing to scientific development.” Flyvbjerg shows the power of a carefully chosen critical case. If a case is selected to occupy the position “if this does not hold for this case, it holds for no case (or vice versa),” a single case can refute a proposition. Drawing on Galileo’s experiment on falling bodies and Popper’s swans, Flyvbjerg argued that it is the assumption—that formal generalization is the only source of scientific knowledge—that is overrated.

Flyvbjerg’s second point is to decouple generalization from learning. Even if a case cannot be formally generalized, this does not mean it contributes nothing to collective knowledge formation. Expertise grows through the accumulation of context-dependent case knowledge, and case study is a legitimate mode of that accumulation. This argument reinforces Yin’s analytic generalization from another angle. To Yin’s procedure of “generalizing toward theory,” Flyvbjerg supplies a grounding: how to select a case with the power to refute and to teach.

Mixed Methods Demand a Logic of Integration

Qualitative and quantitative methods ask about trustworthiness in separate places. So when both are used within one study, how do the two answers become one?

In 2010, Tashakkori and Teddlie edited the SAGE Handbook of Mixed Methods in Social & Behavioral Research (2nd ed.), positioning mixed methods as the third methodological movement in the social and behavioral sciences¹⁰. They systematized mixed methods as an approach that integrates qualitative and quantitative data, analysis, and inference within a single study or a connected program of studies, and grounded its epistemology in a pragmatism that chooses methods by what answers the question. This position tries to dissolve the either/or of quality versus quantity under the higher criterion of fit to the question.

In Designing and Conducting Mixed Methods Research (2011, 2nd ed.), Creswell and Plano Clark brought mixed methods down to workable designs¹¹. They present core design types. The explanatory sequential and exploratory sequential designs, which conduct qualitative and quantitative phases at different times, and the convergent parallel design, which conducts both at once and compares the results. In every type, the key is to make explicit, within the design, the point of interface at which the qualitative and quantitative results are integrated. Merely using two methods together does not make mixed methods. A juxtaposition lacking a logic of integration is no more than two independent studies placed side by side.

Mixed methods connect the other frameworks of this literature map into one process. At the exploration stage, qualitative methods describe phenomenon and context (Lincoln & Guba’s vocabulary applies); at the evaluation stage, quantitative methods measure effect (Cook & Campbell’s vocabulary applies). Creswell and Plano Clark’s sequential designs give a concrete form that links these two stages along a time axis. Whether integration can be justified by pragmatism alone, however, remains an open point. The criterion of what works can be a reason to choose a method, but it does not by itself supply the logic that binds findings from two epistemologies into a description of the same reality.

The Division of Labor Across the Questions

The literature above answers, each in a different place, the three questions posed at the outset, together with the question of binding them into a single study.

What can be said from a single case is answered by Yin’s analytic generalization and Flyvbjerg’s critical case. Generalization runs toward a theory rather than a population, and a carefully chosen case has the power to refute and to teach.

What is produced when a work is completed is answered by Frayling’s three-part division and Zimmerman et al.’s four criteria. What is produced is the work and, if documented in a form on which others can build, knowledge as well. Koskinen et al.’s three sites show that the trustworthiness of that knowledge rests on different criteria depending on whether the site is Lab, Field, or Showroom.

Whether a structure described in language can be retraced is answered by Lincoln and Guba’s dependability and confirmability. Reproduction in qualitative research is replaced not by reproducing the same result but by making the path of inquiry auditable.

And how to bundle a design research that contains both exploration and evaluation in one study is answered by the mixed methods of Tashakkori and Teddlie and of Creswell and Plano Clark, together with the demand to design the point of integration. The one tension that remains lies in how far pragmatism can justify that integration. This question shows that the logic binding two separately forged vocabularies—the validity of quantitative research and the trustworthiness of qualitative research—into a description of the same reality is not yet fully in hand.

References

Unverified Items

• The DOI for ⁸ may have been assigned to an Elsevier review article about the book rather than the book itself; whether it is accurate as the identifier for the book proper has not been primary-verified ([verification needed]). The book’s bibliographic record (SAGE, 1985) is established.
• The DOI for ⁶ (an Elsevier book identifier) may be represented differently across editions and printings; not primary-verified ([verification needed]).
• The DOI for ¹⁰ refers to the electronic edition on the SAGE Knowledge platform; the identifier differs between the first edition (2003) and the second (2010). Whether it is accurate as the second-edition identifier has not been primary-verified ([verification needed]).
• ¹, ⁷, ¹¹, and ³ carry no DOI, being books or a research paper. The bibliographic records are established, but the exact page range of Frayling (1993) may vary by edition.

Footnotes

1. Yin, R. K. (2014). Case Study Research: Design and Methods (5th ed.). SAGE Publications. ↩ ↩² ↩³

2. Eisenhardt, K. M. (1989). Building Theories from Case Study Research. Academy of Management Review, 14(4), 532–550. https://doi.org/10.5465/amr.1989.4308385 ↩

3. Frayling, C. (1993). Research in Art and Design. Royal College of Art Research Papers, 1(1), 1–5. ↩ ↩²

4. Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through Design as a Method for Interaction Design Research in HCI. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘07), 493–502. https://doi.org/10.1145/1240624.1240704 ↩

5. Zimmerman, J., Stolterman, E., & Forlizzi, J. (2010). An Analysis and Critique of Research through Design: Towards a Formalization of a Research Approach. Proceedings of the 8th ACM Conference on Designing Interactive Systems (DIS ‘10), 310–319. https://doi.org/10.1145/1858171.1858228 ↩

6. Koskinen, I., Zimmerman, J., Binder, T., Redström, J., & Wensveen, S. (2011). Design Research Through Practice: From the Lab, Field, and Showroom. Morgan Kaufmann. https://doi.org/10.1016/C2010-0-65896-2 ↩ ↩²

7. Cook, T. D., & Campbell, D. T. (1979). Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin. ↩ ↩²

8. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. SAGE Publications. https://doi.org/10.1016/0147-1767(85)90062-8 ↩ ↩²

9. Flyvbjerg, B. (2006). Five Misunderstandings About Case-Study Research. Qualitative Inquiry, 12(2), 219–245. https://doi.org/10.1177/1077800405284363 ↩

10. Tashakkori, A., & Teddlie, C. (Eds.). (2010). SAGE Handbook of Mixed Methods in Social & Behavioral Research (2nd ed.). SAGE Publications. https://doi.org/10.4135/9781506335193 ↩ ↩²

11. Creswell, J. W., & Plano Clark, V. L. (2011). Designing and Conducting Mixed Methods Research (2nd ed.). SAGE Publications. ↩ ↩²