Where the Question Lies

The preceding note (qualitative-quantitative-design-research) concluded with four claims: (1) Design is an exploratory activity with deep epistemological affinity to qualitative research. (2) Qualitative methods are suited to the exploratory phase, while quantitative methods are suited to the evaluative phase. (3) Mixed methods provide the methodological scaffolding that bridges both phases. (4) The choice of method depends on the nature of the research question. Is this view mainstream in the academic field of design research? If so, what sustains it? Are there opposing positions? And how does a research agenda focused on “designing conditions that enable people to try again” relate to this view?

The Methodological Consensus in Design Research

The recognition that design research possesses a distinctive mode of knowledge, irreducible to the natural sciences, became established from the 1990s onward. In a 1993 Royal College of Art research paper, Frayling classified research in design into three categories: “research into design,” “research through design,” and “research for design”¹. In 1995, Archer positioned “research through practice” as a form of inquiry specific to design, alongside the traditions of science and the humanities². In his 2006 monograph, Cross formulated “designerly ways of knowing” as a “third culture” distinct from both scientific and humanistic methods, empirically documenting designers’ distinctive cognition through protocol analysis³.

What these arguments share is the claim that design knowledge cannot be subsumed under hypothesis-testing positivism. In a 2011 paper, Dorst characterized the core of design thinking as a particular form of abduction — not ordinary abduction (abduction-1), which applies a known solution frame to a novel situation, but “abduction-2,” in which both the problem frame and the solution are generated simultaneously⁴. The defining feature that the problem frame itself is not given in advance entails epistemological premises fundamentally different from those of quantitative methods that operationally define variables and test hypotheses.

The fact that the representative methods of design research all belong to the qualitative tradition further corroborates this affinity. Chai and Xiao’s bibliometric analysis (2012), covering fifteen years of the journal Design Studies (1996—2010), classified research themes and found that “design process” and “design cognition” constituted the two dominant themes, and that protocol analysis was the most widely employed method in design cognition research⁵. Protocol analysis, case studies, and ethnography are all qualitative methods that construct data from language and action rather than numerical measurement.

Research through design (RtD), an approach that generates knowledge through design artifacts, was methodologically articulated in the HCI context by Zimmerman, Forlizzi, and Evenson in 2007⁶. In 2008, Fallman modeled design research as a triangle of “design practice,” “design studies,” and “design exploration,” in which the researcher dynamically traverses these activities⁷. Stappers argued in 2007 that the act of designing is itself part of inquiry, and that research questions are refined through the process of prototyping⁸. These arguments share many premises with the epistemology of qualitative research: description from within the phenomenon, inductive concept generation, and the mutual constitution of researcher and subject.

Taken together, conclusion (1) of the preceding note — that design is exploratory and therefore epistemologically consonant with qualitative research — represents a mainstream view in the academic field of design research. Bayazit’s 2004 overview of forty years of design research in Design Issues confirms that methodological discourse in the field has consistently revolved around establishing independence from scientific method and articulating a distinctive research paradigm⁹.

Countervailing Positions and Tensions

Even granted its mainstream status, the proposition that “qualitative methods are suited to design research” faces at least three sources of tension.

The Quantitative Orientation in HCI

HCI is a field rooted in psychology and computer science, and at CHI (ACM Conference on Human Factors in Computing Systems), quantitative methods such as controlled experiments, usability testing, and surveys are established as standard evaluation procedures. A survey of accessibility research at CHI found that 94.3% of papers included user studies, with interviews (42.1%), usability testing (41.7%), and controlled experiments (34.6%) as the three most common methods, and 56.4% of papers employing multiple methods¹⁰. This distribution reveals a landscape in which qualitative and quantitative methods coexist. HCI does not exclude qualitative methods; rather, it maintains a norm that demands quantitative methods when comparing effects or objectively evaluating interfaces.

An epistemological tension exists between this norm and the qualitative orientation of design research. In their 2015 volume, Bardzell and Bardzell proposed “humanistic HCI,” arguing for the value of introducing humanistic methods — literary criticism, critical interpretation, feminist design — into HCI¹¹. That such a proposal needed to be made explicitly testifies both to the existence of a countercurrent against the quantitative orientation within HCI and to the strength of HCI’s quantitative norms.

The Quantitative Approach of Evidence-Based Design

In the domains of architecture and healthcare facilities, evidence-based design (EBD) has developed its own methodological tradition. EBD quantitatively verifies the physical and psychological effects of the built environment on occupants through hypothesis formulation, measurement, and post-occupancy evaluation¹². Randomized controlled trials (RCTs) are regarded as the most rigorous form of evidence, and quantitative methods take precedence in evaluating the effects of healthcare facility design.

The existence of EBD delimits the scope of the proposition that “qualitative methods are mainstream in design research.” In domains where the effects of an intervention can be objectively measured — such as architecture and healthcare — quantitative methods can occupy the center of design research. Conclusion (2) of the preceding note (that quantitative methods are suited to the evaluative phase) is consistent with EBD practice. EBD, however, does not take the exploratory process of design itself as its object of study; it measures the effects that design outcomes (built environments) produce on users. The difference in method follows from the difference in object of inquiry.

Positions That Transcend the Qualitative-Quantitative Dichotomy

In 2012, Gaver and Bowers proposed “annotated portfolios,” arguing that design research should cultivate its own identity from its existing practices and reasoning rather than imitating the mechanisms of theory construction¹³. Annotated portfolios are a representational form that bridges individual artifacts and the broader concerns of the research community by attaching concise annotations to collections of artifacts. This form fits neatly into neither a qualitative nor a quantitative framework.

Koskinen, Zimmerman, Binder, Redstrom, and Wensveen, in their 2011 volume, formulated constructive design research as a methodology¹⁴. They argued that design research requires vocabularies drawn not only from mathematics but from the arts, cultural studies, anthropology, cognitive psychology, and communication studies.

In 2012, Hook and Lowgren proposed “strong concepts,” arguing that intermediate-level knowledge — knowledge abstracted from individual artifacts at a level between instances and theory — constitutes a form of knowledge contribution specific to interaction design research¹⁵.

These arguments share the position that design research should develop its own epistemology and knowledge forms rather than positioning itself within the qualitative-quantitative dichotomy. In relation to the preceding note’s conclusions, claim (1) — that design research has affinity with qualitative research — is not negated, but a more precise account of design research’s epistemological position requires relativizing the qualitative-quantitative axis itself.

Researching Conditions and Epistemological Commitments

How does a research agenda concerned with “studying the conditions under which human potential is opened up,” “analyzing how evaluation and institutional structures affect people’s willingness to take on challenges,” and “designing conditions that enable a second attempt” relate to the methodological landscape outlined in the preceding note?

Qualitative Methods Precede the Description of Conditions

To study “the structure by which evaluation forecloses opportunity” is to describe under what conditions a particular institution or setting enables or impedes people’s willingness to try. Such structures cannot be operationally defined as variables in advance. Understanding why a single evaluation or failure affects the subsequent range of opportunities requires qualitative methods that describe from within — the experience of the person involved, the operation of the institution, the chain of contextual factors.

In sociology, Merton’s 1968 formulation of the “Matthew effect” — cumulative advantage — provides an established theoretical framework for describing structures in which a single evaluative event distorts the subsequent allocation of opportunity¹⁶. The process by which initially trivial differences are amplified over time, such that two once-comparable researchers come to have markedly different careers, cannot be captured by quantitative data alone. Qualitative methods such as case studies and institutional analysis are indispensable for understanding how institutional mechanisms (selection criteria, resource allocation rules, the timing of evaluation) generate cumulative advantage.

Sen’s capability approach carries similar epistemological implications. Sen conceived of development as the expansion of “what a person is actually able to do and to be” (functionings and capabilities), asking not about the quantity of income or resources but about the conditions under which people can exercise substantive freedom¹⁷. Nussbaum applied this framework to education, arguing that institutions bear the responsibility of guaranteeing a threshold level of individual capabilities¹⁸. Aggregate statistics such as enrollment rates cannot reveal what children actually experience in school. Research that interrogates conditions requires qualitative methods capable of describing the intersection of individual experience and institutional context.

The Methodological Tension in Designing Conditions

The methodological challenge confronting research on “designing conditions that enable a second attempt” corresponds to the preceding note’s schema of “qualitative methods for the exploratory phase, quantitative methods for the evaluative phase.” This correspondence, however, is not straightforward.

Between the phase of describing the structure of conditions (qualitative) and the phase of verifying whether designed conditions actually enable people to explore (quantitative), there is an epistemological discontinuity. The description of conditions rests on an interpretivist epistemology. The verification of effects, by contrast, demands some form of positivist procedure. The preceding note argued that mixed-methods pragmatism could bridge this gap, but the pragmatist criterion of “what works” may prove insufficient for research on conditions.

Bhaskar’s critical realism offers one epistemological framework for addressing this tension. Critical realism stratifies reality into three domains — the empirical, the actual, and the real — and positions the elucidation of “generative mechanisms” underlying observable events as the purpose of research¹⁹. Within Bhaskar’s framework, “the structure by which evaluation forecloses opportunity” is located in the “real” domain as a generative mechanism. This structure is not directly observable, yet it possesses causal powers that, under specific conditions, produce events (a person gives up on a challenge, or becomes unable to try again).

The methodological implication of critical realism is that it justifies the use of both qualitative and quantitative methods while grounding their integration not in “pragmatism” (what works) but in “retroduction” (inference from observed events to underlying mechanisms)¹⁹. For research on conditions, this framework holds the potential to integrate qualitative description of structures with quantitative measurement of the effects of altered conditions under a single epistemological roof.

Situating the Research Agenda Within the Preceding Note’s Framework

The preceding note organized methodological choice in design research as “dependent on the nature of the question.” Applied to the study of conditions: the question “What becomes visible when we examine a state of immobility not as a deficit of individual aptitude but as a product of surrounding conditions?” is exploratory and calls for qualitative methods. The question “How should settings and institutions be designed?” involves design intervention and connects to the methodology of research through design. And the question of whether the designed setting or institution actually produces the effect of “enabling a second attempt” requires some form of verification.

The methodological challenges this research agenda faces both support and expose the limits of the preceding note’s framework. What they support is the principle that the choice between qualitative and quantitative methods depends on the nature of the question — a principle that holds here as well. What they expose is that pragmatism alone cannot satisfy the epistemological demands of research on conditions. Integrating qualitative research that describes conditions and quantitative research that measures the effects of altering conditions within a single research program may require an epistemological framework — such as critical realism — that admits the existence of structures and mechanisms.

References

Unverified Items

• Author names, exact publication year, and conference name for ¹⁰ have not been confirmed against the primary source. The entry is based on search results; the DOI has not been directly verified.
• For ¹², Ulrich et al. (2008) is cited as a representative review in evidence-based design, but full bibliographic details (volume, issue, page numbers, DOI) have not been confirmed against the primary source.
• Dorst’s cited paper was published in 2011, but a separate 2008 paper (“Design research: a revolution-waiting-to-happen,” Design Studies, 2008) may exist. This note relies on the 2011 paper. The existence and content of the 2008 paper have not been verified.
• No meta-study reporting the overall distribution of methods (qualitative vs. quantitative) across CHI as a whole was identified in this survey. ¹⁰ covers the accessibility research subset and is not representative of CHI as a whole.

Footnotes

1. Frayling, C. (1993). Research in Art and Design. Royal College of Art Research Papers, 1(1). ↩

2. Archer, B. (1995). The Nature of Research. Co-design, 2, 6—13. ↩

3. Cross, N. (2006). Designerly Ways of Knowing. Springer. https://doi.org/10.1007/1-84628-301-9 ↩

4. Dorst, K. (2011). The core of ‘design thinking’ and its application. Design Studies, 32(6), 521—532. https://doi.org/10.1016/j.destud.2011.07.006 ↩

5. Chai, K.-H., & Xiao, X. (2012). Understanding design research: A bibliometric analysis of Design Studies (1996—2010). Design Studies, 33(1), 24—43. https://doi.org/10.1016/j.destud.2011.06.004 ↩

6. Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘07), 493—502. https://doi.org/10.1145/1240624.1240704 ↩

7. Fallman, D. (2008). The Interaction Design Research Triangle of Design Practice, Design Studies, and Design Exploration. Design Issues, 24(3), 4—18. https://doi.org/10.1162/desi.2008.24.3.4 ↩

8. Stappers, P. J. (2007). Doing design as a part of doing research. In R. Michel (Ed.), Design Research Now: Essays and Selected Projects (pp. 81—91). Birkhauser. https://doi.org/10.1007/978-3-7643-8472-2_6 ↩

9. Bayazit, N. (2004). Investigating Design: A Review of Forty Years of Design Research. Design Issues, 20(1), 16—29. https://doi.org/10.1162/074793604772933739 ↩

10. Mack, K., et al. (2021). What Do We Mean by “Accessibility Research”? A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. [Citation to be verified: author names and publication year are inferred from search results. DOI has not been directly verified.] ↩ ↩² ↩³

11. Bardzell, J., & Bardzell, S. (2015). Humanistic HCI. Morgan & Claypool (Synthesis Lectures on Human-Centered Informatics). https://doi.org/10.1007/978-3-031-02214-2 ↩

12. Ulrich, R. S., et al. (2008). A review of the research literature on evidence-based healthcare design. Health Environments Research & Design Journal, 1(3), 61—125. [Citation to be verified: DOI and full bibliographic details have not been confirmed against the primary source.] ↩ ↩²

13. Gaver, W., & Bowers, J. (2012). Annotated Portfolios. Interactions, 19(4), 40—49. https://doi.org/10.1145/2212877.2212889 ↩

14. Koskinen, I., Zimmerman, J., Binder, T., Redstrom, J., & Wensveen, S. (2011). Design Research Through Practice: From the Lab, Field, and Showroom. Morgan Kaufmann. ↩

15. Hook, K., & Lowgren, J. (2012). Strong Concepts: Intermediate-Level Knowledge in Interaction Design Research. ACM Transactions on Computer-Human Interaction, 19(3), Article 23. https://doi.org/10.1145/2362364.2362371 ↩

16. Merton, R. K. (1968). The Matthew Effect in Science. Science, 159(3810), 56—63. https://doi.org/10.1126/science.159.3810.56 ↩

17. Sen, A. (1999). Development as Freedom. Oxford University Press. ↩

18. Nussbaum, M. C. (2011). Creating Capabilities: The Human Development Approach. Harvard University Press. ↩

19. Bhaskar, R. (1975). A Realist Theory of Science. Leeds Books. For the methodological implications of critical realism, see also Danermark, B., et al. (2002). Explaining Society: Critical Realism in the Social Sciences. Routledge. ↩ ↩²