# Problematizing the wiki corpus: an assumption ledger for four domains and twelve challenges

- Author: Shuichiro Ogawa (Design Researcher / Design Consultant), https://jamjet-iqbal.com/
- Published: 2026-08-08; updated: 2026-08-09
- Canonical (HTML): https://jamjet-iqbal.com/en/notes/wiki-corpus-problematization/
- Japanese version: https://jamjet-iqbal.com/notes/wiki-corpus-problematization/ (Markdown: https://jamjet-iqbal.com/notes/wiki-corpus-problematization.md)
- Language: en
- Keywords: problematization, research-questions, meta-research, research-strategy, assumptions

> A research-strategy note that applies Alvesson and Sandberg's problematization to the bodies of literature summarized in wiki/note across four domains (AI and design practice, generative AI and learning, the evaluation and measurement of creativity, and design process research and methodology). We surfaced 45 assumptions that each domain shares without questioning them, sorted into five layers (in-house theory, root metaphor, paradigm, ideology, and field), and developed the 12 assumptions worth challenging into an alternative ground, an RQ, a qualitative study design, and target venues. The note responds to an earlier audit, which found that gap-spotting novelty cannot be defended in peer review, by strengthening the candidates through assumption-reversing question generation. Questions that restate the author's own theory were excluded. An external novelty audit added later judged the 12 challenges (9 survive with limits, 3 substantially pre-empted) and revised the ranking substantially. The main lesson of the audit is that almost every problematization of an assumption already has a precedent, and novelty remains only on the side of empirical execution.

## Purpose and method

The novelty audit in [Extracting candidates for peer-reviewed papers: a cross-wiki analysis that excludes the author's own theory](https://jamjet-iqbal.com/en/notes/paper-candidate-extraction/) showed that gap-spotting claims ("no study has done X") cannot be defended in peer review.
In response, this note changes the method of question generation to **problematization**, which identifies and challenges the assumptions that a field's literature shares (Alvesson and Sandberg 2011).
The procedure follows the five layers organized in [Where Do Novel Research Questions Come From? The Four Loci and Their Combinations, Examined Against the Literature](https://jamjet-iqbal.com/en/notes/novel-research-questions-literature/): in-house assumptions of a theory, root metaphors shared by a field, paradigm, ideology, and field assumptions.

The material was the literature that wiki/note summarizes in four domains, and an independent subagent examined each domain.
We excluded questions that would reinvent assumptions already challenged within the corpus: the symmetry of subtraction in [The Creativity of Subtraction: The Additive Bias, and the Hypothesis of Designing Absence](https://jamjet-iqbal.com/en/notes/subtractive-creativity-design/), deviation as error in [Recombination of the Known and Nonhuman Reasoning: Can Hallucination Be Treated as a Resource for Creativity?](https://jamjet-iqbal.com/en/notes/hallucination-recombination-diversity/), the in-cognition assumption of co-evolution in RQ-11 of [Connecting design theory and sociology: where does novelty get lost?](https://jamjet-iqbal.com/en/notes/design-sociology-research-questions/), and the reversal of assumptions in AI research in general in the [Where Are the Gaps in AI Research? Seven Voids Found by Reframing](https://jamjet-iqbal.com/en/notes/ai-research-gaps-abduction/) series.
We also excluded questions that restate the author's own theory.
We surfaced 45 assumptions across the four domains, and this note develops the 12 that passed an assessment of their worth as challenges (breadth of sharing, size of consequences, credibility of the alternative, and distance from the mainstream).

## Domain 1: AI and design practice and the profession

Basis: [90% Adoption x 10% Approval — The Paradox of AI Tool Diffusion and Evaluative Divergence](https://jamjet-iqbal.com/en/notes/adoption-approval-paradox/), [From Maker to Editor: A Structural Analysis of the Designer Role Transition in the Age of AI](https://jamjet-iqbal.com/en/notes/maker-to-editor-paradigm/), [The Academic Foundations of Agentic Experience (AX): Where Human-Agent Interaction Research Stands](https://jamjet-iqbal.com/en/notes/agentic-experience-literature/), [AX and Design: A Cross-Comparison of Academic and Industry Perspectives](https://jamjet-iqbal.com/en/notes/agentic-experience-design-synthesis/), [AI in the Design Industry: An Academic Review (2026)](https://jamjet-iqbal.com/en/notes/ai-in-design-literature/), [Designer Careers That Will Generate Value over the Next Five Years — Academic Review (2026)](https://jamjet-iqbal.com/en/notes/designer-career-value-literature/), [Structural Factors behind Designer Pay Stagnation — Academic Review (2026)](https://jamjet-iqbal.com/en/notes/designer-salary-stagnation-literature/), [Design Systems as AI's Foundation Layer — Structured Design Knowledge Determines Agent Accuracy](https://jamjet-iqbal.com/en/notes/design-systems-ai-infrastructure/), [The Genealogy of Generative AI Engineering and Its Assessment from a Design Perspective](https://jamjet-iqbal.com/en/notes/ai-engineering-taxonomy-design/)

Of the 12 assumptions surfaced in this domain, we dropped two: the norm that adoption and attitude agree (partly mainstream in labor process theory) and de-skilling as loss (low independence from the three selected challenges). We selected three.

### Challenge 1-1: judgment is an authority, not a capability

- **Assumption**: "The human residue that AI cannot replace is judgment and taste, which can be defined, taught, and measured as cognitive capabilities inherent in individuals." This assumption runs through all nine notes (judgment types turned into educational assessment, taste turned into benchmarks, and career advice to cultivate curated taste).
- **Alternative ground**: Judgment is the warrant and answerability that an organization grants to a particular position. Hughes's license and mandate, Abbott's theory of jurisdiction, and workplace studies support this view. An inconsistency within the corpus (the statement that judgment may have value and yet whether it can be billed is a separate matter) disappears if value is treated as a function of the distribution of authority rather than of capability. The reason that the editor role remains is also reversed, from "humans judge better" to "a recipient of answerability for failure is needed."
- **RQ**: In design reviews where AI outputs are adopted or rejected, through which interactional and institutional procedures, and to whom, is the standing to be recognized as "having judged" distributed? How does the introduction of AI reorganize this distribution?
- **Study design**: Workplace ethnography in two or three design organizations (6 to 12 months of recorded observation of review meetings, interactional analysis of adoption and rejection declarations, tracing of disputes over the attribution of judgment, and interviews by job level).
- **Venues**: CSCW / CHI, Human Relations, Work, Employment and Society, Design Studies.

### Challenge 1-2: AI is an endogenous infrastructure that designers themselves lay down, not an exogenous shock

- **Assumption**: "AI is a technological shock that arrives from outside the design profession, and practice is a recipient that adapts to it." DID-style causal inference and acceptance research as a whole assume exogeneity.
- **Alternative ground**: The design profession supplies the conditions under which AI performs design work (design tokens, DESIGN.md, eval criteria). The basis is a disconnect within the corpus: [Design Systems as AI's Foundation Layer — Structured Design Knowledge Determines Agent Accuracy](https://jamjet-iqbal.com/en/notes/design-systems-ai-infrastructure/) records the facts but does not ask about their occupational implication (designers are writing the conditions of their own replacement). The social construction of technology and infrastructure studies give theoretical legitimacy to this view.
- **RQ**: When designers encode design knowledge into machine-readable infrastructure, through what negotiations is the choice of what to encode and what to keep tacit made, and how does it reorganize jurisdiction and the distribution of value within the profession?
- **Study design**: Longitudinal participant observation of design system teams (tracing boundary-drawing decisions through version histories and meeting records, with repeated interviews of those who encode and those who are encoded). The design adapts the methods accumulated in the ethnography of standardization.
- **Venues**: CSCW, Science, Technology, & Human Values, Design Issues, CoDesign.

### Challenge 1-3: the editor is a temporary scaffold that fills a model's deficits, not an end point

- **Assumption**: "The designer's role is moving from maker to editor, and editor-type work is the lasting form of the job in the AI era."
- **Alternative ground**: We apply Anthropic's harness design principle (each component encodes an assumption about what the model cannot do on its own and can be removed as capability improves) to human roles. Human evaluation and oversight of outputs are harness components made of humans, and the corpus already records the progress of automated evaluation. The advice to "cultivate editor skills" may be an investment in a disappearing scaffold.
- **RQ**: In design organizations that have introduced automated evaluation, from which parts are human editing and oversight tasks delegated, and how are the remaining tasks redescribed? Is what remains determined by the scarcity of capability or by the distribution of responsibility?
- **Study design**: A comparative case study of two or three organizations that have introduced or are introducing eval pipelines (repeated visits over 12 to 18 months). The study constructs rival hypotheses that can be discriminated in the same field as Challenge 1-1 (if capability scarcity decides, the tasks disappear with automation; if a recipient of responsibility decides, they remain in a changed form).
- **Venues**: CHI / CSCW, Design Studies, She Ji, Organization Science.

## Domain 2: generative AI, learning, and education

Basis: [Does Early Use of Generative AI Inhibit the Formation of Thought? A Literature Map of Cognitive Offloading and Learning](https://jamjet-iqbal.com/en/notes/ai-cognitive-offloading-learning/), [Patterns of Technological Harm to Education, and Their Refutations: A Robustness Assessment Across 25 Works](https://jamjet-iqbal.com/en/notes/technology-education-harm-patterns/), [Designing Failure Into Learning: From Expectation Failure to Productive Failure and Industry Implementations](https://jamjet-iqbal.com/en/notes/failure-driven-learning/), [The Debate Over Designing Failure Into Learning: Four Lineages Versus Cognitive Load Theory](https://jamjet-iqbal.com/en/notes/failure-driven-learning-debate/), [Research Currents in AI, Education, and the Learning Sciences: Evidence on Generative AI and Learning (2024–2026)](https://jamjet-iqbal.com/en/notes/ai-education-learning-sciences-trends/), [Issues for Design Education (Not Craft Education): From the Evidence on Failure Design and Cognitive Offloading](https://jamjet-iqbal.com/en/notes/design-education-issues-beyond-craft/), [AI Adaptation in Design Education — The Current State and Structural Challenges of Curriculum Reform](https://jamjet-iqbal.com/en/notes/design-education-ai-adaptation/), [Scaffolding Learning and the Design of Interventions: A Literature Map of Cognitive Apprenticeship, Legitimate Peripheral Participation, and Theories of Mediation](https://jamjet-iqbal.com/en/notes/learning-scaffolding-intervention-design/), [Competency Measurement Premised on AI Use — The Aptitude x AI Skill Interaction and Measurement Frameworks](https://jamjet-iqbal.com/en/notes/ai-augmented-competency-measurement/)

Of the 11 assumptions surfaced in this domain, we dropped the acquisition metaphor (mainstream in CSCL and situated learning) and the primacy of RCTs (already countered by DBR), and selected three.

### Challenge 2-1: the validity of the removal-test paradigm

- **Assumption**: "Whether learning has occurred is judged by the performance that remains in the individual learner when AI is removed. Performance with AI may be an illusion, but performance without AI is not considered a possible illusion." This assumption uniformly defines the dependent variable of the harm studies.
- **Alternative ground**: Judging transfer requires specifying a reference environment, and the environment after graduation is saturated with AI. "What remains without AI" may be a construct tied to a reference environment that no longer exists. The decisive point is an unresolved contradiction within the corpus. [Competency Measurement Premised on AI Use — The Aptitude x AI Skill Interaction and Measurement Frameworks](https://jamjet-iqbal.com/en/notes/ai-augmented-competency-measurement/) explicitly rejects "conventional measurement of the individual without AI," whereas the harm studies on the education side treat the residue without AI as true learning. The two clusters coexist in the same field and have never confronted each other over the choice of reference environment.
- **RQ**: What does the residue of learning measured under AI removal predict, and fail to predict, about later performance and learning in AI-saturated environments? If part of it is not predicted, whose harm, and harm to what, did the harm studies detect?
- **Study design**: A longitudinal qualitative study that follows graduates of AI-restricted and AI-permitted programs into their first year of employment (retaining their in-program assessment results without AI and linking them to workplace observation, interviews, and traces of AI logs). As a supplement, an "AI-enabled transfer task" is added as an outcome to existing RCTs.
- **Venues**: Journal of the Learning Sciences, Educational Researcher, Assessment & Evaluation in Higher Education.

### Challenge 2-2: the moral economy of effort and the politics of naming concepts

- **Assumption**: "Cognitive ease and dependence on support are signs of failed learning, and laziness, debt, and dependence can be used as neutral descriptive concepts."
- **Alternative ground**: These names are moral verdicts rather than descriptions, and they can spread into policy discourse before the evidence is settled. The corpus itself holds a history of refutation (harm discourses about calculators, laptops, and screen time were named and refuted repeatedly; [Patterns of Technological Harm to Education, and Their Refutations: A Robustness Assessment Across 25 Works](https://jamjet-iqbal.com/en/notes/technology-education-harm-patterns/)). The theory of the cycle of technology panics (Orben) gives a general precedent for this ground.
- **RQ**: At what speed and with what transformations is the morally loaded vocabulary of generative AI learning research taken into policy documents, institutional guidelines, and teachers' practical discourse, independent of the empirical status of each concept, and which design decisions (prohibition, surveillance, staged restriction) does it justify?
- **Study design**: A three-part discourse study (qualitative coding of the passages where the concepts are cited, discourse analysis of policy and guideline documents, and teacher interviews).
- **Venues**: Learning, Media and Technology, Postdigital Science and Education, BJET.

### Challenge 2-3: the controllability of exposure

- **Assumption**: "Researchers and educational institutions can assign learners' exposure to AI as a condition and can open and close it as a stage of the curriculum."
- **Alternative ground**: Access to AI is an environmental infrastructure rather than an assignable treatment, and learners organize their own AI ecology across sanctioned and unsanctioned channels. Evidence within the corpus (the failure of detectors, the inverse correlation between policy awareness and reported use, and use that is not recognized as a violation) already supports this view, and the evidence of uncontrollability coexists without dialogue with design arguments that assume controllability in the same field. A "no-AI condition" may be another name for an "unsanctioned-AI condition" (contamination of the control group in an RCT).
- **RQ**: How do learners in an AI-restricted phase organize their use across sanctioned and unsanctioned channels? Given that de facto ecology, what does staged design actually design (norms, disclosure, and practice rather than access)?
- **Study design**: Digital ethnography and diary methods across several courses under restrictive policies (recording use on all devices with consent for two to three months and producing a map that contrasts de jure and de facto use).
- **Venues**: Computers & Education, CHI, Studies in Higher Education.

## Domain 3: the evaluation and measurement of creativity

Basis: [Recent Currents in LLM-as-a-Judge: A Literature Map of 53 Core Studies and a Standalone Chapter on Creativity Evaluation](https://jamjet-iqbal.com/en/notes/llm-as-a-judge-literature/), [The Genesis of LLM-as-a-Judge: A Confluence of Three Lineages and Four Demands](https://jamjet-iqbal.com/en/notes/llm-as-a-judge-history/), llm-judge-creativity-scoping-review-outline (private), [The Scholarly Lineage of Generative Art: A 94-Item Literature Map from Information Aesthetics to the Post-LLM Era](https://jamjet-iqbal.com/en/notes/generative-art-literature/), [An Academic Map of Research That Treats Hallucination as a Resource for Creativity](https://jamjet-iqbal.com/en/notes/hallucination-as-creativity-resource-literature/), [Creativity, Situated Cognition, and Environment: A Literature Map of 4E Cognition, Affordances, and Situated Learning](https://jamjet-iqbal.com/en/notes/creativity-situated-cognition-environment/), [Activating Reframing in Design: Cognitive Mechanisms and Practical Methods](https://jamjet-iqbal.com/en/notes/reframing-activation-in-design/)

Of the 11 assumptions surfaced in this domain, we dropped three: the decomposability of rubrics (the alternative reinvents CAT), the artifact-only view (mainstream in situated cognition, although we use it as a component of Challenge 3-3), and measurability itself (mainstream in the systems model). We selected three.

### Challenge 3-1: the assumption of consensus calibration (disagreement is a signal, not error)

- **Assumption**: "The higher the agreement among raters and with human ratings, the better the evaluation, and calibration to human consensus makes an evaluator valid. Disagreement is error to be minimized." Judge meta-evaluation as a whole is organized around agreement indices.
- **Alternative ground**: A transformatively new artifact is transformative precisely because existing evaluative norms cannot adjudicate it, and the structured disagreement of raters who share established norms may be a primary signal that the artifact touches the boundary of those norms. The consistent failure that judge research itself reports ("originality and expert quality are not reached") may reflect a wrong choice of calibration target rather than a lack of capability.
- **RQ**: Does the structure of disagreement among experts (who splits from whom, on which dimension, and in what vocabulary) predict later changes in an artifact's evaluation (canonization, reevaluation) better than consensus-based scores?
- **Study design**: Conversation analysis of crits and competition juries (describing the grammar by which split judgments are handled) combined with a historical cohort study of jury record archives (the relation between the structure of disagreement at the time and later reevaluation).
- **Venues**: Creativity Research Journal, PACA, Design Studies, NeurIPS Datasets and Benchmarks (for its implications for redesigning agreement indices).



### Challenge 3-3: the assumption of provenance-blind evaluation (sensitivity to provenance is not bias)

- **Assumption**: "Evaluation should be independent of an artifact's provenance, and an evaluator's sensitivity to provenance information is a bias to be removed." This assumption is the core of the debiasing program.
- **Alternative ground**: In the philosophy of art, provenance is a constituent of aesthetic judgment (Dutton's account of forgery, in which evaluation is an evaluation of achievement and achievement is relative to the conditions of making, and Walton's account of categories). On this ground, sixty years of label effects (ratings fall when raters learn that a machine made the work) are reread as evidence that raters evaluate achievement rather than as a distortion on the receiving side, and a judge stripped of provenance measures a different construct (the surface properties of the output).
- **RQ**: In the judgments of expert critics, which judgment functions (attribution of achievement, reconstruction of intention, choice of comparison class) does provenance information serve? As the measurement of what construct should provenance-blind evaluation be redescribed?
- **Study design**: Think-aloud and discourse analysis (having critics review the same set of works under three conditions, with no provenance, the true provenance, and a false provenance, and analyzing changes in the vocabulary of judgment and the frame of reference rather than the scores).
- **Venues**: PACA, Empirical Studies of the Arts, Journal of Aesthetics and Art Criticism, CHI.

## Domain 4: design process research and methodology

Basis: design-exploration-evaluation-genealogy (private), [Connecting design theory and sociology: where does novelty get lost?](https://jamjet-iqbal.com/en/notes/design-sociology-research-questions/), design-sociology-blindspots-literature (private), [Is the Primacy of Qualitative Methods in Design Research Academically Mainstream?](https://jamjet-iqbal.com/en/notes/design-research-methodology-mainstream/), [Methods and Validity in Design Research: A Literature Map of Case Study, Research through Design, and Mixed Methods](https://jamjet-iqbal.com/en/notes/design-research-methods-validity/), [Qualitative and Quantitative Research: In the Context of Design](https://jamjet-iqbal.com/en/notes/qualitative-quantitative-design-research/), [The Epistemological Premises and Methodological Foundations of Design: A Literature Map of Wicked Problems, Abduction, and Set-Based Exploration](https://jamjet-iqbal.com/en/notes/design-epistemology-methodology-foundations/), [The Nexus of Design and Social Science: A Genealogy of Methods, Theories, and Institutions](https://jamjet-iqbal.com/en/notes/design-social-science-nexus/), [The Many Dimensions of Design Value: A Literature Map from Instrumental to Relational and Recognitive Value](https://jamjet-iqbal.com/en/notes/design-value-dimensions/)

Of the 12 assumptions surfaced in this domain, we dropped three: the window assumption of protocol analysis (already challenged by Lloyd and colleagues), the search metaphor (already countered by C-K theory), and design as a positive value (already called into question by critical design research). We selected three.

### Challenge 4-1: the third-culture thesis

- **Assumption**: "Design knowledge is a distinct mode that differs from both the natural sciences and the humanities, and this distinctness grounds methodological autonomy" (the lineage of Cross's designerly ways).
- **Alternative ground**: This claim of distinctness rests on an asymmetric comparison. The design side was described through empirical observation of practice, whereas science, the term of contrast, was represented by its textbook normative self-description (the hypothetico-deductive method). The laboratory studies of STS, which the field itself imported, show that science as practice is also opportunistic, converses with its materials, and leaps between frames. If the terms of comparison are aligned as practice against practice, the features regarded as "designerly" may be features of skilled practice under uncertainty in general.
- **RQ**: When design practice and experimental science practice are observed with the same coding framework, can the indicators regarded as "designerly" (reframing the problem, simultaneous updating of problem and solution, abductive leaps) discriminate between the two? If they cannot, to what should the argument for methodological autonomy be moved?
- **Study design**: A paired focused ethnography of design studios and experimental laboratories. Alternatively, blind application of the same coding scheme to published DTRS-type protocols and transcripts of scientific practice (testing whether coders can identify the domain from the transcript).
- **Venues**: Design Issues, Design Studies, She Ji, Social Studies of Science, DRS.

### Challenge 4-2: the assumption of the project as the unit

- **Assumption**: "The design process occurs in units of projects that have a start and an end, and observing it in those units captures the properties of the process." This is the working assumption of protocol studies, co-evolution models, and the whole validity apparatus of case studies.
- **Alternative ground**: Project boundaries are not natural seams in design activity. They are containers that the accounting practices of contracting, billing, and organizing impose on continuous activity. The absence of a stopping rule (Rittel and Webber) supports this view from the inside: if the problem itself has no stopping rule, every observed end point was supplied by the institution. Some of the features that process research has reported as properties of the process may be properties of the container. Infrastructuring and design-after-design (Ehn) are mainstream on the side of use, but they have not reached the validity apparatus of design cognition and process research.
- **RQ**: How are problem frames and constraints carried over, mutated, or lost across project boundaries (delivery, dissolution of the team, change of vendor)? How much of the findings of process research are properties of the project as an accounting container?
- **Study design**: A longitudinal embedded case study that follows a single ecology of artifacts across three or more generations of subsequent projects (reconstructing a "biography of the problem frame" through archival analysis and serial interviews with those in charge across generations).
- **Venues**: Design Studies, CoDesign, JCSCW, Organization Studies.

### Challenge 4-3: the exemption from reflexivity

- **Assumption**: "The theories of valuation that design research applies to its objects (commensuration, reactivity, evaluative infrastructure) need not be applied to the apparatus of design research's own knowledge production (trustworthiness, the four RtD criteria, peer review norms)."
- **Alternative ground**: Methodological criteria are also evaluative infrastructure and are reactive. However, the mechanism of reactivity has been formulated together with commensuration (compression into a common scale), whereas trustworthiness and the four RtD criteria are non-commensurating qualitative criteria designed to resist quantification, and the base theory does not guarantee that reactivity occurs in an evaluation apparatus that lacks commensuration. This failure of the mapping is the core of the question (a design that does not fall into the covering type under the demotion norm of [Connecting design theory and sociology: where does novelty get lost?](https://jamjet-iqbal.com/en/notes/design-sociology-research-questions/)).
- **RQ**: Do qualitative methodological criteria designed to resist quantification nevertheless reshape the questions and forms of submitted research? If they do, how is the mechanism described as reactivity without commensuration?
- **Study design**: A diachronic discourse analysis of the justification sections of accepted papers before and after the four RtD criteria were proposed (2007), together with semi-structured interviews with reviewers and editors of design research journals (a description of evaluative cultures modeled on Lamont's study of peer review).
- **Venues**: Design Studies, Design Issues, Valuation Studies, Science, Technology, & Human Values, Minerva.

## Cross-cutting synthesis

The 12 challenges were generated independently, but they converge in three ways.
The first convergence is on **the reactivity of evaluation apparatus**. Challenge 3-2 (evaluators change generation), Challenge 4-3 (the reactivity of methodological criteria), and the existing RQ-11 (evaluative infrastructure drives co-evolution) belong to the same family: each maps Espeland and Sauder's reactivity onto a different object (a machine evaluator inside the generation loop, qualitative methodological criteria, and the evaluative infrastructure of design organizations).
Taken together, the three constitute one research program (the co-evolution of evaluation apparatus and its objects).
The second is that the strongest challenges are those that target **unresolved contradictions within the corpus**. For Challenge 2-1 (harm studies and competence measurement studies have not confronted each other over the choice of reference environment) and Challenge 1-2 (the fact that designers themselves build the context layer of AI coexists with the view of an exogenous shock), the evidence for the alternative ground can be supplied from within the corpus.
The third is that all 12 selected challenges can be carried out with qualitative study designs (ethnography, conversation and discourse analysis, discourse studies, longitudinal case studies, think-aloud), which is consistent with the narrowing to qualitative research in [Extracting candidates for peer-reviewed papers: a cross-wiki analysis that excludes the author's own theory](https://jamjet-iqbal.com/en/notes/paper-candidate-extraction/).

Following the lesson of the earlier audit, these RQs also require an external novelty audit (identification of the closest prior work) before submission.
The check in this note on whether an assumption has become mainstream is limited to what the corpus describes.

## Ranking by promise (working hypothesis; provisional before the audit, and the external novelty audit below takes precedence)



2. **Challenge 2-1, the validity of the removal test**: It has the most defensible footing, an unresolved contradiction within the field itself.
3. **Challenge 1-1, judgment as authority**: It is highly feasible (an ethnography of review meetings), and it can discriminate rival hypotheses with Challenge 1-3 in the same field.
4. **Challenge 1-2, endogenous infrastructure**: The evidence for the alternative ground can be supplied from within the corpus, and it has a clear field in design system teams.
5. Then, in order: Challenge 4-2 (the project as the unit), Challenge 2-3 (the controllability of exposure), Challenge 3-3 (provenance-blind evaluation), Challenge 4-3 (the exemption from reflexivity), Challenge 3-1 (disagreement as signal), Challenge 2-2 (moral economy), Challenge 1-3 (the editor as scaffold), and Challenge 4-1 (the third-culture thesis).

## External novelty audit (added 2026-08-09)

For all 12 challenges, we conducted an audit that identified the closest prior work through an external literature search.
Nine survived with limits, three were substantially pre-empted, and none survived intact.

| Challenge | Judgment | Closest prior work | Form that survives |
|---|---|---|---|
| 1-1 Judgment as authority | Survives with limits | Elish's moral crumple zone, the liability sink argument, Pachidi and colleagues' regime of knowing, the AMJ study of workplace jurisdiction | The theoretical core is pre-empted. Bring it down to an empirical test at the interactional level of design reviews (the grammar of adoption and rejection declarations) |
| 1-2 Endogenous infrastructure | Survives with limits | Theories of knowledge codification, the CWA record of self-automation, the CSCW 2025 study of reconfigured division of labor, Grant's educational data infrastructuring | Limit the target of the attack on the exogeneity assumption to DID-style economics and acceptance-model research. Longitudinal observation of design systems being made AI-readable is a gap |
| 1-3 The editor as scaffold | Substantially pre-empted | Gray and Suri's paradox of the last mile, Bainbridge, the 2025 application of Bainbridge to design | Does not stand alone. Absorb it into the discriminating study of 1-1 (capability scarcity vs. distribution of responsibility) |
| 2-1 Removal test | Survives with limits | Dawson, the assessment reform school of Lodge and colleagues (TEQSA discussion paper), Bearman and colleagues' account of evaluative judgement | The normative confrontation has already taken place. Limit it to "an empirical test of the predictive validity of performance without AI" and "a Messick-type critique of the dependent variable of the harm studies" |
| 2-2 Moral economy | Substantially pre-empted | Bearman and colleagues' discourse analysis, the JIME special issue on metaphors, Feeling AI, critiques of the dependency concept | Every component is already occupied. Absorb it as a section of 2-1, or reduce it to a measurement of the speed at which vocabulary travels |
| 2-3 Control of exposure | Survives with limits | Beane's shadow learning, the Hong Kong study of policy trajectories, empirical evidence of detector failure | Conceptualize it as shadow AI learning. A longitudinal description of usage ecologies through device-level diary methods is a gap |
| 3-1 Disagreement as signal | Survives with limits | Perspectivism in NLP and CrowdTruth (institutionalized), two negative results on scalar disagreement (in PNAS 2022, reviewer disagreement was unrelated to novelty; in F1000 2018, disagreement was unrelated to later citations) | Start from the negative precedents and limit it to "structured disagreement × art × diachronic canonization" |



| Challenge | Judgment | Closest prior work | Form that survives |
|---|---|---|---|
| 3-3 Provenance-blind evaluation | Theory pre-empted | Dutton and Walton are canonical, and Lopes 2026 and Nannicelli 2025 have applied them to AI in JAAC | Cite the philosophical thesis as a premise, and limit the contribution to an empirical decomposition of experts' judgment functions and its translation into judge design |
| 4-1 Third culture | Concept pre-empted | The debate between Farrell and Hooker (2012, 2013) and Galle and Kroes exists in Design Studies | Reframe it as "the first empirical adjudication of the debate" (a test of the discriminant validity of coding schemes) |
| 4-2 The project as the unit | Survives with limits | Pollock and Williams's biography of artifacts, critical project studies, infrastructuring | All prior work can be turned into allies. Targeting the validity apparatus of design cognition with the "biography of the problem frame" as the unit has not been claimed |
| 4-3 Exemption from reflexivity | Substantially pre-empted | Buus and colleagues on COREQ performativity (2025), Dahler-Larsen's constitutive effects, Gaver 2012 | Withdraw the gap claim of "reactivity without commensuration." Limit it to epistemic criteria or integrate it into 3-2 |

### The ranking after the audit

The audit almost reversed the working ranking.
The top two before the audit (3-2 and 2-1) faced heavy precedents for their components or normative claims, and 1-2 and 2-3, which were in the middle before the audit, became the best.

1. **1-2 Endogenous infrastructure**: The field (design system teams) and the phenomenon (negotiation over where to draw the line of encoding) are highly specific, and academic research has not started on it despite the accumulation of practitioner discourse.
2. **2-3 Shadow AI learning**: The destination for transferring Beane's theoretical frame is vacant, and it bridges education research and workplace studies.
3. **4-2 Biography of the problem frame**: A direct attack on the target (the validity apparatus of process research) has not been claimed. The condition is a design that will not be read as a simple application of BOAP.



5. **1-1 Attribution practices of judgment** (absorbing the discrimination of 1-3).
6. **2-1 Predictive validity of the removal test** (as an empirical program).
7. **4-1 Empirical adjudication of the debate**.
8. **3-1 Structured disagreement**.
9. **3-3 Empirical decomposition of the judgment functions of provenance**.

Dropped: 1-3 (absorbed into 1-1), 2-2 (absorbed into 2-1 or reduced), 4-3 (integrated into 3-2 or limited to epistemic criteria).

### The main lesson of the audit

For almost all 12 challenges, someone had already put the problematization itself (the exposure that "the field takes X for granted") into print.
Novelty that survives peer review remains only on the side of empirical execution in a specific field and unit of analysis (adjudication, longitudinal description, discrimination), not in the claim of problematization.
Question generation in the manner of Alvesson and Sandberg raises the quality of candidates, but external audits and empirical designs still have to secure novelty.
[Writing Material for a Review Article: The Assumption Ledger and Nearest-Neighbour Literature Left by 17 Novelty Audits](https://jamjet-iqbal.com/en/notes/review-paper-writing-material/) rearranges this lesson and the closely related literature that the audit identified as material for a review paper.

## Unverified items

The external audit created the following checks that are required before starting.

- The scope of the TiCS 2026 review and the DIS 2024 paper (decides the judgment on 3-2)
- The full text of Black Box Assessment 2026 (2-1)
- The full text of the AMJ paper on workplace jurisdiction (1-1)
- Forward citation checks of Farrell and Hooker 2013, Hyysalo and colleagues 2019, and Buus and colleagues 2025 (4-1, 4-2, 4-3)
- Confirming the bibliographic record of the CSCW 2025 study of reconfigured division of labor (1-2)

The audit relied on general web searches in English; systematic searches of Scopus, Web of Science, and the ACM DL, literature in languages other than English, and paywalled full texts have not been checked.

All the evidence for this analysis rests on what wiki/note describes.
A claim of absence ("not present in the corpus") is an absence within the literature that the notes summarized, not a proof of absence in the field as a whole.
The literature outside the corpus cited as alternative ground was cited from memory, and its primary sources have not been checked.
These are Hughes, Perkins's person-plus, Sfard, Orben's cycle of technology panics, Espeland and Sauder, Messick, Dutton, Walton, Latour and Woolgar, Knorr Cetina, Ehn, and Lamont.
When a challenge is chosen for work, an external novelty audit of the same type as in [Extracting candidates for peer-reviewed papers: a cross-wiki analysis that excludes the author's own theory](https://jamjet-iqbal.com/en/notes/paper-candidate-extraction/) must be conducted.

## References

This list gives the external literature named in the text, grouped by area, with bibliographic records checked against Crossref, OpenAlex, and arXiv.
Notes in this wiki are linked from the text and are not listed here.

### Methods of question generation

- Alvesson, M. & Sandberg, J. (2011). Generating research questions through problematization. *Academy of Management Review* 36(2), 247–271. https://doi.org/10.5465/amr.2009.0188

### Judgment, authority, and professional jurisdiction (Challenges 1-1 and 1-3)

- Hughes, E. C. (1958). *Men and Their Work*. Free Press.
- Abbott, A. (1988). *The System of Professions: An Essay on the Division of Expert Labor*. University of Chicago Press. https://doi.org/10.7208/chicago/9780226189666.001.0001
- Elish, M. C. (2019). Moral crumple zones: Cautionary tales in human-robot interaction. *Engaging Science, Technology, and Society* 5, 40–60. https://estsjournal.org/index.php/ests/article/view/260
- Lawton, T., Morgan, P., Porter, Z. et al. (2024). Clinicians risk becoming "liability sinks" for artificial intelligence. *Future Healthcare Journal* 11(1), 100007. https://doi.org/10.1016/j.fhj.2024.100007
- Pachidi, S., Berends, H., Faraj, S. & Huysman, M. (2021). Make way for the algorithms: Symbolic actions and change in a regime of knowing. *Organization Science* 32(1), 18–41. https://doi.org/10.1287/orsc.2020.1377
- Kim, B., Rezazade Mehrizi, M. H. & Bailey, D. E. (2025). Interlacing situated and algorithmic modes of knowledge work: A workplace jurisdiction perspective. *Academy of Management Journal* 68(5), 907–938. https://doi.org/10.5465/amj.2023.1286
- Gray, M. L. & Suri, S. (2019). The paradox of automation's last mile. *Ghost Work*. https://ghostwork.info/2019/01/paradox-of-automations-last-mile/
- Bainbridge, L. (1983). Ironies of automation. *Automatica* 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
- Shukla, P., Bui, P., Levy, S. S., Kowalski, M. et al. (2025). De-skilling, cognitive offloading, and misplaced responsibilities: Potential ironies of AI-assisted design. *CHI EA '25*. https://doi.org/10.1145/3706599.3719931
- Anthropic (2026). Harness design for long-running apps. https://www.anthropic.com/engineering/harness-design-long-running-apps

### Codification of knowledge and infrastructure (Challenge 1-2)

- Cowan, R. & Foray, D. (1997). The economics of codification and the diffusion of knowledge. *Industrial and Corporate Change* 6(3), 595–622. https://doi.org/10.1093/icc/6.3.595
- Communications Workers of America (2025). Ghost workers in the AI machine. https://cwa-union.org/ghost-workers-ai-machine
- Clarke, M., Joffe, M. et al. (2025). Beyond replacement or augmentation: How creative workers reconfigure division of labor with generative AI. arXiv:2505.18938. https://arxiv.org/abs/2505.18938 `[要出典確認: 書誌未確定 Crossref と arXiv で CSCW 2025 の掲載を確認できず]`
- Grant, L. (2024). Data infrastructuring in schools: New forms of professional edu-data expertise and agency. *Research in Education* 120(1), 54–71. https://doi.org/10.1177/00345237241257177

### Assessment of learning and AI (Challenges 2-1, 2-2, and 2-3)

- Dawson, P. (2020). Cognitive offloading and assessment. In *Re-imagining University Assessment in a Digital World* (pp. 37–48). Springer. https://doi.org/10.1007/978-3-030-41956-1_4
- Lodge, J. M. et al. (2023). *Assessment Reform for the Age of Artificial Intelligence*. TEQSA. https://www.teqsa.gov.au/sites/default/files/2023-09/assessment-reform-age-artificial-intelligence-discussion-paper.pdf
- Bearman, M., Tai, J., Dawson, P., Boud, D. & Ajjawi, R. (2024). Developing evaluative judgement for a time of generative artificial intelligence. *Assessment & Evaluation in Higher Education* 49(6), 893–905. https://doi.org/10.1080/02602938.2024.2335321
- Winstone, N., Gravett, K. & Elkington, S. (2026). Black box assessment: Rethinking integrity and learning for a time of generative AI. *Assessment & Evaluation in Higher Education*. https://doi.org/10.1080/02602938.2026.2661365
- Messick, S. (1995). Validity of psychological assessment. *American Psychologist* 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
- Perkins, D. N. (1993). Person-plus: A distributed view of thinking and learning. In G. Salomon (Ed.), *Distributed Cognitions: Psychological and Educational Considerations*. Cambridge University Press.
- Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. *Educational Researcher* 27(2), 4–13. https://doi.org/10.3102/0013189X027002004
- Orben, A. (2020). The Sisyphean cycle of technology panics. *Perspectives on Psychological Science* 15(5), 1143–1157. https://doi.org/10.1177/1745691620919372
- Bearman, M., Ryan, J. & Ajjawi, R. (2023). Discourses of artificial intelligence in higher education: A critical literature review. *Higher Education* 86(2), 369–385. https://doi.org/10.1007/s10734-022-00937-2
- Ferreira, G., Costello, E., Farrow, R., Lee, K. et al. (2025). Metaphors of AI in education: Discourses, histories and practices. *Journal of Interactive Media in Education* 2025(1). https://doi.org/10.5334/jime.1077
- Heinsfeld, B. & Veletsianos, G. (2025). The language on GenAI: A critical exploration of personification metaphors in UNESCO's guidance for generative AI. *Journal of Interactive Media in Education* 2025(1). https://doi.org/10.5334/jime.983
- Oberg, G., Liang, Y., Bearman, M., Fawns, T. et al. (2026). Feeling AI: Circulating emotions, institutional climates, and moral boundaries in student use of AI. *Higher Education*. https://doi.org/10.1007/s10734-026-01658-6
- He, X., Li, S., Cha, R., Wang, L. et al. (2026). Artificial intelligence dependency in college students: A critical narrative review. *Frontiers in Psychology* 17. https://doi.org/10.3389/fpsyg.2026.1808268
- Beane, M. (2019). Shadow learning: Building robotic surgical skill when approved means fail. *Administrative Science Quarterly* 64(1), 87–123. https://doi.org/10.1177/0001839217751692
- Tsao, J. W. (2025). Trajectories of AI policy in higher education: Interpretations, discourses, and enactments of students and teachers. *Computers and Education: Artificial Intelligence* 9, 100496. https://doi.org/10.1016/j.caeai.2025.100496
- Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T. et al. (2023). Testing of detection tools for AI-generated text. *International Journal for Educational Integrity* 19(1). https://doi.org/10.1007/s40979-023-00146-z

### Evaluation of creativity, disagreement, and reactivity (Challenges 3-1, 3-2, and 4-3)

- Xu, Y. & Jurgens, D. (2026). Beyond consensus: Perspectivist modeling and evaluation of annotator disagreement in NLP. *EMNLP 2026*, 53785–53805. https://doi.org/10.18653/2026.emnlp-main.2370
- Aroyo, L. & Welty, C. (2015). Truth is a lie: Crowd truth and the seven myths of human annotation. *AI Magazine* 36(1), 15–24. https://doi.org/10.1609/aimag.v36i1.2564
- Teplitskiy, M., Peng, H., Blasco, A. & Lakhani, K. R. (2022). Is novel research worth doing? Evidence from peer review at 49 journals. *PNAS* 119(47). https://doi.org/10.1073/pnas.2118046119
- Barnett, A. G., Glisson, S. R. & Gallo, S. A. (2018). Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study. *F1000Research* 7, 1030. https://doi.org/10.12688/f1000research.15479.1
- Espeland, W. N. & Sauder, M. (2007). Rankings and reactivity: How public measures recreate social worlds. *American Journal of Sociology* 113(1), 1–40. https://doi.org/10.1086/517897
- Cheon, E. (2024). Examining algorithmic metrics and their effects through the lens of reactivity. *DIS '24*, 3179–3192. https://doi.org/10.1145/3643834.3660676
- Sourati, Z., Ziabari, A. S. & Dehghani, M. (2026). The homogenizing effect of large language models on human expression and thought. *Trends in Cognitive Sciences* 30(9), 805–816. https://doi.org/10.1016/j.tics.2026.01.003



### Provenance and aesthetic judgment (Challenge 3-3)

- Dutton, D. (1979). Artistic crimes: The problem of forgery in the arts. *British Journal of Aesthetics* 19(4), 302–314. https://doi.org/10.1093/bjaesthetics/19.4.302
- Walton, K. L. (1970). Categories of art. *The Philosophical Review* 79(3), 334–367. https://doi.org/10.2307/2183933
- Lopes, D. M. (2026). AI art and artists: What they are, what they could be, what they should be. *Journal of Aesthetics and Art Criticism* 84(2), 247–258. https://doi.org/10.1093/jaac/kpag001
- Nannicelli, T. (2025). Mass AI-art: A moderately skeptical perspective. *Journal of Aesthetics and Art Criticism* 83(4), 343–357. https://doi.org/10.1093/jaac/kpaf026

### Design process research and methodology (Challenges 4-1, 4-2, and 4-3)

- Cross, N. (1982). Designerly ways of knowing. *Design Studies* 3(4), 221–227. https://doi.org/10.1016/0142-694X(82)90040-0
- Farrell, R. & Hooker, C. (2012). The Simon–Kroes model of technical artifacts and the distinction between science and design. *Design Studies* 33(5), 480–495. https://doi.org/10.1016/j.destud.2012.05.001
- Farrell, R. & Hooker, C. (2013). Design, science and wicked problems. *Design Studies* 34(6), 681–705. https://doi.org/10.1016/j.destud.2013.05.001
- Galle, P. & Kroes, P. (2014). Science and design: Identical twins? *Design Studies* 35(3), 201–231. https://doi.org/10.1016/j.destud.2013.12.002
- Latour, B. & Woolgar, S. (1986). *Laboratory Life: The Construction of Scientific Facts* (2nd ed.). Princeton University Press. https://doi.org/10.1515/9781400820412
- Knorr-Cetina, K. D. (1981). *The Manufacture of Knowledge: An Essay on the Constructivist and Contextual Nature of Science*. Pergamon Press.
- Rittel, H. W. J. & Webber, M. M. (1973). Dilemmas in a general theory of planning. *Policy Sciences* 4(2), 155–169. https://doi.org/10.1007/BF01405730
- Binder, T., De Michelis, G., Ehn, P., Jacucci, G., Linde, P. & Wagner, I. (2011). *Design Things*. MIT Press. https://doi.org/10.7551/mitpress/8262.001.0001
- Karasti, H. (2014). Infrastructuring in participatory design. *PDC '14*, 141–150. https://doi.org/10.1145/2661435.2661450
- Pollock, N. & Williams, R. (2010). e-Infrastructures: How do we know and understand them? Strategic ethnography and the biography of artefacts. *Computer Supported Cooperative Work* 19(6), 521–556. https://doi.org/10.1007/s10606-010-9129-4
- Hyysalo, S., Pollock, N. & Williams, R. (2019). Method matters in the social study of technology: Investigating the biographies of artifacts and practices. *Science & Technology Studies* 32(3), 2–25. https://doi.org/10.23987/sts.65532
- Zimmerman, J., Forlizzi, J. & Evenson, S. (2007). Research through design as a method for interaction design research in HCI. *CHI '07*, 493–502. https://doi.org/10.1145/1240624.1240704
- Gaver, W. (2012). What should we expect from research through design? *CHI '12*, 937–946. https://doi.org/10.1145/2207676.2208538
- Buus, N., Ong, B., Einboden, R., Juel, A. et al. (2025). Constructing research quality: On the performativity of the COREQ checklist. *Qualitative Health Research* 36(6), 619–629. https://doi.org/10.1177/10497323251323225
- Dahler-Larsen, P. (2014). Constitutive effects of performance indicators: Getting beyond unintended consequences. *Public Management Review* 16(7), 969–986. https://doi.org/10.1080/14719037.2013.770058
- Lamont, M. (2009). *How Professors Think: Inside the Curious World of Academic Judgment*. Harvard University Press. https://doi.org/10.4159/9780674054158