Thinking in Problem Definition: Rereading the Game Board of Existing Research

Skill at Answering and Skill at Questioning Are Different Abilities

A researcher’s craft is usually measured by the quality of the answers. More precise methods, higher performance, more robust demonstration. But line up the studies that moved a field, and there is a group that skill at answering alone cannot explain. They did not produce good answers. They turned phenomena that had not until then been recognized as problems into researchable form.

Call this ability Problem Definition. It is an ability distinct from knowing the object in detail, and it can be defined as follows.

The thinking that alters the observation apparatus, evaluation axes, time horizon, units, and boundaries that existing research implicitly sets, and turns phenomena not until then recognized as problems into researchable form.

The observation itself, that the design of the question shapes the outcome, has been confirmed across several lineages. In a longitudinal study of art students, Getzels and Csikszentmihalyi showed that the quality of problem finding, before any task is given, predicts the success of creative work years later¹. Schön located the professional’s core skill not in solving a given problem but in problem setting, the re-setting of the problem². Simon argued that in ill-structured problems the process of structuring is itself the bulk of the solving³, and Rittel and Webber stated that in wicked problems the formulation of the problem is itself the problem⁴. The work of Sandberg and Alvesson, which typologized ways of constructing research questions, shows the contrast most directly. Gap-spotting, which searches out and fills blanks in the literature, inherits the premises of existing research as they stand, whereas influential theories have come from challenging premises (problematization)⁵⁶. What follows is a toolkit that breaks this problematization down into operational parts that are easy to transfer.

The Game Board: What a Research Field Implicitly Fixes

A researcher strong in Problem Definition looks not at individual studies but at the system of certification the field shares. Call that system the game board. The board implicitly fixes at least the following eight things.

What counts as a phenomenon
What counts as the thing to be explained
What unit comparisons are made in
What counts as evidence of success and failure
Over what span of time evaluation runs
Whose viewpoint counts as a legitimate position of observation
What is discarded as noise
What state counts as the endpoint

Within any individual paper, each of these terms is a constant, not an argument. What a system of classification renders visible and what it renders invisible has been shown repeatedly by research that takes classification itself as its object⁷. Because each term of the board is inscribed in the field’s measuring apparatus and publishing institutions, it never becomes an object of questioning so long as one plays on the board. Problem Definition is the operation that turns these constants back into variables. The operation has repeatable patterns. Ten patterns that are easy to transfer follow, each shown as a pair of the ordinary RQ and the RQ after redefinition.

Pattern 1: Ask Not About the Outcome but About Its Conditions of Possibility

Ordinary research explains differences in outcomes. Why A succeeded and B failed, which method yields higher outcomes, which factor raises performance. Strong Problem Definition asks about the stage before that. What was needed in the first place for that outcome to be possible?

The ordinary RQ asks, “Does generative AI improve novices’ app-building outcomes?” After redefinition, it asks, “Through what sources of response and structures of record does the state come about in which a novice can adopt, revise, and reject what is generated?” The object has shifted from the presence or absence of outcomes to the conditions under which judgment comes about.

Pattern 2: Ask Not About Wins and Losses but About the Apparatus That Certifies Them

Existing research compares on already-defined criteria of evaluation. Grades, accuracy, completion rate, satisfaction, frequency of use, productivity. What should be asked here is why that measure holds the authority to certify “success.”

The ordinary RQ asks, “Which raises learning outcomes more, no-code or generative AI?” After redefinition, it asks, “Across different low-barrier making environments, what is treated as evidence that a learner’s judgment has come about?” A comparison of tools turns into a comparison of the apparatuses that certify outcomes.

Pattern 3: Give the Lead Role to Boundary Cases, Not Averages

Theories break not at average cases but at boundaries. That what shakes a paradigm is not the average but the anomaly is a classic observation in the history of science⁸. Boundary cases include the following.

The artifact was completed, but there is no process of judgment
The artifact is unfinished, but important learning is happening
It earned high marks, but cannot be reproduced
It failed, but exploration took place
The AI’s output is correct, but the learner does not understand
The learner understands, but the artifact’s quality is low

The RQs born here take a single form. In what kinds of cases do existing success metrics misrecognize success? For problematizing the blind spots of existing metrics, this pattern is strong.

Pattern 4: Change Not the Object but the Unit That Cuts the Object Out

A study’s novelty sometimes comes from the unit of analysis rather than from the object itself. The existing units are the student, the class, the artifact, the session, the team, the prompt. Other units can be taken. A single judgment, a single discrepancy, a single revision, a single source of response, the turning point from adoption to rejection, the connection from observation to update.

The ordinary RQ asks, “How did students use generative AI?” After redefinition, it asks, “By what evidence can the moments be reconstructed at which a learner recognized a mismatch in the generated result and identified what to revise?” It is a change from the person as unit to the judgment event as unit.

Pattern 5: Open Static States into Processes

If existing research treats capability, satisfaction, outcome, and quality as static attributes, redefine them as processes.

Capability: the process of recognizing differences and updating
Understanding: not the state of being able to explain but the state of being able to revise
Success: not completion but the state in which the next update is possible
Design: not an artifact but a chain of observation, comparison, and update
Learning: not the acquisition of knowledge but a change in the criteria of judgment

An example RQ becomes, “Through what observations, comparisons, and updates is a learner’s judgment that ‘it worked’ formed?” This is the pattern that shifts result-centered research toward process.

Pattern 6: Bring In Variables from Off the Board, Not On It

As a strong player reads the weather and the tournament rules, bring in variables from outside the target domain. In education research: the evaluation regime, the authority to assign grades, time constraints, the tool’s terms of use, class structure, submission formats, the composition of pairs, the publicness of feedback. In AI research: the UI, usage limits, model updates, log-retention specifications, the allocation of user responsibility, in-house rules.

An example RQ runs like this. Not differences in generative AI’s capability, but how log-retention specifications and the arrangement of responses within a class determine the recordability of learners’ judgments. What becomes the problem is not the technology’s performance but the institutions and environments that decide observability.

Pattern 7: Swap Subject and Apparatus

Usually, we think of humans as using tools. Invert it.

Not the learner receiving evaluation, but the evaluation regime shaping the learner’s behavior
Not people using AI, but AI’s output format constraining humans’ problem setting
Not teachers designing assignments, but the assignment format constraining what teachers can observe
Not researchers analyzing data, but the format of records pre-selecting which phenomena can be analyzed

An example RQ becomes, “Which learner judgments does the format of class records render visible, and which does it render invisible?” Because it questions not the object itself but the apparatus that generates knowledge, this pattern is quite strong.

Pattern 8: Turn Failure into a Diagnostic Apparatus for Broken Premises

Most research on failure searches for the causes of failure. A stronger definition asks which premise’s breakdown the failure exposed.

The AI answered wrongly: it became visible that the request was ambiguous
The student could not operate it: the designer’s tacit knowledge was exposed
The artifact was unfinished: it was exposed that the evaluation regime was biased toward finished work
The data could not be connected: the differing premises of record design were exposed

An example RQ runs like this. How do unconnectable records show, not mere missing data, but that each practice presupposed a different unit of judgment? Noise and gaps turn into theoretical resources.

Pattern 9: Shift the Axis of Competition

If existing research competes over “which is superior,” introduce a different axis of comparison. The usual axes are efficiency, accuracy, speed, completeness, satisfaction. Alternative axes include revisability, the visibility of where judgment resides, the continuity of exploration, openness to others’ responses, the reconstructability of records, and the updatability of evaluation axes.

An example RQ runs like this. How do low-barrier making environments differ not in the quality of the finished work but in how they make learners’ own judgments updatable?

Pattern 10: Doubt the Endpoint of the Game

Most research places an endpoint implicitly. At submission, at the end of the class, at the test, at product completion, right after adoption, at the moment of survey response. A researcher strong in Problem Definition asks why that point counts as the end.

An example RQ becomes, “To what extent does evaluation at the point of completion conceal later revisability and the continuity of exploration?” This is a pattern that can directly address what an evaluation taken at a slice of time ends up settling.

The Meta-Conversion Table: Pick One Existing Study and Apply the Conversion

The ten patterns compress into a single conversion table. It maps what existing research looks at (left column) onto what to ask after conversion (right column).

What existing research looks at	What to ask after conversion
Outcomes	The conditions that make outcomes possible
Success / failure	The criteria that certify success / failure
Individual differences	The arrangements that produce individual differences
Tool performance	The environment that determines the possibility of judgment
The finished artifact	The revision history
Averages	Boundary cases
Explicit actions	Discarded options
Data	The recording regime that generated the data
Capability	The process of updating
Errors	The exposure of broken premises
Objects of comparison	Units of comparison
Point-in-time evaluation	Temporal chains
The subject	The apparatus that forms the subject
In-game moves	The conditions that make the game possible

The Generation Protocol: From a Description of the Board to a Problem Statement

The conversion table is the seed of generation, and the procedure runs in five steps.

Step 1: Write out the game board of existing research. What it aims at, what it counts as success, what it measures, what it takes as the unit of analysis, at what point it stops evaluating, from whose viewpoint it describes, and what it discards as noise.

Step 2: Write out what has been pushed off the board. The unfinished, the interrupted, reasons for revision, options not adopted, others’ responses, the passage of time, the evaluation regime, what could not be recorded, judgments that went unobserved.

Step 3: Choose the most unnatural asymmetry. Artifacts are recorded, but reasons for revision are not. The AI’s outputs are saved, but the human’s adopt-or-reject judgments are not. Success cases are compared, but unconnectable cases are excluded. Learners are evaluated, but the evaluation regime is not.

Step 4: Convert it into a problem statement. A strong problem statement takes the following form.

Existing research has explained Y on the premise of X. But under that premise, Z cannot be discerned.

For example, one can write this. Existing research has used finished artifacts and self-reports as indicators of learning outcomes. But by those alone, one cannot discern which responses a learner took as grounds and what the learner revised.

Step 5: Lower it into an RQ. The RQ directly fills the lack in the problem statement. For example, it becomes this. From heterogeneous records of practice, by what grounds of connection can the relations among a learner’s operations, responses, discrepancies, and updates be reconstructed?

Habits of Thought: Five Redirections of the Gaze

Read not the answers of existing research but the design of its questions
Look not at the metric but at what the metric has rendered invisible
Look not at the average but at the boundary cases where the theory breaks
Look not at the object but at the apparatus that makes the object possible
Look not at the result but at the rules that certify the result

Compressed into one sentence, it is this. A researcher exceptionally good at Problem Definition is one who, before explaining a phenomenon, doubts what is being counted as a phenomenon.

The Game Board of This Pattern Collection Itself

A tool can be turned on the tool itself. The ten patterns and the conversion table implicitly fix the certification that “questions which question apparatus and institutions are strong.” In the vocabulary of the board, this pattern collection is itself a game board of what counts as a “strong question.” What that board cannot count, for example the value of research that perfects the precision of answers on the board without changing the apparatus, is invisible from this table. If one more move that rereads the board were to be added outside this table, what would it be? That question is left open.

ai-research-gaps-problem-definition — the application piece that rereads the AI research-gaps essays themselves through this note’s patterns
ai-research-gaps-abduction — the practice of searching for gaps in AI research by inverting premises (abduction-2)
ai-research-gaps-revisited — the practice of disciplining the inversion into the A–G operational worksheet

References

The material is a working memo provided in conversation (2026-07-20); the body’s patterns, conversion table, and generation protocol are its organized form. The literature used for the lineage is given with DOI/URL/ISBN.

Alvesson, M. & Sandberg, J. (2011). Generating Research Questions Through Problematization. Academy of Management Review 36(2):247–271. https://doi.org/10.5465/amr.2009.0188
Sandberg, J. & Alvesson, M. (2011). Ways of constructing research questions: gap-spotting or problematization? Organization 18(1):23–44. https://doi.org/10.1177/1350508410372151
Rittel, H. W. J. & Webber, M. M. (1973). Dilemmas in a General Theory of Planning. Policy Sciences 4(2):155–169. https://doi.org/10.1007/BF01405730
Simon, H. A. (1973). The Structure of Ill Structured Problems. Artificial Intelligence 4(3–4):181–201. https://doi.org/10.1016/0004-3702(73)90011-8
Getzels, J. W. & Csikszentmihalyi, M. (1976). The Creative Vision: A Longitudinal Study of Problem Finding in Art. Wiley. ISBN 9780471014867.
Schön, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action. Basic Books. ISBN 9780465068746.
Bowker, G. C. & Star, S. L. (1999). Sorting Things Out: Classification and Its Consequences. MIT Press. ISBN 9780262024616.
Kuhn, T. S. (1962). The Structure of Scientific Revolutions. University of Chicago Press. ISBN 0-226-45808-3.
Dorst, K. (2011). The Core of ‘Design Thinking’ and Its Application. Design Studies 32(6):521–532. https://doi.org/10.1016/j.destud.2011.07.006
Dorst, K. (2015). Frame Innovation: Create New Thinking by Design. MIT Press. ISBN 9780262324311.

Unverified Items

None. All references were confirmed by reaching the publisher’s page, the DOI, or multiple bibliographic sources (2026-07-20).

Getzels, J. W. & Csikszentmihalyi, M. (1976). The Creative Vision: A Longitudinal Study of Problem Finding in Art. Wiley. ISBN 9780471014867. ↩
Schön, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action. Basic Books. ISBN 9780465068746. ↩
Simon, H. A. (1973). The Structure of Ill Structured Problems. Artificial Intelligence 4(3–4):181–201. https://doi.org/10.1016/0004-3702(73)90011-8 ↩
Rittel, H. W. J. & Webber, M. M. (1973). Dilemmas in a General Theory of Planning. Policy Sciences 4(2):155–169. https://doi.org/10.1007/BF01405730 ↩
Sandberg, J. & Alvesson, M. (2011). Ways of constructing research questions: gap-spotting or problematization? Organization 18(1):23–44. https://doi.org/10.1177/1350508410372151 ↩
Alvesson, M. & Sandberg, J. (2011). Generating Research Questions Through Problematization. Academy of Management Review 36(2):247–271. https://doi.org/10.5465/amr.2009.0188 ↩
Bowker, G. C. & Star, S. L. (1999). Sorting Things Out: Classification and Its Consequences. MIT Press. ISBN 9780262024616. ↩
Kuhn, T. S. (1962). The Structure of Scientific Revolutions. University of Chicago Press. ISBN 0-226-45808-3. ↩