Does Being Able to Build Mean Being Able to Learn? Rereading the Learning Outcomes of End-User Development Through Motive

An integrative summary that rereads end-user development (EUD) along the two axes of who builds, and why and what is learned there, collecting 185 sources through a lightweight scoping review and mapping them into thirty-three clusters. The starting point was “what is learned by novice learners who hold no domain knowledge”. To that was added civic tech (the practice in which citizens build technology themselves for public problems), and to that, further, the motive of building for one’s own need. The last of these is the motive that EUD research, for all that it is the most basic one, has not dealt with. The sister note end-user-development-literature maps the same object from 152 sources along the axes of history, social demands, and limits. This note takes up what was thin there: what building brings to learning. Below, sources from the existing corpus are referred to by E number and sources from this corpus by N number. The blank this note identified at the intersection of making-for-oneself × generative AI is filled by the sequel genai-personal-software-literature (59 sources), which also re-examines how gaps 1, 12, 13, and 14 of this note fared in the generative AI era. Prior notes on the learning sciences side: learning-scaffolding-intervention-design for the theory of scaffolding and support for expertise, failure-driven-learning for the design of desirable difficulties, ai-cognitive-offloading-learning for empirical work on the learning harms of generative AI, technology-education-harm-patterns for the replicability of harm claims, and ai-augmented-competency-measurement for the methodology of competence measurement. Collection pipeline: three parallel source-researcher runs (profile: scholarly). Protocol: .claude/rules/collection-protocol.md (zero fabrication, provenance tracking). The internal ledger with provenance tracking is at source/review/end-user-development-novice-learning/papers.md.

Survey Metadata

• Collection date: 2026-08-14 / Count: 185 (66 from the first round plus 64 from the second plus 55 from the third; 207 candidates at the exploration stage, 4 merged as duplicates, 18 excluded)
• Period: 1981–2026
• Design of the first round (learning outcomes, 66 items): the work was split into three parallel strands — empirical studies of transfer and learning outcomes (25 items), the image of the novice and domain knowledge (28 items), and learning outcomes in the AI and low-code period (18 items). The first strand was instructed to include, without fail, negative evidence against the claim that learning to program raises thinking ability; the second was instructed to collect both sides of contested points (minimal guidance, the Dunning-Kruger effect) in pairs; the third was instructed to record immediate performance indicators separately from post-tests.
• Design of the second round (civic tech, 64 items): three strands likewise — the practice and learning of civic tech (24 items), digital participation platforms (24 items), and critical research (20 items). The first strand was instructed to include, without fail, studies reporting what did not work; the second was instructed to look for peer-reviewed evaluations of well-known cases rather than success stories; the third was instructed to include at least three sources on the side defending against the criticism.
• Exclusion for conflict of interest: in the second round, two papers on vTaiwan and Polis whose authors include central designers of the platform or members of the developing organization were excluded, narrowing the corpus to independent evaluation research. A consequence of this judgment is that no independent peer-reviewed study quantitatively verifying vTaiwan’s degree of policy uptake exists in this corpus.
• Design of the third round (building for one’s own need, 55 items): this round was carried out because, at the point when the first and second rounds were finished, it emerged that not one source among the 282 — the 152 of the companion note plus the 130 of this corpus — in total took this motive as its subject. It was split into three strands, collecting user innovation (26 items), making for its own sake and learning (22 items), and the persistence and abandonment of self-built artifacts (20 items). The second strand was instructed to include, without fail, negative evidence against the claim that learning proceeds when the matter is one’s own, together with peer-reviewed studies that doubt the effects of maker education; the first strand was instructed to include, without fail, methodological criticism of the estimates of scale.
• Verification of the collection stage’s descriptions: the first strand of the third round returned descriptions of principal contributions for 24 of its 26 items without having reached the abstract. Since the ground for these was that they are “findings established in the field”, that is generation from memory, and it conflicts with the zero-fabrication protocol. The parent therefore re-obtained abstracts through the OpenAlex API (succeeding for 17 of 22) and deleted, in 4 items, descriptions that the abstracts did not support. Conversely, of the figures the collection stage had left unconfirmed, two (the sample of 1,173 and the 6.1% in the UK survey, and the 12 interviewees) could be confirmed directly in the abstracts. For items whose abstracts could not be obtained, the principal contribution was limited to what can be said with certainty from the title, and no figures were written at all.
• Limits of coverage: all three strands of the third round were carried out with the WebSearch budget exhausted, and exploration relied on direct queries to the Crossref and OpenAlex APIs. The second strand declares that “new exploration was effectively impossible, and the work became a matter of corroborating already known candidates”. Coverage cannot be claimed for this cluster group.
• Exclusions: seven items in total. For five of them the bibliography could be confirmed in Crossref but the abstract was unreachable, and neither the principal contribution nor the method could be confirmed. One was out of scope (moderation behavior on social media). In addition, one item suspected of coming from a predatory journal, along with non-peer-reviewed postings and out-of-scope sources, was rejected at the collection stage.
• Machine verification of the bibliography: the 128 items carrying a DOI were queried against the Crossref REST API. For the first round, 62 agreed, 1 showed a discrepancy in year, and 1 is unregistered in Crossref although its DOI resolves. For the second round, all 64 agreed on author surname and year of publication. Fabricated DOIs: zero.
• Venue distribution: 15 items from educational psychology venues, 26 from computing education venues, 6 from cognitive science venues, 11 from newer venues of the AI period (the above from the first round), 15 from HCI and CSCW venues (CHI, CSCW, PACM HCI, DIS, PDC), 13 from political science and public administration venues (American Political Science Review, European Journal of Political Research, Government Information Quarterly, Public Administration Review), 10 from urban studies and sociology venues (Urban Studies, New Media & Society, The Sociological Review), 8 from citizen science and community informatics venues (the above from the second round), and 26 others.

TL;DR

The users that the end-user development literature has assumed were accountants, clinicians, and researchers. People who do not know programming but know their own field better than anyone. What Nardi saw in spreadsheet users was likewise the expert who commands the tool precisely because they can think in the vocabulary of the domain (E17).

The novice without domain knowledge is a different being from that image of the user. Whereas experts categorize problems by principle, novices categorize them by the surface features of the problem statement (N49). At the stage of deciding what to build — that is, at the stage of handling an ill-structured problem — what is needed is domain knowledge rather than operation of the tool (N50). And the way of teaching that works for novices becomes counterproductive for learners who have acquired knowledge (N27). As long as EUD’s design knowledge has been accumulated on the premise of the domain expert, there is no ground for transferring it as it stands to novices with zero domain knowledge.

As for what is learned by building, forty years of answers are in. Effects on skills close to programming (near transfer) are supported relatively robustly, while effects on general thinking ability (far transfer) have yielded only negative or limited evidence ever since Pea & Kurland (1984, N01). Mayer et al. (1986, N03) and Palumbo (1990, N05) pointed to methodological weakness, and the meta-analysis of Scherer et al. (2019, N07) reports not only that far transfer effects are weak but also that effect sizes tend to be overestimated the more a design is quasi-experimental.

The doubt about whether what is being measured is secure has not gone away either. Scores on computational thinking tests correlate highly with existing reasoning tests, so it cannot be said that they measure a distinct learning outcome (N22). A systematic review of measurement instruments concludes that many studies lack validity evidence (N24). Self-efficacy does rise. But that rise does not correspond correctly to actual performance (N16), and what predicted performance more strongly was prior experience and organizational ability (N14).

Generative AI has not dissolved this configuration. In an introductory Arduino experiment, the group that did not use AI was significantly superior both on the post-test and on self-efficacy (N54, N=33). Meanwhile, a one-semester comparison of 182 students found no significant difference by whether ChatGPT was permitted (N55). Neither “AI harms learning” nor “AI does not harm learning” can be asserted on the current evidence. What can be said is that not a single study in this corpus measures learning outcomes under AI assistance with a delayed test administered several weeks or more afterwards.

Civic tech brings a third condition into this. In a setting that is neither work nor school, where one builds unpaid for the public good, the very calculation of whether to pay the learning cost changes. And in this setting the location of domain knowledge moves away from the builder. In workplace EUD the accountant builds their own spreadsheet, whereas in civic tech the resident who knows the local problem and the technologist who can build tend to be different people. The condition EUD has presupposed, that the person who knows the domain builds it themselves, does not hold here.

On whether participation changes citizens, the evidence is split. There is a report that the quality of the deliberative experience is related to subjective outcomes (N111), while what determined whether a proposal was implemented was not the quality of deliberation but cost and political support within the municipality (N108, a multilevel analysis of 571 proposals). When a minipublic’s recommendation is ignored by government, perceptions of legitimacy in fact fall (N112, n=3,102). On the learning side, reports that civic tech grows capability are confined to small-scale cases or self-report, and not a single quantitative study with a control group exists in this corpus.

The three conditions seen so far (work, education, the public) all share one thing that is missing. It is the motive of building because one is oneself in trouble. At the point when the first and second rounds were finished, not one source among the 282 — the 152 of the companion note plus the 130 of this corpus — took this motive as its subject. This appears to be a consequence of EUD research having begun its account from the organizational imbalance of supply and demand and from a shortage of experts.

What has dealt with this motive was a lineage outside EUD research. User innovation research after von Hippel treats the case in which the developer themselves is the only user. In a survey of a representative UK sample of 1,173 people, 6.1% of consumers (an estimated roughly 2.9 million) had developed or modified a product themselves (N137). There is criticism, however, directed at the very framework of measurement behind estimates of this kind (N152, N153).

And seen from the side of learning, this motive is not a simple facilitator. A meta-analysis of 41 studies on the provision of choice reports that there is an effect but that it weakens according to conditions (N157), and if choices become too numerous motivation instead falls (N155). The most suggestive is a comparative study that gave a personal subject matter in the form of storytelling. Motivation rose and the time spent programming was 42% longer, yet acquisition of basic syntax showed no significant difference from the group given an assigned task (N164). What making the matter one’s own increases is first of all the time and the willingness directed at learning, not attainment per unit of time.

Who Is Learning?

The Presupposition of Domain Knowledge

The image of the user in EUD research counts on domain knowledge as a cognitive resource. Requirements engineering research shows that this resource does in fact work. At the same time, however, a dysfunction is reported in which domain knowledge makes tacit premises impossible to question. Hadar et al. (2012/2014, N52) argued that the richer a requirements engineer’s domain knowledge, the easier problem understanding becomes, while points taken as self-evident in that domain may be overlooked. Niknafs & Berry (2012, N51) supported the converse hypothesis in a controlled experiment: adding analysts without knowledge of the domain to a team compensates for experts’ oversights (the authors themselves reserve that the result is not decisive).

What follows is that the presence or absence of domain knowledge is not a matter of higher or lower ability but a difference in the arrangement of what is visible and what is not. Novices can be in a position to notice points that experts miss. But at the stage of converting that noticing into a design, a different problem is waiting.

The Stage of Categorizing by Surface Features

The classic experiment of Chi, Feltovich & Glaser (1981, N49) showed that experts categorize physics problems by principle whereas novices categorize them by the surface features of the problem statement. In the EUD context this difference bears directly on the structure of what gets built. Without being able to tell which problems are the same and which are different, one cannot judge when an existing solution may be carried over.

Jonassen’s (1997, N50) distinction between well-structured and ill-structured problems bears on this point. Solving ill-structured problems requires domain knowledge, epistemological beliefs, and the ability to reframe the problem. Building something for the sake of one’s work is, in most cases, an ill-structured problem. Novices are prone to treat it as a well-structured one.

The Expertise Reversal Effect

The expertise reversal effect defined by Kalyuga et al. (2003, N27) is the phenomenon whereby instructional methods effective for novices instead raise the load and impede learning for learners who have acquired knowledge. On Sweller’s (1988, N25) cognitive load theory the reason is plain. Tackling problem solving without a schema of domain-specific knowledge consumes working memory in means-ends analysis, leaving no resources for schema acquisition.

This effect is a reason why EUD’s design knowledge cannot be carried over to novices as it stands. Blackwell’s attention investment model (E61) explained users’ refusal to pay the learning cost as a rational judgment rather than a deficit of ability. That rationality is the rationality of a user who holds domain knowledge as a resource. When a novice without that resource makes the same judgment, the result is to keep building without learning.

Yet the simplification that prior knowledge is all one needs is not supported either. Simonsmeier et al. (2022, N30), in a meta-analysis of 8,776 effect sizes, reported that the stability of individual differences in prior knowledge is high (rP+=.534) while the power of prior knowledge to predict learning gains is on average close to zero (rNG+=−.059), with a 95% prediction interval as wide as [−.688, .621]. Both the claim that knowledge is power and the claim that the effect of prior knowledge is negligible become crude in the face of an interval that wide.

Does Building Amount to Learning?

Far Transfer Is Not Supported

The claim that learning to program grows general thinking ability has underpinned EUD’s educational demand (E46, E47, E48). Testing of the claim began forty years ago and was negative from the first answer.

Pea & Kurland (1984, N01) reported that children who received a year of Logo instruction showed no significant difference from a control group on a far transfer task of planning ability. Mayer, Dyck & Vilberg (1986, N03) synthesized prior work and argued that transfer can be confirmed only within a range close to programming-specific skills. Palumbo (1990, N05) examined the methodology of the studies that reached positive conclusions and concluded that the absence of control groups, the brevity of instruction, and insufficient validity of the measures make the conclusions inconsistent.

The account of Salomon & Perkins (1987, N04) recast this negative finding as a matter of conditions. Transfer has a route through automation by repetition (low road) and a route through deliberate abstraction (high road), and neither occurs automatically. That is, one does not learn by building; one learns only when the conditions for transfer are built into the design.

Recent meta-analysis confirms this configuration. Scherer, Siddiq & Sánchez Viveros (2019, N07) reported that near transfer is relatively robust while far transfer effects are weak, and moreover that effect sizes tend to be overestimated the more quasi-experimental the study design. Taken together with the small-to-moderate positive effect sizes reported by Liao & Bright (1991, N06), the estimated effect is inversely related to the rigor of the research design.

Does Understanding of the Domain Deepen?

If what is learned by building is not general thinking ability, what about the concepts of the target domain? Positive evidence on this question comes from design research in the constructionist lineage.

Wilensky & Reisman (2006, N08) reported that having students construct predator-prey models themselves in StarLogo led them to understand the causal mechanisms of biological phenomena more deeply than lecture-style instruction did. Blikstein & Wilensky (2009, N09) showed a similar result in materials science. These, however, are design studies without control groups, and are not cast in a form that allows the magnitude of the effect to be compared with other instructional methods.

Studies that followed concept learning longitudinally in environments such as Scratch draw a more cautious picture. Meerbaum-Salant et al. (2013, N10), in a two-year study across several schools, reported that concepts of sequential execution and event handling took hold while difficulties with the concepts of variables and concurrency persisted after instruction. Grover, Pea & Cooper (2015, N11) likewise confirmed significant pre-post gains on a concept test while reporting that misconceptions such as those about variables remained after instruction. What is learned differs from concept to concept.

One gap becomes visible here. The evidence on domain concept learning in this corpus is skewed toward educational research on agent-based modeling. No study measuring whether building business models such as accounting or inventory management oneself deepens understanding of that domain was found within the scope of the search. That means there is no empirical work on concept learning for the spreadsheet, the tool most widely used in EUD’s practical contexts.

Confidence Is Not a Proxy for Competence

Self-efficacy is the learning outcome most often measured in the EUD context. The scale of Compeau & Higgins (1995, N13) is used as the standard, and rises in its level before and after training are reported (N12).

A raised confidence does not necessarily represent competence, however. Wiedenbeck (2005, N14), tracking non-majors over one semester, reported that self-efficacy rose significantly but that what more strongly predicted course performance and debugging performance was prior experience and organizational ability. Gorson & O’Rourke (2020, N16) examined CS1 students at three universities and showed that self-assessments diverge from actual performance and are not calibrated. Accumulating experience does not automatically grow accurate self-perception.

Interest and collaboration attitudes have also been measured as constructs adjacent to self-efficacy. Kong et al. (2018, N15) reported, with primary school students, that programming empowerment, interest, and collaboration attitude are positively related to one another. These are indicators of attitude, however, and sit on a different layer from what one has become able to do.

What Has Been Measured?

In discussing learning outcomes, doubt about measurement itself is unavoidable.

Román-González et al. (2017, N22) reported that scores on a computational thinking test correlate highly with existing tests of reasoning and problem solving. This raises a question of construct validity: does the test measure a distinct ability newly grown by instruction, or is it merely a proxy for existing cognitive ability? The systematic review of Tang et al. (2020, N24) concluded that many of the instruments used to assess CT lack evidence of reliability and validity. Popat & Starkey (2018, N23) likewise criticized much of the effectiveness research on learning to code for not measuring learning outcomes rigorously. The immaturity of the measurement foundation is also a problem the field has been aware of from early on. Grover & Pea (2013, N20), taking stock of CT education research, already listed the vagueness of definitions and the weakness of the measurement foundation as principal problems.

Ingenuity on the measuring side has advanced as well. The commutative assessment of Weintrop & Wilensky (2015, N62) presents the same program in both block and text notation, a method for separating the measurement of conceptual understanding from familiarity with a notation. The problem that what is being measured becomes contaminated by notation arises without fail when evaluating environments that hide abstraction.

Using artifacts as evidence of learning has its limits too. Denner, Werner & Ortiz (2012, N18) matched features of games built by middle school girls against their scores on a separately administered test of conceptual understanding, and showed that the artifact is only partially valid as a proxy for understanding. That the finished thing runs does not mean its maker understands how it works.

Lye & Koh (2014, N21) point to this skew at the level of research design. Much CT instruction research leans toward the “building” stage, and assessment at the “reflection” stage that measures the depth of conceptual understanding is thin. The assessment framework proposed by Brennan & Resnick (2012, N19) across the three dimensions of concepts, practices, and perspectives amounts to a response to this skew.

How Novices Misjudge Their Own Understanding

Being able to build produces the sensation of having understood. The illusion of explanatory depth of Rozenblit & Keil (2002, N42) showed across twelve experiments that people believe they understand the workings of complex things far more deeply than they in fact do. This illusion is especially strong for explanatory knowledge, more so than for factual or procedural knowledge. Being able to operate but unable to explain is the typical position in which the EUD novice is placed.

The review of Bjork, Dunlosky & Kornell (2013, N43) sets out that learners tend to misjudge their own state of learning on the basis of the subjective sense of current fluency. Research on novices’ program comprehension has specified where concretely this error arises. Spohrer & Soloway (1986, N38) showed that novices’ errors are rooted in misunderstandings at the level of plan composition rather than in a lack of syntactic knowledge, and Soloway (1986, N37) recast learning to program as the acquisition of stereotypical solution patterns and of explanations. Pea (1986, N36) in the same period classified novices’ errors into three kinds of conceptual bug — parallelism, intentionality, and egocentrism — and reported that these appear across age groups and languages. The notional machine of du Boulay (1986, N35) and Sorva (2013, N39) is a concept that treats, at the level of learning objectives, the fact that being able to write code and understanding the mechanism of execution are different things. This forty years of accumulation is gathered in the reviews of Robins et al. (2003, N17) and Qian & Lehman (2017, N40). The latter organizes novices’ difficulties into three layers of syntactic, conceptual, and strategic knowledge, and concludes that correcting misconceptions requires the theory of conceptual change and teachers’ pedagogical content knowledge. Neither condition exists in the EUD setting, where there is no teacher.

Yet the simplification that novices are systematically overconfident is itself contested within the peer-reviewed literature. The original paper of Kruger & Dunning (1999, N41) attributed the large overestimation of the bottom-scoring group to a lack of metacognitive ability, but Nuhfer et al. (2017, N44) argued that its graphical method is distorted by random noise and a plotting convention, and Gignac & Zajenkowski (2020, N45) reported that with data from 929 participants the heteroscedasticity predicted by the DK hypothesis could not be detected at statistical significance. Nuhfer & Fleisher (2026, N48), with N=11,229, analyze much of the apparent overestimation by low scorers as arising from disengaged guessing rather than genuine overconfidence. If novices’ errors of self-assessment are to be used in an argument, it is safer to ground them in research on calibration (N16, N43) or in the illusion of explanatory depth (N42) rather than in the DK effect.

Prather et al. (2020, N46) and Loksa et al. (2022, N47) point out that such metacognitive difficulties have not been treated explicitly in programming education, and confirm the thinness of the theoretical foundation in a systematic review.

How Much Guidance to Give

Should novices be left to explore freely, or should procedures be shown to them? This question has long been disputed in the learning sciences.

Kirschner, Sweller & Clark (2006, N32) argued, from findings on cognitive architecture and expert-novice differences, that minimally guided instructional methods such as constructivist discovery learning and problem-based learning run counter to half a century of empirical research. Hmelo-Silver, Duncan & Chinn (2007, N33) rebutted this, pointing out that inquiry learning and problem-based learning in fact involve extensive scaffolding and are being conflated with unguided discovery learning.

The meta-analysis of Alfieri et al. (2011, N34), covering 164 studies, settles this opposition conditionally. Unassisted discovery learning is inferior to explicit instruction (d=−0.38), while “enhanced discovery learning” accompanied by feedback, worked examples, scaffolding, and elicited explanations is superior to other instructional methods (d=0.30). Mayer (2004, N31) likewise summarized that in each of three waves of research, guided discovery learning proved more effective than pure discovery learning.

EUD, in this classification, is close to unassisted exploration. The scene of building something for the sake of one’s work has no teacher, no worked examples, and no designed feedback. The fading of Renkl & Atkinson (2003, N28) — the procedure of replacing worked examples with problem solving in stages — and the adaptive design according to knowledge level of Kalyuga & Sweller (2004, N29) are, first of all, not built into EUD’s tools. The same holds for the finding of Paas & van Merriënboer (1994, N26) that sets of worked examples with high contextual variability promote acquisition of schemas with greater transferability. This is a place where room for design remains, if learning is set as the goal.

What AI and Low-Code Changed

The Range of What Can Be Built and the Range of What Can Be Learned

Generative AI has certainly widened the range within which novices without domain knowledge can build something that runs. The question is whether that widening of range is accompanied by a widening of the range of learning.

The evidence is split. In the introductory Arduino experiment of Johnson, Doss & Estepp (2024, N54, N=33), the self-programming group was significantly superior to the ChatGPT group both on post-test scores (p=.03, d=0.76) and on self-efficacy (p=.03, d=0.75). In the one-semester comparison of Kosar et al. (2024, N55, N=182), by contrast, no significant difference by whether ChatGPT was permitted appeared either on lab assignments or on the midterm exam. Melanou & Beege (2026, N57) compared three conditions of scaffolding strength and report that knowledge and critical thinking improved significantly in every condition while there was no significant difference between conditions.

Kazemitabaar et al. (2023, N53) show that this divergence occurs at the level of how the tool is used. Observing 33 learners, they found that a strategy combining generation with hand-written work showed a positive trend with post-test scores while a strategy of generating whole solutions showed a negative trend. The finding that what matters is not whether AI is used but how it is used agrees with the conclusion set out in ai-cognitive-offloading-learning, that the presence or absence of harm divides according to whether the system is answer-providing or guardrailed. Scholl & Kiesler (2024, N58), in a survey of 298 students, describe how usage diverges widely from uncritical acceptance to critical engagement, showing that this divergence is the norm rather than the exception.

If divergence occurs, the next question is what determines which way it falls. Hou et al. (2024, N59) reported that students compare generative AI with conventional support resources along the axes of quality, response speed, and reliability, and that generative AI has not entirely displaced instructors and TAs. The judgment of where to seek help is itself a metacognitive activity, and whether one can make that judgment is, for the novice, not a precondition of learning but an object of it.

Haindl & Weinberger (2024, N56) reported that code submitted by a ChatGPT-assisted group had both fewer rule violations and lower cyclomatic complexity. This study, however, does not measure retention of learning. Artifact quality and learning outcomes are different objects of measurement.

Consequences of Hiding Abstraction

On what environments that hide abstraction do to concept formation, there are clues in pre-generative-AI research on block-based programming. Weintrop & Wilensky (2017, N63) reported that in a high school CS class the block-based group showed higher gains on an achievement test while the text-based group tended to rate their own experience as “professional and real”.

What matters is the sequel. Weintrop & Wilensky (2019, N64) followed the process of transitioning to Java over ten weeks after a five-week introductory environment, and reported that the conceptual advantage observed at introduction had disappeared ten weeks later. No significant difference remained in programming practices or in attitudes. This is the only study in this corpus that carries out a follow-up several weeks or more later, and that sole long-term follow-up yields the result that a short-term advantage does not persist.

For environments combining generative AI with low-code, no follow-up of the same kind yet exists. No study measuring which concepts are formed in a user of an environment where instructing in natural language produces something that runs was found within the scope of the search.

Vocational Training Contexts Are Not Connected to the Learning Sciences

Research addressing the training of citizen developers does exist. Matook et al. (2025, N66) quantitatively surveyed the influence of experiential learning factors and team learning on metacognitive reflection. McHugh et al. (2023/2024, N65) examine the introduction of low-code in secondary education, but their object is teachers’ perceptions rather than students’ learning outcomes.

Research in this area is written within the frameworks of organization theory and information systems research, and designs involving pre-post tests are almost absent. In EUD’s own context — adults learning for work purposes, in a situation with neither credits nor grades — studies that measure learning remain thin.

Civic Tech as a Third Condition

Building Unpaid for the Public Good

The learners seen so far were people who build for a salary or people who build for course credit. Participants in civic tech are neither.

This difference bears directly on the cost-benefit calculation that Blackwell’s attention investment model (E61) addresses. At work it could be rational not to pay the learning cost, but in a setting without reward, participation itself has already been chosen for a different reason. A study following citizen science volunteers (N78) reported that what sustains continuation is interest in the subject domain and past experience, while what invites withdrawal is constraints on time and the difficulty of digital platforms. The unusability of a tool is something to be endured and got past at work, but in unpaid activity it becomes a reason to leave.

The hackathon is the format most studied in this setting. Porter et al. (2017, N69), from interviews with participants, nonprofits, and organizers, reported that philanthropic hackathons bring, beyond code, technical skill, networks, exposure to design methods, and the formation of professional identity. What works and how, however, has not been sorted out. Falk et al. (2021, N71) analyzed sixteen cases across the board with program theory, and the field has only just reached the stage of a framework that reassembles the causal relations from input through to effect.

And the artifacts do not remain. When Falk et al. (2024, N73) listed six challenges for hackathon research, one of them was that code is abandoned when the event ends. Lodato & DiSalvo (2016, N67) followed issue-oriented hackathons through two years of ethnography, theorizing them as experimental material participation while critically examining the formality of participation. Schrock (2016, N68), by contrast, rebutted the received view that treats civic hacking as depoliticized technologism, and showed from history its political lineage as data activism.

Fundraising, too, falls to the participants themselves in this setting. Gooch et al. (2020, N87) analyzed four cases of community-led crowdfunding and reported that while the sense of empowerment of those who led the projects rose, the burden on time, the dependence on existing personal networks, and the exclusion of those not inside them remained as problems. Whether funds can be raised is skewed toward those who already hold connections.

Domain Knowledge Moves Away from the Builder

In workplace EUD, the person who built knew the domain. In civic tech, these two are often different people.

Once that is so, the requirements engineering findings from the existing corpus bear directly. Adding analysts without knowledge of the domain to a team compensates for experts’ oversights (N51), while those who know the domain become unable to question tacit premises (N52). The barriers reported by a study of 15 civic hackers in Seattle (N70) arise precisely from this separation. Both the quality of data and its metadata are self-evident inside government and invisible to builders who come from outside.

Rothschild et al. (2022, N80), from interviews with 19 data workers in government and nonprofits, made visible the practical knowledge of contextualization held by practitioners positioned at the periphery. Those who know what the data is, how it was collected, and what is missing are, more often than not, on the non-technologist side. The motivations for and barriers to open data engagement that Dove et al. (2023, N81) investigated across students, commercial users, civic tech participants, and government staff likewise rest on this asymmetry.

Collaborative design, which ought to bridge the separation, has difficulties of its own. Harrington, Erete & Piper (2019, N86) examined participatory design in low-income communities and identified tensions of the reproduction of privilege, the neglect of historical context, and unintended harm. The method of participatory design does not itself guarantee equity. Lodato & DiSalvo (2018, N84) draw out three institutional constraints from three cases within smart city initiatives: sandboxing, administrative voids, and ideological misalignment. The series of studies by Corbett & Le Dantec (N83, N88) reported, from two years of co-design, that the understanding of the very word trust diverges between government officials and citizens.

Seen from the development side, the difficulties concentrate on requirements and handover. Knutas et al. (2019, N85) tracked two Code for Ireland projects and reported that while the quality standards they set for themselves were met, they stumbled over requirements definition, stakeholder involvement, and handover to government. The practical guidance the same authors later compiled for engineers (N89) also rests on this experience. A framework that divides the design of community technology into the three levels of community, group, and individual (N82) was proposed early on because design changes according to whom one builds with.

Has Learning Been Measured?

Studies that measured learning as their subject do exist in this area too. Bhargava et al. (2016, N75) and D’Ignazio & Bhargava (2016, N76) reported mastery of basic data concepts, from data literacy education using the arts and from the design of tools for beginners. Ballard et al. (2016, N74) followed the development of environmental-science agency in young people who took part in citizen science, through pre- and post-interviews and observation. Phillips et al. (2018, N79) presented a conceptual framework for measuring the individual learning outcomes of citizen science.

All of these, however, are small-scale case studies or self-report. Not one of the 64 civic tech sources in this corpus carries out quantitative measurement of effects with a control group. On the quality of the artifacts there is a report that combining training, validation, and statistical modeling can yield expert-level accuracy (N77), but that concerns the quality of the data produced, not what its producers learned.

Does Participation Change Citizens?

Gathering the empirical research on participation platforms, one notices that an increase in participants and a change in citizens’ capabilities are measured separately. Much of what is in operation does not even reach a stage that could be called co-production. When Falco & Kleinhans (2018, N92) classified 113 platforms, about a quarter showed realistic potential for online and offline co-production. Bartocci et al. (2022, N98), who systematically reviewed 139 studies of participatory budgeting, likewise organize a framework by stage of implementation while leaving many research gaps open.

A study of Decidim across several Catalan municipalities (N96) reported that while participants and newly participating groups increased, the administration prioritized information gathering and proposal management, and a managerial continuity that stops short of transferring sovereignty predominated. In Decide Madrid, after high initial expectations produced a high participation rate, a lack of transparency and dysfunction damaged performance in later years (N93). Menéndez-Blanco & Bjørn (2022, N97) followed the practices of bicycle activists involved in participatory budgeting and point out that the design of the platform produced competitive rather than collaborative interaction, and in fact limited the possibility of collective self-governance.

That design choices change outcomes has been confirmed with larger data as well. Moore et al. (2020, N110) used roughly 45 million comments to compare three identity disclosure regimes: anonymity, pseudonyms, and real names. Cognitive complexity was highest under pseudonyms, and in fact declined after the move to a real-name requirement. The received idea that requiring real names raises the quality of discussion is not supported by a natural experiment at this scale. Moats & Tseng (2023, N99) analyzed vTaiwan’s Pol.is and showed that while the aggregation algorithm organizes controversy into the binary of for and against, the original comments contained a more diverse range of positions. There is also a report that merely changing the display of discussion from linear to hierarchical improved the structure of argumentation (N91), and the range over which details of design bear on outcomes is wide. What Tseng (2022, N95) saw in comparing vTaiwan and Decide Madrid was that gamified devices do not automatically raise participation but produce the frictions of emotional difficulty and tactical misuse. The tendency Aragón et al. (2017, N90) found in Decidim Barcelona, that comments taking a negative position attract more discussion, likewise shows that the volume of engagement is not the same as the quality of deliberation. A proposal for algorithms bridging voting and deliberation (N101) lies along this line, but it does not directly measure outcomes on the citizens’ side.

The coldest result comes from research that followed where proposals end up. Font et al. (2018, N108) subjected 571 proposals emerging from participatory processes to multilevel analysis and reported that what determined whether they were implemented was cost, the degree of challenge to existing policy, and political support within the municipality. The causal claim that deliberating well gets a proposal through is not supported here. And the survey experiment of van Dijk & Lefevere (2022, N112, n=3,102) showed that while political support rises when a minipublic’s recommendation is adopted, perceptions of legitimacy can fall when it is ignored. Creating opportunities for participation is not in itself neutral.

On who wants to participate, there is a finding at odds with the received view. Neblo et al. (2010, N104) combined surveys of two national samples with an experiment inviting people to online deliberative town halls held by sitting members of Congress, and showed that willingness to deliberate is distributed more widely than assumed, and that those least inclined toward conventional partisan political participation are the more interested in deliberation. A follow-up from the same research program (N106) also reports a multiplier effect in which participants spread the discussion to non-participants around them. This, however, concerns the distribution of those who want to participate, and is a separate point from whether those who actually keep participating are representative (N107).

Connection to the Critique of Democratization, and the Distinctive Points

The critique of civic tech shares the same rhetorical structure as the critique of democratization discourse in the existing corpus (E52–E56). The point is that a narrative of liberation through participation in fact conceals a particular rationality of governance or of the economy. Kitchin (2014, N113) pointed to the danger of data-driven urban management reducing the running of cities to positivist measurement, and Shelton et al. (2015, N115) described the divergence between the uniformly narrated ideal and the uneven implementations determined by each city’s political economy as the “actually existing smart city”. Sadowski & Pasquale (2015, N116) theorize these technical infrastructures as working as a spectrum of control that normalizes surveillance. Vanolo (2014, N114) analyzed with the concept of governmentality the mechanism by which smart city discourse promotes citizens’ self-discipline, and Cardullo & Kitchin (2019, N118) argued that European policy documents professing to be “citizen-focused” presuppose an image of the citizen as consumer or entrepreneur. The “transactional citizen” of Johnson et al. (2020, N119) names the mechanism by which the design of interfaces reduces the citizen from a political subject to an entity counted in clicks. Cardullo & Kitchin (2018, N117) and Haughton & McManus (2020, N124) use Arnstein’s ladder of participation to follow the process by which participation stays at a tokenistic rung and contested issues are absorbed back into the institution.

What is distinctive lies in the way civic tech cuts into the very body that delivers public services. Whereas EUD’s democratization discourse took as its problem the distribution of programming capability within an organization, the critique of civic tech asks about the shift of responsibility for producing public services onto citizens’ unpaid labor. Clark et al. (2013, N122) demonstrated, from request data to the City of Boston’s 311 system, distributional biases in request behavior by race, education, and income. A system anyone can report to is not a system in which everyone reports alike. Crivellaro et al. (2021, N125) argued that digital welfare services under austerity can amplify existing injustices, and the systematic review of Voorberg et al. (2015, N128) points out that the discourse of co-creation has circulated affirmatively while remaining theoretically vague.

Criticism, however, is not the only evidence. Peixoto & Fox (2016, N129) reviewed across cases in which ICT-enabled citizen voice did and did not raise government responsiveness, and set out the conditions under which the effect appears. Erete (2015, N127) demonstrates the pathway by which online exchange in a neighborhood community prompts real-world behavior such as residents’ attendance at meetings. The typology of we-government by Linders (2012, N126) amounts to an early attempt to organize the forms in which citizens take part in producing public services. From within participatory design as well come proposals that criticize the consensus-formation model and counterpose an agonistic public space admitting conflict (N121), and theoretical criticism of design’s presupposition of a single consensual public (N120).

Building for One’s Own Need

The Motive EUD Research Passed By

As reasons for end users building things themselves, EUD research has given the organizational imbalance of supply and demand (D1 in the existing corpus), a shortage of experts, educational literacy, and the public good. In every one of them, the reason for building lies outside the person. Only the reason that one builds because one is oneself in trouble has been left unexplained.

What took up this motive directly was the side of management and technology management. von Hippel (1976, N131) discussed the role of users in the innovation process for scientific instruments, and in 1986 (N132) positioned lead users, who hold strong needs ahead of the market at large, as a source of new product concepts, on the ground that for highly novel products one cannot rely on the judgment of ordinary users. von Hippel & Katz (2002, N133) reorganize the very division of labor in which the manufacturer first explores user needs and then develops, discussing designs that delegate part of development to users.

Estimates of scale came in the 2010s. von Hippel, de Jong & Flowers (2012, N137) measured, in a representative sample of 1,173 UK consumers aged 18 and over, that 6.1% (an estimated roughly 2.9 million people) develop or modify products themselves. It is the first survey of its kind.

There are criticisms, however, of taking this figure at face value. Bogers, Afuah & Bastian (2010, N152) reviewed and criticized the literature that grew rapidly after the argument that users too can be a source of innovation, and set out a research agenda from there. Gault (2012, N153) points out from a different angle that because the OECD and Eurostat Oslo Manual defines innovation with reference to the market, consumers who modify products for their own use are structurally excluded from official statistics. The point is that what has not been measured has been treated as not existing.

What Is Built for Oneself Does Not Spread

Some of what is built for oneself passes to others. Harhoff, Henkel & von Hippel (2003, N134) discussed the conditions under which revealing an innovation freely can be to the innovator’s own benefit, and Baldwin & von Hippel (2011, N136) assessed the economic viability of producer innovation in contrast with innovation by single users and by open collaboration.

But more often it does not spread. de Jong et al. (2015, N138), building on the demonstration that millions of people develop new products and services for themselves, examined the patterns of diffusion from Finnish data and conceptualize this as a market failure. The phenomenon of something being built and yet not spreading is a point that does not exist at all for an EUD that begins from an external demand. A business system is rolled out by the organization, and what is made for the public is settled by whether government takes it over. Only what is built for oneself has no one taking on the question of whether it spreads.

Some of it turns into entrepreneurship. Shah & Tripsas (2007, N140) built a process model of how users create, evaluate, share, and commercialize ideas, and emphasized, in contrast with the classical entrepreneurial process, that it is emergent and collective. Ogawa & Pongtanalert (2013, N141) examine the characteristics and motives of consumer innovators by type.

Open Source Became a Laboratory of Motives

The setting in which what was built for oneself passes to others has been observed by more researchers in open source than anywhere else.

Shah’s (2006, N144) starting point is candid. Open source depends on the voluntary effort of thousands of developers, yet why they participate is not sufficiently understood. From there a framework for understanding participation from the viewpoint of the individual developer was derived inductively. Roberts, Hann & Slaughter (2006, N145), taking what motivates participation to be the central subject of the field, analyze the relations among motivation, participation, and performance longitudinally. Hertel et al. (2003, N142) and Lakhani & Wolf (2005, N143) are likewise surveys of the motives for participation. The latter surveyed 684 developers working on 287 projects. von Krogh et al. (2012, N146) presented a framework that integrates these and grasps them as social practice.

The same configuration appears on a small scale in everyday development as well. Rosson, Ballin & Nash (2004, N147) analyzed tool use and problem-solving strategies from interviews with 12 end users who develop and maintain their organization’s web content themselves. Brandt et al. (2009, N148) observed how programmers work by interleaving foraging on the web, learning, and writing code. Building and learning are not separated. The “expert amateur” that Kuznetsov & Paulos (2010, N149) saw in DIY communities occupies the same position. This, however, is not an empirical study but an argument that current technology design overlooks the direction in which people’s curiosity and inquiry would be grown.

What Happens to Learning When the Problem Is the Learner’s Own

This is the point of contact with this note’s subject.

The effect of giving choice has been examined in meta-analysis. Patall, Cooper & Robinson (2008, N157) synthesized 41 studies and reported that providing choice raises intrinsic motivation, effort, task performance, and perceived competence. It is conditional, however. The effect weakens when choices are imposed successively, after a reward has been given, when the participants are adults, and outside the laboratory. Iyengar & Lepper (2000, N155) showed from the other side that when choices become too numerous, satisfaction and motivation instead decline.

The effect of personalizing content has been measured more directly. Cordova & Lepper (1996, N154) reported that in arithmetic tasks for primary school children, contextualization, personalization, and the provision of choice increased not only motivation but also the depth of engagement and the amount learned within a fixed time. Guzdial & Forte’s (2005, N156) course for non-majors adopts a task design in which students manipulate their own photographs and music.

However, motivation rising and attainment rising are two different things. Kelleher, Pausch & Kiesler (2007, N164) compared an environment built around storytelling with one without story elements. The storytelling group was more motivated and spent 42% longer programming. Even so, acquisition of basic syntax was equivalent in the two groups. What a subject matter of one’s own increased was the time directed at learning, not the efficiency of learning per unit of time.

Primary research on settings where people build what they themselves want to build has long been referenced in this field. Bruckman’s (1998, N162; 2000, N163) work on virtual worlds microanalyzed the process by which one learner built what she wanted to build, and argued that situated support embedded in identity and social connection helped learning more than decontextualized instructional content itself did. Maloney et al. (2008, N165) analyzed 536 projects made over 18 months by 8- to 18-year-olds at an after-school facility, and reported that use of the main programming concepts was observed even without formal instruction or expert mentors. Neither, however, is a direct measurement of attainment with a control group.

On how interest develops there is theory and description. The four-phase model of Hidi & Renninger (2006, N158) formulated the conditions under which externally triggered interest moves to self-directed, sustained engagement. Barron (2006, N159) identified, from the cases of three adolescents, five kinds of pathway by which an interest sparked at school spreads to self-initiated learning outside school. Ito et al. (N160, N161) described in large-scale ethnography the process by which self-directed, deep immersion leads to the formation of specialized skill.

Evidence in the opposite direction needs to be looked at as well. Begel & Simon (2008, N166) shadowed new-graduate developers over their first six months and reported that a university curriculum in which problems are given explicitly does not adequately prepare the transition to self-directed problem setting. Growing up on given problems alone has a vulnerability of its own.

Measuring for Oneself

What one builds for oneself is not only tools. Self-tracking is the practice of collecting data for oneself and reading it oneself.

Choe et al. (2014, N167) analyzed 52 presentation videos and reported that while sustained practice does come about, there are pitfalls: tracking too many variables, not recording context, and lacking scientific rigor. Epstein et al. (2015, N169) position lapsing and subsequently resuming as a normal component of the model rather than as failure. The finding that persistence is not the premise matters for thinking about the lifespan of what is built for oneself. Rooksby et al. (2014, N168) reported that self-measurement is in many cases carried on as a social and collaborative practice rather than an individual act.

How Far Does the Evidence for Maker Education Go?

What has claimed learning by building most strongly is maker education. In this corpus, peer-reviewed studies that doubt its effects were deliberately collected.

What comes into view as a result is the structure that the empirical foundation of this area is itself thin. Halverson & Sheridan (2014, N171) is a positioning paper that organizes the context of the research and presents no empirical data on learning outcomes. Martin (2015, N173) organizes the field into the three elements of tools, community infrastructure, and mindset, while positioning the building of an evidence base as a task for the future. The comparative case study of three makerspaces by Sheridan et al. (2014, N172) describes how participants find problems, build models, apply skills, and exchange knowledge, but involves neither a control group nor quantitative pre-post measurement. The systematic review of Papavlasopoulou et al. (2017, N175) also takes the empirical studies of this area as its object.

Vossoughi, Hooper & Escudé (2016, N174) argue further that equity-oriented maker education can either narrow or widen the learning possibilities of working-class students and students of color depending on how it is framed. The question is who is made visible as a “maker” and who is marginalized.

Where What Is Built for Oneself Ends Up

On what happens after something is built, there is an area where quantitative empirical work is thickest. An important qualification attaches, however.

Avelino et al. (2017, N177) computed the truck factor for popular GitHub applications and reported that 46% have a factor of 1 and 28% a factor of 2. Nearly half of the projects become dysfunctional if one person leaves. Coelho et al. (2020, N182) followed 2,927 projects for a year and reported that about one sixth became unmaintained, and Ait et al. (2022, N183) showed, from a survival analysis of 1,127 repositories started in 2016, that the survival rate after five years is under 50%. Coelho & Valente (2017, N180) identified nine failure factors from a survey of 104 maintainers of discontinued projects, and Miller et al. (2025, N185) analyzed responses to the abandonment of dependencies and reported that removal is faster when the ending is explicitly announced.

The qualification is this. These are all studies of published GitHub projects, not tracking of self-use tools that were never even intended to be published. Large-scale empirical work following how long personal scripts, macros built during work, or researchers’ private code survive does not exist in this corpus. Strictly speaking, what this area has made clear is the fate not of “what was built for oneself” but of “what came to be carried by one person”.

For self-built artifacts that remain inside an organization there is a separate lineage. Avdic (2018, N181) distinguished civil servants’ spreadsheet development into primary appropriation as the use of a tool and secondary appropriation as continuing modification accompanying users’ learning and changes in organizational requirements. Spierings et al. (2016, N179) theorized the practice of filling the gaps of official systems with self-built tools and continuing to use them, presenting the four types of acceptance, refusal, hidden workaround, and open workaround. McGill (2002, N176) reported that inexperienced end-user developers tend to rate the quality of their own applications higher than experts’ assessments do.

Research addressing attachment to self-built artifacts itself could not be found however far it was searched. Exploring from the angles of the IKEA effect, psychological ownership, and digital possessions alike, sources tying any of them to self-built software numbered zero. The intuition that one finds it hard to let go of what one built oneself has not yet been measured in this field.

Implications for the Review Paper

The question “is EUD good for learning?” cannot be answered as it stands. The answer changes with what is called a learning outcome, and moreover that answer has been divided since forty years ago. The usable way of posing the question is: which learning outcome arises, for which learner, under which conditions?

Dividing into three layers makes the differences in evidential strength plain.

• Near transfer (skills close to programming): supported relatively robustly (N07). Where measurement depends on artifacts, however, validity as a proxy for understanding is limited (N18).
• Understanding of domain concepts: positive in design studies without control groups (N08, N09), but retention differs by concept (N10, N11), and there is no empirical work in business contexts.
• Far transfer (general thinking ability): negative or limited since 1984 (N01, N03, N05, N07). Using this claim in a paper cannot be justified on the current evidence.

Dividing learners works just as well. EUD’s design knowledge has been accumulated on the premise of the domain expert, and the expertise reversal effect (N27) supplies the reason why that knowledge cannot be transferred to novices. The configuration the sister note end-user-development-literature set out — that what generative AI lowered is the cost of learning, not the cost of verification — bears doubly hard on novices. Verification requires knowledge of the target domain. Novices without domain knowledge, even as the range of what they can build widens, hold no clue for judging whether what they built is correct. The report that business users failed to detect serious flaws in generated code even when warned that errors were possible (E81) is about people who do hold domain knowledge.

Once civic tech was added, the learner’s conditions divided into three. The setting of building for a salary, the setting of building for course credit, and the setting of building unpaid for the public good. But these three are divided along one axis only, that of whom one builds for. Putting in the motive of building from one’s own need calls for one more axis.

	Starting from one’s own need	Given from outside
Building for oneself	self-built tools, hobby making, self-tracking (E1–E5, E10, E12)	(scarcely arises)
Building for others	building on seeing someone in trouble, part of civic tech (D1)	orders at work, school assignments, contract development (existing D1, C1–C4)

What the mainstream of EUD research has dealt with is the lower right of this table. Both the organizational imbalance of supply and demand and the shortage of experts presuppose that the reason for building lies outside the person. What has dealt with the upper left was not EUD research but the separate lineages of user innovation research and research on motivation in open source. The two barely cite each other. The same disconnection that divided EUC and EUD in the existing corpus along the vocabularies of control and liberation is here again in another form.

What differs across the four conditions is the reason for paying the learning cost (pay, assessment, interest, one’s own trouble), the location of domain knowledge (the builder themselves, the teaching material, another person), and the destination of the artifact (a business system, a submission, government and the local community, one’s own hands). In discussing the learning outcomes of EUD, these four cannot be spoken of within a single frame.

In civic tech in particular, one condition EUD has presupposed comes off. It is the condition that the person who knows the domain builds it themselves. What happens once it comes off has been described by requirements engineering in a different context (N51, N52). The one who does not know opens up the questions, and the one who knows closes off the premises. The difficulties that collaborative design in civic tech has repeatedly reported (N83, N86, N88) can be read as a consequence of this structure.

Gaps (Unaddressed Threads)

1. Measurement of retention by delayed testing is all but absent. Of the five items addressing learning outcomes under AI assistance, zero build a delayed test administered several weeks or more later into their design; the longest is a within-semester midterm exam (N55). Across the whole corpus, the only genuine long-term follow-up is N64 (ten weeks later), and that single study yields the result that a short-term advantage disappears. No study measuring whether “a novice who has become able to build with AI becomes able to build without assistance” was found.
2. There is no research on concept formation in environments combining generative AI with low-code. Empirical work on the effects of environments that hide abstraction stops at the pre-AI comparison of blocks and text.
3. There is no empirical work on concept learning in spreadsheet modeling in business contexts. Evidence for learning through the construction of domain concepts is skewed toward educational research on agent-based modeling, and there is no empirical work on the tool most used in EUD.
4. There is no learning-sciences design in vocational training contexts. Research on the training of citizen developers is centered on the frameworks of organization theory, and only one item involves pre-post testing, and that one depends on self-report.
5. There is no EUD research targeting adult learners with zero domain knowledge. The image of the learner in this corpus is centered on K-12 and university students. There is no study tracking novices’ learning under EUD’s own conditions — adults learning for work purposes, in a situation with neither credits nor grades.
6. There is no empirical work in the Japanese-language sphere. The asymmetry between academic and practitioner discourse that the sister note observed appears in the same form in the domain of learning outcomes as well.

Gaps Made Visible by Adding Civic Tech

7. There is no quantitative research with a control group on the learning outcomes of civic tech. The studies that measured learning as their subject (N69, N74, N75, N76, N79) are confined to small-scale cases or self-report. The evidence that civic tech grows individual capability and the disconfirming evidence that participation and the persistence of artifacts do not come easily cannot be set side by side at the same strength. The latter has been obtained convergently from more varied contexts.
8. There is no research tracking the persistence of artifacts across cases. The abandonment of code after a hackathon ends has been noted as a research problem (N73), but there is no empirical work following survival rates or the reality of maintenance across multiple citizen-led projects.
9. There is no study of those who do not participate. Research addressing participants’ barriers exists (N70), but no study taking as its object the people who did not participate was found. What demonstrated the inequality of participation was an analysis of request data on the government side (N122), not the accounts of those involved.
10. There is no independent study verifying vTaiwan’s degree of policy uptake. Excluding accounts by the developers themselves revealed that, for this well-known case, no independent peer-reviewed study quantitatively verifying how much was actually reflected in policy exists.
11. There is no research comparing the learning outcomes of civic tech and workplace EUD. No study examining how learning differs when the same person uses the same tool in paid work and in unpaid public activity exists across either corpus.

Putting gaps 1 and 5 together makes visible where the review paper’s contribution can be placed. No study exists that measures retention by delayed testing in a setting where adults without domain knowledge build something for work purposes under AI assistance. Proposing that design is itself a step beyond where the current literature has arrived.

Gaps Made Visible by Adding the Motive of One’s Own Need

12. There is no empirical measurement of attachment to self-built artifacts. Exploring from the angles of the IKEA effect, psychological ownership, and digital possessions alike, sources tying any of them to self-built software numbered zero. The intuition that one finds it hard to let go of what one built oneself has not yet been measured.
13. There is no empirical work tracking non-public self-use tools. The large-scale empirical work on abandonment rates all takes published GitHub projects as its object, and there is no study following the survival of personal scripts or of macros built during work. What this area has made clear is the fate of “what came to be carried by one person”, not the fate of “what was built for oneself”.
14. There is no learning research directly comparing building for one’s own need with building on assignment. Research on self-selection deals with the choice of task content, and the case studies deal only with settings of self-selection. There is no design comparing the two conditions within the same population of learners. Kelleher et al. (N164) come closest, but what they compare is the personal meaning of the subject matter, not the origin of the motive.
15. The scale estimates of user innovation and EUD research are not connected. The estimate that 6.1% of consumers develop or modify products (N137) and the population estimates of end-user developers that EUD research works with (Scaffidi et al. in the existing corpus) coexist with different methodologies and different definitions. No study bringing the two together was found.

Gaps 7 and 11 add an axis of comparison to that. If the reason for paying the learning cost changes with the presence or absence of pay, then what is learned should change even with the same tool. If this comparison can be designed, the learning outcomes of EUD can be discussed as a function of the conditions of building rather than of the properties of the tool.

Gap 14 gives that design its concrete form. Building that began from one’s own need and building given from outside are compared within the same population of learners, and retention is measured by delayed testing. Given that Kelleher et al. (N164) produced a difference in motivation and in time without producing a difference in attainment, the prediction for this comparison would be that time increases while efficiency per unit of time does not change. Whether that is right, no one has yet confirmed.

Unverified Items

All 185 items were re-checked on 2026-08-14. The 182 items carrying a DOI were queried against the Crossref REST API for machine collation of their bibliographic records, and wherever an abstract could be retrieved, its statements about samples and methods were extracted. Books and proceedings, the review standards of the journals, and the existence of peer-reviewed versions of preprints were each confirmed against publisher and society pages.

The main figures settled by the re-check are as follows.

• N02 (Clements & Gullo 1984): eighteen 6-year-olds randomly assigned to a 12-week intervention. The pretest covered receptive vocabulary, impulsivity/reflectivity, and divergent thinking
• N06 (Liao & Bright 1991): 432 comparisons, 89% of them with positive effect sizes, and a study-weighted mean of 0.41
• N12 (Torkzadeh & Koufteros 1994): a 30-item computer self-efficacy scale administered to 224 undergraduates at the start and the end of an introductory course
• N30 (Simonsmeier et al. 2022): a meta-analysis over 8,776 effect sizes
• N118 (Cardullo & Kitchin 2019): twenty interviews with city officers and corporate exhibitors at the 2017 Smart City Expo
• N122 (Clark et al. 2013): one year of service requests to the City of Boston across 2010–2011, analyzed with geospatial analysis and negative binomial regression
• N124 (Haughton & McManus 2020): interviews with 25 key informants, plus attendance at six public consultation and protest events
• N127 (Erete 2015): a three-year study across five communities, with interviews of 45 residents and qualitative content analysis of over 7,000 online messages
• N143 (Lakhani & Wolf 2005): a survey of 684 developers across 287 projects. This figure, deleted during the first pass, has been restored
• N156 (Guzdial & Forte 2005): roughly 90% of enrolled students earned an A, B, or C, and 80% reported an effect more than a semester later
• N178 (Avelino et al. 2016): developers of 67 target systems were asked to confirm the results, and 84% agreed or partially agreed
• N48 (Nuhfer & Fleisher 2026): the volume and issue are settled as Numeracy 19(1), art.1479
• N61 (Paludo & Montresor 2026): a peer-reviewed article published in Italian Journal of Educational Technology 34(1) on February 26, 2026, with 40 students as its subjects. It is not registered with Crossref, but the DOI resolves
• N161 (Ito et al. 2013): a research synthesis report issued by the Digital Media and Learning Research Hub (ISBN 978-0-9887255-0-8), not a peer-reviewed publication
• N177 (Avelino et al. 2017): still a PeerJ Preprint, never published through peer review; the service stopped accepting new submissions in September 2019. It is a separate output from N178, with a different set of authors
• N160 (Ito et al.): the DOI points to the 2019 MIT Press Open edition (the tenth-anniversary edition), whose text is described as identical to the 2009 edition apart from an added foreword

The items that remain unverified are listed below; in every case the figure is absent from the abstract and the full text could not be reached.

• Sample or methodological detail that the abstract does not settle: N29 (number of participants), N51 (statistical detail), N66 (sample size), N79 (extent of its own primary data), N94 (external validity of the effect), N96 (survey sample size; its non-disclosure is confirmed), N100 (size of the pilot), N102 (sample size and format; the abstract itself could not be retrieved), N103 (whether the experiment was online or face-to-face; only its running in Finland in November 2006 is settled), N105 (which format came out ahead), N107 (data collection method), N109 (survey sample size), N123, N130, N138, N140, N141, N144, N145, N148, N150, N151, N152, N154, N155, N166, N168, N175, N176, N181, N184, N185 [unverified]
• Breakdowns within meta-analyses and scales: the number of studies N06 covered (the count of 432 comparisons is settled), what changed in N12’s factor structure between pre- and post-training, and the period of the studies N30 included [unverified]
• Bibliographic record settled, but neither the full text nor a detailed abstract reached: N117, N119, N131, N134, N135, N142, N162, N170 (volume, issue, and pages settled through Crossref for all of them; the description of each contribution stays within what the title and the record support) [unverified]
• Preprints with no peer-reviewed version: N60 (Ma et al. 2025; the arXiv journal-ref field is empty) [unverified]
• N72 (Almeida & de Souza 2022): an autoethnographic workshop paper (n=1) with limited generalizability (a known limitation rather than an unverified item)
• Across all 185 items, each work’s “main contribution” is a summary based on the bibliographic record and the abstract, not on a reading of the full text. When writing the paper, at minimum every work used as evidence in the argument will have to be read in the original

For 17 items the year of online-first publication differs from the year of the print issue (N23, N40, N52, N63, N65, N74, N95, N96, N98, N99, N110, N111, N112, N117, N139, N160, N179). This note cites the online-first year throughout; in the reference list, the print issue is given alongside only for N74 and N179, the two whose years were queried in the first pass.

References

Verification of transfer (A1)

• N01. Pea, R.D. & Kurland, D.M. (1984). On the cognitive effects of learning computer programming. New Ideas in Psychology, 2(2), 137–168. https://doi.org/10.1016/0732-118x(84)90018-7
• N02. Clements, D.H. & Gullo, D.F. (1984). Effects of computer programming on young children’s cognition. Journal of Educational Psychology, 76(6), 1051–1058. https://doi.org/10.1037/0022-0663.76.6.1051
• N03. Mayer, R.E., Dyck, J.L. & Vilberg, W. (1986). Learning to program and learning to think: What’s the connection? Communications of the ACM, 29(7), 605–610. https://doi.org/10.1145/6138.6142
• N04. Salomon, G. & Perkins, D.N. (1987). Transfer of cognitive skills from programming: When and how? Journal of Educational Computing Research, 3(2), 149–169. https://doi.org/10.2190/6f4q-7861-qwa5-8pl1
• N05. Palumbo, D.B. (1990). Programming language/problem-solving research: A review of relevant issues. Review of Educational Research, 60(1), 65–89. https://doi.org/10.3102/00346543060001065
• N06. Liao, Y.C. & Bright, G.W. (1991). Effects of computer programming on cognitive outcomes: A meta-analysis. Journal of Educational Computing Research, 7(3), 251–268. https://doi.org/10.2190/E53G-HH8K-AJRR-K69M
• N07. Scherer, R., Siddiq, F. & Sánchez Viveros, B. (2019). The cognitive benefits of learning computer programming: A meta-analysis of transfer effects. Journal of Educational Psychology, 111(5), 764–792. https://doi.org/10.1037/edu0000314

Learning through constructing domain concepts (A2)

• N08. Wilensky, U. & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories. Cognition and Instruction, 24(2), 171–209. https://doi.org/10.1207/s1532690xci2402_1
• N09. Blikstein, P. & Wilensky, U. (2009). An atom is known by the company it keeps: A constructionist learning environment for materials science using agent-based modeling. International Journal of Computers for Mathematical Learning, 14(2), 81–119. https://doi.org/10.1007/s10758-009-9148-8
• N10. Meerbaum-Salant, O., Armoni, M. & Ben-Ari, M. (2013). Learning computer science concepts with Scratch. Computer Science Education, 23(3), 239–264. https://doi.org/10.1080/08993408.2013.832022
• N11. Grover, S., Pea, R. & Cooper, S. (2015). Designing for deeper learning in a blended computer science course for middle school students. Computer Science Education, 25(2), 199–237. https://doi.org/10.1080/08993408.2015.1033142

Self-efficacy and attitudes (A3)

• N12. Torkzadeh, G. & Koufteros, X. (1994). Factorial validity of a computer self-efficacy scale and the impact of computer training. Educational and Psychological Measurement, 54(3), 813–821. https://doi.org/10.1177/0013164494054003028
• N13. Compeau, D.R. & Higgins, C.A. (1995). Computer self-efficacy: Development of a measure and initial test. MIS Quarterly, 19(2), 189–211. https://doi.org/10.2307/249688
• N14. Wiedenbeck, S. (2005). Factors affecting the success of non-majors in learning to program. Proc. ICER 2005, 13–24. https://doi.org/10.1145/1089786.1089788
• N15. Kong, S.C., Chiu, M.M. & Lai, M. (2018). A study of primary school students’ interest, collaboration attitude, and programming empowerment in computational thinking education. Computers & Education, 127, 178–189. https://doi.org/10.1016/j.compedu.2018.08.026
• N16. Gorson, J. & O’Rourke, E. (2020). Why do CS1 students think they’re bad at programming? Investigating self-efficacy and self-assessments at three universities. Proc. ICER 2020, 170–181. https://doi.org/10.1145/3372782.3406273

Measurement and construct validity (A4)

• N17. Robins, A., Rountree, J. & Rountree, N. (2003). Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137–172. https://doi.org/10.1076/csed.13.2.137.14200
• N18. Denner, J., Werner, L. & Ortiz, E. (2012). Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts? Computers & Education, 58(1), 240–249. https://doi.org/10.1016/j.compedu.2011.08.006
• N19. Brennan, K. & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. Proc. AERA 2012 Annual Meeting, Vancouver, BC, April 13–17, 2012, 25pp. https://scratched.gse.harvard.edu/ct/files/AERA2012.pdf
• N20. Grover, S. & Pea, R. (2013). Computational thinking in K-12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189x12463051
• N21. Lye, S.Y. & Koh, J.H.L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012
• N22. Román-González, M., Pérez-González, J.C. & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb.2016.08.047
• N23. Popat, S. & Starkey, L. (2018). Learning to code or coding to learn? A systematic review. Computers & Education, 128, 365–376. https://doi.org/10.1016/j.compedu.2018.10.005
• N24. Tang, X., Yin, Y., Lin, Q., Hadad, R. & Zhai, X. (2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019.103798

Cognitive load and domain knowledge (B1)

• N25. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4
• N26. Paas, F.G.W.C. & van Merriënboer, J.J.G. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86(1), 122–133. https://doi.org/10.1037/0022-0663.86.1.122
• N27. Kalyuga, S., Ayres, P., Chandler, P. & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31. https://doi.org/10.1207/s15326985ep3801_4
• N28. Renkl, A. & Atkinson, R.K. (2003). Structuring the transition from example study to problem solving in cognitive skill acquisition: A cognitive load perspective. Educational Psychologist, 38(1), 15–22. https://doi.org/10.1207/s15326985ep3801_3
• N29. Kalyuga, S. & Sweller, J. (2004). Measuring knowledge to optimize cognitive load factors during instruction. Journal of Educational Psychology, 96(3), 558–568. https://doi.org/10.1037/0022-0663.96.3.558
• N30. Simonsmeier, B.A., Flaig, M., Deiglmayr, A., Schalk, L. & Schneider, M. (2022). Domain-specific prior knowledge and learning: A meta-analysis. Educational Psychologist, 57(1), 31–54. https://doi.org/10.1080/00461520.2021.1939700

The controversy over guidance (B2)

• N31. Mayer, R.E. (2004). Should there be a three-strikes rule against pure discovery learning? American Psychologist, 59(1), 14–19. https://doi.org/10.1037/0003-066x.59.1.14
• N32. Kirschner, P.A., Sweller, J. & Clark, R.E. (2006). Why minimal guidance during instruction does not work. Educational Psychologist, 41(2), 75–86. https://doi.org/10.1207/s15326985ep4102_1
• N33. Hmelo-Silver, C.E., Duncan, R.G. & Chinn, C.A. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99–107. https://doi.org/10.1080/00461520701263368
• N34. Alfieri, L., Brooks, P.J., Aldrich, N.J. & Tenenbaum, H.R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017

Novice program comprehension (B3)

• N35. du Boulay, B. (1986). Some difficulties of learning to program. Journal of Educational Computing Research, 2(1), 57–73. https://doi.org/10.2190/3lfx-9rrf-67t8-uvk9
• N36. Pea, R.D. (1986). Language-independent conceptual “bugs” in novice programming. Journal of Educational Computing Research, 2(1), 25–36. https://doi.org/10.2190/689t-1r2a-x4w4-29j2
• N37. Soloway, E. (1986). Learning to program = Learning to construct mechanisms and explanations. Communications of the ACM, 29(9), 850–858. https://doi.org/10.1145/6592.6594
• N38. Spohrer, J.C. & Soloway, E. (1986). Novice mistakes: Are the folk wisdoms correct? Communications of the ACM, 29(7), 624–632. https://doi.org/10.1145/6138.6145
• N39. Sorva, J. (2013). Notional machines and introductory programming education. ACM Transactions on Computing Education, 13(2), 1–31. https://doi.org/10.1145/2483710.2483713
• N40. Qian, Y. & Lehman, J. (2017). Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education, 18(1), 1–24. https://doi.org/10.1145/3077618

Metacognition and the calibration of self-assessment (B4)

• N41. Kruger, J. & Dunning, D. (1999). Unskilled and unaware of it. Journal of Personality and Social Psychology, 77(6), 1121–1134. https://doi.org/10.1037/0022-3514.77.6.1121
• N42. Rozenblit, L. & Keil, F. (2002). The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive Science, 26(5), 521–562. https://doi.org/10.1207/s15516709cog2605_1
• N43. Bjork, R.A., Dunlosky, J. & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444. https://doi.org/10.1146/annurev-psych-113011-143823
• N44. Nuhfer, E., Fleisher, S., Cogan, C., Wirth, K. & Gaze, E. (2017). How random noise and a graphical convention subverted behavioral scientists’ explanations of self-assessment data. Numeracy, 10(1), art.4. https://doi.org/10.5038/1936-4660.10.1.4
• N45. Gignac, G.E. & Zajenkowski, M. (2020). The Dunning-Kruger effect is (mostly) a statistical artefact. Intelligence, 80, art.101449. https://doi.org/10.1016/j.intell.2020.101449
• N46. Prather, J., Becker, B.A., Craig, M., Denny, P., Loksa, D. & Margulieux, L. (2020). What do we think we think we are doing? Metacognition and self-regulation in programming. Proc. ICER 2020, 2–13. https://doi.org/10.1145/3372782.3406263
• N47. Loksa, D., Margulieux, L., Becker, B.A., Craig, M., Denny, P., Pettit, R. & Prather, J. (2022). Metacognition and self-regulation in programming education: Theories and exemplars of use. ACM Transactions on Computing Education, 22(4), 1–31. https://doi.org/10.1145/3487050
• N48. Nuhfer, E. & Fleisher, S. (2026). Unskilled but aware of it: Understanding self-assessment by estimating guessing from cognitive-metacognitive paired measures. Numeracy, 19(1), art.1479. https://doi.org/10.5038/1936-4660.19.1.1479

Domain knowledge and problem definition (B5)

• N49. Chi, M.T.H., Feltovich, P.J. & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121–152. https://doi.org/10.1207/s15516709cog0502_2
• N50. Jonassen, D.H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65–94. https://doi.org/10.1007/bf02299613
• N51. Niknafs, A. & Berry, D.M. (2012). The impact of domain knowledge on the effectiveness of requirements idea generation during requirements elicitation. Proc. IEEE RE 2012, 181–190. https://doi.org/10.1109/re.2012.6345802
• N52. Hadar, I., Soffer, P. & Kenzi, K. (2012/2014). The role of domain knowledge in requirements elicitation via interviews: An exploratory study. Requirements Engineering, 19(2), 143–159. https://doi.org/10.1007/s00766-012-0163-2

Learning outcomes under AI assistance (C1)

• N53. Kazemitabaar, M., Hou, X., Henley, A., Ericson, B.J., Weintrop, D. & Grossman, T. (2023). How novices use LLM-based code generators to solve CS1 coding tasks in a self-paced learning environment. Proc. Koli Calling 2023. https://doi.org/10.1145/3631802.3631806
• N54. Johnson, S., Doss, W. & Estepp, C. (2024). Using ChatGPT with novice Arduino programmers. Journal of Research in Technical Careers, 8(1). https://doi.org/10.9741/2578-2118.1152
• N55. Kosar, T., Ostojić, D., Liu, Y.D. & Mernik, M. (2024). Computer science education in the ChatGPT era. Mathematics, 12(5), 629. https://doi.org/10.3390/math12050629
• N56. Haindl, P. & Weinberger, G. (2024). Does ChatGPT help novice programmers write better code? A comparative study. IEEE Access. https://doi.org/10.1109/access.2024.3445432
• N57. Melanou, E. & Beege, M. (2026). Scaffolding generative AI as a tutor: A quasi-experimental study of learning outcomes and motivational, cognitive and metacognitive processes. Education Sciences, 16(4). https://doi.org/10.3390/educsci16040651

Metacognition and help-seeking (C2)

• N58. Scholl, A. & Kiesler, N. (2024). How novice programmers use and experience ChatGPT when solving programming exercises. Proc. IEEE FIE 2024. https://doi.org/10.1109/fie61694.2024.10893442
• N59. Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C. & MacNeil, S. (2024). The effects of generative AI on computing students’ help-seeking preferences. Proc. ACE 2024. https://doi.org/10.1145/3636243.3636248
• N60. Ma, B., Li, Y., Li, X., Chen, T., Tang, Y., Xie, T., Gu, X., Shimada, A. & Konomi, S. (2025). Scaffolding metacognition in programming education: Understanding student-AI interactions and design implications. arXiv:2511.04144. https://arxiv.org/abs/2511.04144 [unverified]
• N61. Paludo, M. & Montresor, A. (2026). Toward a new paradigm of AI-powered programming education through metacognition. Italian Journal of Educational Technology, 34(1), published February 26, 2026 (peer-reviewed; not registered with Crossref, so the DOI appears to come from another registration agency such as mEDRA). https://doi.org/10.17471/2499-4324/1457

Environments that hide abstraction, and concept formation (C3)

• N62. Weintrop, D. & Wilensky, U. (2015). Using commutative assessments to compare conceptual understanding in blocks-based and text-based programs. Proc. ICER 2015. https://doi.org/10.1145/2787622.2787721
• N63. Weintrop, D. & Wilensky, U. (2017). Comparing block-based and text-based programming in high school computer science classrooms. ACM Transactions on Computing Education, 18(1), 1–25. https://doi.org/10.1145/3089799
• N64. Weintrop, D. & Wilensky, U. (2019). Transitioning from introductory block-based and text-based environments to professional programming languages in high school computer science classrooms. Computers & Education, 142, 103646. https://doi.org/10.1016/j.compedu.2019.103646

Vocational training and reskilling (C4)

• N65. McHugh, M., Carroll, N. & Connolly, C. (2023/2024). Low-code and no-code development platforms: A case study in secondary education. Computers in the Schools, 41(4), 399–424. https://doi.org/10.1080/07380569.2023.2256729
• N66. Matook, S., Wang, Y. & Axelsen, M. (2025). Experiential learning for citizen developers: Training IT talent in low-code development and metacognitive reflection. Business & Information Systems Engineering, 67(1), 7–30. https://doi.org/10.1007/s12599-024-00921-3

Civic hackathons and civic hacking (D1)

• N67. Lodato, T.J. & DiSalvo, C. (2016). Issue-oriented hackathons as material participation. New Media & Society, 18(4), 539–557. https://doi.org/10.1177/1461444816629467
• N68. Schrock, A.R. (2016). Civic hacking as data activism and advocacy: A history from publicity to open government data. New Media & Society, 18(4), 581–599. https://doi.org/10.1177/1461444816629469
• N69. Porter, E., Bopp, C., Gerber, E. & Voida, A. (2017). Reappropriating hackathons: The production work of the CHI4Good Day of Service. Proc. CHI 2017. https://doi.org/10.1145/3025453.3025637
• N70. Young, M. & Yan, A. (2017). Civic hackers’ user experiences and expectations of Seattle’s open municipal data program. Proc. HICSS-50. https://doi.org/10.24251/hicss.2017.324
• N71. Falk, J., Kannabiran, G. & Hansen, N.B. (2021). What do hackathons do? Understanding participation in hackathons through program theory analysis. Proc. CHI 2021. https://doi.org/10.1145/3411764.3445198
• N72. Almeida, L.A. & de Souza, C.R.B. (2022). A social-cognitive analysis of a female-focused hackathon. Proc. GEICSE Workshop (ICSE 2022). https://doi.org/10.1145/3524501.3527597
• N73. Falk, J., Nolte, A., Huppenkothen, D., Weinzierl, M., Gama, K., Spikol, D., Tollerud, E., Chue Hong, N., Knäpper, I. & Hayden, L.B. (2024). The future of hackathon research and practice. IEEE Access, 12. https://doi.org/10.1109/access.2024.3455092

Data literacy and learning in citizen science (D2)

• N74. Ballard, H.L., Dixon, C.G.H. & Harris, E.M. (2016). Youth-focused citizen science: Examining the role of environmental science learning and agency for conservation. Biological Conservation, 208, 65–75 (print issue April 2017). https://doi.org/10.1016/j.biocon.2016.05.024
• N75. Bhargava, R., Kadouaki, R., Bhargava, E., Castro, G. & D’Ignazio, C. (2016). Data murals: Using the arts to build data literacy. Journal of Community Informatics, 12(3). https://doi.org/10.15353/joci.v12i3.3285
• N76. D’Ignazio, C. & Bhargava, R. (2016). DataBasic: Design principles, tools and activities for data literacy learners. Journal of Community Informatics, 12(3). https://doi.org/10.15353/joci.v12i3.3280
• N77. Kosmala, M., Wiggins, A., Swanson, A. & Simmons, B. (2016). Assessing data quality in citizen science. Frontiers in Ecology and the Environment, 14(10), 551–560. https://doi.org/10.1002/fee.1436
• N78. Frensley, T., Crall, A., Stern, M., Jordan, R., Gray, S., Prysby, M., Newman, G., Hmelo-Silver, C., Mellor, D. & Huang, J. (2017). Bridging the benefits of online and community supported citizen science. Citizen Science: Theory and Practice, 2(1), art.4. https://doi.org/10.5334/cstp.84
• N79. Phillips, T., Porticella, N., Constas, M. & Bonney, R. (2018). A framework for articulating and measuring individual learning outcomes from participation in citizen science. Citizen Science: Theory and Practice, 3(2). https://doi.org/10.5334/cstp.126 [unverified]
• N80. Rothschild, A., Meng, A., DiSalvo, C., Johnson, B., Shapiro, B.R. & DiSalvo, B. (2022). Interrogating data work as a community of practice. PACM HCI, 6(CSCW2), art.371. https://doi.org/10.1145/3555198
• N81. Dove, G., Shanley, J., Matuk, C. & Nov, O. (2023). Open data intermediaries: Motivations, barriers and facilitators to engagement. PACM HCI, 7(CSCW1), art.135. https://doi.org/10.1145/3579511

Collaborative design, trust, and institutional constraints (D3)

• N82. Erete, S.L. (2013). Community, group and individual: A framework for designing community technologies. Journal of Community Informatics, 10(1). https://doi.org/10.15353/joci.v10i1.2671
• N83. Corbett, E. & Le Dantec, C.A. (2018). Exploring trust in digital civics. Proc. DIS 2018. https://doi.org/10.1145/3196709.3196715
• N84. Lodato, T. & DiSalvo, C. (2018). Institutional constraints: The forms and limits of participatory design in the public realm. Proc. PDC 2018. https://doi.org/10.1145/3210586.3210595
• N85. Knutas, A., Palacin, V., Maccani, G. & Helfert, M. (2019). Software engineering in civic tech: A case study about Code for Ireland. Proc. ICSE-SEIS 2019, 41–50. https://doi.org/10.1109/icse-seis.2019.00013
• N86. Harrington, C., Erete, S. & Piper, A.M. (2019). Deconstructing community-based collaborative design: Towards more equitable participatory design engagements. PACM HCI, 3(CSCW), art.216. https://doi.org/10.1145/3359318
• N87. Gooch, D., Kelly, R.M., Stiver, A., van der Linden, J., Petre, M., Richards, M., Klis-Davies, A., MacKinnon, J., Macpherson, R. & Walton, C. (2020). The benefits and challenges of using crowdfunding to facilitate community-led projects in the context of digital civics. International Journal of Human-Computer Studies, 134. https://doi.org/10.1016/j.ijhcs.2019.10.005
• N88. Corbett, E. & Le Dantec, C.A. (2021). Designing civic technology with trust. Proc. CHI 2021. https://doi.org/10.1145/3411764.3445341
• N89. Knutas, A., Palacin, V., Maccani, G., Aragon, P., Wolff, A. & Mocek, L. (2022). Civic code for social change: Lessons in civic tech grassroots for software engineers. IEEE Software, 39(6), 64–72. https://doi.org/10.1109/ms.2022.3179670

The design and empirical study of participation platforms (D4)

• N90. Aragón, P., Kaltenbrunner, A., Calleja-López, A., Pereira, A., Monterde, A., Barandiaran, X.E. & Gómez, V. (2017). Deliberative platform design: The case study of the online discussions in Decidim Barcelona. Social Informatics (LNCS 10540). https://doi.org/10.1007/978-3-319-67256-4_22
• N91. Aragón, P., Gómez, V. & Kaltenbrunner, A. (2017). Detecting platform effects in online discussions. Policy & Internet, 9(4), 420–443. https://doi.org/10.1002/poi3.158
• N92. Falco, E. & Kleinhans, R. (2018). Digital participatory platforms for co-production in urban development. International Journal of E-Planning Research, 7(3). https://doi.org/10.4018/ijepr.2018070105
• N93. Royo, S., Pina Martínez, V. & García-Rayado, J. (2020). Decide Madrid: A critical analysis of an award-winning e-participation initiative. Sustainability, 12(4). https://doi.org/10.3390/su12041674
• N94. Arana-Catania, M., Van Lier, F.-A., Procter, R., Tkachenko, N., He, Y., Zubiaga, A. & Liakata, M. (2021). Citizen participation and machine learning for a better democracy. Digital Government: Research and Practice, 2(3). https://doi.org/10.1145/3452118 [unverified]
• N95. Tseng, Y.-S. (2022). Rethinking gamified democracy as frictional: A comparative examination of the Decide Madrid and vTaiwan platforms. Social & Cultural Geography, 24(8). https://doi.org/10.1080/14649365.2022.2055779
• N96. Borge, R., Balcells, J. & Padró-Solanet, A. (2022). Democratic disruption or continuity? Analysis of the Decidim platform in Catalan municipalities. American Behavioral Scientist, 67(7). https://doi.org/10.1177/00027642221092798 [unverified]
• N97. Menéndez-Blanco, M. & Bjørn, P. (2022). Designing digital participatory budgeting platforms: Urban biking activism in Madrid. Computer Supported Cooperative Work, 31(4). https://doi.org/10.1007/s10606-022-09443-6
• N98. Bartocci, L., Grossi, G., Mauro, S.G. & Ebdon, C. (2022). The journey of participatory budgeting: A systematic literature review and future research directions. International Review of Administrative Sciences, 89(3). https://doi.org/10.1177/00208523221078938
• N99. Moats, D. & Tseng, Y.-S. (2023). Sorting a public? Using quali-quantitative methods to interrogate the role of algorithms in digital democracy platforms. Information, Communication & Society, 27(5). https://doi.org/10.1080/1369118x.2023.2230286
• N100. Kermanshahi, S.N., van der Tuin, I., Dirks, S., Overkempe, T., Hardman, L. & Pieters, T. (2024). Re-mediatization as a new digital approach towards local political deliberation. Digital Society, 3(3). https://doi.org/10.1007/s44206-024-00154-7 [unverified]
• N101. Yang, J.C. & Bachmann, F. (2025). Bridging voting and deliberation with algorithms: Field insights from vTaiwan and Kultur Komitee. Proc. ACM FAccT 2025. https://doi.org/10.1145/3715275.3732205

Deliberative effects, representativeness, and policy uptake (D5)

• N102. Fishkin, J.S. & Luskin, R.C. (2005). Experimenting with a democratic ideal: Deliberative polling and public opinion. Acta Politica, 40(3), 284–298. https://doi.org/10.1057/palgrave.ap.5500121 [unverified]
• N103. Grönlund, K., Setälä, M. & Herne, K. (2010). Deliberation and civic virtue: Lessons from a citizen deliberation experiment. European Political Science Review, 2(1), 95–117. https://doi.org/10.1017/s1755773909990245 [unverified]
• N104. Neblo, M.A., Esterling, K.M., Kennedy, R.P., Lazer, D.M.J. & Sokhey, A.E. (2010). Who wants to deliberate—and why? American Political Science Review, 104(3), 566–583. https://doi.org/10.1017/s0003055410000298
• N105. Monnoyer-Smith, L. & Wojcik, S. (2012). Technology and the quality of public deliberation: A comparison between a citizen forum and an online forum. International Journal of Electronic Governance, 5(1). https://doi.org/10.1504/ijeg.2012.047443 [unverified]
• N106. Lazer, D., Sokhey, A.E., Neblo, M.A., Esterling, K.M. & Kennedy, R. (2015). Expanding the conversation: Multiplier effects from a deliberative field experiment. Political Communication, 32(4), 552–573. https://doi.org/10.1080/10584609.2015.1017032
• N107. Su, C. (2017). From Porto Alegre to New York City: Participatory budgeting and democracy. New Political Science, 39(1), 67–75. https://doi.org/10.1080/07393148.2017.1278854 [unverified]
• N108. Font, J., Smith, G., Galais, C. & Alarcón, P. (2018). Cherry-picking participation: Explaining the fate of proposals from participatory processes. European Journal of Political Research, 57(3), 615–636. https://doi.org/10.1111/1475-6765.12248
• N109. Pow, J., van Dijk, L. & Marién, S. (2020). It’s not just the taking part that counts: ‘Like me’ perceptions and support for the legitimacy of minipublics. Journal of Deliberative Democracy, 16(2). https://doi.org/10.16997/jdd.368 [unverified]
• N110. Moore, A., Fredheim, R., Wyss, D. & Beste, S. (2020). Deliberation and identity rules: The effect of anonymity, pseudonyms and real-name requirements on the cognitive complexity of online news comments. Political Studies, 69(1), 45–65. https://doi.org/10.1177/0032321719891385
• N111. Knobloch, K.R. & Gastil, J. (2021). How deliberative experiences shape subjective outcomes: A meta-analysis of citizens’ initiative reviews. Journal of Deliberative Democracy, 18(1). https://doi.org/10.16997/jdd.942
• N112. van Dijk, L. & Lefevere, J. (2022). Can the use of minipublics backfire? A survey experiment. European Journal of Political Research, 62(1). https://doi.org/10.1111/1475-6765.12523

Critiques of the smart city and governmentality (D6)

• N113. Kitchin, R. (2014). The real-time city? Big data and smart urbanism. GeoJournal, 79(1), 1–14. https://doi.org/10.1007/s10708-013-9516-8
• N114. Vanolo, A. (2014). Smartmentality: The smart city as disciplinary strategy. Urban Studies, 51(5), 883–898. https://doi.org/10.1177/0042098013494427
• N115. Shelton, T., Zook, M. & Wiig, A. (2015). The ‘actually existing smart city’. Cambridge Journal of Regions, Economy and Society, 8(1), 13–25. https://doi.org/10.1093/cjres/rsu026
• N116. Sadowski, J. & Pasquale, F. (2015). The spectrum of control: A social theory of the smart city. First Monday, 20(7). https://doi.org/10.5210/fm.v20i7.5903
• N117. Cardullo, P. & Kitchin, R. (2018). Being a ‘citizen’ in the smart city: Up and down the scaffold of smart citizen participation in Dublin, Ireland. GeoJournal, 84(1), 1–13. https://doi.org/10.1007/s10708-018-9845-8 [unverified]
• N118. Cardullo, P. & Kitchin, R. (2019). Smart urbanism and smart citizenship: The neoliberal logic of ‘citizen-focused’ smart cities in Europe. Environment and Planning C: Politics and Space, 37(5), 813–830. https://doi.org/10.1177/0263774X18806508 [unverified]
• N119. Johnson, P.A., Robinson, P.J. & Philpot, S. (2020). Type, tweet, tap, and pass: How smart city technology is creating a transactional citizen. Government Information Quarterly, 37(1), 101414. https://doi.org/10.1016/j.giq.2019.101414 [unverified]

The depoliticization of participation, and equity (D7)

• N120. DiSalvo, C. (2009). Design and the construction of publics. Design Issues, 25(1), 48–63. https://doi.org/10.1162/desi.2009.25.1.48
• N121. Björgvinsson, E., Ehn, P. & Hillgren, P.-A. (2012). Agonistic participatory design: Working with marginalised social movements. CoDesign, 8(2–3), 127–144. https://doi.org/10.1080/15710882.2012.672577
• N122. Clark, B.Y., Brudney, J.L. & Jang, S.-G. (2013). Coproduction of government services and the new information technology: Investigating the distributional biases. Public Administration Review, 73(5), 687–701. https://doi.org/10.1111/puar.12092 [unverified]
• N123. Gabrys, J. (2017). Citizen sensing, air pollution and fracking: From ‘caring about your air’ to speculative practices of evidencing harm. The Sociological Review, 65(2_suppl), 172–192. https://doi.org/10.1177/0081176917710421 [unverified]
• N124. Haughton, G. & McManus, P. (2020). Participation in postpolitical times. In Hou, J. & Knierbein, S. (eds.) Learning from Arnstein’s Ladder. Routledge, 228–250. https://doi.org/10.4324/9780429290091-19 [unverified]
• N125. Crivellaro, C., Coles-Kemp, L. & Wood, K. (2021). ‘Computer says no’: Exploring social justice in digital services. In Hope Under Neoliberal Austerity. Policy Press, ch.7. https://doi.org/10.1332/policypress/9781447356820.003.0007

Co-production and conditional defense (D8)

• N126. Linders, D. (2012). From e-government to we-government: Defining a typology for citizen coproduction in the age of social media. Government Information Quarterly, 29(4), 446–454. https://doi.org/10.1016/j.giq.2012.06.003
• N127. Erete, S.L. (2015). Engaging around neighborhood issues: How online communication affects offline behavior. Proc. ACM CSCW 2015, 1590–1601. https://doi.org/10.1145/2675133.2675182 [unverified]
• N128. Voorberg, W.H., Bekkers, V.J.J.M. & Tummers, L.G. (2015). A systematic review of co-creation and co-production: Embarking on the social innovation journey. Public Management Review, 17(9), 1333–1357. https://doi.org/10.1080/14719037.2014.930505
• N129. Peixoto, T. & Fox, J. (2016). When does ICT-enabled citizen voice lead to government responsiveness? IDS Bulletin, 47(1). https://doi.org/10.19088/1968-2016.104
• N130. Falco, E. & Kleinhans, R. (2018). Beyond technology: Identifying local government challenges for using digital platforms for citizen engagement. International Journal of Information Management, 40. https://doi.org/10.1016/j.ijinfomgt.2018.01.007 [unverified]

Theory and empirical work on user innovation (E1)

• N131. von Hippel, E. (1976). The dominant role of users in the scientific instrument innovation process. Research Policy. https://doi.org/10.1016/0048-7333(76)90028-7 [unverified]
• N132. von Hippel, E. (1986). Lead users: A source of novel product concepts. Management Science, 32(7), 791. https://doi.org/10.1287/mnsc.32.7.791
• N133. von Hippel, E. & Katz, R. (2002). Shifting innovation to users via toolkits. Management Science, 48(7), 821. https://doi.org/10.1287/mnsc.48.7.821.2817
• N134. Harhoff, D., Henkel, J. & von Hippel, E. (2003). Profiting from voluntary information spillovers: How users benefit by freely revealing their innovations. Research Policy. https://doi.org/10.1016/s0048-7333(03)00061-1 [unverified]
• N135. Franke, N. & Shah, S. (2003). How communities support innovative activities: An exploration of assistance and sharing among end-users. Research Policy. https://doi.org/10.1016/s0048-7333(02)00006-9 [unverified]
• N136. Baldwin, C. & von Hippel, E. (2011). Modeling a paradigm shift: From producer innovation to user and open collaborative innovation. Organization Science. https://doi.org/10.1287/orsc.1100.0618
• N137. von Hippel, E., de Jong, J.P.J. & Flowers, S. (2012). Comparing business and household sector innovation in consumer products: Findings from a representative study in the United Kingdom. Management Science. https://doi.org/10.1287/mnsc.1110.1508
• N138. de Jong, J.P.J., von Hippel, E., Gault, F., Kuusisto, J. & Raasch, C. (2015). Market failure in the diffusion of consumer-developed innovations: Patterns in Finland. Research Policy. https://doi.org/10.1016/j.respol.2015.06.015 [unverified]
• N139. von Hippel, E. (2017). Free Innovation. MIT Press (hardcover released November 18, 2016). ISBN 978-0-262-03521-7. An open-access edition is available under CC BY-NC-ND 4.0. https://doi.org/10.7551/mitpress/9382.001.0001

Consumer innovators and entrepreneurship (E2)

• N140. Shah, S.K. & Tripsas, M. (2007). The accidental entrepreneur: The emergent and collective process of user entrepreneurship. Strategic Entrepreneurship Journal. https://doi.org/10.1002/sej.15 [unverified]
• N141. Ogawa, S. & Pongtanalert, K. (2013). Exploring characteristics and motives of consumer innovators: Community innovators vs. independent innovators. Research-Technology Management. https://doi.org/10.5437/08956308x5603088 [unverified]

Motivation in open source (E3)

• N142. Hertel, G., Niedner, S. & Herrmann, S. (2003). Motivation of software developers in open source projects: An internet-based survey of contributors to the Linux kernel. Research Policy. https://doi.org/10.1016/s0048-7333(03)00047-7 [unverified]
• N143. Lakhani, K.R. & Wolf, R.G. (2005). Why hackers do what they do: Understanding motivation and effort in free/open source software projects. In Feller, J., Fitzgerald, B., Hissam, S. & Lakhani, K.R. (eds.) Perspectives on Free and Open Source Software, MIT Press, 3–22. ISBN 978-0-262-06246-4. https://doi.org/10.7551/mitpress/5326.003.0005
• N144. Shah, S.K. (2006). Motivation, governance, and the viability of hybrid forms in open source software development. Management Science. https://doi.org/10.1287/mnsc.1060.0553
• N145. Roberts, J.A., Hann, I.-H. & Slaughter, S.A. (2006). Understanding the motivations, participation, and performance of open source software developers: A longitudinal study of the Apache projects. Management Science. https://doi.org/10.1287/mnsc.1060.0554 [unverified]
• N146. von Krogh, G., Haefliger, S., Spaeth, S. & Wallin, M.W. (2012). Carrots and rainbows: Motivation and social practice in open source software development. MIS Quarterly. https://doi.org/10.2307/41703471

Programming in everyday life (E4)

• N147. Rosson, M.B., Ballin, J. & Nash, H. (2004). Everyday programming: Challenges and opportunities for informal web development. Proc. IEEE VL/HCC 2004. https://doi.org/10.1109/vlhcc.2004.26
• N148. Brandt, J., Guo, P.J., Lewenstein, J., Dontcheva, M. & Klemmer, S.R. (2009). Two studies of opportunistic programming: Interleaving web foraging, learning, and writing code. Proc. CHI 2009. https://doi.org/10.1145/1518701.1518944 [unverified]

Makers and personal fabrication (E5)

• N149. Kuznetsov, S. & Paulos, E. (2010). Rise of the expert amateur: DIY projects, communities, and cultures. Proc. NordiCHI 2010. https://doi.org/10.1145/1868914.1868950
• N150. Tanenbaum, T.J., Williams, A.M., Desjardins, A. & Tanenbaum, K. (2013). Democratizing technology: Pleasure, utility and expressiveness in DIY and maker practice. Proc. CHI 2013. https://doi.org/10.1145/2470654.2481360 [unverified]
• N151. Hui, J.S. & Gerber, E.M. (2017). Developing makerspaces as sites of entrepreneurship. Proc. CSCW 2017. https://doi.org/10.1145/2998181.2998264 [unverified]

Critiques of scale estimates and measurement (E6)

• N152. Bogers, M., Afuah, A. & Bastian, B. (2010). Users as innovators: A review, critique, and future research directions. Journal of Management. https://doi.org/10.1177/0149206309353944
• N153. Gault, F. (2012). User innovation and the market. Science and Public Policy. https://doi.org/10.1093/scipol/scs005

Self-selection and learning outcomes (E7)

• N154. Cordova, D.I. & Lepper, M.R. (1996). Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. Journal of Educational Psychology, 88(4), 715–730. https://doi.org/10.1037/0022-0663.88.4.715 [unverified]
• N155. Iyengar, S.S. & Lepper, M.R. (2000). When choice is demotivating: Can one desire too much of a good thing? Journal of Personality and Social Psychology, 79(6), 995–1006. https://doi.org/10.1037/0022-3514.79.6.995 [unverified]
• N156. Guzdial, M. & Forte, A. (2005). Design process for a non-majors computing course. Proc. SIGCSE 2005. https://doi.org/10.1145/1047344.1047468 [unverified]
• N157. Patall, E.A., Cooper, H. & Robinson, J.C. (2008). The effects of choice on intrinsic motivation and related outcomes: A meta-analysis of research findings. Psychological Bulletin, 134(2), 270–300. https://doi.org/10.1037/0033-2909.134.2.270

The development of interest and self-directed learning (E8)

• N158. Hidi, S. & Renninger, K.A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. https://doi.org/10.1207/s15326985ep4102_4
• N159. Barron, B. (2006). Interest and self-sustained learning as catalysts of development: A learning ecology perspective. Human Development, 49(4), 193–224. https://doi.org/10.1159/000094368
• N160. Ito, M., Baumer, S., Bittanti, M., boyd, d., Cody, R. et al. (2009/2019). Hanging Out, Messing Around, and Geeking Out: Kids Living and Learning with New Media. MIT Press. https://doi.org/10.7551/mitpress/11832.001.0001
• N161. Ito, M., Gutiérrez, K.D., Livingstone, S., Penuel, B., Rhodes, J.E., Salen, K., Schor, J., Sefton-Green, J. & Watkins, S.C. (2013). Connected Learning: An Agenda for Research and Design. Connected Learning Alliance. ISBN 9780988725508. https://clalliance.org/publications/connected-learning-an-agenda-for-research-and-design/

Settings for building what one wants to build (E9)

• N162. Bruckman, A. (1998). Community support for constructionist learning. Computer Supported Cooperative Work, 7(1–2), 47–86. https://doi.org/10.1023/a:1008684120893 [unverified]
• N163. Bruckman, A. (2000). Situated support for learning: Storm’s weekend with Rachael. Journal of the Learning Sciences, 9(3), 329–372. https://doi.org/10.1207/s15327809jls0903_4
• N164. Kelleher, C., Pausch, R. & Kiesler, S. (2007). Storytelling Alice motivates middle school girls to learn computer programming. Proc. CHI 2007, 1455–1464. https://doi.org/10.1145/1240624.1240844 [unverified]
• N165. Maloney, J., Peppler, K., Kafai, Y., Resnick, M. & Rusk, N. (2008). Programming by choice: Urban youth learning programming with Scratch. Proc. SIGCSE 2008. https://doi.org/10.1145/1352135.1352260
• N166. Begel, A. & Simon, B. (2008). Novice software developers, all over again. Proc. ICER 2008, 3–14. https://doi.org/10.1145/1404520.1404522 [unverified]

Self-tracking as a tool for oneself (E10)

• N167. Choe, E.K., Lee, N.B., Lee, B., Pratt, W. & Kientz, J.A. (2014). Understanding quantified-selfers’ practices in collecting and exploring personal data. Proc. CHI 2014, 1143–1152. https://doi.org/10.1145/2556288.2557372
• N168. Rooksby, J., Rost, M., Morrison, A. & Chalmers, M. (2014). Personal tracking as lived informatics. Proc. CHI 2014, 1163–1172. https://doi.org/10.1145/2556288.2557039 [unverified]
• N169. Epstein, D.A., Ping, A., Fogarty, J. & Munson, S.A. (2015). A lived informatics model of personal informatics. Proc. UbiComp 2015, 731–742. https://doi.org/10.1145/2750858.2804250

The empirical foundation of maker education (E11)

• N170. Blikstein, P. (2013). Digital fabrication and ‘making’ in education: The democratization of invention. In FabLabs: Of Machines, Makers and Inventors. transcript Verlag, 203–222. https://doi.org/10.14361/transcript.9783839423820.203 [unverified]
• N171. Halverson, E.R. & Sheridan, K. (2014). The maker movement in education. Harvard Educational Review, 84(4), 495–504. https://doi.org/10.17763/haer.84.4.34j1g68140382063
• N172. Sheridan, K., Halverson, E.R., Litts, B., Brahms, L., Jacobs-Priebe, L. & Owens, T. (2014). Learning in the making: A comparative case study of three makerspaces. Harvard Educational Review, 84(4), 505–531. https://doi.org/10.17763/haer.84.4.brr34733723j648u
• N173. Martin, L. (2015). The promise of the maker movement for education. Journal of Pre-College Engineering Education Research, 5(1), art.4. https://doi.org/10.7771/2157-9288.1099
• N174. Vossoughi, S., Hooper, P.K. & Escudé, M. (2016). Making through the lens of culture and power: Toward transformative visions for educational equity. Harvard Educational Review, 86(2), 206–232. https://doi.org/10.17763/0017-8055.86.2.206
• N175. Papavlasopoulou, S., Giannakos, M.N. & Jaccheri, L. (2017). Empirical studies on the Maker Movement, a promising approach to learning: A literature review. Entertainment Computing, 18, 57–78. https://doi.org/10.1016/j.entcom.2016.09.002 [unverified]

The persistence and abandonment of self-built artifacts (E12)

• N176. McGill, T.J. (2002). User-developed applications: Can end users assess quality? Journal of Organizational and End User Computing. https://doi.org/10.4018/joeuc.2002070101 [unverified]
• N177. Avelino, G., Valente, M.T. & Hora, A. (2017). What is the truck factor of popular GitHub applications? A first assessment. PeerJ Preprints. https://doi.org/10.7287/peerj.preprints.1233v3
• N178. Avelino, G., Passos, L., Hora, A. & Valente, M.T. (2016). A novel approach for estimating truck factors. Proc. ICPC 2016, 1–10. https://doi.org/10.1109/icpc.2016.7503718
• N179. Spierings, A., Kerr, D. & Houghton, L. (2016). Issues that support the creation of ICT workarounds: Towards a theoretical understanding of feral information systems. Information Systems Journal, 27(6), 775–794 (print issue November 2017). https://doi.org/10.1111/isj.12123
• N180. Coelho, J. & Valente, M.T. (2017). Why modern open source projects fail. Proc. ESEC/FSE 2017. https://doi.org/10.1145/3106237.3106246
• N181. Avdic, A. (2018). Second order interactive end user development appropriation in the public sector: Application development using spreadsheet programs. Journal of Organizational and End User Computing. https://doi.org/10.4018/joeuc.2018010105 [unverified]
• N182. Coelho, J., Valente, M.T., Milen, L. & Silva, L.L. (2020). Is this GitHub project maintained? Measuring the level of maintenance activity of open-source projects. Information and Software Technology. https://doi.org/10.1016/j.infsof.2020.106274
• N183. Ait, A., Cánovas Izquierdo, J.L. & Cabot, J. (2022). An empirical study on the survival rate of GitHub projects. Proc. MSR 2022. https://doi.org/10.1145/3524842.3527941
• N184. Kaur, R. & Chahal, K.K. (2022). Exploring factors affecting developer abandonment of open source software projects. Journal of Software: Evolution and Process. https://doi.org/10.1002/smr.2484 [unverified]
• N185. Miller, C., Jahanshahi, M., Mockus, A., Vasilescu, B. & Kästner, C. (2025). Understanding the response to open-source dependency abandonment in the npm ecosystem. Proc. ICSE 2025. https://doi.org/10.1109/icse55347.2025.00004 [unverified]

(Access dates are 2026-08-14 for the first-round collection, and the same day for both the civic tech items and the items on building for one’s own need. Sources cited in the text by E number appear in the References section of the sister note end-user-development-literature. Details of the reachability checks are recorded in the Provenance section of the internal ledger source/review/end-user-development-novice-learning/papers.md.)