Learning by Practice with Knowledge Withheld, Learning by Knowledge with Practice Withheld: The Literature on Whether Withholding One Side Has Meaning

The claim that output matters more than input is supported by solid experiments. After reading the same passage, students who took a recall test remembered more one week later than students who reread it. Rereading won after five minutes (81% against 75%), and testing won after two days (68% against 54%) and after one week (56% against 42%) (Roediger and Karpicke 2006, p. 251). However, in this comparison every participant first saw the same material. The experiment compared adding or not adding retrieval practice on top of given knowledge. It did not withhold the knowledge itself.

The question of this note lies one step further. Nobody denies that mixing knowledge (verbal explanation, instruction, worked examples) with practice (solving, learning from experience, moving the body) is useful. The question is whether withholding one side has meaning in itself. Is there something that cannot be learned once knowledge is given? Is there something that cannot be learned once practice is allowed? Before reading the literature, my expectation was that both exist. After reading it, the answer is that both exist, that their conditions are specific, and that a prescription to withhold which does not state those conditions does not hold.

The starting point is one experiment from 1984. Berry and Broadbent had participants practice a screen-based task in which they adjusted the output of a sugar factory toward a target. With practice, control performance improved, but scores on a questionnaire about the relationships inside the task did not. Conversely, when the relationships were explained in detail, questionnaire scores improved, but control performance did not (abstract of Berry and Broadbent 1984; Dienes and Berry 1997, pp. 10–11, report the details). Practice and instruction were developing different things. Instruction did not lower control performance. It simply had no effect on control, and it did no harm. A double dissociation can also be read as a case for mixing, since each side develops something. Withholding can be said to have meaning only when there is evidence that one side interferes with the other, or evidence that one side cannot reach what the other reaches. The following sections read these two kinds of evidence separately for withholding knowledge and for withholding practice.

Conditions under Which Withholding Knowledge Has Meaning

The evidence for withholding knowledge falls into four conditions.

The first condition is that the regularity is not salient. In 1988, Berry and Broadbent built a salient version and a non-salient version of the relationships in the task and, after the first half of the trials, told half of the participants that the responses followed a fixed pattern and that searching for the rule would make control easier. In the salient version this instruction raised performance; in the non-salient version it lowered performance. When two non-salient tasks resembled each other, teaching the key relationship impaired transfer, whereas leaving it untaught produced positive transfer (abstract of Berry and Broadbent 1988; Dienes and Berry 1997, pp. 11–12). The harm is restricted to non-salient relationships, and for salient relationships instruction is an advantage. Therefore this evidence does not say that instruction in general is harmful. Instruction becomes harmful when a learner who sets out to search for a rule overlooks a regularity for which there is nothing to search.

The second condition is that the skill is already well practiced. Flegal and Anderson had 80 undergraduate golfers practice a putting task until they reached a criterion, after which half spent five minutes writing a detailed description of how they putted while the other half rated unrelated words. Among golfers classified as higher skill by their self-reported scores, those who wrote needed 21.2 trials to reach the criterion again, compared with 10.6 trials for those who did not write (F(1,76) = 5.55, p = .02). Among lower-skill golfers the figures were 19.9 and 22.4, with no difference (Flegal and Anderson 2008, p. 929). Describing the skill in words erased the effect of practice. The effect disappeared only on the well-practiced side. Beilock and colleagues also reported that experienced golfers putted worse when attending to components of their swing, whereas novices were not harmed by attending to components (abstract of Beilock et al. 2002). For novices, verbal instruction is not harmful. It becomes harmful when verbal knowledge is laid over a skill that has already consolidated as a procedure. Flegal and Anderson propose competition between procedural and declarative memory as the mechanism, but the authors themselves list an alternative explanation, that verbalization leaves attention on components, and the mechanism remains undecided (p. 931).

Masters’s research on implicit motor learning belongs to the same line. Masters reported that a group that learned putting while a secondary task prevented hypothesis testing about rules broke down less under pressure than a group that learned explicitly (abstract of Masters 1992). In a longitudinal study of 3,000 trials, Maxwell and colleagues found that the implicit group performed worse during learning, that there was no group difference on a delayed retention test, and that in the explicit group participants who reported more rules performed worse (abstract of Maxwell et al. 2000). A group taught by analogy pointed in the same direction (Liao and Masters 2001). However, this line shrank in a 2018 systematic review. Kal and colleagues collected 25 controlled trials and 39 comparisons from 4,125 reports. Most comparisons showed no group difference in dual-task performance, the implicit group was better in only some comparisons, the risk of bias was unclear in every trial, and the strength of evidence was level 3 (Kal et al. 2018, abstract and sections 4.1 to 4.3). Analogy learning reduced declarative knowledge in all 12 comparisons, whereas errorless learning failed to reduce it sufficiently in 7 of 12 (p. 18). The claim that implicitly learned skills break down less currently rests on weak evidence.

The third condition is that the object of learning is perceptual differentiation. Kellman and Garrigan argue that much of expert work rests on implicit pattern recognition and that the structures experts extract often cannot be explained in words. The rules of chess can be taught verbally in a short time, yet, in their words, no verbal instruction suffices to produce grandmasters (Kellman and Garrigan 2009, section II). Formal education leans toward declarative knowledge and procedural learning, and perceptual learning is nearly absent from it (section VI, “The Container Metaphor”). With perceptual-learning training, which repeats many short classification trials, novices in reading flight instruments matched pilots with about 1,000 hours of flight time after about one hour, and experienced pilots cut their reading time by 60% (reported in the review as the result of Kellman and Kaiser 1994). In medical imaging, perceptual and procedural measures improved while declarative tests improved in no group (reported as the result of Guerlain et al. 2004). This is evidence that words do not reach the skill, not evidence that words interfere with it. The authors do not claim that verbal rules actively hinder learning.

This condition has a counterexample. Sexing day-old chicks was a skill that experts had described as a feel that could not be put into words. Biederman and Shiffrar reported that when the position of the shape contrast that serves as the cue was made explicit in a short training, novices reached expert-level judgments (abstract of Biederman and Shiffrar 1987). At least some perceptual skills believed to be beyond words can be reached once the right words are found. “Words cannot reach it” is sometimes indistinguishable from “nobody has yet found the words.”

The fourth condition is that instruction is delayed. Schwartz and Martin compared ninth-grade statistics students who were taught a formula first and then practiced (tell-and-practice) with students who first invented their own index and were then taught. When the transfer problem came without a learning resource (an example of how to solve it), the two groups were equal; when it came with the resource, the invention group did better (Schwartz and Martin 2004, p. 129). In the meta-analysis by Sinha and Kapur, problem solving followed by instruction had an advantage of g = 0.36 over instruction followed by problem solving, rising to 0.58 when the design followed the principles of productive failure. In contrast, for lower grades (grades 2 to 5) and for domain-general skills, instruction first had the advantage (abstract of Sinha and Kapur 2021). It is tempting to read this line as evidence for withholding knowledge, but it is not. Invention alone performed the same as tell-and-practice, and what works is invention combined with teaching afterward. What this design moves is the order of knowledge; the knowledge itself is given later.

The evidence for withholding knowledge also faces a fundamental objection. Shanks and St. John reviewed research on implicit learning and argued that unconscious learning has not been established and that knowledge of rules which cannot be reported is, in most cases, memory for instances or fragments rather than rules (Shanks and St. John 1994, abstract). Dienes and Berry also concluded that, when the questions used to elicit knowledge are set up properly (the objective threshold), the knowledge lies above the threshold. Knowledge can be called implicit only at the level where people answer while believing they are guessing (the subjective threshold) (Dienes and Berry 1997, p. 3 and conclusion). On the philosophical side, Stanley and Williamson reject the distinction between knowing that and knowing how itself and argue that knowing how is a species of knowing that (abstract of Stanley and Williamson 2001). If these positions are accepted, no learning exists in which knowledge is withheld; what exists is learning in which knowledge in verbal form is bypassed. The meaning of withholding then shifts to how one chooses the form of knowledge, whether rules that can be stated in words or memory for instances and procedures.

Conditions under Which Withholding Practice Has Meaning

The evidence for withholding practice falls into five conditions.

The first condition is that the learner is a novice. Sweller and Cooper reported that students beginning algebra learned more from reading worked examples than from solving equivalent problems themselves (abstract of Sweller and Cooper 1985). Alfieri and colleagues summarized 164 studies in two meta-analyses: unassisted discovery learning was inferior to explicit instruction (580 comparisons, d = −0.38, 95% confidence interval −.44 to −.31), and assisted discovery was superior to other forms of instruction (360 comparisons, d = 0.30, .23 to .36). What worked was feedback, worked examples, scaffolding, and elicited explanations (Alfieri et al. 2011, abstract and p. 11).

Mayer reviewed three research histories, rule discovery in the 1960s, conservation concepts in the 1970s, and LOGO in the 1980s, and found pure discovery inferior to guided discovery in all three. He cites children who had used LOGO for more than 50 hours and still had difficulty with basic concepts (Mayer 2004, p. 16). Mayer’s emphasis is that what works is cognitive activity, not behavioral activity as such (pp. 15 and 17). Kirschner and colleagues write that this advantage recedes only when learners have sufficiently high prior knowledge to provide internal guidance for themselves (Kirschner et al. 2006, abstract). The worked-example effect disappears and reverses with expertise (abstract of Kalyuga et al. 2003). The meaning of withholding practice from novices and having them read worked examples lies in working memory: when it is filled by searching the problem, no attention is left for the structure to be learned.

The second condition is that the concept runs against intuition. When McCloskey and colleagues asked university students to draw the path of a moving object, a majority made marked errors, and some students who had completed a physics course were among them (abstract of McCloskey et al. 1980). Intuitions formed by everyday experience remain after instruction. Chi summarizes that misconceptions about emergent processes such as diffusion resist instruction and that even confrontational interventions, which show students that their answers conflict with the observed outcome, have generally failed to improve understanding (Chi 2005, pp. 161–162). In Chi’s explanation, as long as learners classify the phenomenon as a direct process, added observations are interpreted under the same classification, so that correction requires re-representation across ontological categories. In Japan, Magara pointed out that counterexamples have been thought effective for correcting wrong knowledge (the “le bar” of Nagano and Hosoya), but that some learners who meet a counterexample ignore it, exclude it, or make only peripheral changes to their theory. He proposed a double-inference method in which the learner first reads the knowledge that provides the cue (Magara 1999, pp. 33–34).

Vygotsky wrote that scientific concepts acquired through instruction and everyday concepts acquired through experience develop in opposite directions. Scientific concepts move downward from verbal definition to the concrete, and everyday concepts move upward from the concrete to abstraction. Instruction plays a decisive role in development during the school years, and the level of scientific concepts forms a zone of proximal possibilities for everyday concepts (Vygotsky 1986, chapter 6). Ichikawa also argues that concepts in science and mathematics are a kind of artificial concept, that people learn concepts efficiently when an intensional definition is conveyed in words, and that generalization from instances alone does not produce them (Ichikawa 2000, p. 363). Here the meaning of withholding practice lies in the fact that experience cannot form the concept. Experience supplies instances, but it does not supply the category to which the instances belong.

The third condition is that environmental feedback is wrong or missing. Hogarth and colleagues divided environments for learning from experience into kind environments, in which feedback is accurate and experience promotes correct learning, and wicked environments, in which feedback is wrong or missing and experience produces mistaken learning (abstract of Hogarth et al. 2015). Kahneman and Klein agreed that the reliability of intuition is determined by the regularity of the environment and the opportunity to learn that regularity, and that subjective confidence is not an indicator of accuracy (abstract of Kahneman and Klein 2009). The title of Brehmer’s paper is “In One Word: Not from Experience” (Brehmer 1980; bibliographic record only). In medicine, Choudhry and colleagues collected 62 evaluations: in 32 (52%) performance declined on all measures as years of practice increased, in 13 it declined on some measures, and in 13 there was no association (abstract of Choudhry et al. 2005). In mental health as well, professional experience generally did not improve the validity of clinical judgment (abstract of Garb 1989), and actuarial judgment was more accurate than clinical judgment (abstract of Dawes et al. 1989).

Ericsson and colleagues’ research on deliberate practice claimed that expertise is not mere experience (abstract of Ericsson et al. 1993). In the replication of the original violin study, however, the variance explained by practice fell from 48% to 26%, and the best and good violinists did not differ in practice (Macnamara and Maitra 2019). Accumulating experience and gaining expertise are different matters. The meaning of withholding practice lies in the fact that experience accumulated under wrong feedback consolidates errors together with confidence.

The fourth condition is that the event is rare and irreversible. March and colleagues argued that organizations end up learning about rare events from samples of one or fewer (abstract of March et al. 1991). Fukushima summarizes that in high-reliability organizations such as nuclear power plants and emergency departments, failure is not permitted, and the more safety is raised, the fewer opportunities for learning remain (Fukushima 2010; according to Ishimura’s 2013 review, p. 152). Here there is no choice about whether to withhold practice. Experience cannot take place, so learning comes only from knowledge, that is, records of others’ experience, history, and simulation.

The fifth condition is that the object of learning is formal reasoning or grammar. Fong and colleagues reported that brief statistical training increased the frequency and quality of statistical reasoning about everyday problems and that the effect appeared outside the training context (abstract of Fong et al. 1986). Nisbett and colleagues argued, against the twentieth-century psychological view that abstract rules of reasoning develop only through self-discovery, that even brief formal training improves everyday reasoning (abstract of Nisbett et al. 1987). For second-language grammar, the meta-analysis by Norris and Ortega (49 studies) found explicit instruction more effective than implicit instruction, with durable effects (abstract of Norris and Ortega 2000). Rules of statistics and rules of grammar rarely appear in the situations where they are used, and when they appear they are not visible at a single encounter. When experience does not show the regularity, the rule can be reached only by instruction.

The withholding-practice side also has counterexamples. Kudo reported that when a rule was taught to 93 university students through instances, about half interpreted the instruction as not containing any general information, and those who interpreted it that way generalized less. Even when a rule is taught explicitly, inductive learning based on instances occurs (abstract of Kudo 2003). In learning Hooke’s law, the instances from the measurement experiment restricted the abstractness of the rule, and teaching a proposition at a higher level of abstraction did not fully resolve this (abstract of Kudo 2013). Instruction alone cannot control how learners interpret the instruction. In their response to Kirschner and colleagues, Hmelo-Silver and colleagues argued that problem-based and inquiry learning are not minimally guided but are designed with scaffolding (abstract of Hmelo-Silver et al. 2007), and the results of Alfieri and colleagues support this. Smith and colleagues argued, against the view that misconceptions are to be removed, that intuitive knowledge is a resource for change (abstract of Smith et al. 1994). Withholding practice has meaning only for unassisted practice.

What the Japanese Literature Has Said “Not Teaching” Is

The Japanese literature has spoken of not teaching in three forms. The first is the withholding of verbal instruction in the practice of traditional arts (waza). The second is the situation in which verbalization lowers performance. The third is the educational policy of avoiding strong directions.

Ikuta’s Learning from Waza described practice in the classical performing arts as centered on imitating the master’s form (kata), with evaluation limited to good or bad without stated reasons (non-transparency) and with no progression from basics to application (non-gradedness). Ikuta explains that the master does not directly instruct how to move body parts because the intention is an environment that lets the learner enter the world and become entrained to the master’s breathing (Ikuta 1987; according to Suwa’s 2008 review, pp. 714–715). In a lecture on craft transmission, Ikuta observes that an apprentice carpenter is told to clean the storehouse and the full-scale drawing floor so that the master’s work becomes visible, and that the master “makes a groove” so that the learner falls into it naturally. Ikuta calls this the intentionality of unintentional teaching (Ikuta 2004, pp. 28–30). What is withheld here is verbal instruction, not the intention to teach. The intention to teach has been moved into the design of the setting. Ikuta also argued that the phrase “knowing but unable to do” looks contradictory only because knowing is identified with acquiring information, and that acquiring a rule as knowledge means being able to apply it in concrete operations (Ikuta 1992, pp. 33–35). In this position, the opposition between knowledge and practice itself results from mistaking knowing for verbal information.

On the situation in which verbalization lowers performance, Suwa opposes the received view that verbalization is an obstacle to acquiring embodied knowledge. Suwa cites the result of Beilock and colleagues that professional golfers say less than novices immediately after a putt and say more when using a deformed putter with a bent shaft. He accepts that verbalization temporarily breaks an automated state, and he reads this as preparation for rebuilding toward the next level of expertise (Suwa 2005, pp. 526 and 528). A phase that cannot be verbalized always exists, but it is a temporary state in the course of expertise.

Suzuki and Yokoyama list that using chopsticks or riding a bicycle cannot be executed immediately after hearing however detailed an explanation, and that verbal description interferes with memory for faces and with insight problem solving. They write that excessive dependence on words is not only ineffective for learning but sometimes obstructive (Suzuki and Yokoyama 2016, pp. 5–7). Murakoshi introduced Reber’s result that instructions making learners strategic interfere with rule learning that was possible implicitly, together with Masters’s putting results, and stated that becoming too strategic may interfere with learning, citing Ikuta for the non-strategic acquisition of traditional skills (Murakoshi 1996, pp. 330 and 332). In Japanese motor-learning research, there is also a report in which an internal-focus group improved with practice, against the premise that an external focus of attention is always superior (abstract of Oki and Kokubu 2021).

The Japanese literature on withholding practice is found in science education and educational psychology. In 1970, Matsuura compared an expository method that presents the law first with a discovery method based on learner exploration and reported that, in a balance-beam experiment with 150 fourth graders, the law-first group was superior in learning and retention while no difference in transfer appeared (abstract of Matsuura 1970). Kawakami and Watanabe summarized Japanese research on meaningful reception learning since 1985 and positioned reception learning as suited to abstract content rather than as a replacement for discovery learning. They collected domestic criticisms of discovery learning: Shindo’s “naive discoverism,” Kaburagi’s “making students think in situations where teaching would be fine,” Ichikawa’s “a lesson that makes students think without teaching is a very risky form of lesson,” and Nishikawa’s “doing experiments and observations does not mean understanding correct concepts” (Kawakami and Watanabe 2010, pp. 2–3).

Literature on the integrating side also exists in Japanese. Hatano criticized the traditional lesson of presenting a solution and then mechanically practicing its application, while recommending an intermediate strategy: first encourage learner exploration, and only when it seems unlikely to lead to a productive result should the teacher suggest an explanation (Hatano 1995, pp. 354 and 356). Mizukoshi pointed out that American comparisons of discovery learning and expository methods used transfer and retention as criteria yet reached widely divergent conclusions, and that the cause lay in the lack of unified basic concepts and terms (Mizukoshi 1970, pp. 21 and 24). What becomes visible when the Japanese literature is laid side by side is that almost no author has set up withholding as a principle. Ikuta withholds verbal instruction but not the intention to teach, Suwa does not call verbalization harmful, and the reception-learning side attaches the condition of abstract content.

What Separates the Two Sides

When the conditions of the two sides are placed together, it becomes clear that the meaning of withholding is not decided by which of knowledge and practice matters more. It is decided by the nature of the object of learning, the learner’s stage, and the conditions of performance.

This table shows what the general claim that mixing is good leaves out. What it leaves out is the judgment of which row the object of learning falls into. Adding rule instruction to a motor skill with non-salient regularities and adding experience to a concept on which intuition errs are both cases of mixing, but the former can be harmful and the latter does not work. The claim that withholding has meaning can be read as the claim that this judgment must come first.

The remaining problem is that experiments directly comparing withholding with giving are few. On the knowledge side, direct comparisons between a withheld group and an instructed group are limited to the exploration instructions of Berry and Broadbent 1988, the verbalization of Flegal and Anderson 2008, and the comparison of direct instruction with discovery by Klahr and Nigam 2004. In that study, among 112 third and fourth graders, far more children learned the control-of-variables strategy under direct instruction, and the few who reached it by discovery performed comparably on transfer (abstract). Masters’s line showed no group difference in most comparisons in the systematic review, and the original verbal-overshadowing effect (25 percentage points) shrank in a preregistered replication to 4 points immediately after description and 16 points after a 20-minute delay (abstract of Alogna et al. 2014). The evidence on the practice side consists of comparisons between solving and reading and between assisted and unassisted conditions, and conditions in which practice is completely withheld are rare. The claim that experience creates errors is supported by observational studies and theory. The claim that withholding has meaning stands under stated conditions, but the number of experiments that have tested those conditions is small compared with how widely the claim circulates.

What Could Not Be Confirmed This Time

• Berry and Broadbent 1984 and 1988, Masters 1992, Maxwell et al. 2000, Liao and Masters 2001, Beilock et al. 2001 and 2002, Schooler and Engstler-Schooler 1990, Alogna et al. 2014, Cohen and Squire 1980, Biederman and Shiffrar 1987, Kapur 2008, Schwartz and Bransford 1998, Sinha and Kapur 2021, Sweller and Cooper 1985, Kalyuga et al. 2003, Hmelo-Silver et al. 2007, Klahr and Nigam 2004, Lazonder and Harmsen 2016, McCloskey et al. 1980, Hogarth et al. 2015, Kahneman and Klein 2009, March et al. 1991, Einhorn and Hogarth 1978, Ericsson et al. 1993, Choudhry et al. 2005, Garb 1989, Dawes et al. 1989, Nisbett et al. 1987, Fong et al. 1986, Norris and Ortega 2000, Bereiter 1985, and Stanley and Williamson 2001 are behind subscription walls and were used within the scope of their abstracts. Only figures stated in the abstracts were used.
• Reber 1967, 1976, and 1989, Berry and Broadbent 1987, Hayes and Broadbent 1988, Corkin 1968, Vosniadou and Brewer 1992, Brehmer 1980, and Meyer and Land 2005 were confirmed as bibliographic records only; their positioning follows the accounts in Dienes and Berry 1997, Chi 2005, and Murakoshi 1996.
• Vygotsky’s chapter 6 was read in the translation on marxists.org (translator unidentified) and was not checked against the pages of the 1986 MIT Press edition.
• Ikuta 1987, Ikuta and Kitamura 2011, Fukushima 2010, and Ichikawa 2008 were used within the scope of bibliographic records and reviews (Suwa 2008; Nishihira 2012; Ishimura 2013) or secondary citation (Kawakami and Watanabe 2010). Ichikawa’s 2005 phrase about a very risky form of lesson follows the citation in Kawakami and Watanabe.
• Ryle 1949 and Polanyi 1966 are listed only as the sources of the terms knowing how and knowing that and tacit knowledge; they were not used for the claims in the body.
• No Japanese primary study in motor learning that explicitly withholds verbal instruction (for example by Kudo Kazutoshi, Shinya Masahiro, or Higuchi Takahiro) was found in J-STAGE and CiNii searches.
• The experiments that directly compare withholding with giving are the few listed above, and the search was not exhaustive.

Related Notes

• Designing Failure Into Learning: From Expectation Failure to Productive Failure and Industry Implementations: the lineage of designs that deliberately withhold instruction (productive failure, desirable difficulties) and their industrial implementations
• What Should Juniors Do When Generative AI Takes Away the Entry-Level Work? (Academic Research): skill formation when entry-level work shrinks; the limits of deliberate practice and the path of learning beside experts
• Scaffolding Learning and the Design of Interventions: A Literature Map of Cognitive Apprenticeship, Legitimate Peripheral Participation, and Theories of Mediation: a map of scaffolding, cognitive apprenticeship, and legitimate peripheral participation
• Is Expertise a One-Way Street? Unlearning and the Blind Spot of Reversible Mastery: the reversibility of expertise and unlearning
• Does Early Use of Generative AI Inhibit the Formation of Thought? A Literature Map of Cognitive Offloading and Learning: the conditions under which early use of answer-giving generative AI impairs learning
• What Has Design Education Carried So Far? (Academic Research): the roles that the studio and the critique have played in design education

Unverified Items

• The sources whose full text could not be obtained are listed above under “What Could Not Be Confirmed This Time.” All were written with attribution within the scope of abstracts or bibliographic records, and the figures in the main claims are limited to sources read in full text (Roediger and Karpicke 2006, Flegal and Anderson 2008, Alfieri et al. 2011, Kal et al. 2018, Macnamara and Maitra 2019, Schwartz and Martin 2004, Kellman and Garrigan 2009, the translation of Vygotsky 1986, and 16 Japanese papers) or to figures stated in abstracts.
• The section numbers for Kellman and Garrigan 2009 follow the author manuscript in PMC and were not checked against the pages of the published version.

References

Implicit Learning and the Double Dissociation

• Reber, Arthur S. 1967. “Implicit Learning of Artificial Grammars.” Journal of Verbal Learning and Verbal Behavior 6 (6): 855–863. https://doi.org/10.1016/S0022-5371(67)80149-X
• Reber, Arthur S. 1976. “Implicit Learning of Synthetic Languages: The Role of Instructional Set.” Journal of Experimental Psychology: Human Learning and Memory 2 (1): 88–94. https://doi.org/10.1037/0278-7393.2.1.88
• Reber, Arthur S. 1989. “Implicit Learning and Tacit Knowledge.” Journal of Experimental Psychology: General 118 (3): 219–235. https://doi.org/10.1037/0096-3445.118.3.219
• Berry, Dianne C., and Donald E. Broadbent. 1984. “On the Relationship between Task Performance and Associated Verbalizable Knowledge.” Quarterly Journal of Experimental Psychology Section A 36 (2): 209–231. https://doi.org/10.1080/14640748408402156
• Berry, Dianne C., and Donald E. Broadbent. 1987. “The Combination of Explicit and Implicit Learning Processes in Task Control.” Psychological Research 49 (1): 7–15. https://doi.org/10.1007/BF00309197
• Berry, Dianne C., and Donald E. Broadbent. 1988. “Interactive Tasks and the Implicit–Explicit Distinction.” British Journal of Psychology 79 (2): 251–272. https://doi.org/10.1111/j.2044-8295.1988.tb02286.x
• Hayes, Nigel A., and Donald E. Broadbent. 1988. “Two Modes of Learning for Interactive Tasks.” Cognition 28 (3): 249–276. https://doi.org/10.1016/0010-0277(88)90015-7
• Stanley, William B., Robert C. Mathews, Ray R. Buss, and Susan Kotler-Cope. 1989. “Insight without Awareness: On the Interaction of Verbalization, Instruction and Practice in a Simulated Process Control Task.” Quarterly Journal of Experimental Psychology Section A 41 (3): 553–577. https://doi.org/10.1080/14640748908402382
• Mathews, Robert C., Ray R. Buss, William B. Stanley, Fredda Blanchard-Fields, Jeung Ryeul Cho, and Barry Druhan. 1989. “Role of Implicit and Explicit Processes in Learning from Examples: A Synergistic Effect.” Journal of Experimental Psychology: Learning, Memory, and Cognition 15 (6): 1083–1100. https://doi.org/10.1037/0278-7393.15.6.1083
• Shanks, David R., and Mark F. St. John. 1994. “Characteristics of Dissociable Human Learning Systems.” Behavioral and Brain Sciences 17 (3): 367–395. https://doi.org/10.1017/S0140525X00035032
• Dienes, Zoltán, and Dianne Berry. 1997. “Implicit Learning: Below the Subjective Threshold?” Psychonomic Bulletin & Review 4 (1): 3–23. https://doi.org/10.3758/BF03210769

Procedural and Declarative Memory

• Corkin, Suzanne. 1968. “Acquisition of Motor Skill after Bilateral Medial Temporal-Lobe Excision.” Neuropsychologia 6 (3): 255–265. https://doi.org/10.1016/0028-3932(68)90024-9
• Cohen, Neal J., and Larry R. Squire. 1980. “Preserved Learning and Retention of Pattern-Analyzing Skill in Amnesia: Dissociation of Knowing How and Knowing That.” Science 210 (4466): 207–210. https://doi.org/10.1126/science.7414331
• Squire, Larry R. 2004. “Memory Systems of the Brain: A Brief History and Current Perspective.” Neurobiology of Learning and Memory 82 (3): 171–177. https://doi.org/10.1016/j.nlm.2004.06.005

Explicit Knowledge, Motor Skill, and Verbalization

• Masters, Richard S. W. 1992. “Knowledge, Knerves and Know-How: The Role of Explicit versus Implicit Knowledge in the Breakdown of a Complex Motor Skill under Pressure.” British Journal of Psychology 83 (3): 343–358. https://doi.org/10.1111/j.2044-8295.1992.tb02446.x
• Maxwell, Jon P., Richard S. W. Masters, and Frank F. Eves. 2000. “From Novice to No Know-How: A Longitudinal Study of Implicit Motor Learning.” Journal of Sports Sciences 18 (2): 111–120. https://doi.org/10.1080/026404100365180
• Liao, Chu-Min, and Richard S. W. Masters. 2001. “Analogy Learning: A Means to Implicit Motor Learning.” Journal of Sports Sciences 19 (5): 307–319. https://doi.org/10.1080/02640410152006081
• Beilock, Sian L., and Thomas H. Carr. 2001. “On the Fragility of Skilled Performance: What Governs Choking under Pressure?” Journal of Experimental Psychology: General 130 (4): 701–725. https://doi.org/10.1037/0096-3445.130.4.701
• Beilock, Sian L., Thomas H. Carr, Clare MacMahon, and Janet L. Starkes. 2002. “When Paying Attention Becomes Counterproductive: Impact of Divided versus Skill-Focused Attention on Novice and Experienced Performance of Sensorimotor Skills.” Journal of Experimental Psychology: Applied 8 (1): 6–16. https://doi.org/10.1037/1076-898X.8.1.6
• Flegal, Kristin E., and Michael C. Anderson. 2008. “Overthinking Skilled Motor Performance: Or Why Those Who Teach Can’t Do.” Psychonomic Bulletin & Review 15 (5): 927–932. https://doi.org/10.3758/PBR.15.5.927
• Kal, Elmar, Rens Prosée, Marco Winters, and John van der Kamp. 2018. “Does Implicit Motor Learning Lead to Greater Automatization of Motor Skills Compared to Explicit Motor Learning? A Systematic Review.” PLOS ONE 13 (9): e0203591. https://doi.org/10.1371/journal.pone.0203591
• Schooler, Jonathan W., and Tonya Y. Engstler-Schooler. 1990. “Verbal Overshadowing of Visual Memories: Some Things Are Better Left Unsaid.” Cognitive Psychology 22 (1): 36–71. https://doi.org/10.1016/0010-0285(90)90003-M
• Alogna, Victoria K., et al. 2014. “Registered Replication Report: Schooler and Engstler-Schooler (1990).” Perspectives on Psychological Science 9 (5): 556–578. https://doi.org/10.1177/1745691614545653

Perceptual Learning and Embodiment

• Held, Richard, and Alan Hein. 1963. “Movement-Produced Stimulation in the Development of Visually Guided Behavior.” Journal of Comparative and Physiological Psychology 56 (5): 872–876. https://doi.org/10.1037/h0040546
• Kellman, Philip J., and Patrick Garrigan. 2009. “Perceptual Learning and Human Expertise.” Physics of Life Reviews 6 (2): 53–84. https://doi.org/10.1016/j.plrev.2008.12.001
• Biederman, Irving, and Margaret M. Shiffrar. 1987. “Sexing Day-Old Chicks: A Case Study and Expert Systems Analysis of a Difficult Perceptual-Learning Task.” Journal of Experimental Psychology: Learning, Memory, and Cognition 13 (4): 640–645. https://doi.org/10.1037/0278-7393.13.4.640

Delaying Instruction and Generation

• Kapur, Manu. 2008. “Productive Failure.” Cognition and Instruction 26 (3): 379–424. https://doi.org/10.1080/07370000802212669
• Schwartz, Daniel L., and John D. Bransford. 1998. “A Time for Telling.” Cognition and Instruction 16 (4): 475–522. https://doi.org/10.1207/s1532690xci1604_4
• Schwartz, Daniel L., and Taylor Martin. 2004. “Inventing to Prepare for Future Learning: The Hidden Efficiency of Encouraging Original Student Production in Statistics Instruction.” Cognition and Instruction 22 (2): 129–184. https://doi.org/10.1207/s1532690xci2202_1
• Sinha, Tanmay, and Manu Kapur. 2021. “When Problem Solving Followed by Instruction Works: Evidence for Productive Failure.” Review of Educational Research 91 (5): 761–798. https://doi.org/10.3102/00346543211019105
• Slamecka, Norman J., and Peter Graf. 1978. “The Generation Effect: Delineation of a Phenomenon.” Journal of Experimental Psychology: Human Learning and Memory 4 (6): 592–604. https://doi.org/10.1037/0278-7393.4.6.592
• Roediger, Henry L., III, and Jeffrey D. Karpicke. 2006. “Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention.” Psychological Science 17 (3): 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x

Worked Examples, Discovery Learning, and the Expertise Reversal

• Sweller, John, and Graham A. Cooper. 1985. “The Use of Worked Examples as a Substitute for Problem Solving in Learning Algebra.” Cognition and Instruction 2 (1): 59–89. https://doi.org/10.1207/s1532690xci0201_3
• Kalyuga, Slava, Paul Ayres, Paul Chandler, and John Sweller. 2003. “The Expertise Reversal Effect.” Educational Psychologist 38 (1): 23–31. https://doi.org/10.1207/S15326985EP3801_4
• Kirschner, Paul A., John Sweller, and Richard E. Clark. 2006. “Why Minimal Guidance during Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching.” Educational Psychologist 41 (2): 75–86. https://doi.org/10.1207/s15326985ep4102_1
• Hmelo-Silver, Cindy E., Ravit Golan Duncan, and Clark A. Chinn. 2007. “Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006).” Educational Psychologist 42 (2): 99–107. https://doi.org/10.1080/00461520701263368
• Mayer, Richard E. 2004. “Should There Be a Three-Strikes Rule against Pure Discovery Learning? The Case for Guided Methods of Instruction.” American Psychologist 59 (1): 14–19. https://doi.org/10.1037/0003-066X.59.1.14
• Klahr, David, and Milena Nigam. 2004. “The Equivalence of Learning Paths in Early Science Instruction: Effects of Direct Instruction and Discovery Learning.” Psychological Science 15 (10): 661–667. https://doi.org/10.1111/j.0956-7976.2004.00737.x
• Alfieri, Louis, Patricia J. Brooks, Naomi J. Aldrich, and Harriet R. Tenenbaum. 2011. “Does Discovery-Based Instruction Enhance Learning?” Journal of Educational Psychology 103 (1): 1–18. https://doi.org/10.1037/a0021017
• Lazonder, Ard W., and Ruth Harmsen. 2016. “Meta-Analysis of Inquiry-Based Learning: Effects of Guidance.” Review of Educational Research 86 (3): 681–718. https://doi.org/10.3102/0034654315627366

Naive Theories and Conceptual Change

• McCloskey, Michael, Alfonso Caramazza, and Bert Green. 1980. “Curvilinear Motion in the Absence of External Forces: Naive Beliefs about the Motion of Objects.” Science 210 (4474): 1139–1141. https://doi.org/10.1126/science.210.4474.1139
• Vosniadou, Stella, and William F. Brewer. 1992. “Mental Models of the Earth: A Study of Conceptual Change in Childhood.” Cognitive Psychology 24 (4): 535–585. https://doi.org/10.1016/0010-0285(92)90018-W
• Chi, Michelene T. H. 2005. “Commonsense Conceptions of Emergent Processes: Why Some Misconceptions Are Robust.” Journal of the Learning Sciences 14 (2): 161–199. https://doi.org/10.1207/s15327809jls1402_1
• Smith, John P., III, Andrea A. diSessa, and Jeremy Roschelle. 1994. “Misconceptions Reconceived: A Constructivist Analysis of Knowledge in Transition.” Journal of the Learning Sciences 3 (2): 115–163. https://doi.org/10.1207/s15327809jls0302_1
• Vygotsky, Lev S. 1986. Thought and Language. Translation newly revised and edited by Alex Kozulin. Cambridge, MA: MIT Press. https://mitpress.mit.edu/9780262720106/thought-and-language/

Environments That Mislead Experience, and Experience and Expertise

• Hogarth, Robin M., Tomás Lejarraga, and Emre Soyer. 2015. “The Two Settings of Kind and Wicked Learning Environments.” Current Directions in Psychological Science 24 (5): 379–385. https://doi.org/10.1177/0963721415591878
• Kahneman, Daniel, and Gary Klein. 2009. “Conditions for Intuitive Expertise: A Failure to Disagree.” American Psychologist 64 (6): 515–526. https://doi.org/10.1037/a0016755
• Brehmer, Berndt. 1980. “In One Word: Not from Experience.” Acta Psychologica 45 (1–3): 223–241. https://doi.org/10.1016/0001-6918(80)90034-7
• Einhorn, Hillel J., and Robin M. Hogarth. 1978. “Confidence in Judgment: Persistence of the Illusion of Validity.” Psychological Review 85 (5): 395–416. https://doi.org/10.1037/0033-295X.85.5.395
• March, James G., Lee S. Sproull, and Michal Tamuz. 1991. “Learning from Samples of One or Fewer.” Organization Science 2 (1): 1–13. https://doi.org/10.1287/orsc.2.1.1
• Ericsson, K. Anders, Ralf T. Krampe, and Clemens Tesch-Römer. 1993. “The Role of Deliberate Practice in the Acquisition of Expert Performance.” Psychological Review 100 (3): 363–406. https://doi.org/10.1037/0033-295X.100.3.363
• Macnamara, Brooke N., and Megha Maitra. 2019. “The Role of Deliberate Practice in Expert Performance: Revisiting Ericsson, Krampe & Tesch-Römer (1993).” Royal Society Open Science 6 (8): 190327. https://doi.org/10.1098/rsos.190327
• Choudhry, Niteesh K., Robert H. Fletcher, and Stephen B. Soumerai. 2005. “Systematic Review: The Relationship between Clinical Experience and Quality of Health Care.” Annals of Internal Medicine 142 (4): 260–273. https://doi.org/10.7326/0003-4819-142-4-200502150-00008
• Garb, Howard N. 1989. “Clinical Judgment, Clinical Training, and Professional Experience.” Psychological Bulletin 105 (3): 387–396. https://doi.org/10.1037/0033-2909.105.3.387
• Dawes, Robyn M., David Faust, and Paul E. Meehl. 1989. “Clinical versus Actuarial Judgment.” Science 243 (4899): 1668–1674. https://doi.org/10.1126/science.2648573

Reasoning Transmitted by Instruction, Threshold Concepts, and the Learning Paradox

• Nisbett, Richard E., Geoffrey T. Fong, Darrin R. Lehman, and Patricia W. Cheng. 1987. “Teaching Reasoning.” Science 238 (4827): 625–631. https://doi.org/10.1126/science.3672116
• Fong, Geoffrey T., David H. Krantz, and Richard E. Nisbett. 1986. “The Effects of Statistical Training on Thinking about Everyday Problems.” Cognitive Psychology 18 (3): 253–292. https://doi.org/10.1016/0010-0285(86)90001-0
• Norris, John M., and Lourdes Ortega. 2000. “Effectiveness of L2 Instruction: A Research Synthesis and Quantitative Meta-Analysis.” Language Learning 50 (3): 417–528. https://doi.org/10.1111/0023-8333.00136
• Meyer, Jan H. F., and Ray Land. 2005. “Threshold Concepts and Troublesome Knowledge (2): Epistemological Considerations and a Conceptual Framework for Teaching and Learning.” Higher Education 49 (3): 373–388. https://doi.org/10.1007/s10734-004-6779-5
• Bereiter, Carl. 1985. “Toward a Solution of the Learning Paradox.” Review of Educational Research 55 (2): 201–226. https://doi.org/10.3102/00346543055002201

Japanese Sources

• Ikuta, Kumiko. 1987. “Waza” kara shiru [Learning from Waza]. Tokyo: University of Tokyo Press (in Japanese). https://ci.nii.ac.jp/ncid/BN0131370X
• Suwa, Masaki. 2008. “Shohyō: Ikuta Kumiko ‘Waza’ kara shiru” [Book review: Ikuta, Learning from Waza]. Cognitive Studies 15 (4): 714–715 (in Japanese). https://doi.org/10.11225/jcss.15.714
• Ikuta, Kumiko. 1992. “‘Shitte ite mo dekinai’ to wa nani ka” [What is “knowing but unable to do”?]. Studies in the Philosophy of Education 65: 30–35 (in Japanese). https://doi.org/10.11399/kyouikutetsugaku1959.1992.30
• Ikuta, Kumiko. 2004. “‘Chishiki’ to ‘waza’ no kyōshigaku: Shokunin no ‘waza’ no denshō katei ga shisa suru koto” [Teacher studies of knowledge and waza: What the transmission of craftsmen’s skills suggests]. Journal of the Japanese Association for the Study of Teacher Development 5–6: 24–34 (in Japanese). https://doi.org/10.20590/jaehd.5.6.0_24
• Ikuta, Kumiko, and Katsuro Kitamura, eds. 2011. Waza gengo: Kankaku no kyōyū o tōshite no “manabi” e [The Language of Waza: Toward Learning through Shared Sensation]. Tokyo: Keio University Press (in Japanese). https://ci.nii.ac.jp/ncid/BB05343402
• Nishihira, Tadashi. 2012. “Tosho shōkai: Ikuta Kumiko, Kitamura Katsuro hen ‘Waza gengo’” [Book notice: Ikuta and Kitamura, eds., The Language of Waza]. The Japanese Journal of Educational Research 79 (1): 120–121 (in Japanese). https://doi.org/10.11555/kyoiku.79.1_120
• Suwa, Masaki. 2005. “Shintaichi kakutoku no tsūru to shite no metaninchiteki gengoka” [Metacognitive verbalization as a tool for acquiring embodied knowledge]. Journal of the Japanese Society for Artificial Intelligence 20 (5): 525–532 (in Japanese). https://doi.org/10.11517/jjsai.20.5_525
• Suzuki, Hiroaki, and Taku Yokoyama. 2016. “Kotoba o koeta chi o umidasu: Shintaisei ninchi kagaku kara mita komyunikēshon to jukutatsu” [Producing knowledge beyond words: Communication and expertise from embodied cognitive science]. Organizational Science 49 (4): 4–15 (in Japanese). https://doi.org/10.11207/soshikikagaku.49.4_4
• Suzuki, Hiroaki, and Haruaki Fukuda. 2013. “Dōsatsu mondai kaiketsu no muishikiteki seishitsu” [The unconscious nature of insight problem solving]. Cognitive Studies 20 (3): 353–367 (in Japanese). https://doi.org/10.11225/jcss.20.353
• Murakoshi, Shin. 1996. “Senzai gakushū paradaimu to undō gakushū” [The implicit learning paradigm and motor learning]. Japan Journal of Physical Education, Health and Sport Sciences 40 (5): 329–332 (in Japanese). https://doi.org/10.5432/jjpehss.KJ00003391417
• Oki, Yuta, and Masahiro Kokubu. 2021. “Chūi no shōtenzuke ga entō undō no gakushū ni ataeru eikyō” [Effects of attentional focus on learning a long-distance throw]. Japanese Journal of Sport Psychology 48 (2): 37–49 (in Japanese). https://doi.org/10.4146/jjspopsy.2021-2003
• Fukushima, Masato. 2010. Gakushū no seitaigaku: Risuku, jikken, kōshinraisei [The Ecology of Learning: Risk, Experiment, High Reliability]. Tokyo: University of Tokyo Press (in Japanese). https://ci.nii.ac.jp/ncid/BB03005045
• Ishimura, Gensei. 2013. “Shohyō: Fukushima Masato ‘Gakushū no seitaigaku’” [Book review: Fukushima, The Ecology of Learning]. Japanese Journal for Science, Technology and Society 10: 151–154 (in Japanese). https://doi.org/10.24646/jnlsts.10.0_151
• Hatano, Giyoo. 1995. “Rikai to kyōju no sōgo sayō” [The interaction of understanding and instruction]. Journal of the Japanese Society for Artificial Intelligence 10 (3): 354–360 (in Japanese). https://doi.org/10.11517/jjsai.10.3_354
• Ichikawa, Shin’ichi. 2000. “Gainen, zushiki, tetsuzuki no gengoteki kijutsu o unagasu gakushū shidō” [Instruction that encourages verbal description of concepts, schemas, and procedures]. Japanese Journal of Educational Psychology 48 (3): 361–371 (in Japanese). https://doi.org/10.5926/jjep1953.48.3_361
• Ichikawa, Shin’ichi, ed. 2008. “Oshiete kangaesaseru jugyō” o tsukuru [Creating Lessons That Teach and Then Make Students Think]. Tokyo: Toshobunka (in Japanese). https://ci.nii.ac.jp/ncid/BA86113489
• Magara, Keiichi. 1999. “Gakushūsha no ayamatta chishiki o dono yō ni shūsei suru ka” [How to correct learners’ erroneous knowledge]. Journal of Science Education in Japan 23 (1): 33–41 (in Japanese). https://doi.org/10.14935/jssej.23.33
• Kudo, Yoshifumi. 2003. “Gainen juyō gakushū ni okeru chishiki no ippanka kanōsei ni oyobosu kyōji jōhō kaishaku no eikyō” [Effects of interpretation of instructional information on the generalizability of knowledge in concept reception learning]. Japanese Journal of Educational Psychology 51 (3): 281–287 (in Japanese). https://doi.org/10.5926/jjep1953.51.3_281
• Kudo, Yoshifumi. 2013. “Rūru gakushū ni okeru chishiki hyōshō no fujūbun na chūshōka to sono mondai” [Insufficient abstraction of knowledge representation in rule learning and its problems]. Japanese Journal of Educational Psychology 61 (3): 239–250 (in Japanese). https://doi.org/10.5926/jjep.61.239
• Matsuura, Hiroshi. 1970. “Hōsoku gakushū ni okeru setsumeiteki hōhō to hakkenteki hōhō no kenkyū” [A study of expository and discovery methods in learning laws]. Japanese Journal of Educational Psychology 18 (3): 129–138 (in Japanese). https://doi.org/10.5926/jjep1953.18.3_129
• Mizukoshi, Toshiyuki. 1970. “Amerika ni okeru hakken gakushū ni tsuite no ichi kōsatsu” [A study of discovery learning in the United States]. The Japanese Journal of Educational Research 37 (1): 21–31 (in Japanese). https://doi.org/10.11555/kyoiku1932.37.21
• Kawakami, Shogo, and Koichiro Watanabe. 2010. “Nihon ni okeru yūimi juyō gakushū no tenkai” [The development of meaningful reception learning in Japan]. Journal of Research in Science Education 50 (3): 1–14 (in Japanese). https://doi.org/10.11639/sjst.KJ00006659483
• Tanaka, Yoshifumi. 2025. “Geijutsuka wa ikani manabi, ikani oshieru ka? Ongaku daigaku kyōin e no intabyū ni motozuku kenkyū” [How do artists learn and teach? A study based on interviews with music college faculty]. Cognitive Studies 32 (1): 129–151 (in Japanese). https://doi.org/10.11225/cs.2024.056

Philosophy

• Ryle, Gilbert. 1949. The Concept of Mind. London: Hutchinson (bibliographic record confirmed from the Routledge 2009 reprint). https://ci.nii.ac.jp/ncid/BA91415909
• Polanyi, Michael. 1966. The Tacit Dimension. Garden City, NY: Doubleday (bibliographic record confirmed from the University of Chicago Press 2009 reprint). https://ci.nii.ac.jp/ncid/BA89793619
• Dreyfus, Hubert L., and Stuart E. Dreyfus. 1986. Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. New York: Free Press. https://archive.org/details/mindovermachinep00drey
• Dreyfus, Stuart E. 2004. “The Five-Stage Model of Adult Skill Acquisition.” Bulletin of Science, Technology & Society 24 (3): 177–181. https://doi.org/10.1177/0270467604264992
• Stanley, Jason, and Timothy Williamson. 2001. “Knowing How.” Journal of Philosophy 98 (8): 411–444. https://doi.org/10.2307/2678403