Shuichiro Ogawa
日本語

Notes · updated 2026-08-07

Scope and Method

The question of this note is how many learning services deliberately under-specify instructions so that learners fall into the gaps, and what supports that design. The starting point is the Cloud Resume Challenge design confirmed in the scenario-learning note (its creator explicitly states the deliberate under-instruction), but this design philosophy is not one service’s invention; it has an established lineage in the learning sciences. Two streams were therefore collected. The scholarly stream: 13 core works on design theories that deliberately build failure in (source/review/failure-driven-learning/papers.md; a lineage grounding, not an exhaustive review). The industry stream: 13 technical learning services whose makers explicitly state intentional difficulty in first-person sources (source/review/failure-driven-learning/industry.md). Explicit statement was the inclusion criterion, to separate materials that are merely unhelpful from materials whose difficulty was designed. For Schank’s expectation failure, the GBS literature map is canonical.

The Scholarly Stream: Four Lineages of Designed Failure

Theories that build failure into the learning path rather than removing it are established in at least four independent lineages.

Productive Failure (Kapur 2008) showed in quasi-experiments that having learners tackle complex problems without support structures (where they mostly fail), followed by direct instruction in canonical solutions, improves transfer over teaching first. The design theory (Kapur & Bielaczyc 2012) is emphatic that failure alone produces no learning: the failure phase exists to generate diverse solutions and awareness of knowledge gaps, and the subsequent consolidation instruction leads learners to canonical solutions using their own attempts as scaffolds. A meta-analysis of 53 studies (Sinha & Kapur 2021) reports a moderate significant effect for the problem-solving-first design, increasing with fidelity to PF’s design principles.

Desirable difficulties (Bjork 1994) is the memory-psychology lineage formalizing training conditions (spacing, interleaving, retrieval practice) that depress short-term performance but improve long-term retention and transfer. Impasse-driven learning (VanLehn et al. 2003) showed from analyses of human tutoring that learning occurs only when the learner reaches an impasse, supplying the mechanism by which being stuck prepares the reception of instruction. Error management training (Frese et al. 1991; meta-analysis Keith & Frese 2008) is the workplace-training lineage: training that encourages active error experience under minimal instruction, paired with explicit emotion-regulation messages (“errors are informative”), outperforms conventional training on acquisition and transfer (d ≈ 0.44). Schwartz’s preparation-for-future-learning work (“A Time for Telling” 1998; invention activities 2004) is the adjacent lineage sharing PF’s serial design, emphasizing generative activity before telling rather than failure as such.

The lineage has an established opposing school. Kirschner, Sweller & Clark (2006) argued from cognitive load theory that minimal guidance overloads novices’ working memory and does not work. Its direct target is unstructured discovery learning, so PF, which always includes consolidation instruction, sits at the edge of its range; but the boundary-condition review by Loibl et al. (2017) identified that effects can vanish unless the prior problem solving generates contrasting cases and the instruction builds on learners’ solutions. Designed difficulty, in other words, is not always desirable; effectiveness depends on the quality of post-failure consolidation. The debate with this opposing school (response literature, direct empirical confrontations, unresolved conflicts) is organized by contention point in The Debate Over Designing Failure Into Learning. The reverse face of this lineage — the claim that generative AI harms learning by preemptively erasing opportunities for failure and difficulty — is covered empirically in Does Early Use of Generative AI Inhibit the Formation of Thought?.

The Industry Stream: Services That State Intentional Difficulty

Technical learning services whose makers state the intentional design of difficulty fall into four types by the form of difficulty.

Under-instruction: give the goal, withhold the steps. The Cloud Resume Challenge’s creator writes in the FAQ that he “didn’t give you enough instructions to figure any of this out without going down some late-night rabbit holes,” positioning “the ability to learn fast and google well” as the core skill. CodeCrafters (build Redis or Git from scratch) provides only an initial codebase and high-level requirements, and its About page states “If it feels easy, you’re not learning.” The Odin Project declares the course “very research based,” requiring learners to do their own research, and recommends human community dialogue over direct AI answers. On the vendor side, AWS Jam (“real-world scenarios that don’t have step-by-step instructions,” hints traded for score deductions) and Google Cloud Challenge Labs (“instead of following step-by-step instructions, you will… figure out how to complete the tasks on your own”) take the same form.

Broken artifacts: hand over something already broken and have the learner fix it. Rustlings (official Rust project) and Ziglings are exercises that block progress until compile-error-laden programs are fixed, explicitly aiming to acclimate learners to reading unaltered compiler output. SadServers’ author writes that “showing the solution to the scenarios defeats the purpose,” providing only a problem description and a pass/fail test. The author of Learn Code the Hard Way builds break-and-fix demonstrations into the material and likens repetition and self-directed research to martial-arts drill.

Hint-refusal culture: withholding support is itself the doctrine. OffSec’s “Try Harder” is an official pedagogy whose CEO states “we don’t hand out easy answers because that’s how real-world skills are built,” defined by self-directed exploration, resourcefulness through adversity, and learning from failure. Hack The Box’s founder describes creating the platform out of frustration with reading-centric training, so that “you don’t follow a set of instructions, you must figure it out and carve new paths.” Advent of Code has no official hint mechanism; its stated design commitments are the difficulty curve and the guarantee that every problem has a solution completing within 15 seconds on old hardware.

Solo-first then feedback differs from the other types by institutionalizing post-failure support. Exercism’s co-founder says “you don’t know what you don’t know… it’s very hard to identify your own knowledge gaps” and that the gap between working code and great code “is too big just to jump,” making the two-stage design (solve alone, then human mentor review) the core.

Theory Against Industry: The Thinness After Failure

Checking the implementations against the scholarly lineages, the failure phase is richly designed but the post-failure consolidation that theory makes the condition of effectiveness is thin.

In Productive Failure’s design theory, failure is preparation; the effect arrives only with consolidation instruction that scaffolds from learners’ attempts to canonical solutions. Yet of the 13 industry cases, only Exercism (mentor review) and the recently shifted OffSec (from pure Try Harder to weekly mentoring alongside) institutionalize post-failure support; most offer pass/fail tests, community reliance, or nothing. The emotion regulation that error management training identified as a mediator (explicitly messaging that errors are informative) is likewise not systematically implemented; OffSec’s caveat that Try Harder “does not glorify burnout or blind persistence” is about as far as it goes. Layering Kirschner’s critique and Loibl’s boundary conditions, the faithful scholarly reading is conditional: these services’ intentional difficulty can work as desirable difficulty for learners with prior knowledge, but can become unconsolidated cognitive overload for novices. Indeed, the creator of the Cloud Resume Challenge himself targets people who have finished the basics and want practical experience, so industry practice implicitly encodes this boundary condition.

No evidence was found that any of these services were designed with reference to PF or EMT. The makers’ motives are uniformly their own practice (“real skills were built through struggle”); learning science and industry implementation converge on the same design independently. This is the same configuration as the industrial convergence of scenario equipment confirmed in the scenario-learning note.

Implications for Student Education

Applying this to students requires transferring the lineage’s two conditions into the design. First, deliberate under-instruction acquires its theoretical backing only when paired with post-failure consolidation (explanations scaffolded on students’ attempts, mentor review). If assigning Challenge-Lab-style tasks, collecting “where you got stuck and what you tried” after submission and connecting it to canonical solutions in lecture corresponds to PF’s design theory. Second, calibrate the amount of difficulty to prior knowledge. For novices, the broken-artifact type (Rustlings-style, where the error’s location is bounded and the compiler gives immediate feedback) controls cognitive load better than under-instruction; the under-instruction type (Cloud Resume Challenge-style) fits as a capstone for students past the basics, per the boundary conditions. The scenario-design framework (GBS’s seven components) is covered in the scenario-learning note, and design-review tools in Tools for Learning System Design.

Gaps

  • No independent study validating the learning effectiveness of the industry services that state intentional difficulty was found; the scholarly evidence comes from schooling and workplace training, and extrapolation to self-paced online services is unverified.
  • No peer-reviewed work directly bridging Schank’s expectation failure and Kapur’s productive failure was found (the structural similarity exists, but the bibliographic connection is unverified).
  • No public data shows dropout rates for these services (how many learners the intentional difficulty sheds); the emotion-regulation gap identified by EMT may surface as dropout, but this cannot be verified.
  • Intentional-failure design specific to system design education is thin beyond parts of AWS Jam and Challenge Labs; combining design exercises (Architectural Katas and the like) with failure design is an open space.

Unverified Items

  • Chapter pages for Bjork (1994, 2011) (no DOIs; confirmed by book ISBN)
  • Full text of Frese et al. (1991) (paywalled; bibliography confirmed via Crossref)
  • The explicit statement that OffSec deliberately includes rabbit holes in exams
  • A primary URL for Eric Wastl’s remark that looking up hints is fine (podcast transcript unpublished)
  • Publication date of the Hack The Box interview (cynthiacorsetti.com) and details of official hint features
  • SadServers’ paid-plan details; Exercism’s primary documentation of Practice Mode design intent (official site returns 403)
  • Design statements under the personal names of Erik Trautman (The Odin Project) and Katrina Owen (Exercism)

References

All accessed 2026-08-07. For ledger details, see source/review/failure-driven-learning/papers.md (scholarly) and source/review/failure-driven-learning/industry.md (industry).

Scholarly (T1)

  • Kapur, M. (2008). Productive Failure. Cognition and Instruction, 26(3), 379–424. https://doi.org/10.1080/07370000802212669
  • Kapur, M., & Bielaczyc, K. (2012). Designing for Productive Failure. Journal of the Learning Sciences, 21(1), 45–83. https://doi.org/10.1080/10508406.2011.591717
  • Sinha, T., & Kapur, M. (2021). When Problem Solving Followed by Instruction Works: Evidence for Productive Failure. Review of Educational Research, 91(5), 761–798. https://doi.org/10.3102/00346543211019105
  • Kapur, M. (2024). Productive Failure: Unlocking Deeper Learning Through the Science of Failing. Jossey-Bass. https://onlinelibrary.wiley.com/doi/book/10.1002/9781394308712 (non-peer)
  • Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about Knowing (pp. 185–205). MIT Press. ISBN 0-262-13298-2.
  • Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way. In Psychology and the Real World (pp. 56–64). Worth Publishers.
  • VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. B. (2003). Why Do Only Some Events Cause Learning During Human Tutoring? Cognition and Instruction, 21(3), 209–249. https://doi.org/10.1207/s1532690xci2103_01
  • Schwartz, D. L., & Bransford, J. D. (1998). A Time for Telling. Cognition and Instruction, 16(4), 475–523. https://doi.org/10.1207/s1532690xci1604_4
  • Schwartz, D. L., & Martin, T. (2004). Inventing to Prepare for Future Learning. Cognition and Instruction, 22(2), 129–184. https://doi.org/10.1207/s1532690xci2202_1
  • Frese, M., Brodbeck, F., Heinbokel, T., Mooser, C., Schleiffenbaum, E., & Thiemann, P. (1991). Errors in Training Computer Skills: On the Positive Function of Errors. Human–Computer Interaction, 6(1), 77–93. https://doi.org/10.1207/s15327051hci0601_3
  • Keith, N., & Frese, M. (2008). Effectiveness of Error Management Training: A Meta-Analysis. Journal of Applied Psychology, 93(1), 59–69. https://doi.org/10.1037/0021-9010.93.1.59
  • Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why Minimal Guidance During Instruction Does Not Work. Educational Psychologist, 41(2), 75–86. https://doi.org/10.1207/s15326985ep4102_1
  • Loibl, K., Roll, I., & Rummel, N. (2017). Towards a Theory of When and How Problem Solving Followed by Instruction Supports Learning. Educational Psychology Review, 29(4), 693–715. https://doi.org/10.1007/s10648-016-9379-x

Industry (T3)

← All Notes · Home