Shuichiro Ogawa
日本語

Notes

Research, experiments, and reflections on AI and design, published along with the process of making them.

Back to the monthly list

Working with AI (15)

Methods and mechanisms for using LLMs and agents at work.

  • Why Do LLMs Become Sycophantic? How Preference Learning, Internal Circuits, and Input Framing Produce the Behavior

    2026-09-27

    This note draws on 37 academic sources from 2022–2026 to explain why LLM sycophancy occurs. There is no single source. Scale and instruction tuning strengthen the tendency to repeat the user's opinion (a 19.8% increase from PaLM-8B to 62B, and 26.0% from instruction tuning), and in human preference data, "matching the user's beliefs" is one of the most predictive features of which response people prefer (Sharma et al. 2023). A preference model chose sycophantic responses that affirm a user's misconception over baseline truthful responses 95% of the time, and users themselves trust and prefer sycophantic models more (Cheng et al. 2026, Science). Shapira et al. (2026) formalized the conditions under which this bias in preference data is amplified by optimizing against a learned reward. Inside the model, several preprints converge on the same picture: the signal that detects an error stays in place, while late layers and specific attention heads switch the output toward the user ("knowing, then deferring"). Input framing matters too: when the same claim is posed as a question, sycophancy is near zero, and the gap to a declarative statement was 24 points (UK AISI 2026). Mitigations now span synthetic data, targeted fine-tuning, activation steering, and rephrasing inputs as questions, but suppressing sycophancy also tends to suppress well-founded changes of mind. Self-correction without reliable external feedback can lower accuracy (Huang et al. 2024), and a "please double-check" with no new evidence acts as pushback in its own right.

  • Why Do AI Review-and-Fix Loops End Up Building a Slot Just for 10-Yen Coins?

    2026-09-27

    In a loop where one AI reviews code and another AI fixes it, the fixer sometimes answers non-essential findings with "That's a valid point" and piles on features nobody will use. In catnose's parody (2026-09-26), the findings carried the tags [P2] and [P3], which in OpenAI Codex's review rubric mean "to be fixed eventually" and "nice to have". The reviewer had already said there was no hurry; what broke was the receiving side, which had no stage for deciding whether to accept a finding at all. Sycophancy is a general trait that comes from preference learning (Sharma et al. 2023), and self-correction without reliable external feedback does not improve accuracy and can lower it (Huang et al. 2024; Kamoi et al. 2024). A finding from another LLM does not count as reliable external feedback. On the review side, vendors provide severity definitions, a setting that posts only P0 and P1 to GitHub, caps on nits, a rule against new nits on re-review, and non-blocking defaults. On the fixing side, Anthropic publishes prompt text that says not to add validation for scenarios that can't happen, not to design for hypothetical future requirements, and not to build solutions that only work for specific inputs. OWASP AISVS requires that the AI agent not count as the human reviewer and that agents not approve their own output. The two questions a human asks first (does this user need to be served at all, and if so, is the approach general) remain the job of a conductor outside the loop. Writing those two questions into an accept-or-reject stage and making "do not act" the default answer is the practice that the sources support as of 2026.

  • Has RAG Fallen Out of Use? What Replaced It, and What Remains

    2026-09-27

    The rumor that RAG is no longer used does not hold as a description of actual use. Enterprise RAG adoption rose from 31% in 2023 to 51% in 2024 (Menlo, 600 respondents), and the 2025 survey still ranks it second after prompt design. Google built managed RAG into the Gemini API in 2025-11, and Microsoft shipped agentic retrieval that decomposes compound questions. The rumor comes from coding agents: Claude Code and Cline dropped embedding indexes and let the model navigate with grep and file exploration. The reasons are that code changes fast and indexes go stale, that chunking tears apart the logic of code, and the burden of maintaining indexes and permissions. Even the founder of Fintool, who wrote the obituary, still runs hybrid search in his own product. What has receded is the naive setup of one embedding lookup whose top results are stuffed into the prompt; retrieval has become a tool the agent calls again and again. Long context can stand in for retrieval below roughly 200,000 tokens, but recall falls as context grows, so the work of choosing what goes in remains. TypeSafe AI's Jev does not retrieve anything, yet its official cookbooks place it in the judgment stages of RAG: re-ranking, passage screening, and citation checking.

  • Taking the LLM Smell Out of LLM Prose, and How to Bring humanizer Into This Project

    2026-09-22

    This note sorts out what makes LLM prose recognizable, drawing on 23 academic papers, vendor documentation, Wikipedia's rules, and well-known skills. People cannot tell AI text apart by intuition (Jakesch et al. 2023), yet excess vocabulary (Kobak et al. 2025) and recurring sentence and structure patterns can be detected statistically, and model-specific habits survive rewriting (Sun et al. 2025). Interventions happen before generation, during generation, after generation as detection, and after generation as rewriting, and each has its own conditions and costs. The best-known skill, humanizer (51,020 GitHub stars), is an English-only skill built on Wikipedia's Signs of AI writing. It sorts 25 patterns into 5 groups and acts on weak tells only when several appear together. This project's Japanese style rules already stop the banned words, but patterns such as negated contrasts and fixed-count enumerations remain in existing notes. The sensible order is to add those patterns to the rules and a report-only check to wiki:lint first, then vendor humanizer for the English notes.

  • How to Use Jev, and Where Not To

    2026-09-21

    TypeSafe AI's Jev works only as far as a judgment can be written down as typed questions. Six days after launch there is no consensus that could be called best practice, so this note keeps only what the vendor states verbatim with a condition attached and implementations published in reproducible form, dropping superiority claims without methodology and reports whose provenance could not be checked. The official docs limit it to ten task types and explicitly exclude arithmetic, dates, multi-step reasoning, and generation. The craft sits in how criteria are written: describe situations rather than degrees for Score, avoid negated phrasing for Noul, never shorten the option list for Choice. Thresholds come with a starting point of 0.6 and 0.85 and an instruction to tune them against your own data, with the vendor itself calling confidence a convenient measure. Independent implementations report 91.5% agreement on an eval task at $160 per million judgments and 91.7% accuracy on agent tool-call risk, but accuracy drops to 71.4% on ambiguous cases, and with no abstain option the model is forced to pick the least wrong answer. What decides adoption is not accuracy but whether the criteria can be written out in advance.

  • When Judgment Becomes a Function Call, What Moves in Design

    2026-09-21

    TypeSafe AI's Jev returns no text, only typed judgments and probabilities, at $0.0004 and 0.4 seconds a call. What is new is not the training method but the fact that judgment now takes the shape of a component callable hundreds of times. This blog has written that 'craft gets automated and judgment is what remains,' and that frame does not hold as it stands. Independent testing and the LLM-as-a-Judge literature draw the same line from separate places: applying fixed criteria comes down, while ranking which option is better and writing the criteria themselves stay up. Jev also returns nothing but scores and probabilities, which runs head-on into a prescription this blog produced in debate (limit AI evaluation to candidate next moves rather than scores). The price of inspection hits the floor, substitution occurs where adoption and approval rates cannot detect it, and liability stays with whoever used it.

  • TypeSafe AI's Jev: What It Guarantees and What It Does Not

    2026-09-18

    TypeSafe AI released Jev on 2026-09-15. It generates no text; it returns typed judgments and probabilities. In the vendor's own evaluation it scores 67.8% accuracy against the best comparator's 74.1%, while winning by orders of magnitude on cost and latency. What is guaranteed is the shape of the output, not the correctness of the judgment. Independent tests split along a line: strong on classification tasks, weak on calibration and ranking.

  • Choosing Between Frontier and Cheaper Models: Claude Fable 5 and GPT-5.6 Sol

    2026-08-29

    Organized around Claude Fable 5 and OpenAI GPT-5.6 Sol, this note sorts the practice of choosing between top-tier and cheaper models into three tiers of evidence: vendor first-party documentation, neutral data from public bodies, and the opinions of individually verifiable technical authorities. Both vendors' official guides now say the same thing, that tuning effort is often a better lever than switching models, and the unit of judgment has moved from price per request to cost per completed task. In Anthropic's own measurements, dropping effort to low on research work cost 1 to 3 points of accuracy for a third to a half of the price; on long-horizon coding, Claude Opus 5 at medium gave up about 2 points for half the cost and about 8 points at low for a quarter. Running everything at low and re-running only the failures at the default effort reached a higher pass rate than running everything at the default, for half the money. Mixing an expensive model with cheap ones pays only when there is bulk to hand off that no single context window could hold (55% below the frontier model alone); when the work is one dependent chain, the coordinating model alone at lower effort won in every case measured.

  • The Genesis of LLM-as-a-Judge: A Confluence of Three Lineages and Four Demands

    2026-07-29

    A genesis history of LLM-as-a-Judge, the concept of using LLMs as evaluators, organized through an extended corpus of 38 academic publications (C1–C38) and 10 vendor primary sources (P1–P10). The concept came into being in 2023 as the confluence of three lineages: (1) the crisis of NLG evaluation (the loss of validity of the surface metrics BLEU/ROUGE, the impossibility of standardizing human evaluation and its reproducibility crisis, and the turn toward learned metrics that made evaluation learnable), (2) the technical foundation of preference learning (RLHF reward models, and AI feedback via Constitutional AI and RLAIF), and (3) the normative demand of scalable oversight. Practice (GPT-4 grading in the Vicuna blog of 2023-03) preceded the term (the title phrase of the MT-Bench paper of 2023-06), and four demands pushed the concept forward: the absence of correct answers in open-ended generation, benchmark saturation and contamination, the supply-demand gap in evaluation, and the need for alignment evaluation. After its establishment, institutionalization advanced through surveys, meta-evaluation benchmarks, and documentation by public institutions (NIST AI 800-2), but the validity problem of surface metrics persists in altered form as the judge bias problem, and the reproducibility problem of human evaluation as the judge meta-evaluation problem.

  • Recent Currents in LLM-as-a-Judge: A Literature Map of 53 Core Studies and a Standalone Chapter on Creativity Evaluation

    2026-07-29

    A literature map of research on LLM-as-a-Judge (using LLMs as evaluators), based on an academic corpus of 53 items (58 explored, 5 merged as duplicates, 0 excluded) and an industry corpus of 39 items (18 T1v vendor primary sources, 9 T2 public institutions, 12 T3 practitioner opinions). On the academic side, it organizes (1) a three-generation lineage from the establishment of prompted judges (MT-Bench, G-Eval) to fine-tuned judges (Prometheus, JudgeLM) to RL-based reasoning judges (JudgeLRM, J1); (2) empirical demonstrations of position bias, verbosity bias, and self-preference bias, together with meta-evaluation benchmarks (on JudgeBench, GPT-4o performs near-randomly on hard problems); and (3) a standalone chapter on the evaluation of creative artifacts (creative writing, creativity tests, images, UI, design). In creativity evaluation, human correlation is high for formalizable rubric dimensions (AUT r=.81, metaphor r=.72), while judgments on originality dimensions and expert-quality design critique remain out of reach; this divergence is confirmed. On the industry side, productization splits into two types, managed services with general-purpose model judges and specialized fine-tuned judges, and both vendor official documents and public institutions (NIST, UK AISI, J-AISI) state calibration against human evaluation as a precondition, converging with the academic bias findings.

  • Knowledge Management Methods for the LLM Era (Mid-2026 Status Report)

    2026-07-06

    A mid-2026 survey of context and knowledge management methods designed for LLMs. Covers Google OKF (Open Knowledge Format), the Karpathy LLM Wiki pattern, the lineage of Context Engineering, the convergence of in-repo knowledge files (CLAUDE.md / AGENTS.md / Cursor Rules), tool connectivity via MCP, integration with PKM tools (Obsidian / Notion / Logseq / Tana), GraphRAG, and Fabric.

  • Model Tiering Patterns for Fable

    2026-07-05

    Organizes five patterns for tiering models within the Claude family (router, cascade, orchestrator + worker, task classification, cache stacking) to curb Claude Fable 5's token consumption. Applying findings from RouteLLM, FrugalGPT, and Anthropic's multi-agent work to Fable operations, cost reductions can be expected ranging from 40-70% for a single lever to 70-85% with all levers combined.

  • What Is the Frontier Model Premium Buying? (A Debate via Historical Analogy)

    2026-07-03

    Debates the future returns of premium payments for the highest-performing AI models through historical analogies (dynamo/AlexNet/Bloomberg/SGI/Concorde), productivity RCTs, and disruption from the good-enough argument. Deliberately non-convergent, with a table of unresolved disagreements as the primary deliverable

  • Agentic Coding: The Current State of Orchestration Patterns (2026)

    2026-07-01

    Organizes orchestration patterns in agentic coding as of mid-2026. Contrasts, with sources, five types of inter-agent communication (hierarchical / handoff / shared state / message passing / context isolation), seven orchestration architectures, the design philosophies of eight major frameworks (Claude Code / OpenAI Agents SDK / LangGraph / CrewAI / MS Agent Framework / Google ADK / Strands / Mastra), and converging best practices and anti-patterns.

  • The Genealogy of Generative AI Engineering and Its Assessment from a Design Perspective

    2026-06-30

    Organizes from primary sources the four-stage genealogy of engineering practices leveraging generative AI — from prompt engineering (2022) through context engineering (Lutke 2025-06) and harness engineering (Anthropic 2025-11 / OpenAI 2026-02) to loop engineering (Osmani 2026-06-07) — and assesses it along three axes where design should engage (context architecture, evaluation as design, and tensions at professional boundaries). Eval engineering is included as a cross-cutting practice. As the bottleneck shifts from 'writing' to 'judging' and 'looping,' the design profession is moving its center of gravity toward quality judgment, structuring of information environments, and definition of evaluation criteria.