Shuichiro Ogawa
日本語

Notes · updated 2026-09-21

When Judgment Becomes a Function Call, What Moves in Design

TypeSafe AI's Jev returns no text, only typed judgments and probabilities, at $0.0004 and 0.4 seconds a call. What is new is not the training method but the fact that judgment now takes the shape of a component callable hundreds of times.

Contents (15)
  1. Selling Judgment Alone at $0.0004 a Call
  2. What Is New Is Not the Training Method
  3. The Type Is Guaranteed, the Judgment Is Not
  4. Has Reliability Judgment Been Outsourced?
  5. Judgments That Come Down, and Judgments That Do Not
  6. Writing the Criteria Moves to the Front
  7. Scores Come Back, Next Moves Do Not
  8. A Bundle of Passed Checks Is Not Quality
  9. When Verification Gets Cheap, What Gets Verified Is Fixed
  10. How to Count Cheapness
  11. The Price of Inspection Hits the Floor
  12. Substitution That Adoption Rates Cannot See
  13. What Happens on the Learning Side
  14. Liability Stays with Whoever Used It
  15. What Moves and What Does Not

Selling Judgment Alone at $0.0004 a Call

This blog has held that what AI takes is the craft, and what remains for the designer is the judgment. The shift from making to editing shows up in the numbers as well. In a survey of 217 practitioners across 43 countries, 71% spent more time evaluating and curating AI output than producing work themselves, and among senior practitioners that figure reached 91% (Rivera & Russi 2026, From Maker to Editor: A Structural Analysis of the Designer Role Transition in the Age of AI).

Jev, which TypeSafe AI released on 2026-09-15, sells the half that was supposed to remain at $0.0004 and 0.4 seconds a call. It returns no prose. What comes back is an option with its probabilities, a score on an ordered scale, or a truth value from 0 to 1 (TypeSafe AI's Jev: What It Guarantees and What It Does Not).

The thing expected to remain now has a price on it.

What Is New Is Not the Training Method

The idea of training against calibration as the reward came first. Work using a logarithmic scoring rule as the reward to make an LLM produce calibrated confidence was accepted at ICLR 2026 (arXiv:2503.02623, first version 2025-03-04), reporting ECE 0.0226 on TriviaQA. Anthony Maio also writes that “Reinforcement learning for calibration is also not new,” and concludes that Jev’s strongest claim is “architectural rather than algorithmic.”

What is new is that judgment has taken the shape of a function. Input is text, output is a typed value, the API is a single endpoint, output tokens are free, and latency runs from 70ms to 500ms. Questions are evaluated in parallel, so adding questions does not add latency. TypeSafe’s own manifesto says “Intelligence today is like databases before SQL.”

Whether the analogy holds is a separate question; the change it points at is clear. It is the change from judgment as something consulted one call at a time to judgment as a component that can be invoked hundreds of times. An implementation that ran a 370-rule diff check in under 2 seconds, and a record of 777 judgments across 37 documents and 21 questions processed in under 0.7 seconds for about 0.25 cents, show that this way of handling it actually works.

The Type Is Guaranteed, the Judgment Is Not

TypeSafe’s landing page says “Zero Hallucinations.” That is not a measurement. The launch post states: “Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots.” The reasoning is that a type match is guaranteed by construction, so 0% can be written in.

Maio’s one sentence is shorter. “Jev constrains the shape of the output. It does not constrain the judgment.”

Put back into design work: the fact that a review result comes back matching the schema says nothing about whether it is correct as a review. An error in good form is harder to find than an error in broken form.

Has Reliability Judgment Been Outsourced?

This blog has recorded the new types of judgment emerging where AI has entered the work of making. They are the agency-allocation judgment (how much to hand to AI) and the reliability judgment (how far to trust AI output) that Naik et al. (2025) identified from the reflection records of 33 teams in an HCI design course (From Maker to Editor: A Structural Analysis of the Designer Role Transition in the Age of AI).

Jev’s confidence looks like that reliability judgment sold as a number. It even comes with three bands: high for automated action, medium for proceeding with attention, low for not acting.

The official definition, however, reads as follows. “confidence is a statistic computed from the probability distribution the answer already gives you.” It is a statistic computable from the distribution the answer already carries, not a separately learned number. Archer Hume’s black-box audit of roughly 10,000 API calls likewise reports that confidence was an arithmetic value for how far the top answer sat from a uniform distribution (he states explicitly that his reconstruction is speculative).

What comes back is the answer rewritten in different units. It is not independent evidence backing that answer up. Setting out to outsource the reliability judgment, what you get is the same judgment restated.

Deciding automated action by threshold already has a cautionary precedent. From the measured finding that 93% of permission prompts are approved, Anthropic judged that per-action approval was not functioning as oversight, and changed its design toward a model that moves freely inside a boundary (AX and Design: A Cross-Comparison of Academic and Industry Perspectives). Assigning high confidence to automated action reproduces that 93% in a form with no human in it at all.

Judgments That Come Down, and Judgments That Do Not

The independent results split cleanly. Strong on classification. In a pre-registered pilot (300 examples, jev-1.13.0), AG News came out Jev 0.910 against GLiNER2.5’s 0.700, and Banking77/BTZSC 0.870 against 0.610. Weak on ranking. In a 53-day news-triage test, mean daily AUC was 0.653 for Jev, 0.697 for Sonnet 5, and 0.637 for a position-based baseline. Losing on calibration. On DAIR Emotion the Brier score was 0.846 against 0.668, and 16% of examples were assigned probability 0 on the true label.

There is evidence of the same line being drawn from a different place. The literature map for LLM-as-a-Judge shows high human correlation on fixed rubric dimensions (r=.81 on AUT, r=.72 on metaphor) alongside a gap: judging originality dimensions and producing expert-quality design critique remain out of reach (Recent Currents in LLM-as-a-Judge: A Literature Map of 53 Core Studies and a Standalone Chapter on Creativity Evaluation). UICrit (UIST 2024), a dataset of 3,059 designer critiques, improved LLM feedback by 55% with few-shot prompting and still states outright that it does not reach expert quality.

The independent tests of a judgment model and the line of judge research draw the same boundary without knowing about each other. Judgment that applies fixed criteria comes down. Judgment that ranks which option is better does not.

Restated at the grain of practice: detecting guideline violations, classifying components, judging the tone of copy, and checking against accessibility rules fall on this side. Which of three proposals to put in front of the client does not.

Writing the Criteria Moves to the Front

Calling Jev requires writing criteria. For Choice, a map from option name to description; for Score, descriptions of at least 2 and at most 10 levels. The precondition for invoking a judgment is that the evaluation axis has already been written down in prose.

This shape is the same one this blog has described for design systems. Given a formal specification declaring module boundaries and constraints, an AI coding agent’s navigation steps fell by 33 to 44%, and accuracy reached 100% even with auto-generated descriptors (Jin 2026, Wilcoxon p=0.009, d=0.92; Design Systems as AI's Foundation Layer — Structured Design Knowledge Determines Agent Accuracy). The reading is that the meaning of investing in a design system moves from maintaining consistency to setting the ceiling on agent accuracy.

Jev put a price tag on that. Criteria written down become an asset enforceable hundreds of times at $0.0004 a call. Criteria not written down never reach the enforcement circuit.

Scores Come Back, Next Moves Do Not

This blog produced one prescription in its debate on AI and design. Limit AI evaluation to feed-forward, meaning candidate next moves, rather than to scores. The design intent was to present AI output as a provisional viewpoint rather than a settled one, and to leave the step of taking the evaluation and choosing what comes next on the human side.

Jev returns nothing but scores, choices and probabilities. It is not designed to return candidate next moves. That runs head-on into the prescription.

Parallel evaluation shows a similar kind of mismatch. The official documentation says questions are evaluated independently of one another and do not contaminate each other’s context, which is close to holding several evaluation axes side by side without converging them. What comes back, though, is a probability per axis, not an operation that recomposes the axes. Whoever wrote the criteria is holding the axes.

Reframing can only be placed outside this apparatus. The back-and-forth that Dorst and Cross described as the co-evolution of problem and solution proceeds by the evaluation axis itself moving (Activating Reframing in Design: Cognitive Mechanisms and Practical Methods). Jev works only while the evaluation axis is held still.

There is one more asymmetry in the shape of the output. Choice picks one of up to 255 options, and Noul returns the truth of a proposition. Neither carries “not choosing” or “not making” as a first-class output. Seen from the argument that carried into design the finding that people skew toward adding elements when given no cue (Adams et al. 2021, Nature, 8 experiments), the judgment model too is encoded on the additive side (The Creativity of Subtraction: The Additive Bias, and the Hypothesis of Designing Absence).

A Bundle of Passed Checks Is Not Quality

Bundling cheap judgments does not make the result more certain in proportion to the bundle. Maio’s point goes as far as the mechanism. “individually calibrated judgments do not automatically compose into a calibrated workflow.” The reason is one sentence: “Correlated mistakes survive composition.”

TypeSafe itself does not guarantee arithmetic consistency between questions. The official documentation supplies a counterexample: on the same state, Noul returns 0.22 while Choice returns yes at 0.01 and no at 0.99. In another, refund is 0.72 and not_refund is 0.47, summing to 1.19. It says “don’t hold the model to arithmetic identities between separate questions.” Hume’s audit also observed that adding an irrelevant fifth option lowered the log-odds of the existing binary across every block.

Translated into design operations, this is what it means. Passing all 370 rules of a design system is neither a set of 370 independent passes nor proof that the design is good. Reading a bundle of checks as quality is an operation that gets adopted more readily the cheaper Jev is, and cheapness is no ground for correctness.

When Verification Gets Cheap, What Gets Verified Is Fixed

This blog has read AI slop not as poor quality but as a distribution of the burden of verification (AI Slop: Reading It as Outsourced Verification, Not Low Quality). What the term put at its core when it first appeared was pushing onto others what you have not verified yourself (Willison 2024). The skeleton is an asymmetry of cost: only the marginal cost of low-quality generation falls, while the cost of verification does not (Zhang & Zhang 2025).

Jev looks like a tool that breaks that asymmetry from the verification side. Except that calibration was where it came out weakest in independent testing. What falls is the price of verification, not the state of having verified.

The break comes with a condition. In exchange for lowering the price of verification, Jev requires that the content of the verification be written out in full as criteria beforehand. Criteria written out get enforced; criteria left unwritten do not.

Adjacent empirical work directs attention to the side of fixed criteria. Using generative AI raised evaluations of individual work while lowering novelty at the collective level (Doshi & Hauser 2024), and a single human essay added 2 to 8 times the novelty that GPT-4 did (Moon et al. 2025, N=2,200). Whether selection by cheap, fixed criteria pushes further in that direction has not been measured by anyone yet. This is placed here as conjecture.

How to Count Cheapness

This blog has already settled on the unit for comparing how cheap a model is. It is cost per completed task, not price per request (Choosing Between Frontier and Cheaper Models: Claude Fable 5 and GPT-5.6 Sol). A cheap model’s failure bills you for the tokens, the retries and all the downstream rework. A measured case sits there too: in a 20-question research task, 2 questions accounted for 43% of total spend.

Applied here, the verdict on Jev is held in reserve. Its 67.8% accuracy in the vendor’s own evaluation matches Claude Sonnet 5 and loses to the best comparator, GPT-5.6 Sol, at 74.1%. On unit price it wins by two orders of magnitude. What the 6.3-point gap bills as rework has not been measured by anyone.

The places where it pays off can be narrowed. Work where the rework cost of a wrong pass is small, the volume is high, and the criteria are fixed. In design practice that means the mechanical checks before delivery, not the selection of a proposal.

The Price of Inspection Hits the Floor

On the billing side, this blog has already written something close to a conclusion. AI resistance in a billing model weakens in the order outcome-linked, time, deliverable, and the moment a deliverable is defined as a thing, its price slides toward AI’s marginal cost. And the gatekeeper business splits into an upper tier that designs the rules and a lower tier that inspects against them (AI Commoditization Resilience of Design Billing Models — Output-Based Billing Is Most Vulnerable, and Retainer Stickiness Does Not Mean High Margins).

Jev is the apparatus that gives that lower tier a concrete price. The marginal cost of checking against accessibility rules, inspecting guideline compliance, or confirming the tone of a deliverable moves toward the $0.0004-per-judgment side. There are measurements. In the 53-day news triage, 3,408 requests were billed at $0.22. Running the same work on Sonnet 5 costs 78 seconds and $0.167 a day.

The upper tier, the work of deciding what counts as a pass, remains. How it remains changes, though. Criteria that exist only in someone’s head, rather than as enforceable prose, stop carrying a price.

Substitution That Adoption Rates Cannot See

This blog has written that the adoption rate of an AI tool cannot be used as evidence of approval. There is a gap between 86% usage and 10% approval of the industry’s direction alongside 69% burnout, and that gap reads not as individual inconsistency but as a collision between organizational efficiency pressure and professional identity (90% Adoption x 10% Approval — The Paradox of AI Tool Diffusion and Evaluative Divergence).

Jev sits outside that measurement. It is not a tool a designer touches on screen; it is a judgment component called from code. The occasion to be asked whether you used it, and to answer that you did, never arises.

Even if part of the judgment is replaced, the adoption and approval rates used as indicators of that gap cannot detect it. To detect it, what to look at is not people’s answers but which judgments in the workflow have been raised into typed questions.

What Happens on the Learning Side

Design education already has an axis for judging harm. The axis is that harm appears when the process being delegated is itself the current learning objective (Issues for Design Education (Not Craft Education): From the Evidence on Failure Design and Cognitive Offloading). What that axis flagged about generative AI was that it takes over the generation and exploration of alternatives by default.

What Jev takes over is not generation but evaluation. Evaluative judgment is among the learning objectives of design education, so on this axis it falls on the harm side. Because the delegated process differs, however, the evidence gathered about generative AI cannot be applied to it as is.

A distinction matters here. In an experiment with about 1,000 Turkish high school mathematics students, plain GPT-4 raised practice scores by 48% while lowering exam scores, with AI removed, 17% below the control group. Under a GPT Tutor condition that asked questions and provided scaffolding, practice rose 127% with no significant drop on the exam (Bastani et al. 2025, PNAS; Does Early Use of Generative AI Inhibit the Formation of Thought? A Literature Map of Cognitive Offloading and Learning). What worked was not withholding answers but returning questions. Jev withholds answers of a kind, and returns no questions either.

There is an advantage for teaching too. Choice, Score and Noul make the scope of what is delegated visible as a type. The agency-allocation judgment of what to hand to AI and how far becomes easier to teach to the extent that the type is written as a contract (AI Adaptation in Design Education — The Current State and Structural Challenges of Curriculum Reform). Conversely, a design that decides automated action by a confidence threshold replaces the moments where a student judges for themselves with the setting of a threshold. Which way it falls has not been observed yet.

Liability Stays with Whoever Used It

The obligations on the model side are thin. The EU AI Act treats models and systems separately, and model-level obligations attach to providers of general-purpose AI models. The indicative criteria for applicability given in the European Commission’s FAQ are twofold: training compute, and the ability to generate language, images or video. A decision-only model with no generative capability fails one of them. Applying this is inference, and no public source naming Jev could be confirmed.

The contractual side is thin too. The Terms of Service disclaim all warranties under AS IS, and cumulative liability is capped at $100 USD. There is no special clause for wrong decisions.

And design, as a profession, holds no institutional barricade. Its intellectual conditions largely satisfy the requirements of a profession, while it has arrived at the present day without licensure to enter or a monopoly of jurisdiction (Design and the Professions: A Literature Map of Design's Position Seen Through the Theory of Professions). Liability for a wrong pass stays with whoever used it. The more cheap judgments are run at volume, the larger the total liability that stays.

What Moves and What Does Not

This blog’s reading gets partially updated.

The frame that craft gets automated and judgment remains does not hold as it stands. Within judgment, the application of criteria comes down. What remains is the work of writing the criteria themselves, and the work of ranking which option is better.

What lies beyond the shift from maker to editor is not the editor. It is the author of the criteria. And the work of the author of the criteria is not only writing criteria down into the shape Jev is good at. It is also continuing to handle what cannot be written down, without writing it down.

Unverified Items

  • [primary source unverified] The 71% figure (217 respondents across 43 countries) from Rivera & Russi (2026): as of From Maker to Editor: A Structural Analysis of the Designer Role Transition in the Age of AI, the paper URL could not be reached.
  • [primary source unverified] Creative Boom (2025), the source of 86% usage against 10% approval, is an online self-selected survey with n=882, and 90% Adoption x 10% Approval — The Paradox of AI Tool Diffusion and Evaluative Divergence notes the publisher-side bias explicitly.
  • [primary source unverified] The case count behind Jev’s own eval is not shown on the dashboard, and the “711 examples” in reporting cannot be confirmed in a primary source (TypeSafe AI's Jev: What It Guarantees and What It Does Not).
  • [primary source unverified] The formula for confidence. Hume’s reconstruction is flagged by its author as speculative.
  • [primary source unverified] Applying the EU AI Act here is inference; no primary source from a public authority naming Jev or TypeSafe AI could be confirmed.
  • [unconfirmed] Whether selection by cheap, fixed criteria pushes the homogenization of output further is placed in this note as conjecture. No measurement targeting Jev exists.
  • [unconfirmed] There is no measurement of what the 6.3-point accuracy gap adds to cost per completed task.

References

All accessed 2026-09-21. The primary sources for the Jev specifications and figures this note cites are given in full in TypeSafe AI's Jev: What It Guarantees and What It Does Not.

Primary sources and independent testing on Jev

Sources on the design and learning side

Related notes: TypeSafe AI's Jev: What It Guarantees and What It Does Not, How to Use Jev, and Where Not To, From Maker to Editor: A Structural Analysis of the Designer Role Transition in the Age of AI, Recent Currents in LLM-as-a-Judge: A Literature Map of 53 Core Studies and a Standalone Chapter on Creativity Evaluation, Design Systems as AI's Foundation Layer — Structured Design Knowledge Determines Agent Accuracy, Choosing Between Frontier and Cheaper Models: Claude Fable 5 and GPT-5.6 Sol, AI Commoditization Resilience of Design Billing Models — Output-Based Billing Is Most Vulnerable, and Retainer Stickiness Does Not Mean High Margins, 90% Adoption x 10% Approval — The Paradox of AI Tool Diffusion and Evaluative Divergence, AX and Design: A Cross-Comparison of Academic and Industry Perspectives, Issues for Design Education (Not Craft Education): From the Evidence on Failure Design and Cognitive Offloading, AI Adaptation in Design Education — The Current State and Structural Challenges of Curriculum Reform, Does Early Use of Generative AI Inhibit the Formation of Thought? A Literature Map of Cognitive Offloading and Learning, Activating Reframing in Design: Cognitive Mechanisms and Practical Methods, The Creativity of Subtraction: The Additive Bias, and the Hypothesis of Designing Absence, Design and the Professions: A Literature Map of Design's Position Seen Through the Theory of Professions, AI Slop: Reading It as Outsourced Verification, Not Low Quality.

Author: Shuichiro Ogawa (Design Researcher / Consultant) About me →