Shuichiro Ogawa
日本語

Notes · updated 2026-09-21

How to Use Jev, and Where Not To

TypeSafe AI's Jev works only as far as a judgment can be written down as typed questions.

Contents (12)
  1. There Is No Best Practice Yet
  2. The Difficulty Is in Writing the Questions
  3. The Vendor Narrows the Target to Ten Task Types
  4. The Craft Is Concentrated in the Criteria
  5. Thresholds Come With a Starting Point and Nothing More
  6. What Was Kept and What Was Dropped
  7. Six Patterns Carry the Weight
  8. Four Implementations Are Public
  9. Adoption Turns on Whether the Criteria Can Be Written First
  10. Shapes to Avoid
  11. What the Outside World Imposes in Production
  12. Reading the Tiers

There Is No Best Practice Yet

Jev was released on 2026-09-15, and the sources here were retrieved six days later. In those six days not a single analyst report has named Jev. There is no mention in Gartner, IDC, Forrester, McKinsey, BCG, or Deloitte. The same holds for public bodies and standards organizations: zero mentions.

A handful of independent implementation records exist. Their samples run to 60 cases, 50 cases, 12 cases, and most authors limit their own claims explicitly. No industry consensus on practice has formed yet.

Without that consensus, only two kinds of material can be put here. What the vendor states verbatim, and what has been published in a form someone else can reproduce.

The Difficulty Is in Writing the Questions

The code that calls Jev is short. You pass state and questions, you get typed answers back, and there is only one API endpoint.

The difficulty sits upstream. The official agent-facing documentation puts it plainly: “Agents aren’t great at writing questions, so expect to edit collaboratively with them.”

Implementation write-ups stall at the same place. When a Japanese cancellation request (“please stop the auto-renewal from next month onward”) produced unstable intent classifications, the author concluded the cause was not model error but ambiguity in the question and the options.

The Vendor Narrows the Target to Ten Task Types

The use case map in the docs limits Jev to the following shapes.

  • Classification: “One known category should win”
  • Detection: “You need a probability that one property is present”
  • Scoring: “The answer belongs on an ordered rubric”
  • Routing: send an input to the right handler
  • Search / Retrieval / Ranking: narrow and order candidates
  • Verification: check another model’s output
  • ML Feature Extraction / Structured Data Extraction: turn judgments into features or structured fields

What is excluded is stated just as plainly. Extended reasoning requiring multi-step analysis, judgments with numerous interconnected factors packed into one question, and free-form text generation. On generation the docs say “Jev deliberately avoids this paradigm.”

The launch post lists the same range: AI workflows, conditional logic, map-reduce over large datasets, latency-critical real-time work, and verification and guardrailing of LLM output.

The Craft Is Concentrated in the Criteria

There are three question types, and each comes with its own instructions.

Choice takes the full option list. “Give the model the full list of teams, categories, or products rather than a shortlist.” Up to 255 options are allowed. Confusable pairs such as return_policy and return_status are separated using the what, not_for, and examples fields.

Score describes situations, not degrees. “Describe situations, not degrees.” Write “Broken or degraded feature, but workaround exists” instead of “moderately severe.” The reason is given: “Every level is evaluated separately… ‘worse than the previous level’ means nothing to it.” Each level is scored on its own, so a description written relative to the previous level carries no meaning. Keep one dimension per Score question, and use as many levels as you can describe distinctly, up to ten.

Noul asks exactly one yes/no question. Phrase it so that a high value means yes, and avoid negated forms such as “Is the message free of personal data?” Where the boundary is fuzzy, supply criteria describing both the true and the false side.

All three end with the same instruction: test against your own data and adjust.

One trap in option design is documented. Without an explicit “none of these” option, out-of-scope inputs get pushed onto one of the existing options. Not generating anything outside the candidate set is a different property from judging correctly.

Thresholds Come With a Starting Point and Nothing More

Confidence attaches only to Choice and Score. By the official definition it is a statistic computed from the shape of the probability distribution the answer already provides.

The starting point is written out in the docs as numbers. Below 0.6, route to a person; between 0.6 and 0.85, ask the user to confirm; above 0.85, act automatically. A classification-granularity example uses 0.9, and an intent-routing example escalates below 0.5.

The same page declines to bind those numbers. “Start with conservative thresholds, test with your own data, and adjust as you observe results.” Confidence itself is called a “convenient measure,” and the docs say you may switch to a different measure if it serves you better.

The vendor also warns against attaching thresholds everywhere. When the task is simply to pick the best option, take the highest confidence rather than applying a threshold, and reserve thresholds for specific statistical algorithms. Questions and threshold constants should live in one place so that a human can review them.

Practitioners have published their own rules. One cuts off below 0.05 (keeping at least one candidate), concentrates on a single candidate above 0.8, and reads below 0.5 as a split judgment. In a 60-case hand-labeled benchmark on operation risk, all 40 cases returning confidence 1.000 were correct, and every incorrect answer came in below 1.000. On the ambiguous subset of that same benchmark, accuracy fell to 71.4%.

What Was Kept and What Was Dropped

The wiring listed below was selected on four conditions.

  • The vendor attaches a reason or a precondition verbatim: items marked only as “recommended” were not kept. Fan-out was kept because the docs state the condition: “Speculative questions only provide value when the overhead of including them is negligible compared to savings from avoiding follow-up requests.”
  • The number comes with a methodology: figures missing the model version, the number of questions, the size of the state, or the comparison baseline do not appear in the body. The 12.2x batching figure was usable because all of it was disclosed: jev-1.12, thirteen questions, 54,000 characters, compared against sequential sending.
  • It is published in a reproducible form: only implementations publishing code, a benchmark procedure, or measured billing were used as examples.
  • Failures and limits are reported in the same material: records that close on success alone were not kept on their own.

What was dropped is worth naming too.

  • “193.6x faster,” “444.6x cheaper,” “mathematically cannot hallucinate”: the comparison baseline and measurement conditions are undisclosed, and the author qualifies some of the numbers as “potentially inflated.” These are separated in the corpus as marketing-claim and kept out of the body.
  • Cost reduction rates for routing and cascading: figures above 85% and up to 98% exist, but they come from academic papers. No industry source with a disclosed methodology was found, so they cannot serve as a practical selection criterion.
  • Implementation repositories whose authors could not be verified: one self-described router repository returned 404, and the organization behind another could not be identified. A gist with no execution record was dropped for the same reason.
  • Vendor blog claims of “50 to 90% cost reduction”: self-reported by companies selling guardrail or evaluation tooling, with no sample or period given. Fact and promotion cannot be separated, so these were excluded.

Six Patterns Carry the Weight

The wiring that recurs across both official docs and independent implementations reduces to six shapes.

Speculative fan-out. Send every question that could matter in one call, ahead of the branch, and let your code read only the answers it needs. Questions are evaluated in parallel, so adding more barely moves latency. The docs attach a condition: “Speculative questions only provide value when the overhead of including them is negligible compared to savings from avoiding follow-up requests.”

Batching. Bundle multiple questions against the same state. The official cookbook reports 12.2x cheaper and 10.0x faster with the methodology disclosed: jev-1.12, thirteen questions (eight Noul, two Choice, three Score), five repetitions, with a roughly 54,000-character Wikipedia article on GDPR as the state. The comparison is against sequential sending, so the speed advantage narrows against concurrent single calls.

Confidence-gated routing. “The answer tells you what; confidence tells you whether to act.” The answer decides what; confidence decides whether you are allowed to act on it.

Composite scoring. Split a complex judgment into independent Score dimensions, normalize, weight, and combine in code. Because the weights sit outside the model, they can be retuned when results come back wrong.

Cascade. Run the cheap path first, have Jev verify field by field, and call the expensive reasoning model only when a flag crosses the threshold. The official cookbook uses gpt-5.4-mini for extraction, jev-1.12 for verification, and gpt-5.5 for re-runs, with the any_flag threshold at 0.7. The design reason given there is what matters: “we do not use structured outputs, tool calls, or json mode, because: a schema following mistake is not the mistake we expect an LLM to make.” The failure you expect from an LLM is not a schema violation, so a mechanism that enforces the schema does not verify anything.

Guardrails on both sides. “Run this TypeSafe check both on LLM inputs, and on LLM outputs, because even ordinary-looking prompts can lead to harmful generated replies.”

Four Implementations Are Public

Replacing an evaluator. Langfuse ran user-disagreement detection on LLM output as a single Noul question. It matched Claude Fable 5.1’s verdict 91.5% of the time at $160 per million graded answers. In the same comparison Claude Fable 5.1 cost $33,000, GPT-5.6 Luna $400, and DeepSeek V4.1 Flash $260 at 93.5% agreement. They ran a 0.7 threshold and collapsed consecutive generations sharing the same latest user message into one turn. The same post names two limits: accuracy degrading with longer context (“Jev suffers from context rot”), and the inability to abstain. “It cannot abstain. A forced binary with no unknown or needs_review option makes Jev pick the least wrong answer.”

Selecting functions for an agent. The maintainer of typia applied Jev to picking, per turn, which of potentially hundreds of controller functions are needed. A single Choice was not used. “One turn may need several functions, and choice picks exactly one.” Because a turn may require several functions, the design places an independent Noul per candidate. Where there are too many functions, a two-stage selection (group first, then individual) caps the turn at two requests. The same author also recorded the failure. A custom fetch client, an SDK wrapper, and conditional types had grown to roughly 550 lines plus 1,000 lines of tests before being cut back to a single boolean-to-noul conversion. “The package’s whole reason to exist is one conversion.”

Judging agent tool-call risk. A benchmark classifying 60 cases as readonly, destructive, privileged, or exfiltration reported 91.7% accuracy, p50 of 421.6ms, p95 of 542.0ms, and about $0.0000173 per call. The author notes that the frontier-model comparison column was never run because they had no API key.

Checking rules in bulk. One implementation ran 370 rules written in Markdown against a diff in under two seconds. What actually broke was max_tokens_exceeded, caused not by the number of rules but by the size of the diff.

Adoption Turns on Whether the Criteria Can Be Written First

Compared against structured output from a general model, two write-ups point the same way. Matching the type is not the same as judging correctly, but on the Jev side a probability comes attached. In a weather example, “was a unit specified” came back as 0.02, while the general model gave no way to tell whether a value had been supplied or filled in. The stated motivation was to avoid two problems: values appearing outside the option set, and inference taking tens of seconds per call.

No measured comparison against small classifiers or rule-based systems turned up in this search. There is one comparison against embedding retrieval. In typia’s design notes, an extra LLM call and an embedding search (which requires index maintenance and handles multi-intent poorly) were both rejected in favor of Jev.

Industry data supplies exactly one axis for selection. The rate at which the price of reaching a fixed performance level falls ranges from 9x to 900x per year depending on the benchmark. In real usage by task category, cost spans roughly a thousandfold, from $0.036 to $34.965 per million tokens. Cheapness is not uniform; it is decided by which task you point it at.

Applying the criterion this blog already uses, the decision comes down to cost per completed task rather than cost per request. Jev being two orders of magnitude cheaper per call only pays where the rework from a wrong pass is small.

The conditions where it pays come out the same shape from both the vendor and the independent implementations. The criteria can be written out as text in advance, the judgment decomposes into a single dimension, the rework from an error is small, and the volume is high.

Shapes to Avoid

Collected from what the vendor lists itself and what surfaces in implementation records.

  • Handing it arithmetic or counting: “Jev is not a calculator.” Keep the math in code.
  • Handing it date ordering or durations: “Jev reads dates as text, not as ordered quantities.” Extract the components with Choice and compute in code.
  • Packing multi-step reasoning into one question: reduce the hops and pass the relevant state directly.
  • Asking it to generate: chaining Choice to force text out is called “slow and ineffective.”
  • Letting the state swell with irrelevant content: “Include only the context relevant to the current questions.” Degradation with longer context is reported independently too.
  • Omitting an abstain option: a forced binary without unknown or needs_review produces the least wrong answer.
  • Omitting a “none of these” option: out-of-scope inputs land on an existing candidate.
  • Expecting arithmetic consistency across questions: “P(noul) ≠ 1 - P(not_noul).”
  • Attaching a threshold to every routing decision: when you only need the best option, take the highest confidence.
  • Writing a thick integration layer: a custom client and wrapper can inflate a single-conversion responsibility into hundreds of lines.
  • Assuming free output means cheap: input tokens are billed. One measurement recorded about $0.0015 for 48 short Japanese sentences.

What the Outside World Imposes in Production

No public-sector document naming Jev turned up in this search either. Every application below is inference, but the norms governing automated judgment in operations already exist.

EU AI Act Art 14(4) requires that the overseeing person be able to disregard, override, or reverse the output, and to intervene and stop the system. Art 13(3)(b) requires accuracy metrics, known risks, and oversight measures in the instructions for use, and Art 15 makes declaring the accuracy metrics an obligation in itself. Art 26 obliges deployers to retain logs for at least six months and to inform affected persons.

There is a matching norm for thresholds. NIST AI 600-1 action GV-1.3-002 calls for establishing minimum thresholds for performance or assurance criteria and reviewing them as part of deployment approval. The same document notes that confidently presented erroneous content warrants particular monitoring when integrated into consequential decision making.

Where decisions go against a person, notice and appeal are required. OMB M-24-10 requires notifying negatively affected individuals with information on their right to appeal, maintaining human consideration and remedy processes, and where practicable maintaining an opt-out to a human alternative. California’s ADMT regulations remove the opt-out duty where a business provides a way to appeal to a human reviewer with authority to overturn the decision, and require that the reviewer understand how to interpret the output.

There are prohibitions as well. EU AI Act Art 5(1)(d) bans predicting criminal risk based solely on profiling. Uses falling under the eight Annex III domains cannot rely on the lightweight exemption as long as profiling is involved.

Reading the Tiers

This note keeps three tiers apart.

  • T1v vendor primary: the patterns, primitives, and cookbooks on docs.typesafe.ai, the launch post, and the official Vercel and Netlify documentation. Treated as the primary authority on usage and constraints. Claims inside the same material (“40x-200x faster,” “444.6x cheaper,” “mathematically cannot hallucinate,” “up-and-left of every single model”) are separated in the corpus as marketing-claim.
  • T2 public bodies and research: no analyst report names Jev six days after launch, so only adjacent facts are recorded. No industry source with disclosed methodology quantifies the cost effect of routing or cascading; the figures that exist sit in the academic literature.
  • T3 individual opinion and independent implementation: authority is confirmed for the typia maintainer and the Langfuse engineer. The dev.to, Qiita, and Zenn implementation posts carry real names and real artifacts but no externally verifiable track record, so they are treated as medium-or-lower confidence and read as single records.

Vendor and independent sources agree on the order of magnitude for speed and cost, and on question design deciding the outcome. They diverge on accuracy for ambiguous cases, where the independent numbers come in lower.

Related notes: TypeSafe AI's Jev: What It Guarantees and What It Does Not, When Judgment Becomes a Function Call, What Moves in Design, Choosing Between Frontier and Cheaper Models: Claude Fable 5 and GPT-5.6 Sol, Recent Currents in LLM-as-a-Judge: A Literature Map of 53 Core Studies and a Standalone Chapter on Creativity Evaluation, Agentic Coding: The Current State of Orchestration Patterns (2026).

Unverified Items

  • 要一次検証 https://typesafe.ai/pricing returns 404. Pricing was confirmed from the launch post instead.
  • 要一次検証 The official Python SDK README was not reached directly.
  • 要一次検証 Fourteen cookbook pages (consistency_noul, rerank, semantic_find, function_calling and others) were confirmed to exist but not retrieved. They likely contain further usage examples.
  • 要一次検証 Only the model-jaggedness page (2026-09-17) carries a last-reviewed date; other docs pages are recorded as of the retrieval date.
  • 要一次検証 The 12.2x and 10.0x batching figures are the vendor’s own measurement on jev-1.12, compared against sequential sending rather than concurrent single calls.
  • 要一次検証 The result that all 40 confidence-1.000 cases were correct comes from a single 60-case benchmark whose author’s track record could not be externally verified.
  • 要一次検証 Langfuse’s agreement and cost figures were published in the context of its own evals product.
  • 要一次検証 EU AI Act articles were confirmed through a third-party transcription site rather than EUR-Lex. Colorado’s statute returned 403, and the CPPA regulation PDF and the Japanese AI business guidelines PDF could not be text-extracted.
  • 要一次検証 Gartner’s press release and Tokenomics Model source documents are 403 or subscription-only, and the units shift within the relaying article.
  • 要確認 No industry source with disclosed methodology quantifies the cost effect of routing or cascading. The available figures (RouteLLM above 85%, FrugalGPT up to 98%) sit in academic literature and are out of scope here.
  • 要確認 No measured comparison of Jev against small classifiers or rule-based systems was found.

References

All accessed 2026-09-21. Primary sources for the specification and accuracy figures are listed in full in TypeSafe AI's Jev: What It Guarantees and What It Does Not.

Vendor primary (T1v)

Public bodies and research (T2)

Independent implementation and individual opinion (T3)

Author: Shuichiro Ogawa (Design Researcher / Consultant) About me →