Shuichiro Ogawa
日本語

Notes · updated 2026-09-18

TypeSafe AI's Jev: What It Guarantees and What It Does Not

TypeSafe AI released Jev on 2026-09-15. It generates no text; it returns typed judgments and probabilities.

Contents (10)
  1. What Goes In, What Comes Back
  2. The Accuracy Numbers the Vendor Published Against Itself
  3. Where Independent Tests Agree, and Where They Split
  4. What Is Guaranteed Is the Shape, Not the Judgment
  5. The Things the Official Docs List as Out of Reach
  6. Can Judgments Be Composed?
  7. Is Rewarding Calibration New?
  8. The Cost Picture Behind Carving Out This Layer
  9. Pricing and Contract Terms
  10. How to Read the Confidence Levels

What Goes In, What Comes Back

You pass in a state, and you get back answers to a set of questions. What comes back is not prose.

There are three question types. Choice picks one option from a set. You pass criteria as a map from option name to description. You get back choice (the highest-probability option), probabilities (the full distribution, summing to 1), and confidence. One question may carry up to 255 options. Score picks one level from an ordered scale. Criteria is an array of level descriptions, at least 2 and at most 10. The returned score is the probability-weighted mean of the level numbers, so it can land between levels. Noul returns a single value from 0 to 1 for whether a proposition is true. 0 is false, 1 is true, 0.5 is even odds. Noul carries no confidence value, and the official docs give this its own heading: “Noul does not return a separate confidence value.”

confidence appears only on Choice and Score. The official definition is “confidence is a statistic computed from the probability distribution the answer already gives you.” It is a statistic of the distribution, not a separately learned number. The docs suggest three bands: high for automated action, medium for proceeding with attention, low for not acting. The same docs call this a “convenient measure” and note that the thresholds are domain-dependent and not binding.

The API is a single endpoint, POST https://api.typesafe.ai/v1/systemone. The official documentation says all questions are evaluated in parallel, so adding questions does not add latency. Questions are independent of one another and do not contaminate each other’s context. The maximum number of questions per request is not disclosed. Dependent judgments are meant to be split into a second request, which the docs position as “the exception, not the rule.”

One naming point matters. The canonical name used consistently in the official documentation is Noul; the word Boolean does not appear there. Boolean is the name Vercel’s AI SDK gives to its evaluate API. Launch coverage carried the phrase “Boolean (aka Noul),” which conflates the SDK’s name with the vendor’s.

The Accuracy Numbers the Vendor Published Against Itself

TypeSafe publishes its own evaluation dashboard at evals.typesafe.ai. The numbers there largely settle where this model sits.

Averaged with equal weight over 4 workflows, Jev scores 67.8% accuracy at $0.0004 and 0.4 seconds per case. The best comparator, GPT-5.6 Sol, scores 74.1% at $0.0836 and 23.3 seconds. Claude Opus 5 is 73.1% ($0.1761, 37.8 seconds), Claude Sonnet 5 is 67.8% ($0.1174, 78.1 seconds), Claude Haiku 4.5 is 53.6%.

Jev loses on accuracy. It wins by two orders of magnitude on cost and one to two on latency. So Jev is not competing on accuracy.

The table has constraints on how it can be read. The reference label is “the average of answers from GPT-6 Astra and Claude Fable 5.1, both at high thinking,” which is not ground truth. Every comparator ran at its provider’s default reasoning setting, and only the Jev side was tuned. The average is unweighted across 4 workflows, so a different task mix moves the number. Case counts are not shown on the dashboard. (The “711 examples” that several outlets repeat cannot be confirmed in a primary source.)

The vendor itself lists these limits on the page. The reference is two models, so models from those families may be favored; the reasoning budgets are asymmetric; the labels are a consensus rather than a gold standard; and the harness runs on the assumption that “the code is correct.” Its own data also contains a counterexample to the claim that “every model is more accurate,” since on Agent Trace the prompt variant of DeepSeek v4 pro (72.1%) beats its workflow variant (71.6%).

Where Independent Tests Agree, and Where They Split

Several third-party checks are already out. All are small-sample, and the conclusions split.

Strong on classification tasks. In AbdelStark’s pre-registered pilot (300 examples, fixed seed, jev-1.13.0), Jev scored 0.910 against GLiNER2.5’s 0.700 on AG News, with a confidence interval that excludes zero, and 0.870 against 0.610 on Banking77/BTZSC.

Weak on ranking. In a 53-day news-triage test on Zenn, mean daily AUC was 0.653 for Jev, 0.697 for Sonnet 5, 0.676 for Haiku 4.5, and 0.637 for a position-based baseline. The Jev-minus-Sonnet gap of −0.044 has a confidence interval that excludes zero. The strength of this write-up is that its billing is measured rather than estimated: 3,408 requests cost $0.22 against a projected $0.218. Jev ran about 1.0 second and about $0.001 per day; Sonnet 5 ran 78 seconds and $0.167.

Loses on calibration. In the same AbdelStark pilot, DAIR Emotion came out 0.480 against 0.440, statistically indistinguishable, and the calibration scores were far worse: Brier 0.846 against 0.668, NLL 5.588 against 1.381. 16% of examples were assigned probability 0 on the true label. On the central selling point, returning judgments with probabilities, the independent test returns a negative result.

The speed and cost claims reproduce at the order-of-magnitude level. In Every’s test, Jev’s median was 0.35 seconds against 8.83 seconds for Claude Fable 5.1 at high effort, roughly a 25-fold gap. Cost ran about 1/580th (an estimate, not a measurement). It processed 777 judgments across 37 documents and 21 questions in under 0.7 seconds for about 0.25 cents. Another implementation ran a 370-rule diff check in under 2 seconds.

But the accuracy cost shows up. Every’s test used 12 passages: 6 clean and 6 with planted defects. Jev missed 1 of the 7 planted defects, and missed the same one on three consecutive runs. Fable 5.1 caught all 7. The authors write that “much more thorough accuracy check” is needed.

In a chess application, Jev landed at Elo 242.9, 59th on the leaderboard. It completed all 80 games under protocol, but scored 5 losses and 5 draws against Dragon, never reducing its 50% draw rate. This use wires legal moves into Choice’s 255-option limit, and the author limits how valid a benchmark it is.

What Is Guaranteed Is the Shape, Not the Judgment

TypeSafe’s landing page says “Zero Hallucinations.” That is not a measurement.

The launch post states it plainly: “Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots.” Schema match is guaranteed by construction, so 0% can go in the chart. The vendor states on the page that the number is not empirical.

The distinction matters and is easy to lose. A matched schema and a correct judgment are two different things. CEO Diogo Almeida wrote on Hacker News that “it’s also possible to be confidently wrong.” He also wrote, “Would you say a linear classifier hallucinates?”

The Register’s reporter put it as “TypeSafe claims that Jev is hallucination-free, which really isn’t a fair comparison as its output is not natural language,” and added “and that does not preclude the possibility of being incorrect.”

Anthony Maio’s phrasing is the shortest. “Jev constrains the shape of the output. It does not constrain the judgment.”

The Things the Official Docs List as Out of Reach

The official documentation has a page at model-jaggedness/jev-1.13. It is the vendor’s own list of known rough edges, and it states that jev-1.13 is not trained to generate text. Chaining Choice to force generation is “slow and ineffective,” and for text “there are other models for that.”

It cannot do arithmetic. “Jev is not a calculator,” and “jev-1.13 does not count reliably.” Error grows with the magnitude of the quantity. Score outputs must not be used to compute exact amounts.

It cannot read dates. “jev-1.13 reads dates as text, not as ordered quantities.” Ordering, distance and duration are all unreliable.

The underlying principle is one sentence. “Extraction is a judgment, so give it to the model. Arithmetic is not, so keep it in code.”

Input is text only. No images, audio or video. The primary training language is English.

State is not treated as hostile input. “State is data, and jev-1.13 does not treat it as hostile by default.” Injected instructions or misleading framing move the answers.

Nor is arithmetic consistency between questions guaranteed. The vendor supplies its own counterexample: on the same state, Noul returned 0.22 while Choice returned yes at 0.01 and no at 0.99. In another, refund came back 0.72 and not_refund 0.47, summing to 1.19. The docs say “don’t hold the model to arithmetic identities between separate questions.”

The context length is inconsistent inside the official documentation itself. models.md says “64k tokens per request” and “32k tokens for state plus the longest question,” while primitives.md says “around 32,000 tokens, roughly 150,000 characters of English text.” The official documentation does not settle which is right.

Can Judgments Be Composed?

Jev’s pitch is that a judgment becomes a component callable from software. The manifesto’s analogy is that “Intelligence today is like databases before SQL.”

Whether that analogy holds determines where the thing is usable in practice. Maio’s objection aims exactly there: “individually calibrated judgments do not automatically compose into a calibrated workflow.” Calibrated parts do not make a calibrated whole, and he gives the mechanism in one sentence: “Correlated mistakes survive composition.”

Archer Hume’s black-box audit of roughly 10,000 API calls supplies the mechanics. confidence was not a learned calibration but an arithmetic value measuring how far the top answer sat from a uniform distribution. Questions do not attend to each other: a secret string planted in a sibling question scored 0.00, and moved to state it scored 0.90 to 0.92. Adding an irrelevant fifth option lowered the log-odds of the existing binary everywhere, which should not happen if the options were scored independently. The ceiling was about 32,768 tokens per branch and about 65,536 per request. The author flags his reconstruction as speculative and notes that an attention mask cannot be uniquely identified.

Maio bounds the evaluation too: “the evaluation did not establish probability calibration, general intelligence, or production reliability.”

Is Rewarding Calibration New?

TypeSafe calls its training method RLCD, Reinforcement Learning for Calibrated Decisions. The official docs describe three output contracts: it does not generate text, it returns decisions and probabilities, and higher probability corresponds to higher accuracy. RLHF is contrasted as something that “can reward sycophancy and confidently hallucinated answers,” and RLVR as “strong on math and similar tasks but slow and expensive.”

The idea itself is not new. Work using a logarithmic scoring rule as the reward to train LLMs toward calibrated confidence was accepted at ICLR 2026 (arXiv:2503.02623, first version 2025-03-04). It reports ECE 0.0226 on TriviaQA. Maio writes that “Reinforcement learning for calibration is also not new,” and concludes that Jev’s strongest case is “architectural rather than algorithmic.”

The architecture is not disclosed. Parameter count, layers, training data and weights are all non-public. The CEO wrote on Hacker News that “architecture is close to the chest for now, but we have talked about writing a paper.” There is no document stating that these are withheld; they are simply absent.

Public leaderboards are also avoided. Saplin, who put it on a chess board, writes that “TypeSafe is openly anti-benchmaxxing,” describing a stated preference for dated internal snapshots over standard public scoreboards on the grounds that existing LLM benchmarks do not apply. That characterization could not be confirmed verbatim on a TypeSafe primary page.

The Cost Picture Behind Carving Out This Layer

Whether a judgment-only model has a market depends on how much of a problem judgment cost has become. The numbers for that background sit on the research-firm side.

Gartner predicts inference cost per agentic workflow will rise more than fivefold through 2028 (its Tokenomics Model, computed across 12 AI models). It puts advanced reasoning agents at up to 150x for a single task, and agents at 5x to 30x the tokens of an equivalent task. Deloitte predicts inference will account for roughly two-thirds of AI compute in 2026, with post-training at 30x the compute of base-model training and long thinking at over 100x simple inference (relayed from an Nvidia blog). A relayed McKinsey piece reports that about 60% of agentic task cost goes to verifying and refining responses.

Pressure in the other direction is running at the same time. Epoch AI’s measurements show inference price for a fixed level of performance falling between 9x and 900x per year depending on the milestone, and 40x per year at the GPT-4 level. A joint a16z and OpenRouter study covering over 100 trillion tokens across 13 months finds reasoning models taking more than 50% of tokens.

So the motive for carving out a judgment layer sits in the composition of spend, not its absolute size. And as of 2026-09-18 there is no analyst report covering Jev itself. Gartner, IDC, Forrester, Constellation, McKinsey, BCG, Deloitte and a16z all lack a mention. Nor is there a market size for “using LLMs only for judgment and classification,” though sizing exists for the adjacent LLM router and small-language-model markets.

Public bodies are the same. A primary source from a public authority, regulator or standards body naming Jev or TypeSafe AI came back empty across 19 domains checked for reachability. Nor is there a public document defining a category for “models that only decide and generate no text.”

Adjacent frameworks do exist. EU AI Act Art 3(1) lists “decisions” among the outputs of an AI system, so a decision-only model falls within the wording. The Act nonetheless treats models and systems separately (Recital 97), and the model-level obligations attach to providers of general-purpose AI models. The Commission’s FAQ gives two indicative criteria for GPAI status, training compute above 10^23 FLOP and the ability to generate language, images or video, and a decision-only model lacking generation fails one of them. In the United States, Colorado’s SB26-189 requires developers of automated decision-making technology that materially influences consequential decisions to supply technical documentation from 2027-01-01 (intended use, categories of training data, known limitations, guidance for human review), and California’s ADMT rules define technology that “substantially replaces” human decision-making. None of these names Jev, so applying them is inference.

The evaluation frameworks have not named this kind of model either. No public or standards-based criterion defining or measuring “decision models” or calibration was found. The nearest are NIST AI 800-2 (report statistics with their uncertainty), NIST AI 800-3 (separate benchmark accuracy from generalized accuracy and estimate uncertainty), and NIST AI 700-2’s ARIA pilot (measuring validity with CoRIx).

Pricing and Contract Terms

Input is $0.042 per million tokens and output is free. The documentation states outright that “Output tokens are free.” Official latency is 70ms to 500ms.

Rate limits are 250,000 tokens per second and 1,200 requests per minute. Both come with the qualifiers “adjusting dynamically” and “can change without notice.”

The model ID is jev-1.13.0 with jev-latest as the default alias. Aliases move with releases, so pinning the version is recommended once thresholds have been tuned.

The contract terms are thin. The Terms of Service disclaim everything under “AS IS” and “AS AVAILABLE,” cap cumulative liability at $100 USD, and specify JAMS arbitration, a class-action waiver, and Delaware law. There is no special clause covering wrong decisions. Zero Data Retention is stated to be available to enterprise customers, but the DPA body contains no ZDR provision. No SOC 2, no ISO 27001, and no SLA are disclosed. The offering at launch is early access.

Distribution is broad. Vercel’s AI Gateway (model ID typesafe-ai/jev, via AI SDK 7’s experimental_evaluate) and Netlify’s AI Gateway (no API key required) both supported it the same day, and skills for Claude Code and Codex ship alongside. The evaluations are set up so they can be re-run: the system-one-adapter-python repository publishes the LLM-side adapters.

How to Read the Confidence Levels

This note keeps three tiers of evidence separate.

  • T1v, vendor first-party: what TypeSafe AI states officially about its own model and API (documentation, launch post, manifesto, pricing, legal documents). Treated as the primary authority on what it does, how to use it, its constraints and its price. Numbers asserting competitive superiority without methodology inside those same documents were separated into marketing-claim entries in the corpus (193.6x faster / 444.6x cheaper, Zero Hallucinations, the Doom demo numbers, the landing page’s 0.114s against 8.566s). The Vercel and Netlify changelogs were treated as first-party announcements by the distribution partners.
  • T2, public bodies and research firms: since no analyst report names Jev, only adjacent-category facts were recorded. Gartner, Deloitte, Epoch AI, Stanford HAI, and the a16z with OpenRouter study are the core. VC sources (DCVC, a16z) were judged at least partial for investment interest, and only amounts, dates, and figures with disclosed methodology were taken. Public bodies and standards are the same: no source naming Jev turned up across 19 domains, so only adjacent regulation (the EU AI Act, Colorado’s SB26-189, California’s ADMT rules) and evaluation frameworks (NIST AI 800-2, 800-3, 700-2) were recorded, with any application to Jev treated as inference.
  • T3, individual opinion and independent testing: treated as the views and tests of verifiable technical authorities, never quoted as settled fact. Verifiable factual claims carry a 要一次検証 (primary source unverified) marker. Anonymous-handle posts, including Hacker News comments and reimplementations in anonymous repositories, were not used even where technically interesting. Every’s test is 12 passages, and the authors themselves write that a more thorough accuracy check is needed.

Where vendor claims and independent tests agree is on the order of magnitude for speed and cost. Where they do not is accuracy and calibration, and there the independent tests come out lower.

Related notes: Choosing Between Frontier and Cheaper Models: Claude Fable 5 and GPT-5.6 Sol, Recent Currents in LLM-as-a-Judge: A Literature Map of 53 Core Studies and a Standalone Chapter on Creativity Evaluation, Agentic Coding: The Current State of Orchestration Patterns (2026), Model Tiering Patterns for Fable.

Unverified Items

  • The 0.419-second WikiRace demo figure appears nowhere on typesafe.ai, docs.typesafe.ai, or evals.typesafe.ai. Media reporting only.
  • Context length conflicts between models.md (64k) and primitives.md (about 32k), and the official documentation does not settle which is correct.
  • The $40M raise, $200M valuation, and DCVC lead do not appear on TypeSafe’s own domain. The $40M and the DCVC lead were confirmed in DCVC’s own announcement, but the $200M valuation rests on secondary reporting; the original Forbes URL could not be located.
  • The case count behind the vendor’s own eval. It is not shown on the dashboard, and the “711 examples” figure in reporting cannot be confirmed in a primary source.
  • A verbatim TypeSafe statement that it deliberately avoids public leaderboards. The blog slug the CEO linked on Hacker News returned 404.
  • The formula for confidence. The Zenn reconstruction is a hypothesis matching 5 of 6 samples, and Archer Hume’s own reconstruction is flagged by its author as speculative.
  • The claim that the 193.6x / 444.6x figures are computed against the slowest and most expensive competitor is secondary only. The values in the vendor table are consistent with that level, but the computation itself is unconfirmed.
  • Near Here’s test (96%, 48/50, $0.043 per 1,000 judgments) could not be verified against a primary site, which returns 403.
  • Statements attributed to Nathan Flurry (co-founder of Rivet), including “jev is just a really smart switch lever,” could not be traced to the original X post.
  • Gartner’s Tokenomics Model cost tiers (basic at $0.05 and similar) have units that shift within the article and are implausible as stated. The press release itself returns 403.
  • McKinsey’s breakdown of agentic task cost (about 60% on verification and refinement, 70 to 75% on human oversight, 93% over budget) comes only from relay articles; the primary site timed out on 4 attempts, and neither sample nor question wording is disclosed.
  • SOC 2, ISO 27001, an SLA, and availability guarantees are unmentioned across all domains. Whether they do not exist or are simply unpublished cannot be determined.
  • The contractual status of Zero Data Retention. The docs’ legal page says it is for enterprise customers, but the word ZDR does not appear in the DPA body.
  • Parameter count, layer count, training data and weights are entirely undisclosed, and there is also no statement saying they are withheld, only absence.
  • The publication date, consultation deadline and body of the European Commission’s draft guidelines on high-risk classification (Art 6(5)). Only the existence and title of the print PDF were confirmed.
  • The 2027-01-01 compliance date for California’s ADMT rules. Only the §7001(e) definition could be confirmed here.
  • The publication date of the Stanford HAI AI Index 2026 and whether it addresses calibration. Only the landing page was checked, not the full PDF.
  • The bodies of ISO/IEC 42001:2023 and 22989:2022. Paid standards, so only title, issue date and page count were confirmed.
  • The three body points of the Personal Information Protection Commission’s advisory on generative AI services. The PDF is encrypted and cannot be extracted.

References

All accessed 2026-09-18.

Vendor first-party (T1v)

Public bodies and research firms (T2)

Independent testing and individual opinion (T3)

Author: Shuichiro Ogawa (Design Researcher / Consultant) About me →