Notes · updated 2026-11-21 · Markdown
The Structure of a Claude Code Environment with a Jev Judgment in Every Exchange
This note reads the structure of a Claude Code environment described in an article on ktlog.net from the perspective of the survey of Jev usage. In the article, a request proceeds from the entry points through the harness to four destinations.
Contents (9)
- Scope of This Note and Analytical Framework
- The Overall Environment in the Article
- A Structure That Separates Judgments from Decisions
- Pre-Execution Judgments and Fallback on Failure
- Designing the Judgment That Selects from 147 Skills
- Writing the Judgments That Ask About Quality and Completion
- Independence Where Several Judgments Are Stacked
- Managing Thresholds, Model Versions, and Cost
- Summary of the Structure and Comparisons to Measure Its Benefit
Scope of This Note and Analytical Framework
This note examines the structure described in an article that ktlog.net published on November 8, 2026, “Claude Codeのおすすめ環境を全体図で解説。ハーネス・スキル・MCPの仕組みと実例” [A recommended Claude Code environment explained with an overall diagram: how the harness, skills, and MCP work, with examples] (retrieved November 21, 2026). The note compares the structure of the article with the findings of the survey of Jev usage (Using Jev (TypeSafe AI): Current Practice and Open Problems (November 2026)). The object of discussion is the design of the structure, and the note does not evaluate the author or the publication.
What the article reveals is limited to the structure. The material for this note does not include the question texts or the criteria passed to Jev. Nor does it include the thresholds or how they were set. The number of misjudgments and the behavior when Jev cannot be reached are also unknown. The version of the model being called is not in the material for this note either. Therefore, what this note can check is which parts of the structure correspond to the recommendations of the survey. In addition, the note sets out what needs to be checked when building a similar structure. How much effect the structure has in actual operation is outside the scope of this note.
The following sections trace the flow from judgment to execution, point by point. At each point, the note asks the following three questions.
- What does the judgment look at?
- Who, or which code, decides whether to proceed?
- How does the process fall back when a judgment is wrong or a failure occurs?
Following these three questions, the note proceeds from the overall structure to the individual judgment points.
The Overall Environment in the Article
In the structure of the article, a request flows from top to bottom through the following six layers.
- Entry points
- Command center (Claude Code)
- Harness
- Routing (four destinations)
- Workers and tools (subagents)
- Exit
Knowledge is placed beneath all of these layers as their foundation.
As of November 2026, the command center is Claude Opus 5.5, and the reasoning setting is xhigh. Under the command center are 147 skills and 150 subagents. Seventeen plugins are also installed.
The article lists the following four entry points for requests.
- Smartphone app (remote control)
- Terminal
- Discord
- Scheduled tasks
Besides these, Grok Bot serves as an instruction route to Claude Code running inside a cloud PC. There is also a self-built Discord Bot, which the article states was stopped as of November 5.
The harness is positioned as a mechanism that requests always pass through before work begins. It contains judgment and permissions, as well as safety settings. The harness consists of the following five components.
- Judgment AI
- Hooks
- Permission rules (settings.json)
- API key file
- CLAUDE.md
The judgment AI is Jev from TypeSafe AI. The article states that Jev is inserted into every exchange, and the exchanges covered include those between humans and AI and those between AI agents. Exchanges with subagents are also covered. According to the article, a single judgment takes about 0.2 to 0.35 seconds. Image judgments are handled separately, with OpenAI’s Decisions API.
The four routing destinations are as follows.
- Other AIs (Codex for images, Antigravity for video and Workspace, Codex and Antigravity for review)
- Skills
- MCP
- External APIs
Which destination to use is decided by Claude Code, the command center.
As examples, the article presents the workflows for a blog article, an advertising video, and an invoice. At the end of the work, a review is included as a general mechanism. Only items with no remaining review comments proceed. If there are comments, the item goes back to a subagent and is corrected.
A Structure That Separates Judgments from Decisions
First, this section examines the harness and the examples through the question of who, or which code, decides whether to proceed.
Control over irreversible operations is placed on the side of fixed rules. In the permission rules in settings.json, the default is auto. However, specific dangerous operations are sent for confirmation. The article lists the following three operations as subject to confirmation.
- sudo
- rm -rf
- git push —force
Deletion in Chatwork is prohibited. There are 11 hooks in total, and bash-guard and Jev are placed before execution. Deletion of the contents of a specific folder is forcibly stopped. Another hook, for writing requests, adds an instruction to read the style knowledge and starts Jev. There is also a hook that shows a notice suggesting registration in Obsidian after the work.
A single file holds 27 API keys. Its contents are 1Password secret references. The keys are loaded through a script at run time, and Claude Code does not open the file.
The article explains the difference in roles between settings.json and CLAUDE.md. The settings.json file holds rules that define the boundary between what can and cannot be done. In contrast, CLAUDE.md is described as holding “request-based” rules. CLAUDE.md lists the following five items.
- Polish the Japanese
- Take notes with the Obsidian CLI
- Make judgments with Jev
- Do not use AI-looking designs
- Always review the work once it is complete
Jev sits between settings.json and CLAUDE.md. Jev does not directly enforce by rule whether a process may run, as settings.json does. At the same time, Jev differs from the requests in CLAUDE.md, whose observance is left to the model that reads them. Jev returns an answer of a fixed type with probabilities. TypeSafe states the slogan “Jev makes judgments, code makes decisions” in its blog post “Don’t Call It a Decision Model” of November 3, 2026. The structure that stops irreversible operations on the side of settings.json and the hooks, while layering Jev’s judgments on top, corresponds to the recommendations of the survey.
The examples also include points of human checking. In the invoice example, the flow proceeds from the hooks to skill selection, MCP is used to look up the client, and a confirmation table is created. At that stage, a step in which a human checks the amount is included. The article states a policy that a human should also judge as much as possible, because the work involves money. After that, the invoice and the PDF are created, and the flow reaches the report after Jev’s final judgment. In the blog article example as well, Jev judges the draft, the draft is checked against the source material with EmDash, and a human finally looks at it before publication. The invoice workflow has a point of human checking. However, the description of the structure alone does not allow a firm conclusion about where the final decision is placed.
Separating image judgments into a different API is another part of this division of roles. OpenAI’s Decisions API checks the following three items.
- Garbled text
- Personal information in screenshots (mosaicked if present)
- Overlapping elements in figures
Jev 1.14 added image input as an experiment. However, TypeSafe itself writes in its blog (jev-1-14) that this input is not at a level it recommends for production use. In the comparison that Willison summarized in his November 11, 2026 article, Jev handles only text and JSON. The same comparison describes OpenAI’s decision model as supporting both text and images. The structure that separates image judgments into OpenAI’s Decisions API and keeps Jev on the text side corresponds to the recommendations of the survey.
Pre-Execution Judgments and Fallback on Failure
Next, this section looks at the point before execution. Here, the issues are what the judgment looks at and how the process falls back when a judgment is wrong or a failure occurs.
The pre-execution hooks contain bash-guard and Jev. The article states that whether an operation is reversible is judged before execution.
What we would want to check here is what goes into the state for the pre-execution judgment. If the design puts the string of the command about to run directly into the state, caution is needed. Issue #15 of typesafe-sdk-js, dated September 23, 2026, reports that when the state contains a curl command, even a legitimate request is stopped on the Cloudflare side with an HTML 403. Because this response is an HTML 403, it looks on the surface like an authentication failure. We would want to confirm whether the command string goes into the pre-execution state and then decide in advance how to isolate the cause when a 403 is returned.
Because the structure uses an external API, its behavior during outages is also part of the design. The article states that Jev is inserted into every exchange. If the design has every exchange wait synchronously for Jev’s response, the effect of an outage could extend to all exchanges. Indeed, TypeSafe’s status page records increased latency that lasted about 45 minutes in us-east-1 on October 14. It also records an overload of about 1 hour and 15 minutes on October 30. Through Vercel’s AI Gateway, there was also an incident in which some teams kept receiving 429 responses from around September 26.
Whether to stop the process or let it through when Jev cannot be reached, or when a 403 or 429 is returned, needs to be decided on the code side. In the Spring AI implementation that Tzolov reported on October 2, 2026, the Jev component fails open. In this implementation, when Jev cannot be reached, the process passes without being checked, and only a warning log remains.
The preferable fallback differs by point, even within the same environment. For example, for the pre-execution judgment of whether an operation is reversible, one possible design chooses to stop (fail-closed) when Jev cannot be reached. In contrast, for the judgment of the request type, one possible split returns to the selection made by Claude Code, the command center, when Jev cannot be reached. We would want to decide, as branches in code, at which points to stop the process and at which points to return to another path.
Even when responses return normally, the losses from misjudgments are not symmetric. TypeSafe states in its docs (Noul), “Raise it when acting on a false yes is expensive, such as paging someone or issuing a refund.” The same page recommends framing the question so that a high value means yes. The loss is large when an irreversible operation is mistakenly judged reversible. We would want to check whether the threshold for the pre-execution judgment is set at a level where false yes answers are unlikely.
Designing the Judgment That Selects from 147 Skills
The next point is where a request arrives. Here, the note examines what the judgment looks at to select a skill and how to handle the conditions under which wrong selections are likely.
The structure of the article adopts a policy of creating a skill for each small task and combining them. As of November 2026, the number of skills reaches 147. A subagent is also prepared for each skill. For example, the video skill has a planner subagent, a caption subagent, and a quality-check subagent.
Skills are also divided by project. The article gives an example in which the slide skill is split into one for Company A and one for Company B. Their contents overlap, but the separation keeps the designs and copyrights from overlapping.
When Choice is used for skill selection, the upper limit is 255 options. TypeSafe states in its docs (Choice), “give the model the full list of teams, categories, or products rather than a shortlist.” The recommended policy is to pass the candidates as a full list without narrowing them down. The number 147 fits within the upper limit of 255. However, the description of the structure does not show whether all candidates are passed or a narrowed set is passed, so we would want to check how candidates are passed at design time.
When there are many candidates, bias toward the order of options becomes a problem. TypeSafe’s docs (model jaggedness) state that Jev’s output tends to lean toward the first option. As a countermeasure, the docs recommend reordering the options and checking whether the answer changes. In the evaluation by Song et al. (arXiv 2610.35826), the change in mean accuracy from reordering was less than 2 points. However, the set of problems answered wrongly under some ordering expanded by 27.3% to 28.5% compared with the original ordering. Looking only at mean accuracy may miss wrong selections that occur under specific orderings.
For pairs whose contents overlap, such as the slide skills separated by project, a way of writing that reduces wrong selections is needed. The practices summarized in September (How to Use Jev, and Where Not To) set out a procedure for distinguishing a pair of confusable options with the following three fields.
whatnot_forexamples
To reduce wrong selections while keeping the purpose of separating skills by project, we would want to state the boundary of each pair explicitly with these three fields. A procedure is then needed to test whether the answer stays the same when the options are reordered.
According to the article, at the point where a request arrives, the ambiguity of the request is judged in addition to its type.
TypeSafe states in its docs (Choice), “A none of the above option may not work well in many use cases.”
Instead of placing a “none of the above” option in the list, the docs recommend setting up a separate companion Noul.
The structure that judges the ambiguity of the request separately corresponds to this recommendation.
However, the material does not yet include an evaluation that directly compares the form with a “none of the above” option and the form split into a companion Noul.
Writing the Judgments That Ask About Quality and Completion
At the point before the work is closed and the point of choosing among proposals, the issues are what the judgment looks at and how the pass conditions are written.
The article states that, before the work is closed, a judgment checks whether the work is being marked complete without verification. When proposals are compared, which proposal is better is also judged. The article lists the following four further targets of judgment (our translations).
- “How different is this from the knowledge?”
- “How can the Japanese phrasing be evaluated?”
- “Is the initial plan all right?”
- “Is the final deliverable at 100% quality?”
Here, the phrase “Is the final deliverable at 100% quality?” cannot be taken as the question text actually passed to the API. The wording of the article was written as a description of the structure. The actual question text is not in the material for this note. What we would want to consider in implementation is how, in judgments that ask about quality, the question text should correspond to the pass conditions.
Jev returns its answers as probabilities and scores. TypeSafe defines confidence in its docs (Confidence): “Confidence measures how concentrated the answer’s probabilities are, which is different from the probability that the selected answer is correct.” A high confidence is the concentration of the answer’s probabilities, not the probability that the selected answer is correct. We would want to avoid a design that takes a high value alone as grounds for passing.
On how scores are assigned, TypeSafe’s docs (Score) state, “Describe situations, not degrees.” The recommended policy is to write, as a sentence describing a situation, which shortcomings must be absent for an output to pass, rather than the degree to which something was achieved.
Writing the question itself also has specific constraints. The Literal reading item in TypeSafe’s docs (model jaggedness) states, “Jev answers the question you wrote, not the one you meant.” The Indirection item of the same docs states that accuracy drops for indirect questions that require several steps of reasoning, such as asking about a property of a property. The docs therefore recommend writing questions as directly as possible.
Following the recommendations in TypeSafe’s docs, we would want to write comparison and quality-check questions in a form that states the target and the criterion directly. For example, consider the check “How different is this from the knowledge?” A way of writing that fits the recommendations of the docs is to put the two documents being compared into the state and then split the check into direct questions, one for each check item. In the blog article example, Jev judges the draft, the draft is checked against the source material with EmDash, and the flow returns to the previous step if a problem is found. The more concretely the material to be checked and the conditions for returning are defined at a point, the easier it is to write the judgment at that point as a direct question based on situations.
Independence Where Several Judgments Are Stacked
At points where several checks follow one another on the same output, this section examines whether a later check can stop an error made by an earlier judgment.
In the advertising video example in the article, several checks are lined up within a process that takes 20 to 30 minutes per video. First, Jev judges how to proceed. Next, each one-second element is turned into a picture and checked as a still image, and the work is redone if inspection finds a mismatch. The finished video is also shown to Gemini. Finally, Jev makes the final judgment.
The evaluation by Rao and Callison-Burch (arXiv 2609.29769) bears on how stacked judgments of a similar type behave. The authors examined combinations of Jev and LLM judges on nine evaluation panels that perform rubric-based scoring. For the wrong answers on which Jev’s confidence was highest, 96.0% of the LLM judges’ judgments gave the same answer as Jev. If the errors were independent, this proportion would be 50.3%. Even with an ideal oracle threshold, the accuracy gain from a cascade was at most 2.7 points over the strongest single judge. The authors summarize this result as “a cheap substitute for a flash-tier LLM judge but a poor complement to one.”
However, the range to which this evaluation directly applies is limited. The targets of the evaluation by Rao and Callison-Burch are nine evaluation panels that perform rubric-based scoring. Whether the same thing happens in video inspection has not been confirmed.
With that premise in place, and in light of the rubric-scoring results, we would want to keep in mind the possibility that stacking judgments of a similar type in succession tends not to produce independent checks. On the other hand, the structure of the article also incorporates checks that use different means. The check with still images for each second in the advertising video example is one such check. Image judgment with OpenAI’s Decisions API differs in means from text judgment with a language model. The same applies to the check against the source material with EmDash in the blog article example. When adding a later check, we would want to confirm that it looks at a different target or uses a different means from the earlier judgment.
Managing Thresholds, Model Versions, and Cost
Finally, this section examines thresholds and model versions, as well as the cost structure, in connecting judgment results to decisions on the code side.
The article gives about 0.2 to 0.35 seconds as the time a Jev judgment takes. While response speed is visible within the structure, the way thresholds are set and the way model versions are handled require separate attention.
Li et al. (arXiv 2609.26550) report that thresholds must be set on the target task. The authors present a procedure for choosing such a threshold with about 100 labels. According to the report by Brom (arXiv 2610.36311), thresholds chosen by point estimation fell below the target accuracy in 147 of 190 evaluations. Therefore, thresholds need to be checked for each judgment with the data of that judgment. Setting a threshold once by point estimation alone and stopping there may result in accuracy below the target.
Pinning the model version is also inseparable from threshold management.
On October 14, 2026, jev-1.14.0 was released, and jev-latest came to point to 1.14.0.
If jev-latest continues to be called after a threshold has been chosen with the data of each judgment, the chosen threshold may no longer fit when the version changes.
A procedure is needed that calls Jev with an ID containing the version number and chooses the threshold again whenever the version is changed.
On the cost side, we would want to understand the billing unit. TypeSafe’s docs (Models) state that billing is based on input tokens and that output is free. If the design inserts Jev into every exchange, cost increases with the product of the state length per call and the number of calls. At points where a long context goes into the state, such as pre-execution checks and draft judgments, input tokens tend to increase.
The official materials also describe a way to bundle calls. TypeSafe’s cookbook (parallel_questions) includes the provider’s own measurement that bundling questions about the same state into one call is “12.2x cheaper.” For example, if the type and the ambiguity of a request are both asked about the same state when the request arrives, there is room to bundle them into a single call. However, the figure “12.2x cheaper” is the provider’s measurement, and we would want to measure the actual reduction in the environment at hand.
Summary of the Structure and Comparisons to Measure Its Benefit
Taken together, the points in the structure of the article that correspond to the recommendations of the survey are the following four.
- Placing irreversible operations on the side of fixed rules, namely settings.json and the hooks, separately from Jev’s judgments
- Having points of human checking in the invoice and blog article workflows
- Separating image judgments into OpenAI’s Decisions API and keeping Jev on the text side
- Having a structure that judges ambiguity separately from the type of request
In contrast, the points we would want to check when building a similar structure are the following six.
- The fallback for each judgment, whether to stop or let the process through, when Jev cannot be reached or a 403 or 429 is returned
- Whether command strings such as curl go into the pre-execution state
- Threshold verification with the data of each judgment, and pinning of the model version
- Distinguishing confusable pairs among the 147 skills, and verifying them by reordering
- The correspondence between situation-based questions and pass conditions in quality and completion judgments
- Confirming independence at points where judgments of a similar type follow one another
Whether the structure corresponds to the recommendations of the survey does not by itself reveal the actual benefit. To measure the benefit of this structure, the following three conditions need to be compared on the same set of tasks.
- Points with Jev and points without Jev
- Processing with fixed rules only
- Adding a human check
The measures for comparing these conditions are the following four.
- Errors
- Rework
- Waiting time
- Cost
For example, at the pre-execution point, a configuration with only settings.json and bash-guard is compared with a configuration that adds Jev. The comparison then looks at how much waiting time and cost increase relative to the reduction in errors. At the pre-completion and inspection points, the comparison covers the case with Jev’s judgment, the case without it, and the case routed to a human check. By measuring the change in rework, together with the balance of waiting time and cost, on the same measures, one can judge at which points placing Jev is worthwhile.
Related notes: Using Jev (TypeSafe AI): Current Practice and Open Problems (November 2026), How to Use Jev, and Where Not To, TypeSafe AI's Jev: What It Guarantees and What It Does Not, When Judgment Becomes a Function Call, What Moves in Design, Agentic Coding: The Current State of Orchestration Patterns (2026)
Unverified Items
- The question texts passed to Jev, the thresholds, and the judgment records are not in the material for this note
- This note examines the structure and has not measured effects
- The evaluation by Rao and Callison-Burch covers rubric-based scoring
- Whether the same thing happens in video inspection has not been checked
References
All accessed 2026-11-21.
The Article Examined
- ktlog.net. 「Claude Codeのおすすめ環境を全体図で解説。ハーネス・スキル・MCPの仕組みと実例」. 2026-11-08. https://ktlog.net/posts/claude-code-ai-ecosystem
Vendor primary (T1v)
- TypeSafe AI. Docs: Confidence. https://docs.typesafe.ai/confidence.md
- TypeSafe AI. Docs: Choice. https://docs.typesafe.ai/primitives/choice.md
- TypeSafe AI. Docs: Noul. https://docs.typesafe.ai/primitives/noul.md
- TypeSafe AI. Docs: Score. https://docs.typesafe.ai/primitives/score.md
- TypeSafe AI. Docs: Model jaggedness. https://docs.typesafe.ai/model-jaggedness.md
- TypeSafe AI. Docs: Models. https://docs.typesafe.ai/models.md
- TypeSafe AI. Cookbook: Parallel questions. https://docs.typesafe.ai/cookbooks/parallel_questions.md
- TypeSafe AI. “Jev: For the builders AND dreamers”. 2026-10-14. https://typesafe.ai/blog/jev-1-14
- Laabs, A. (TypeSafe AI). “Don’t Call It a Decision Model”. 2026-11-03. https://typesafe.ai/blog/dont-call-it-a-decision-model
- TypeSafe AI. Status. https://status.typesafe.ai
Individual opinion and independent implementation (T3)
- Tzolov, C. Spring blog, Spring AI modular RAG with TypeSafe Jev. 2026-10-02. https://spring.io/blog/2026/10/02/spring-ai-modular-rag-typesafe-jev
- Willison, S. “Gemini Decisions API”. 2026-11-11. https://simonwillison.net/2026/Nov/11/
- Vercel Community. “TypeSafe AI Jev requests shed with 429 and providerAttemptCount 0”. 2026-09-26. https://community.vercel.com/t/typesafe-ai-jev-requests-shed-with-429-and-providerattemptcount-0-per-team-throttling/49779
- josebreijo. typesafe-sdk-js issue #15, Cloudflare blocks valid /v1/systemone requests. 2026-09-23. https://github.com/typesafe-ai/typesafe-sdk-js/issues/15
Academic (preprints)
- Li, Y., Miao, Y., Krishnan, R., & Padman, R. (2026). JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. arXiv:2609.26550v6. https://arxiv.org/abs/2609.26550
- Rao, D., & Callison-Burch, C. (2026). Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places. arXiv:2609.29769v3. https://arxiv.org/abs/2609.29769
- Brom, P. C. (2026). Confidence Is Not a Promise: A Reference Audit of Typed Decision Models. arXiv:2610.36311. https://arxiv.org/abs/2610.36311
- Song, M., Zheng, M., Ma, Z., & Zhou, X. (2026). Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias? arXiv:2610.35826v3. https://arxiv.org/abs/2610.35826
Author: Shuichiro Ogawa (Design Researcher / Design Consultant) About me →