Shuichiro Ogawa
日本語

Notes · updated 2026-09-22

Taking the LLM Smell Out of LLM Prose, and How to Bring humanizer Into This Project

This note sorts out what makes LLM prose recognizable, drawing on 23 academic papers, vendor documentation, Wikipedia's rules, and well-known skills. People cannot tell AI text apart by intuition (Jakesch et al.

Contents (9)
  1. People Cannot Spot It, Yet They Smell It
  2. What the Smell Is Made Of
  3. Where the Patterns Come From
  4. What the Removal Is For
  5. Where One Can Intervene
  6. What humanizer Does
  7. What This Project’s Rules Stop, and What They Miss
  8. Adoption Steps
  9. Where This Account Falls Short

People Cannot Spot It, Yet They Smell It

People are poor at telling AI-written text from human text by intuition. Across six experiments with 4,600 participants in total, Jakesch and colleagues showed that the cues people rely on for “human-ness”, such as first-person pronouns, contractions, and talk of family, are wrong, and that because they are predictable they can also be exploited (Jakesch et al. 2023).

Even so, people who read a lot of LLM output say they can smell it. Wikipedia editors turned that sense into a catalog, and a rewriting skill for Claude Code built on that catalog has collected more than 50,000 stars on GitHub. Intuition misfires, and still the smell is something many readers agree on. Choosing a way to remove it depends on first pinning down what it is made of.

What the Smell Is Made Of

The first ingredient is vocabulary. Kobak and colleagues examined word frequencies in more than 15 million PubMed abstracts using the same logic epidemiologists use for excess mortality. Words like “delve” rose beyond what earlier trends predicted in 2024, and from that excess they estimate that at least 13.5% of 2024 abstracts were processed with an LLM (Kobak et al. 2025). In peer reviews for AI conferences, Liang and colleagues report that adjectives such as “commendable”, “meticulous”, and “intricate” became 9.8 to 34.7 times more likely (Liang et al. 2024).

The second ingredient is the shape of sentences and documents. A stylometry study on short samples used SHAP to identify grammatical standardization, alongside the overuse of particular words, as a cue separating human from LLM text (Przystalski et al. 2026). The HC3 comparison corpus found that ChatGPT answers tend to move from a definition through step-by-step explanation to a summary (Guo et al. 2023). Practitioner metrics follow the same two layers. EQ-Bench’s Slop Score gives 60% of its weight to word frequency, 25% to “not just X, but Y” contrasts, and 15% to three-word sequences.

The two layers behave differently. Vocabulary can be listed and swapped mechanically. Patterns live in how sentences are assembled, so they survive a change of words. Sun and colleagues report that outputs from five major model families can be attributed to their source with 97.1% accuracy, that these habits are rooted in word-level distributions, and that they persist after another LLM rewrites, translates, or summarizes the text (Sun et al. 2025). A banned-word list can tidy the surface while the model’s fingerprint stays in place.

Readers differ too. Marco and colleagues explain split judgments of AI and human creative writing by a difference between readers who attend to readability and vocabulary and readers who attend to thematic development and rhetorical variety (Marco et al. 2025). The smell is a property of the text meeting a particular way of reading.

Where the Patterns Come From

Several studies show that instruction tuning and preference learning reduce output diversity. In a writing experiment by Padmakumar and He, co-writing with InstructGPT made different writers’ essays more alike, while base GPT-3 made no significant difference (Padmakumar & He 2024). Peeperkorn and colleagues trace most of the diversity loss in OLMo models to the DPO stage (Peeperkorn et al. 2025, preprint). On verbosity, an analysis of RLHF finds that much of the reward improvement is explained by longer responses (Singhal et al. 2024).

This collection found no study that links diversity loss directly to the frequency of specific stock phrases. The claim that RLHF made “delve” common goes beyond the evidence. What has been established, separately, is that preference learning pulls outputs into a narrow range, that length leaks into reward, and that the pulled-in outputs show skewed vocabulary and patterns.

What the Removal Is For

Techniques for removing the smell overlap technically with techniques for evading detectors. This note treats the work as editing for readers, and methods whose purpose is detector evasion, such as adversarial paraphrasing, were left out of the collection.

Outside rules draw a similar line. In a March 2026 RfC (44 in favor, 2 against), English Wikipedia barred using LLMs to generate or rewrite article text, with exceptions only for suggesting light copyedits to one’s own writing, subject to human review, and for help with translation. Its speedy deletion criterion G15 targets objective traces such as leftover chatbot messages and nonexistent references, and states that a subjective impression of AI-ness is not grounds for deletion by itself. Removing the smell leaves the writer exactly as responsible for the content as before, and a lingering smell is not, on its own, evidence that anything improper happened.

Where One Can Intervene

Before Generation, in the Prompt

The prompt is the cheapest place. Anthropic’s prompting documentation suggests stating the desired format positively (“Your response should be composed of smoothly flowing prose paragraphs.”) instead of forbidding markdown, and notes that the prompt’s own formatting shapes the output’s formatting, so removing markdown from the prompt reduces markdown in the response. The guide for Fable 5.1 defines mannered prose as substituting metaphor and flourish for direct statement, gives the fix as saying what you mean and using the literal phrase when one is available, and reports that the short request “Please remove all mannered prose.” tends to work.

Such instructions age. The same Anthropic documentation warns that an older block suppressing bullets and bold can, on Fable 5.1, which already formats less, remove structure the content needs. Vendors also point in different directions. OpenAI’s GPT-5.2 guide advises avoiding long narrative paragraphs in favor of compact bullets and short sections, the opposite of Anthropic’s lean toward prose. Prompt-level fixes have to be revisited for each model version.

During Generation, in Decoding and Training

With access to the model’s probabilities, one can intervene mid-generation. sam-paech’s antislop-sampler waits for a banned phrase to appear, backtracks, lowers the probabilities of the tokens leading to it, and regenerates. A paper by the same author and colleagues reports that this inference-time suppression, combined with token-level preference optimization (FTPO), cut slop by 90% while keeping GSM8K and MMLU performance, and that DPO, used for comparison, sharply reduced writing quality and lexical diversity even at weak suppression strength (Paech et al. 2025). On the diversity side, decoding that steers an instruct model with a more varied base model has been proposed to recover diversity (Peeperkorn et al. 2025).

Two conditions apply here. Hosted API users generally cannot touch logits, so these methods fit local models or teams that train their own. Both papers are preprints, and no independent replication turned up. The antislop-sampler README itself calls the code research grade and notes that it does not support concurrency.

After Generation, as Detection

Tools that flag tells in finished text come in two kinds. The quantitative kind relies on dictionaries and regular expressions, with Slop Score as the main example. Its author describes it as looking for glaringly overused patterns and says it works differently from detectors that classify text as AI or human. On the research side, Shaib and colleagues derived measurement dimensions such as verbosity, readability, generality, exaggeration, and statistical patterns from interviews with experts, while stating that no single agreed definition of slop exists yet (Shaib et al. 2025, preprint).

The qualitative kind is Wikipedia’s Signs of AI writing. It is an advice page from WikiProject AI Cleanup, neither a guideline nor a policy. It lists signs under sections such as Content, Language and grammar, Style, Communication intended for the user, Markup, and Citations, and it sets apart an Ineffective indicators section naming signs that cannot justify a judgment by themselves, which other catalogs lack.

After Generation, as Rewriting

Once tells are found, the usual practice is to have a skill or agent rewrite them. In research, Chakrabarty and colleagues trained a writing-quality reward model on human edits, used it to rerank several candidate revisions, and report a 66% human preference for the result (Chakrabarty et al. 2025, preprint).

Rewriting has a price. Tripto and colleagues showed that repeated paraphrasing pulls text away from the original writer’s style and lowers authorship attribution accuracy (Tripto et al. 2024). Asking an LLM to fix LLM smell can also thin out the original writer’s voice. Marking specific tells and editing only those passages keeps that cost lower than running the rewrite in repeated passes.

What humanizer Does

The best-known skill in this space is humanizer by blader (Siqi Chen). The repository was created on 2026-01-18, and as of 2026-09-22 it has 51,020 GitHub stars, an MIT license, and version 3.0.0. Similar skills sit far below it: skill-deslop, aimed at scientific writing, has 402 stars, and unslop has 0 stars and no license. Stars measure popularity, and say nothing about how well the rewriting works.

The description in the SKILL.md frontmatter reads:

description: |
  Rewrite AI-sounding text so it reads like the writer without changing what it says.
  Use when editing or reviewing prose for AI tells: not-X-but-Y contrasts, one-line
  closers, staged openers, forced triads, dashes everywhere, inflated claims, sales
  language, stock AI words, bold labels, or filler. Based on Wikipedia's "Signs of AI writing."

That single description holds the goal, the tells that should trigger the skill, and the source. Under the Agent Skills specification the description decides when a skill is invoked and is capped at 1,024 characters, so this one is a good model to copy.

The body sorts 25 patterns into five groups. Group A, “Staging instead of stating”, covers not-X-but-Y contrasts, one-line closers, deep-sounding maxims, openers that circle before the point, and arguments with nobody. Group B, “Rhythm by rule”, covers forced triads, repeated openings, dashes everywhere, stacked modifiers, hyphenated compounds, and missing subjects. Group C, “Inflation and borrowed authority”, covers stock AI words, inflated significance, vague associations, shallow -ing clauses, sales language, borrowed authority, and avoidance of is, are, and has. Group D, “Formatting by rule”, covers decorative bold, decorative headings, and curly quotes. Group E, “Leftovers from the chat and the draft”, covers chatbot residue, knowledge-cutoff disclaimers, repeated headings, and references to earlier versions.

The procedure is fixed in four steps: Mark the tells, Draft the rewrite, Check the draft, Write the final version. One operating rule deserves attention. Since a person may use any of these patterns on purpose, weak tells are acted on only when several share a passage. The skill also states that text written before November 30, 2022 is not AI-written.

Its limits are plain. The SKILL.md assumes English and says nothing about other languages. Neither the README nor the SKILL.md discusses overcorrection, where the fix produces a different kind of smell. It can be installed with npx skills add blader/humanizer --global, through /plugin marketplace add blader/humanizer on Claude Code 2.1.142 or later, or by placing the SKILL.md by hand.

For Japanese, m0370 published a 16-item skill on 2026-03-20, adapting 10 items from humanizer_academic and 6 from humanizer. The article gives examples such as phrases that “throw into relief” and then “suggest”, insertions set off with full-width dashes, bullet lists that open with bold labels and colons, and passive constructions like “is a concern” that hide the subject, but it offers no data on how well the 16 items work.

What This Project’s Rules Stop, and What They Miss

This project already has Japanese style rules. .claude/rules/japanese-writing-style.md, based on two gists by k16shikano, bans stock phrases like “the important thing is” and “in summary”, dashes and middle dots in running text, overuse of bold, headings phrased as conclusions, and sentences about the document itself. That covers much of humanizer’s groups A and C, and its companion rules on cognitive rhythm add a guard against shortening sentences just to create rhythm, which the English-language skills do not have.

A read-only audit of existing notes found the banned words almost absent. The patterns are what remain. One published note still contains the banned “in summary”, has three paragraphs in a row ending on a short “is not” sentence, enumerates “first” through “fourth”, and uses the Japanese not-X-but-Y construction 11 times. An unpublished draft contains the banned “head-on” three times, a summary over 2,000 characters, and a title joining two phrases with a colon. The English notes have no style rules at all, and the weekly prompt says explicitly that the Japanese rules do not apply to them.

Patterns the rules do not yet cover:

  • fixed-count enumerations (forced triads, “first” through “fourth”)
  • the frequency of negated contrasts, and paragraphs ending on a short negation
  • inflated significance or lineage (“the theoretical pillar”, “in the same stratum”)
  • vague attribution (“can be read as”, “the researchers themselves assume”)
  • Japanese forms that avoid a plain copula (“is positioned as”, “takes on the role of”)
  • trailing importance claims (“this point is important”)
  • full-width colons in titles and headings
  • chat residue (“below I will…”), knowledge-cutoff remarks, and boilerplate upbeat endings
  • summary length

At one point the rules conflict with Wikipedia. The rules recommend writing term-and-definition lists as a bold term, a colon, and the explanation, while Wikipedia and humanizer count bold-label lists as a tell. Adopting either source means deciding which way to go.

The other gap is how the rules reach the writer. No skill or agent references them, and only two prompts, the weekly and monthly watch prompts, tell the model to read them. No agent or hook runs the post-writing checklist the rules define. A plausible reason the words are gone while the patterns remain is that words can be avoided by reading a list, whereas patterns only show up when someone counts them after the text is written.

Adoption Steps

There are three options.

Option B, folding the patterns into the rules and adding a report-only check, comes first. Add the uncovered patterns above to the “LLM-style expressions” section of japanese-writing-style.md, and decide how to treat the bold-term-and-colon form. Add a category to .claude/skills/wiki-lint/scripts/lint-wiki.ts that counts reappearing banned words, the density of the not-X-but-Y construction, paragraphs ending on a negation, colons in titles and headings, bold per section, and summary length. The existing marker check reports without affecting the exit code, and treating this check the same way keeps CI green. Update the rubric table in wiki-lint/SKILL.md and the wiki:lint description in the README, and add a line to step 6 of source-collection and to the wiki-ingest self-check telling the writer to read the rules before writing. Because rules and skills change, gate G4 requires syncing claude-assets-overview.md. No hooks change, so npm run test:hooks is not needed. It is deterministic and cheap, and it adds the counting step that is currently missing.

Option A, vendoring humanizer, serves the English notes. Place .claude/skills/humanizer/SKILL.md and the MIT LICENSE in the repository. Installing it as a plugin would not carry over to the weekly runs through claude -p on GitHub Actions, so the files have to live in the repo. Add a “finish with humanizer” step to the English-version instructions in the weekly and monthly prompts, and optionally a row in the routing table in orchestration.md. For G4, update the skill count in CLAUDE.md (9 to 10) and the skill table in claude-assets-overview.md, and since scripts change, also sync the weekly automation section of the README. It does nothing for Japanese text, so it cannot replace option B.

Option C, a dedicated editing subagent, waits. It would add a read-only agent, for example prose-editor, that reads a note and returns suggested edits for the parent to apply. That respects the single-writer rule, but the call sites would spread across source-collection, wiki-ingest, critical-review, the roundtable, and the weekly and monthly prompts, and requiring a dispatch header would also touch dispatch-header-gate.py and npm run test:hooks. Each note would cost one more agent run, so it makes sense only after option B’s counts show that the patterns are not going down.

One thing should be checked before option B. The frontmatter of japanese-writing-style.md uses a globs: key for path scoping, while the official Claude Code documentation names paths as the key. There is also a past issue reporting that paths in YAML list form did not work, and whether this rules file is actually loaded always, only for certain paths, or at all has not been tested here. Checking its load state with /memory or a similar command before editing it shows whether the new patterns will reach the writer.

Where This Account Falls Short

This collection found no study that describes or measures LLM smell in Japanese. The one Japanese study found (Takahashi et al. 2025) is about controlling speech style through internal representations, and it does not measure the smell. The Japanese patterns listed above come from the style rules, the audit, and a practitioner’s adaptation, and their frequencies have not been measured.

The mid-generation methods rest on preprints without independent replication. Research on rewriting amounts to two papers, and studies of how readers perceive technical or business writing are scarce. No public data measures how much humanizer actually reduces the smell. Its 51,020 stars show how many people want the problem solved.

Unverified Items

No main claim rests on unverified material. The following remain in peripheral statements and bibliographic details.

  • [要一次検証: 記載なし humanizer の README と CHANGELOG] The release date of humanizer v3.0.0.
  • [要確認: 未試行 本調査の範囲外] Whether Claude Code recognizes the globs: key in japanese-writing-style.md.
  • [要一次検証: 記載なし m0370 の記事本文] Data on how well the 16 Japanese items work.
  • [要出典確認: 書誌未確定 OpenReview の Cloudflare 検証壁] ICLR 2024 acceptance of Kirk et al. (2024). Not used for any claim in the body.
  • [要一次検証: 取得不能 dl.acm.org 403] ACM-side bibliographic data for Anderson et al. (2024) and Chakrabarty et al. (2024, CHI). Content checked on arXiv. Not used for any claim in the body.
  • [要一次検証: 本文未到達 PDF のテキスト抽出不能] The full text of Takahashi et al. (2025). Bibliographic data checked against the ANLP program.

References

All accessed 2026-09-22.

Academic literature

  • Anderson, Shah, & Kreminski (2024). Homogenization Effects of Large Language Models on Human Creative Ideation. Creativity & Cognition 2024. https://doi.org/10.1145/3635636.3656204
  • Chakrabarty, Laban, Agarwal, Muresan, & Wu (2024). Art or Artifice? Large Language Models and the False Promise of Creativity. CHI 2024. https://doi.org/10.1145/3613904.3642731
  • Chakrabarty, Laban, & Wu (2025). AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation. arXiv preprint. https://arxiv.org/abs/2504.07532
  • Guo, Zhang, Wang, Jiang, Nie, Ding, Yue, & Wu (2023). How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. arXiv preprint. https://arxiv.org/abs/2301.07597
  • Jakesch, Hancock, & Naaman (2023). Human heuristics for AI-generated language are flawed. PNAS, 120(11), e2208839120. https://doi.org/10.1073/pnas.2208839120
  • Kirk, Mediratta, Nalmpantis, Luketina, Hambro, Grefenstette, & Raileanu (2024). Understanding the Effects of RLHF on LLM Generalisation and Diversity. https://arxiv.org/abs/2310.06452
  • Kobak, González-Márquez, Horvát, & Lause (2025). Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Science Advances, 11(27). https://doi.org/10.1126/sciadv.adt3813
  • Liang, Izzo, Zhang, Lepp, Cao, Zhao, Chen, Ye, Liu, Huang, McFarland, & Zou (2024). Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. ICML 2024. https://arxiv.org/abs/2403.07183
  • Lin, Zhao, Tian, & Li (2025). Large Language Models Reshape the Language of Science. arXiv preprint. https://arxiv.org/abs/2504.12317
  • Marco, Gonzalo, & Fresno (2025). The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing. Findings of ACL 2025. https://arxiv.org/abs/2506.03310
  • Padmakumar & He (2024). Does Writing with Language Models Reduce Content Diversity? ICLR 2024. https://arxiv.org/abs/2309.05196
  • Paech, Roush, Goldfeder, & Shwartz-Ziv (2025). Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models. arXiv preprint. https://arxiv.org/abs/2510.15061
  • Peeperkorn, Kouwenhoven, Brown, & Jordanous (2025). Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models. arXiv preprint. https://arxiv.org/abs/2507.20956
  • Przystalski, Argasiński, Grabska-Gradzińska, & Ochab (2026). Stylometry recognizes human and LLM-generated texts in short samples. Expert Systems with Applications, 296, 129001. https://doi.org/10.1016/j.eswa.2025.129001
  • Shaib, Chakrabarty, Garcia-Olano, & Wallace (2025). Measuring AI “Slop” in Text. arXiv preprint. https://arxiv.org/abs/2509.19163
  • Singhal, Goyal, Xu, & Durrett (2024). A Long Way to Go: Investigating Length Correlations in RLHF. COLM 2024. https://arxiv.org/abs/2310.03716
  • Sun, Yin, Xu, Kolter, & Liu (2025). Idiosyncrasies in Large Language Models. ICML 2025. https://arxiv.org/abs/2502.12150
  • Takahashi, Yano, Naruse, Takei, Kaji, & Suzuki (2025). LLMにおける内部表現を用いた日本語スタイル制御メカニズムの分析 [An analysis of Japanese style control mechanisms using internal representations in LLMs]. NLP2025 P2-6. https://www.anlp.jp/proceedings/annual_meeting/2025/pdf_dir/P2-6.pdf
  • Tripto, Venkatraman, Macko, Moro, Srba, Uchendu, Le, & Lee (2024). A Ship of Theseus: Curious Cases of Paraphrasing in LLM-Generated Texts. ACL 2024. https://arxiv.org/abs/2311.08374

Vendor primary sources

Public bodies and editorial communities

Skills, tools, and practitioners

Author: Shuichiro Ogawa (Design Researcher / Consultant) About me →