Shuichiro Ogawa
日本語

Notes · updated 2026-10-21 · Markdown

AIO on This Site: What Was Added, and What Was Left Out, So That AI Search Cites the Notes as Sources

This note records how a personal site that publishes academic notes was set up so that AI search and generative AI answers cite the notes as correct sources, and which measures have evidence behind them.

Contents (7)
  1. What this site wants
  2. Where the evidence ends
  3. Separating permission to fetch from preferences on use
  4. Serving the same content in a form that machines read easily
  5. Attaching structured bibliographic data
  6. What was left out
  7. How this will be checked

What this site wants

The notes on this site end with a list of references, with a DOI or a URL wherever one exists. Most readers arrive from search. Since 2025, however, an AI summary has appeared above the search results, and AI chat assistants have begun to read the web and answer directly. The work of getting cited as a source in those answers goes by the names AIO, GEO, AEO, and LLMO, and in industry it has become a contest over measurement and control (Industry Trends in AIO (Optimization for AI Search): What Is Measured, What Is Sold, and What Is Contested). This note starts from that survey and records what was added to this site itself.

The aim was narrowed to two points. First, when an AI answer uses the content of a note, the citation should carry the author, the date, the URL of the note, and the references the note itself cites. Second, the body and the reference list should be the same in the HTML that people read and in the form that AI reads. Recovering clicks was not among the aims. When an AI summary appears, clicks to external sites fall, and few people move from an AI answer to its sources (8% versus 15% in Pew Research Center’s measurement; 4% who always or often click through in the Reuters Institute survey; Industry Trends in AIO (Optimization for AI Search): What Is Measured, What Is Sold, and What Is Contested). For academic notes, correct attribution comes before the number of readers.

Where the evidence ends

Most AIO advice concerns how to write. Adding statistics, adding quotations, cutting text into short chunks, and writing in question-and-answer form are said to make content easier to cite. The paper at the origin of this advice reported, in an experimental setup where GPT-3.5 composed answers from the top five Google documents, that the best technique raised visibility by up to 40% over the baseline (Aggarwal et al. 2024). Later verification did not support this. C-SEO Bench tested techniques of the same kind across several domains of product recommendation and question answering with newer LLMs. Ranking improved significantly in only 3 of 54 conditions. What mattered was the conventional factor: ranking high in search, and therefore being placed at the head of the context (Puerto et al. 2025).

The providers’ official positions point the same way. In May 2026 Google added a guide to its generative AI features. The guide states that, from the viewpoint of Google Search, AEO and GEO are the same as SEO. Machine-readable files such as llms.txt are not used by Google Search and therefore have no effect on visibility, there is no need to chunk content, and collecting inauthentic mentions across the web is less useful than it appears (Google Search Central 2026a). It also states that no new machine-readable file, markup, or special schema.org structured data is needed to appear in AI Overviews and AI Mode (Google Search Central 2026c). Microsoft’s Bing recommends headings, lists, tables, question-and-answer blocks, and structured data, but attaches no figures on their effect (Madhavan 2025).

Given this boundary, the note adopts no writing technique and adds only three measures: telling crawlers separately whether they may fetch and how they may use the content, serving the same content in a form that machines read easily, and attaching structured bibliographic data. For each of the three, a provider’s official document or a standards draft states that the mechanism is actually read. Apart from the token count of Markdown, none of them has a measurement of the size of its effect.

Separating permission to fetch from preferences on use

AI providers separate fetching for search from fetching for training, assign them to different crawlers, and let a site control each in robots.txt. OpenAI writes that a site that allows OAI-SearchBot while disallowing GPTBot appears in ChatGPT search results without being used for training (OpenAI 2026). Anthropic distinguishes ClaudeBot for training, Claude-SearchBot for improving search quality, and Claude-User for fetching pages in response to a user’s question, and writes that disabling Claude-SearchBot or Claude-User may reduce the site’s visibility in user-directed search (Anthropic 2026). Perplexity likewise separates PerplexityBot, which surfaces sites in its search results, from Perplexity-User, which fetches pages for a user’s question, and states that neither is used for training (Perplexity 2026).

Google uses a control token, Google-Extended, with which a site can opt out of training for Gemini and of grounding, that is, supplying content from the search index to the model at prompt time. The token does not affect inclusion or ranking in Google Search (Google Search Central 2026d). AI Overviews and AI Mode, on the other hand, are part of Search, so the control for them is whether Googlebot is allowed (Google Search Central 2026c).

The robots.txt of this site had consisted of two lines that allowed everything. The permission did not change. The search crawlers and user-triggered fetchers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) are now allowed by name, and only the path of the chat API is excluded. Naming them produces the same result as the default permission. The purpose is to make the intention legible in the file itself, so that search fetching is kept even if the default is narrowed in the future.

Apart from permission to fetch, a mechanism for stating preferences on what happens after fetching has been in operation since 2025. In September 2025 Cloudflare published the Content Signals Policy, which adds to robots.txt a line of the form Content-Signal: search=yes, ai-train=no. Here search means building a search index and returning links and short excerpts, and it excludes AI-generated search summaries. ai-input means feeding content into AI, such as retrieval-augmented generation or real-time use in AI search answers. ai-train means training or fine-tuning. A signal that is not written neither grants nor restricts permission; it means that no preference has been expressed (Allen 2025). The IETF aipref working group is drafting a vocabulary that distinguishes training (train-ai), AI use (ai-use), and search (search), together with a way to attach it in HTTP headers and in robots.txt (Keller and Thomson 2026; Illyes and Thomson 2026).

This site states search=yes, ai-input=yes. Being indexed for search and being used as input to AI answers are exactly what the site wants. ai-train was not written. Being used for training leads neither to citation nor to attribution and lies outside the aim, but no reason to refuse it has been examined either. The Content Signals specification allows leaving a preference unexpressed, and that is what was done.

Whether either mechanism is honored is left to the fetching side. The robots.txt standard states that its rules are not a form of access authorization (Koster et al. 2022), and Cloudflare itself writes that content signals express preferences, are not technical countermeasures, and may simply be ignored by some companies (Allen 2025). The statement on this site affects only the providers that honor it.

Serving the same content in a form that machines read easily

In industry, llms.txt is discussed as an AIO technique, but the purpose of the proposal is to gather in one file the information that helps an agent use a website. The proposal specifies the form: the name of the site as an H1, a short summary as a blockquote, and lists of links to pages under H2 headings. It also recommends serving a Markdown version of each page at the page URL with .md appended. The HTML can point to the Markdown version with a link of rel="alternate" and type="text/markdown" (Howard 2026). The developer documentation of OpenAI and of Perplexity places llms.txt and a Markdown version of each page in this form (OpenAI 2026; Perplexity 2026).

The reason for a Markdown version lies in the number of tokens. In February 2026 Cloudflare released a feature that converts HTML to Markdown in response to a request with Accept: text/markdown, and showed that one of its blog posts took 16,180 tokens as HTML and 3,150 tokens as Markdown (Martinho and Allen 2026). When an AI reads a page, the navigation, the scripts, and the decorative elements consume tokens without carrying meaning. Serving the same body as Markdown reduces what the AI has to read, and the structure of the body (headings, lists, the reference section) is conveyed as it is.

A Markdown version of every published note is now served at /notes/<slug>.md (/en/notes/<slug>.md for the English version). Its head lists the title, the author, the dates of first publication and of the last update, the canonical HTML URL, the URL of the version in the other language, the language, the keywords, and the summary, followed by the body. The body is generated under the same rules as the HTML that people read: drafts are not served, the blocks that hide unpublished research are removed, and links between notes are rewritten as absolute URLs. The HTML head carries an alternate link to the Markdown version, and a link to it also appears at the top of the page. llms.txt consists of a description of the site, a list of the Markdown versions of all notes (English first, then Japanese), and an Optional section with the architecture, references, and citation-network pages. sitemap.xml lists the public pages and the HTML and Markdown versions of every note with their update dates, and robots.txt points to it.

Google Search states that it uses neither llms.txt nor Markdown (Google Search Central 2026a). Fetchers that come for the file are also still few. When Ahrefs aggregated the May 2026 logs of the 137,210 domains that run its analytics tool, about 28% published an llms.txt file, but 97% of those files received no request at all in that month (Ahrefs 2026). The count covers requests only, and it does not show whether the fetched content was used. These two measures therefore work only for agents that read llms.txt and for fetchers that come for the Markdown version, and how many there are can only be seen in the logs of this site. For Google, the HTML body is read as it is.

Attaching structured bibliographic data

The third measure attaches the bibliographic data of a note to the page in a machine-readable form. ScholarlyArticle in schema.org is a subtype of Article that can carry the title, the abstract, the author, the language, the dates of publication and modification, the site the article belongs to, and the works it cites (citation) (schema.org n.d.). Google states that Article structured data helps it understand a page, and lists the author’s name and URL, the dates of publication and modification, and the headline among the recommended properties (Google Search Central 2026e). At the same time, Google states that no special structured data is needed to appear in AI Overviews and AI Mode (Google Search Central 2026c). What structured data can be expected to do is therefore not to place a page in AI features but to let the fetching side read without error whose text it is, when it was written, and what it cites.

Every published note now carries ScholarlyArticle JSON-LD in its head. It contains the title, the summary, the language, the keywords, the date of first publication (the first appearance in the wiki’s log), the date of the last update, the author, the site, the location of the Markdown version, the version in the other language, and, as citation, each entry of the reference section. The home page carries JSON-LD for the site (WebSite) and the author (Person), with the author’s other pages (LinkedIn, researchmap) linked through sameAs. The references were placed in citation so that, when an AI answer cites a note, a machine can tell what the note rests on without reading the body.

What was left out

Writing techniques were left out. Cutting text into short chunks, rewriting into question-and-answer form, and collecting mentions across the web rest on evidence limited to laboratory conditions, and Google states that they are unnecessary (Google Search Central 2026a; Puerto et al. 2025). The prose of the notes follows the norms chosen for readers, and it is not altered for AI.

Mechanisms for refusing fetches and charging for them were also left out. In industry, publishers in particular are moving to refuse fetching for training and to demand payment for access (Industry Trends in AIO (Optimization for AI Search): What Is Measured, What Is Sold, and What Is Contested). This site is on the side that wants to be cited, so there is no reason to refuse search crawlers. On training, as stated above, no preference is expressed.

Content negotiation through Accept: text/markdown was left out as well. Cloudflare’s feature is enabled on paid plans and is not part of static delivery. Appending .md to the URL serves the same purpose, so the form of the proposal was followed.

How this will be checked

That the measures work was confirmed in the build output. robots.txt, sitemap.xml, llms.txt, the Markdown version of every note, and the JSON-LD of every page are generated, and a mechanical check confirmed that no wiki-link syntax remains in the Markdown versions.

Whether the measures have an effect has not yet been measured. Google Search Console has a report that counts appearances in generative AI features, and Bing Webmaster Tools has a function that counts citations in AI answers (Industry Trends in AIO (Optimization for AI Search): What Is Measured, What Is Sold, and What Is Contested). Aggregating the user agents in the access logs would show which fetchers come for the Markdown versions and for llms.txt. Results will be added to this note once those measurements begin.

Even if an effect appears, whether a note is cited depends on choices and fluctuations on the providers’ side. What gets cited overlaps little with the top conventional search results, shifts from week to week, and the same question returns the same list of recommendations with a probability below 1 in 100 (Industry Trends in AIO (Optimization for AI Search): What Is Measured, What Is Sold, and What Is Contested). What this site can do ends at preparing a form in which, when a note is cited, it is attributed correctly.

Unverified Items

None. The full text of every document cited was retrieved and checked.

References

Author: Shuichiro Ogawa (Design Researcher / Design Consultant) About me →