Shuichiro Ogawa
日本語

Notes

Research, experiments, and reflections on AI and design, published along with the process of making them.

Back to the monthly list

Business and operations (13)

New business, consulting, and how work gets done.

  • Services That Replace Acceptance Testing with AI Synthetic Respondents: How Agents Are Built, Counted, Shown Stimuli, and Sold

    2026-11-14

    This note examines 38 services that build many AI agents with different traits and run them through the same acceptance testing that would otherwise be done with people (showing concepts, ads, or prices and checking whether they are accepted). The evidence comes from official pricing pages and documentation, developer statements, industry codes, and third-party comparisons, as of November 14, 2026. Agent construction does not split into choosing from a vendor's library versus building from the buyer's own data; most products offer both. The products differ instead in the data that grounds the answers: the vendor's panel and purchase data, public statistics, the buyer's research data, or long interviews with real individuals. Stated limits mix three different quantities: population size (100,000 people, or a scale of about 100 million), responses per study (from 5 to 10,000), and plan quotas. None of them counts independent people. Increasing the number of agents from 500 to 5,000 reduced the error only from 6.8 to 6.7 points (Verasight 2026). Text stimuli work in every product, while support for images and video differs by product, and Qualtrics synthetic panels do not process images, videos, files, or links. Pricing falls into monthly self-serve plans (from USD 20 per month), annual contracts with credits or tokens, flat licenses with unlimited queries, and quoted contracts, and most plans are consumed per response or per interview. All agreement figures with human answers are measured by the vendors themselves. In third-party comparisons, topline estimates still missed by 2 to 12 points, and subgroup results, price logic, and change over time missed further. The codes and reports of ESOMAR, MRS, the Insights Association, and AAPOR require that buyers be told when synthetic data are used and which outputs are synthetic and which come from people.

  • Gate Design for New Businesses: Stopping What Should Stop Without Killing What Could Grow

    2026-11-11

    This note examines gates that decide, stage by stage, whether to keep investing in a new business. It draws on 11 public programs and evaluations, 5 materials from research firms and consultancies, 20 first-hand materials by practitioners who built and run gates, and 14 academic works, and it checks figures and quotations against the full texts. Gate failures fall into three types: projects that should stop do not stop, projects that could grow are stopped too early, and the gate becomes an approval ritual. For the first type, studies of the mechanism are consistent. Executives showed a strong bias toward continuing failing new products; precommitment to self-specified decision rules helped little, whereas a predetermined decision rule or a new decision maker helped relatively more (Boulding et al. 1997). Managers were more reluctant to stop really new products (Schmidt and Calantone 1998), and in teams, hope was more strongly tied to continuing than fear was tied to terminating (Huang et al. 2019, 66 teams). Practitioner remedies separate the people who decide from those who started the project (cross-department review at CyberAgent, independent reviewers in the UK Gateway Review) and release money by validation stage (McGrath, Ries, and Recruit's four stages). However, in a survey of 888 people involved in new businesses at large Japanese firms, 28.5% of written exit criteria were not followed, and criteria written before launch were not followed in 38.1% of cases, compared with 17.2% for criteria written after launch (a survey by a support firm, showing association only). Criteria therefore need to come with a decision on who reviews them and when. There is also evidence for the second and third types. Repeatedly applying strict and objective gate criteria made projects inflexible and lowered the performance of novel products (Sethi and Iqbal 2008, 120 projects), and the transition-rate standards of the US SBIR program were judged to risk limiting exploratory work (National Academies 2026). Since 2022, average investment before a new venture breaks even has fallen from about 125 million to about 77 million US dollars, and the leash has become shorter (McKinsey 2025). Cooper, who developed Stage-Gate, proposes shifting the purpose of gates from optimizing investment decisions to maximizing learning velocity and avoiding the premature killing of promising projects (Cooper and Zhao 2026, a proposal rather than evidence).

  • What Counts as Evidence in New-Business Acceptance Research: From Words to Behavior, and Then Generative AI

    2026-11-11

    This note examines acceptance research for new businesses, that is, research that checks whether customers will accept a business before it is built. It draws on 6 public programs, 5 surveys by research firms and consultancies, 16 first-hand materials by the practitioners who developed the methods, and 14 academic works, and it checks figures and quotations against the full texts. Practitioner best practice converges on treating what customers do as stronger evidence than what they say. Interviews and questionnaires are weak, and simulated sales and post-launch usage are strong (Osterwalder 2017, Gilad 2017). The weakness of stated answers has a measured basis: willingness to pay measured with hypothetical questions is on average 21% higher than real willingness to pay (Schmidt and Bijmolt 2020, a meta-analysis of 77 studies). US public programs make the act of validation mandatory (NSF I-Corps requires 100 interviews with potential customers in seven weeks) and count the conclusion that no market exists as a success. Japan's NEDO and the EU's EIC require a description of the market as part of the application. When randomized experiments measured the effect of hypothesis-driven methods, the result shared by the two largest experiments was the earlier termination of unpromising ideas (Camuffo et al. 2024, 759 firms; Bailey et al. 2026, 553 startups, not yet peer reviewed). In large Japanese firms, about 20% of companies had a project that reached investment recovery (PwC 2025), and 23.6% of those who reported a sense of failure as the voice of actual customers were not heard (Innovation Research Institute 2026, a survey by a support firm). Two changes followed the spread of generative AI. Because prototypes can now be built quickly, a working prototype is no longer evidence of demand (Blank 2026). AI synthetic respondents entered research, and 69% of market researchers had used them in the previous year (Qualtrics 2024, a survey by a vendor). Synthetic answers brought means closer but missed distributions, and overstated willingness to pay for hedonic products (Oetzel and Maiberger 2026). It is reasonable to place synthetic answers below stated human answers in the order of evidence strength.

  • What Is a Playing Manager? Gains and Losses for the Appointee and the Appointing Side in Academic and Industry Sources

    2026-11-01

    This note examines the playing manager, a manager who has subordinates and is also responsible for the results of hands-on work (called a player-coach in recent US usage). It draws on 28 academic works, 11 documents from public bodies and international standards, 14 surveys by research firms and institutes, and 16 first-hand statements and reports by executives and practitioners, and checks figures and claims against the full text wherever the full text could be obtained. Academic research rarely uses the term; the subject is split across studies of first-line managers, hybrid professional-managers, expert leaders, and role overload. In Japanese surveys, 98.5 to 99.5% of section managers do some player work, player work accounts for about half of their work (weighted average 49.1 to 50.1%), and this share did not change across the surveys from 2010 to 2019. The recent trend consists of two different movements. In Japan, a long-standing condition acquired a name; in US technology firms after 2023, executives adopted the player-coach as a policy for removing management layers (for example, Coinbase's "No pure managers"). For the appointee, the gains are continued involvement in hands-on work, a basis for the respect of subordinates, and pay. The losses are a lack of time (59% of section managers report that player work hinders management), role overload and conflict, being pulled back to one's own targets, skills that decay even in the combined role, no time to learn, and promotion gains that do not match the burden. For the appointing side, the gains are two capabilities from one person, fewer layers, and the effect of technically competent bosses on subordinates' satisfaction (Artz et al. 2017). The losses are subordinates who do not develop (the most common worry of section managers), lower team outcomes and lower manager engagement when the share of player work is high (self-rated team outcomes peak at 20 to 30% and are lowest at 80% or more; Gallup finds a difference at 40%), and the cost of promoting the best players (Benson et al. 2019). Academic and industry sources agree that the combined role makes time and attention compete and that management work is cut first. They diverge in that industry measures the share of time and the manager's sense of burden, whereas academia measures the structure of the role and its consequences for subordinates and organizations. No peer-reviewed study was found that examines the relationship between the share of player work and outcomes.

  • Industry Trends in AIO (Optimization for AI Search): What Is Measured, What Is Sold, and What Is Contested

    2026-10-11

    This note examines AIO, the work of getting a company or its information cited and recommended in AI search and generative AI answers (including GEO, AEO, and LLMO). It draws mainly on industry sources (surveys by Pew Research Center and the Reuters Institute; published figures from Ahrefs, Seer Interactive, Semrush, Similarweb, Adobe, Cloudflare, Profound, and SparkToro; official documents from Google, Microsoft, OpenAI, Anthropic, and Perplexity; primary documents from the UK CMA, the European Commission, the Japan Fair Trade Commission, the IETF, and the Japan Newspaper Publishers & Editors Association; and Japanese surveys by CyberAgent and Hakuhodo DY ONE), with academic research as a supplement. Clicks fall when an AI summary appears. In Pew's measurement, users clicked a traditional result link in 8% of visits to searches with an AI summary, about half of the 15% for searches without one. However, the click-through rate also fell by about half over two years for searches without an AI summary (Ahrefs), so the whole decline cannot be attributed to AI summaries. Referrals from AI to websites are growing fast from a small base (4% of respondents say they always or often click through from AI to sources, against 19% for search, Reuters Institute). What gets cited overlaps little with traditional top search results (12% of AI assistant citations), shifts sharply from week to week, and the same question returned the same list of recommended brands with a probability below 1 in 100 (SparkToro). For writing techniques, the paper that named GEO reported gains of up to 40%, but a later benchmark found significant improvement in only 3 of 54 conditions, and gains shrank as more parties adopted the techniques. Google states that AEO and GEO are still SEO and that llms.txt files and chunking content are unnecessary. The industry appears to be shifting its weight away from writing techniques toward measurement that counts citations (Search Console, Bing's citation reports, Profound's USD 35 million round) and toward control over, and payment for, AI use of content (the CMA conduct requirement, Google's worldwide rollout of an exclusion control on August 31, 2026, Cloudflare's paid crawling, and IETF standardization of AI usage preferences).

  • Why Buyers Purchase Design Consulting and How They Choose: Differences from Consulting in General and Changes since 2024 (from Industry Sources)

    2026-10-07

    This note narrows the previous study of why buyers purchase consulting and how they choose to design consulting (outsourced brand, product, UX, and service design). It draws mainly on industry sources: 675 interviews with buyers conducted by Up to the Light with the UK Design Business Association (What Clients Think 2024), the US advertisers' association ANA's survey on in-house agencies, the UK advertisers' association ISBA's survey on agency remuneration, and statements by in-house design leaders and CMOs. Academic research was used to support the mechanisms. In design, the reasons for buying fall into the same four groups, but with different weights. The top advantages of external agencies are wider experience, a greater willingness to challenge, and breadth of skills, and the asset no one else has is not data but experience accumulated across many sectors and brands. The deciding factors are evidence of real client and agency partnership, the ability to answer big strategic challenges, and commercial effectiveness; the top reasons for winning a pitch are a clear and memorable message, a great team with good chemistry, and showing that the agency got the client's problem; price is near the top of none of these lists. In the public sector, the US GSA named the lack of in-house human-centered design expertise as a reason to look outside government, and the UK's digital procurement framework scored cultural fit with a weight of 5 to 20%. 73% of buyers say that design effectiveness is difficult to quantify, and 73% find it difficult to differentiate between agencies at the new business stage. From these sources, the note proposes that the inability to verify quality before purchase works more strongly in design; that design has its own way of addressing it, namely pitches and unpaid creative work that show part of the work before the contract; that the justification of decisions turns toward proof of commercial effectiveness rather than independence; and that internal capability in design is being rebuilt (the share of ANA members with an in-house agency rose from 42% in 2008 to 82% in 2023), so external agencies are moving toward complementing internal capability and challenging from outside. Since 2024, 74% of buyers expect AI to push agency costs down, and the share who consider their agency good value for money fell from 66% in 2014 to 45% in 2024. The note hypothesizes that as AI makes pitch work and production cheaper, polish loses force as a deciding factor and the weight moves to challenge from outside, understanding, and commercial effectiveness.

  • Why Buyers Purchase Consulting and How They Choose: Reasons, Deciding Factors, and the Mechanisms Behind Them (from Industry Sources)

    2026-10-07

    This note examines why buyers purchase consulting, what decides which firm they choose, and why those factors decide, drawing on industry sources: reports by the UK National Audit Office from 2010 and 2025, testimony by buyers at hearings of the Canadian House of Commons and the Auditor General of Canada's audit of contracts with McKinsey, four rounds of buyer surveys by the Hinge Research Institute, statements by corporate CFOs and executives, and a survey by Japan's IPA. It then proposes hypotheses. Academic research was used only to support the mechanisms, within the scope of the abstracts. The reasons for buying fall into four groups: expertise the organization lacks, a shortage of staff and the need for speed, an outside perspective and independent review, and assets no one else has (proprietary data and methods); in the UK, the same structure persisted from 2010 to 2025. In Hinge's surveys, the top deciding factor shifted from reputation (2013) to relevant experience and expertise (2018), to staff skills (2020), and to an existing relationship (2022), and price was never at the top. In the public sector, pre-qualified frameworks and time pressure favor incumbent suppliers. In Canada, about 70% of 97 contracts with McKinsey were non-competitive, the bid evaluations of 10 of the 28 competitive contracts did not contain enough information to support the selection, and about half of the sampled contracts lacked an explanation of the need or gap they were to address. From these sources, the note proposes four hypotheses: because quality cannot be verified before purchase, buyers use people, track record, and relationships as proxies; time pressure and the weight of procedures make pre-qualified and incumbent suppliers the default choice; outside independent review and unique data are chosen because decisions must be justified inside the organization; and because the internal capability gap is structural, buyers keep buying for the same reasons. Since 2024, among companies for which AI has made insourcing a real option, the staffing reason has weakened first and buyers have begun to ask suppliers to share the downside, while reliance on outside firms continues in work such as planning large workforce reductions.

  • Consulting Proposals and Competitive Bids: Their Forms, Why Buyers Award and Sellers Win, and What Generative AI Changes (from Industry Sources)

    2026-10-07

    Drawing on industry sources (procurement rules and public audits in the UK, the US, and Japan, industry surveys of RFP response, and statements by consulting executives on earnings calls), this note sorts out the forms of consulting proposals and the reasons why buyers award and sellers win competitive bids, and examines what has changed since 2024, when generative AI came into use. Academic research was not collected. The form of a proposal is set by the buyer's evaluation framework before the seller writes anything, and the UK's most advantageous tender (MAT) with quality weighted at 50 to 90%, the US tradeoff process, and Japan's proposal method and comprehensive evaluation all predate 2024. The three pricing models (time and materials, fixed price, and payment by results) were already described in a 2022 UK government playbook. Three things moved after 2024: the use of AI by those who write proposals (34%, then 68%, then 79% of RFP response teams), the buyers' rules (disclosure of AI use in the UK, a disclosure provision in a US OMB memorandum, and the Digital Agency's study of drafting specifications with generative AI), and the mix of pricing models (Accenture's fixed-price share reached about 60% in FY25, up about 10 points in three years, and about a third of McKinsey's revenue underwrites outcomes). Meanwhile, in the year AI use rose, the RFP win rate fell from 45% to 39%, and few teams name win rates as a benefit of AI. From these sources, the note proposes three hypotheses about what generative AI changes: the gap between proposal documents narrows and the weight of selection moves outside the document (reputation, existing relationships, interviews); the basis for pricing by time weakens and the share of productivity gains becomes the subject of negotiation; and payment by results spreads only to engagements with measurable outcomes. It sets these against what does not change (the evaluation frameworks, the three pricing models, and the decoupling of revenue from headcount that dates back to RPA).

  • How Should To-Be Principles for Business Processes Be Set? How to Write Them, Where They Come From, and Methods for Redesign

    2026-09-30

    This note draws on 16 academic sources and the TOGAF Standard to explain how design principles for the target (to-be) state are set when business processes are redesigned. Within this search, 'to-be principle' did not appear as an established term in the scholarly literature; the nearest concepts fall into four groups: architecture principles, design principles, redesign best practices, and principles of BPM itself. On how to write them, TOGAF recommends four parts (name, statement, rationale, implications) and five qualities of a good set (understandable, robust, complete, consistent, stable), while Gregor et al. (2020) offer a schema with aim, implementer, user, context, mechanism, enactor, and rationale. Haki and Legner (2021) examined 152 principles in the literature, set aside three kinds of 'nonprinciples' (goals phrased as principles, practices, and low-level standards), and grouped the remaining 45 into nine meta-principles. On where principles come from, the stage-activity framework for process change methods (Kettinger et al. 1997, extended with 98 methods by Gross et al. 2019) places business goals and performance goals before the analysis of the current process. Reijers and Limam Mansar (2005) ask that performance targets be made concrete along time, cost, quality, and flexibility, with attention to trade-offs among them. The current (as-is) process is studied to understand its purpose, but should not be the only source of ideas (Rosemann 2006). Techniques for generating ideas are unstructured, semi-structured, or structured (Vanwersch et al. 2016, 61 studies); structured examples include 29 redesign best practices (Reijers & Limam Mansar 2005) and a design space of 19 dimensions in six layers (Gross et al. 2021). In a practitioner survey, 94% had used task elimination (Limam Mansar & Reijers 2007), yet many best practices lack quantitative support. When AI is added, principles enter the system's 'frame' of goals and constraints, and people decide which parts the system may rewrite (Dumas et al. 2023). Within this search, no study tested whether writing principles itself improves the to-be design.

  • What Replaced What in Test Scoring and Item Writing

    2026-09-18

    A focused industry survey of two stages of Japanese mock exams and large-scale testing: scoring and item writing. What digital scoring replaced was not judgement but the movement of paper, the tallying, and the reconciliation. Partial-credit judgement stayed with people, and a new verification step appeared. The workforce moved from the marking room to the home, but piece-rate pay did not change. Item writing has not been replaced because its hours are not recorded, its review is layered, and its content cannot leak. Generative AI reaches a draft and no further, a position that is both where the technology lands and where the rules draw the line. The defects in AI-written items concentrate in ambiguity and difficulty control rather than duplication, and generating and verifying with the same model lets flawed items pass. Scoring is now counted as a market; item writing is not. METI's final results report was obtained at source: it breaks the saving down (10 to 30 minutes for 40 percent of teachers, no gain for 21 percent, longer for 15 percent), limits the demonstration to the first of three phases, and records nothing about scoring.

  • The Workload of Mock-Exam Operations and Where AI Actually Lands

    2026-09-18

    An industry survey of Japanese mock-exam operations, mapping each stage against where the load concentrates and where AI has actually arrived. The load concentrates on scoring constructed responses; digitalization shortened the stages around scoring (application, collection, digitization, return) rather than scoring itself. Scoring automation remains narrow, and digital scoring created a new verification job rather than removing scoring. Demand shrinks with the birth rate while supply tightens through teacher shortages and a contracting scoring labor market.

  • How do you measure a business analyst's skills?

    2026-09-15

    Japan's Digital Skill Standard added Business Analyst as a role in ver.2.0 (April 2026), so the job title now has a public definition. It has no ruler. The standard states that it sets no detailed level indicators, and instead assigns each skill an importance of a/b/c/d/z per role, which is a weighting rather than a proficiency scale. IIBA's BABOK Guide standardises six knowledge areas, thirty tasks and 29 underlying competencies, but leaves individual proficiency to the Competency Model's self-assessment scale. Its exams publish their weightings while withholding the passing score, and recertification counts activity hours. The public measuring tools are the Information Technology Engineers Examination and ITSS experience judging; the professional bodies lean on hours, exams and CDUs; the hiring market screens on kinds of experience. None of them measure outcomes. The standard gives vocabulary, not evidence.

  • Which standard can design and new-business work actually sit on?

    2026-09-14

    Listing the stages you covered on a portfolio needs a basis, and a self-made taxonomy has none. Japan's obvious candidates both fail: IPA froze Common Frame 2013 and says itself that it no longer aligns with ISO/IEC/IEEE 12207:2017, and the Digital Skill Standard ver.2.0 is a standard for personnel types and skills, not for stages. What held up were ISO 9241-210 (human-centred design, JIS Z 8530) and ISO 56002 (innovation management, JIS Q 56002), both of which state iteration in the standard text. Consulting has ISO 20700 but no JIS and no Japanese edition, marketing has no public standard for the work as a whole, and no public standard defines agile.