Shuichiro Ogawa
日本語

Notes · updated 2026-06-28

The AI Expectation-Capability Gap and UX — Academic Review (2026)

An integrated summary of 32 peer-reviewed papers and major preprints collected and organized through lightweight scoping. For the complete bibliography of each source, see “References” at the end (traceable with DOI/URL). The internal working ledger with provenance tracking and confidence ratings is at source/review/ai-expectation-gap-ux/papers.md (repository-internal, not published). For the industry side, see ai-expectation-gap-ux-industry. Collection lineage: scholarly-search-agent (exploration). Protocol: .claude/collection-protocol.md (zero fabrication, provenance tracking).

Review Metadata

  • Collection date: 2026-06-28 / Count: 32 (lightweight scoping = prioritizing avoidance of omissions)
  • Venue concentration: ACM CHI / CSCW / IUI / FAccT, IJHCS, Human Factors, HDSR, arXiv(cs.HC)
  • Confidence caveats: 2 unrefereed preprints (P20, P32). P22 is a Microsoft Research technical report (non-peer-reviewed, methodology disclosed, treated as T2).

TL;DR

Research on the AI expectation-capability gap, from the 2016 “Gulf of Expectation” problem statement (P02) to the 2024 design principles for generative AI (P29), has consistently aimed to clarify the conditions under which “users correctly grasp what AI can and cannot do and rely on it appropriately.” The findings consolidate into the following five areas.

  1. Trust calibration: Both overtrust and distrust impair performance. “Appropriate reliance” is the design goal (P01, P23, P24, P31).
  2. Mental models: The accuracy of users’ understanding of AI capability determines team performance. Displaying accuracy alone is insufficient; understanding error boundaries is the key (P05, P06, P13).
  3. Design interventions for expectation calibration: Advance disclosure of imperfection (P08), explanation of global properties during onboarding (P07), and cognitive forcing functions (P15) are empirically effective.
  4. The limits of explainability: Merely adding explanations does not reduce overreliance (P15, P17). Contextual explanation design attuned to task difficulty is necessary (P25).
  5. The backfire of anthropomorphism: Endowing AI with human-likeness raises expectations excessively and amplifies disappointment upon failure (P02, P14, P21).

Key Points by Area

A. Trust Calibration (7 papers)

Research in this area addresses not “how much AI should be trusted” but “the conditions for forming trust commensurate with AI’s trustworthiness.”

Lee & See (P01, 2004) laid the foundation by defining trust in automation along three axes: calibration (the correspondence between trust and trustworthiness), resolution (the differentiation of trust by context), and specificity (the granularity of the target); subsequent research has applied and extended this framework to AI. Wischnewski et al. (P23, CHI 2023) closely examined 96 papers out of more than 1,000 and systematized the measurement methods, interventions, and outcomes of trust calibration. Mehrotra et al. (P31, 2024), likewise in a systematic review of 96 papers, organized the trends in “interventions that foster appropriate trust” and pointed out that most interventions skew toward short-term experiments, leaving long-term effects unverified. Schemmer et al. (P24, IUI 2023) defined Appropriateness of Reliance (AoR) as a two-dimensional metric concept distinguishing overreliance from under-reliance, and measured the effect of adding explanations in an experiment with 200 participants. Jacovi et al. (P18, FAccT 2021) conceptually distinguished warranted trust from unwarranted trust and proposed a formalization of contractual trust.

Design implication: Raising trust itself is not the goal. The goal is forming trust commensurate with AI’s actual capability (calibration), and reducing overtrust and dispelling distrust require separate interventions.

B. Mental Models and Expectation Formation (6 papers)

This area addresses what understanding users construct of AI’s capabilities and limits.

Luger & Sellen (P02, CHI 2016) named the divergence between expectations of conversational agents and their actual behavior the “Gulf of Expectation.” Bansal et al. (P05, HCOMP 2019) demonstrated that mental models of error boundaries (where the AI errs), not just AI accuracy, determine the success or failure of human-AI teams. Yin et al. (P06, CHI 2019) showed that both stated accuracy and observed accuracy affect user trust, but that non-experts cannot correctly interpret the context of accuracy. Gero et al. (P13, CHI 2020) classified mental models of an AI agent in a cooperative game into three categories: global behavior, local behavior, and knowledge distribution. Grimes et al. (P14, DSS 2021) applied Expectation Violation Theory and showed in an experiment with 175 participants that the level of prior expectations sways satisfaction beyond actual performance. Yang et al. (P12, CHI 2020) analyzed the structure by which AI’s output complexity, capability uncertainty, and adaptivity make conventional UI design inherently difficult.

Design implication: Numerical accuracy displays alone are insufficient. Global explanations of a model’s properties that let users grasp “where this AI errs” improve the accuracy of mental models.

C. Design Interventions for Expectation Calibration (4 papers)

Empirical research on how to communicate AI’s imperfection and align user expectations with reality.

Kocielnik et al. (P08, CHI 2019) showed in two experiments with 550 participants that advance disclosure of error types (False Positive / False Negative) and accuracy displays are effective for expectation calibration. Cai et al. (P07, CSCW 2019) showed with 21 pathologists that advance explanation of a model’s global properties, strengths, limits, and design objectives is needed, and that this information need resembles understanding a colleague’s point of view. Kinney et al. (P30, Heliyon 2024) constructed a framework for expectation management prior to AI deployment from interviews with 14 participants in healthcare and education. Lobentanzer (P32, arXiv 2026) conducted a meta-analysis across three domains (software development, clinical documentation, and clinical decision support) and showed a quantitative divergence in software development: an expected 24% speedup against an actual 19% slowdown (a 43pp error).

Design implication: Advance disclosure not only of “what it can do” but of “what it cannot do” and “how it errs” is the core of expectation calibration. In healthcare and education, however, pre-deployment context setting matters more than onboarding.

D. The Relationship Between Explainability and Overreliance (7 papers)

This area addresses the impact of XAI (explainable AI) on user experience. Some findings in this area run counter to intuition.

Buçinca et al. (P15, CSCW 2021) showed that adding explanations alone does not reduce overreliance on AI. What proved effective were cognitive forcing functions (thought-prompting UI such as having users make their own judgment before seeing the AI’s answer). Poursabzi-Sangdeh et al. (P17, CHI 2021) reported from a preregistered experiment with 3,800 participants that manipulating model interpretability had no consistent effect on trust or on the detection of model errors. Vasconcelos et al. (P25, CSCW 2023) formalized with a cost-benefit framework, across five studies with 731 participants, that explanations reduce overreliance only when the task is difficult. Liao et al. (P11, CHI 2020) constructed the XAI Question Bank and typologized the questions users typically ask of AI (What-Why-How-What if). Ehsan et al. (P16, CHI 2021) pointed out that social transparency (engagement with organization and context) is needed in addition to algorithmic transparency. Liao & Vaughan (P28, HDSR 2023) identified the problem of missing transparency in the LLM era and presented a human-centered transparency roadmap. Passi & Vorvoreanu (P22, MSR 2022) synthesized roughly 60 papers and systematized the mechanisms, measurement methods, and mitigations of overreliance.

Design implication: The naive expectation that “explaining will calibrate trust” is not empirically supported. UI interventions that prompt thinking (cognitive forcing) and explanation delivery differentiated by task difficulty are necessary.

E. Anthropomorphism and Excess Expectations (3 papers)

The impact on the expectation gap of designs that endow AI with human-likeness.

Crolic et al. (P21, J. Marketing 2022) showed with field data and four experiments that for users in an angry state, chatbot anthropomorphism widens expectation violations and lowers customer satisfaction. Feine et al. (P10, IJHCS 2019) classified conversational agents’ social cues into four major categories and organized the relationship between human-likeness design and expectation formation. The findings of Luger & Sellen (P02) likewise show that the “intelligent assistant” image formed by film and media is a source of excess expectations.

Design implication: Anthropomorphism raises expectations. When AI capability falls short of those expectations, disappointment is greater than in non-anthropomorphized cases. The backfire is especially pronounced in contexts of anger or dissatisfaction.

F. Design Frameworks and Guidelines (5 papers)

Frameworks as implementation guidance.

The 18 guidelines of Amershi et al. (P04, CHI 2019) specify AI behavior across four stages (initial interaction, regular use, when wrong, and change over time) and became the foundation of industry guidelines (Microsoft HAX). Weisz et al. (P29, CHI 2024) presented six principles specific to generative AI (Mental Models / Appropriate Trust & Reliance / Generative Variability / Co-Creation / Imperfection / Responsible Design). That Generative Variability (different output every time) contradicts conventional UX expectations of consistency is a challenge unique to generative AI. Lai et al. (P26, FAccT 2023) systematized the experimental design space of human-AI decision-making along three dimensions: task risk, expertise, and subjectivity. Cheng & Chouldechova (P27, CHI 2023) demonstrated that algorithm aversion is mitigated by granting control over outcomes. Liao & Varshney (P20, 2021) organized a framework for designing XAI in a human-centered way.

Unresolved Issues

  1. Absence of long-term effects: Nearly all empirical studies are one-off experiments. The long-term effects of trust calibration interventions are unverified (noted by P23, P31).
  2. Non-determinism of generative AI: For generative AI, which returns different outputs for the same input, it is unclear whether conventional trust calibration theory applies as-is (P29).
  3. Extension to agentic AI: Expectation management for agentic AI that takes actions differs qualitatively from conversational AI, where viewing outputs is the primary mode (P32).
  4. Cultural differences: The majority of the collected literature targets North American and Western European users; differences in expectation formation across cultural regions remain unexplored.

References

← All Notes · Home