Shuichiro Ogawa
日本語

Notes · updated 2026-09-11

Evidence for Adjudicating the Six Design System Conflicts

The “Unresolved Conflicts” table in design-system-practices writes, for each of the six conflict axes, the “evidence needed to adjudicate” in a single line. That single line is in fact made up of a combination of several measurements. “A comparison measuring, in organizations operating multiple systems, the number of duplicate components and the time to propagate a fix” splits into separate measurements: how duplicates are counted, whether that count is stable against the threshold of the adjudication, how the time until a fix propagates is measured, and whether the number of systems in operation can be placed as an independent variable.

If even one is missing, the conflict cannot be adjudicated. This note decomposes the six single lines into units of measurement and determines how far measurement exists and where it is empty.

Piece: a unit obtained by decomposing an adjudication criterion, which cannot be divided further and can be treated as a single measurement.

The evidence was sought in three layers.

  • A: measurement that takes design systems directly as its object.
  • B: measurement carried out in the context of design and HCI.
  • C: measurement in adjacent fields (software engineering, software ecosystem research, organizational research, public audit).

Even within the same single line, some pieces are filled by A and others only by C.

Conflict axisAdjudication criterionPieces that are filledPieces that are emptyClosest measurement
Single system or multiple systemsA comparison measuring, in organizations operating multiple systems, the number of duplicate components and the time to propagate a fixP1 number of duplicate components, P2 stability of the duplicate judgment, P3 time to propagate a fixP4 a comparison placing the number of systems as an independent variable, P5 cross-organizational and longitudinal comparisonC (the only A measurement is a single qualitative study, and it measures neither counts nor time)
Is consistency the goal or the meansA comparative study that controls the level of consistency and measures task performance and exploration behavior at the same timeP1 control of the level of consistency, P2 task performance, P5 transfer (performance side only)P3 exploration behavior, P4 simultaneous measurement of performance and explorationB (A is only a single comparison of DS reuse)
Should tokens be exhaustive or minimalA controlled experiment measuring the relationship between token count and generation quality on the same taskP2 manipulation of the amount of information, P3 measurement of quality, P4 context length, P5 number of fragments, P6 specificity and compression, P7 dose-response, P8 consumption costP1 manipulation of vocabulary size, P9 vocabulary size and quality taking the DS directly as the objectC (the DS-context comparison is B, and measurement of vocabulary size exists in neither A nor C)
Govern or leave to autonomyA longitudinal comparison of organizations with different levels of governance (adoption rate and the number of external contributions over time)P1 operational definition of governance strength, P3 measurement of boundary-crossing contributions, P4 longitudinal designP2 measurement of adoption rate, P5 parallel comparison of control groups, P6 identification of causalityC (the direct DS longitudinal evidence is only single-point descriptions from industry)
Rename and reorganize, or preserve compatibilityCases tracking, in organizations that chose to preserve compatibility, the survival period of old tokens and the cost of fixesP1 survival period of old names, P2 reaction rate to deprecation, P3 fix cost of migration, P4 rate of actual harm when compatibility was broken, P5 operation of deprecation markersP6 measurement taking the DS directly as the objectC (A is zero, and industry has only estimates and plans)
Does using a DS make you accessibleA comparison of accessibility audit results between services that adopted the DS and services that did notP1 identification of adoption, P2 control design, P3 audit results, P5 control of confoundersP4 implementation fidelity, P6 longitudinal measurement before and after adoptionA (at the level of DS artifacts) and C (at the service level, automated audits only)

Of the 37 pieces, 25 have some form of measurement and 12 have no research that could be confirmed.

Have Duplicate Component Counts and Propagation Times Been Measured?

Adjudication Criterion

A comparison measuring, in organizations operating multiple systems, the number of duplicate components and the time to propagate a fix

What Must Be Measured to Fill It

  • P1 number of duplicate components: count, with a fixed decision rule, how many components with the same role exist across multiple systems.
  • P2 stability of the duplicate judgment: confirm that the count does not change when the threshold of the judgment is moved.
  • P3 time to propagate a fix: measure the time until a fix made in one system reaches the other.
  • P4 a comparison placing the number of systems as an independent variable: place single-system and multiple-system operation side by side and attribute the difference in results to the number in operation.
  • P5 cross-organizational and longitudinal comparison: track multiple organizations under the same definition and subtract organization-specific circumstances.

How Far Measurement Has Reached

The counting of duplicates has been measured in adjacent fields. In clone detection across 17 web applications, the clone rate fell within a range of 17-63%, and no control is described (Rajapakse & Jarzabek 2005, DOI 10.1007/11531371_35). A study that manually classified 493k page pairs concluded that no algorithm detects all functional near-duplicates without sacrificing coverage, even when the threshold is optimized (Yandrapally, Stocco, Mesbah 2020, DOI 10.1145/3377811.3380416). A study that divided one industrial system into 8 components and 10 teams measured the number of clones introduced per team (Sundelin, Gonzalez-Huerta, Torkar, Wnuk 2024, DOI 10.1007/s10664-024-10598-7). The observation period and the number of review participants have not been obtained [requires primary verification].

The closest thing to propagation time is package update delay. A study comparing 890 shared packages across 6 distributions defined two indicators, update delay and time lag, and showed that 1 in 10 was in an outdated state (Legay, Decan, Mens 2021, DOI 10.1109/SoHeal52568.2021.00008). For npm, an analysis of 120K packages, 1.4M releases, and 8M dependencies showed that technical lag grows over time and that adherence to semver mitigates it (Decan, Mens, Constantinou 2018, DOI 10.1109/ICSME.2018.00050).

The only qualitative study taking design systems directly as its object supported a bottom-up model, but it measured neither the number of duplicates nor propagation time (Lamine & Cheng 2022, DOI 10.1007/s10664-022-10181-y). The correspondence between organizational structure and technical structure held in 69% of studies in a review of 102 empirical studies, and weakens in the order of within-firm, between-firm, and open community (Colfer & Baldwin 2016, DOI 10.1093/icc/dtw027).

Measurements That Could Not Be Confirmed

No peer-reviewed study placing the number of systems in operation as an independent variable was found with 9 search terms. There is also no peer-reviewed study measuring the time to propagate a fix for UI components, and none of the four industry surveys checked at the question level (zeroheight 2026 and 2025, Sparkbox 2020 and 2021) has a question on duplicate counts or propagation time. The industry figures are a single case: an average of 3.7 weeks from specification to implementation and 46.3% inconsistency, and the questions and the collection period are undisclosed [requires primary verification] (UX Tools, https://www.uxtools.co/survey/design-systems/overview).

Transfer from Adjacent Fields

Transfer holds when the object of measurement can be narrowed to the part shared by both groups. Limiting to the 890 shared packages made the comparison possible (Legay, Decan, Mens 2021), and if the mechanism by which update delay depends on adherence to convention is the same, the measurement of propagation time can be applied in the same form (Decan, Mens, Constantinou 2018). Because the correspondence between organizational structure and technical structure holds most strongly within a firm, the situation of operating multiple systems inside one firm easily satisfies this condition (Colfer & Baldwin 2016).

Transfer breaks down when the counting of duplicates depends on the procedure. There is no method that detects all near-duplicates even with an optimized threshold, and the 17-63% range of the clone rate, absent controls, includes factors other than the number of systems (Yandrapally, Stocco, Mesbah 2020, Rajapakse & Jarzabek 2005). The correspondence with organizational structure weakens in open communities, and the premise collapses for a design system open to the outside (Colfer & Baldwin 2016).

Can the Level of Consistency Be Controlled While Measuring Performance and Exploration at the Same Time?

Adjudication Criterion

A comparative study that controls the level of consistency and measures task performance and exploration behavior at the same time

What Must Be Measured to Fill It

  • P1 control of the level of consistency: manipulate the degree of agreement as a level or an attribute and hold other conditions fixed.
  • P2 task performance: measure performance indicators such as completion rate, time required, and error count.
  • P3 exploration behavior: measure the discovery of unused features, the spread of attempts, and the sequence of operations.
  • P4 simultaneous measurement of performance and exploration: measure both in the same participants and confirm whether an improvement in one harms the other.
  • P5 transfer between applications and between tasks: measure whether an operation learned on one screen works on another.

How Far Measurement Has Reached

Control of the level of consistency has accumulated findings in the context of design and HCI. An experiment that manipulated consistency across two applications with two attributes (the appearance of the display screen and the syntax of the language) measured the number of actions and accuracy with n=114 (Satzinger & Olfman 1998, DOI 10.1080/07421222.1998.11518190). The task content, the timing measurement, and whether blinding was used have not been obtained [requires primary verification]. An experiment that decomposed the consistency of web pages into three elements, physical, communicational, and conceptual, assigned a total of 40 people in a four-group between-subjects design and measured error rate, time required, and satisfaction (Ozok & Salvendy 2000, DOI 10.1080/001401300184332). No blinding is described.

There is only one comparison taking design systems directly as its object: it compared a GUI reused as-is with a GUI modified along design principles, with n=8 crane operators (Opsahl & Sitompul 2025, DOI 10.1109/CHIuXiD68326.2025.11323756).

The results on the performance side do not converge. Inconsistency in screen appearance raised accuracy and satisfaction, while inconsistency in the command language increased the number of actions (Satzinger & Olfman 1998). For web pages, the hypotheses on performance and satisfaction were not supported (Ozok & Salvendy 2000), and in crane operation the achievement rate was 100% in both conditions with no difference in time, while satisfaction was higher for the modified version (Opsahl & Sitompul 2025).

Transfer was measured only on the performance side, and the prediction that the ease of learning a command language depends on the consistency structure of the rules was supported in a syntax-learning experiment (Payne & Green 1989, DOI 10.1016/s0020-7373(89)80011-2). The sample, the independent variables, the dependent variables, and the controls have all not been obtained [requires primary verification].

Measurements That Could Not Be Confirmed

No study combining the manipulation of consistency with the measurement of exploration behavior was found with 7 search terms. Neither a comparative study controlling the level of consistency in a design system nor a measurement of exploration behavior can be confirmed. No study measured both performance and exploration in the same participants; the count is zero. Industry also has no comparison controlling consistency, and the Sparkbox comparison using Carbon, regarded as the only controlled experiment, has the presence or absence of a DS as its manipulated variable, and its primary URL has not been identified either [requires primary verification]. The closest literature as a method for measuring exploration behavior (Thomas 1996, SIGCHI Bulletin) is not included because Crossref has no matching record and the bibliography cannot be confirmed.

Transfer from Adjacent Fields

Transfer holds when the correspondence between operation and display is fixed and the degree of agreement can be treated as a continuous quantity. The spatial correspondence of stimulus and response codes was formulated in terms of information transmission rate (Fitts & Seeger 1953, DOI 10.1037/h0062827), and three experiments showed that the gain from a device model is limited to cases where the operating procedure can be inferred directly (Kieras & Bovair 1984, DOI 10.1016/s0364-0213(84)80003-8). The number of participants and the number of trials have not been obtained [requires primary verification].

Transfer breaks down when the signs of the things lumped together as “consistency” do not agree. Because inconsistency in screen appearance raised accuracy and satisfaction while inconsistency in the command language increased the number of actions (Satzinger & Olfman 1998), combining appearance and command language into one variable mixes two things that point in opposite directions. Whether the consistency structure of rules can be mapped onto the use of design system components cannot be confirmed, because the sample and controls have not been obtained (Payne & Green 1989).

Has the Relationship Between Token Count and Generation Quality Been Measured?

Adjudication Criterion

A controlled experiment measuring the relationship between token count and generation quality on the same task

What Must Be Measured to Fill It

  • P1 manipulation of vocabulary size: in the same design system, change only the number of token definitions.
  • P2 manipulation of the amount of DS information placed in the context: change how much of the DS documentation goes into the prompt.
  • P3 measurement of quality: score whether the generated artifact complies with the DS.
  • P4 increasing only the context length: pad with tokens unrelated to the task and change only the length.
  • P5 increasing only the number of fragments: hold the overall length fixed and change only the number of documents or examples.
  • P6 raising or compressing the specificity: change whether the same content is written in more detail or more briefly.
  • P7 the shape of the dose-response: estimate whether quality moves monotonically with the increase or decrease, or whether there is an optimum.
  • P8 simultaneous measurement of consumption cost: measure token consumption within the same trial.
  • P9 the relationship between vocabulary size and quality taking the DS directly as the object: satisfy P1 and P3 in the same experiment.

How Far Measurement Has Reached

There is one comparison that passes a design system as context. In a study comparing three ways of passing DS context on the same task, the registry approach had the highest compliance rate at 95.08% (±4.77) and the smallest variance, but because the amount and the structure of the information change at the same time, the effect of amount alone cannot be separated (Cha, Jo, Shin, Seo 2026, 6 screens × 3 repetitions, https://pure.seoultech.ac.kr/en/publications/design-system-compliant-user-interface-generation-with-llm-agents/). The scoring procedure, the raters, and blinding have not been obtained [requires primary verification].

In adjacent fields, controlled experiments that manipulate the amount have accumulated. In an experiment that changed only the amount, type, and position of padding while keeping the task the same, reasoning performance declined at input lengths far shorter than the technical limit, and next-token prediction accuracy was inversely correlated (Levy, M., Jacoby, A., Goldberg, Y. 2024, 5 levels of input length from 250 to 3,000 tokens, DOI 10.18653/v1/2024.acl-long.818). Even with perfect retrieval controlled, input length alone degraded performance by 13.9-85%, and the degradation remained whether irrelevant tokens were replaced with whitespace or masked (Du, Y., Tian, M., Ronanki, S. et al. 2025, DOI 10.18653/v1/2025.findings-emnlp.1264). In an experiment that fixed the context length and the position of the supporting documents and changed only the number of documents, most models degraded by up to 20%, while Qwen2.5 remained stable (Levy, S., Mazor, N., Shalmon, L., Hassid, M., Stanovsky, G. 2025, arXiv:2503.04388). In an experiment that increased the number of examples up to 8,192, XSum degraded beyond 50 shots and MATH declined after peaking at 125, and it was also shown that next-token prediction loss keeps falling and therefore is not a proxy indicator for quality (Agarwal, R., Singh, A., Zhang, L. M. et al. 2024, arXiv:2404.11018). Raising specificity increased pass@1, by up to +0.30 on ParEval-OMP [requires primary verification] (Zi, Y., Menon, H., Guha, A. 2025, DOI 10.18653/v1/2025.ijcnlp-long.128), and compressing the encoding of constraint blocks to cut tokens in the constraint portion by about 71% and the whole prompt by 25-30% produced no significant difference in compliance rate (Tang, H. 2026, 11 models, more than 830 calls, Cliff’s δ < 0.01, arXiv:2604.07192).

Quality is often measured by automated scoring (Cui, Y. 2025, 1,000 tasks, 20 domains, 19 models, arXiv:2505.09027). This paper carries an arXiv administrative note pointing to text overlap with another paper, and the number of models in the conclusion (18) does not match the abstract (19).

The three industry cases point in different directions. In making its own DS machine-readable for AI, usage across all four tasks fell from 19M to 10.4M and fell 39% in the design-to-code scenario, but the authors themselves write that quality was judged by their own engineers and was neither blinded nor automatically scored (BILL, https://www.bill.com/engineering/how-we-made-our-design-system-bilingual). In 27 cases comparing the presence and absence of Code Connect, the median task used 29.5% fewer tokens and 19.6% less time, and quality rose from 2 to 3 on a 1-4 scale, but the judgment was made by an LLM judge, and the model name and whether there was human cross-checking are undisclosed [requires primary verification] (Figma, https://www.figma.com/blog/the-benefits-of-code-connect-in-mcp/). In an in-house experiment comparing four ways of passing context, average tokens were 4.20M with no context, 3.75M with MCP, and 7.21M with DESIGN.md, and the source itself writes of the DESIGN.md condition that it “required ~92% more tokens” and also states that the variance between runs grew about 2.7 times (Atlassian, single task, quality not measured, source is DS-15 in the existing corpus source/review/design-system-practices/industry.md).

Measurements That Could Not Be Confirmed

No controlled experiment placing the vocabulary size (number of definitions) of design tokens as an independent variable was found with 14 search terms or in OpenAlex searches, and no ablation of vocabulary size exists. There is a study dealing with the direct manipulation of design tokens, but visual tokens are a different concept from the token count of a language model, and it does not measure the relationship between vocabulary size and generation quality (Brickify 2025, DOI 10.1145/3706598.3714087). The nearby industry cases also do not take the relationship between token count and quality as an object of measurement. A benchmark that publishes its scores states that the judgment is blind to the treatment condition, but “898 runs” and “+23-26pt” do not exist in the README [requires primary verification] (Open Design System Bench, https://github.com/christophhdesign/open-design-system-bench).

Transfer from Adjacent Fields

Transfer holds when it can be controlled that the increment is unrelated to the task. A design that changed only the amount, type, and position of padding separates the effect of length from the content (Levy, M., Jacoby, A., Goldberg, Y. 2024), and a design that changed only the number of documents fixes the length and the position (Levy, S. et al. 2025). The fact that the degradation remains even with perfect retrieval controlled shows that the cost of length itself remains after retrieval failure is removed, and that portion can appear in the same form in the amount of DS context (Du, Y. et al. 2025).

Transfer breaks down when the increment is not a document to be interpreted but an executable part. The registry approach produced the highest quality with the most tokens (Cha, Jo, Shin, Seo 2026), and within the same study a reversal also occurred in which average tokens relative to no context decreased with MCP and increased with DESIGN.md (Atlassian). Using loss as a proxy indicator for quality misreads the shape of the dose-response (Agarwal, R. et al. 2024), and raising specificity and compressing take opposite signs or become null (Zi, Y. et al. 2025, Tang, H. 2026).

Can Organizations with Different Levels of Governance Be Compared Longitudinally?

Adjudication Criterion

A longitudinal comparison of organizations with different levels of governance (adoption rate and the number of external contributions over time)

What Must Be Measured to Fill It

  • P1 operational definition of governance strength: define the strength of governance as a level or a type.
  • P2 measurement of adoption rate: measure the share of target services that adopted the DS.
  • P3 measurement of the number of external contributions: measure the count and the trend of contributions from outside the organization.
  • P4 longitudinal design: follow the same organization at multiple points in time.
  • P5 control: place organizations with different governance strength side by side at the same time.
  • P6 identification of causality: show, with confounders removed, whether a change in governance moved the result.

How Far Measurement Has Reached

What corresponds to governance strength has been measured as the separation of organizational structure. A multi-case analysis of 3 organizations concluded that the separation of product divisions from the platform organization is the root cause of delivery delays, rising defect rates, and component duplication (Riehle, D., Capraro, M., Kips, D., Horn, L. 2016, DOI 10.1109/TSE.2016.2554553). The period, the data sources, and the quantitative indicators have not been obtained [requires primary verification]. For design systems, there is a study that organized the barriers to adoption in a single organization through 76 interviews, and the adoption rate remains a qualitative statement, “uptake has stayed limited” [requires primary verification] (Richter, K. S., Rüssel, J., Karczewski, K. 2025, DOI 10.1145/3706599.3706705). A method for measuring boundary-crossing contributions was defined, and at one company 47.9% of all contributions were between organizational units and 42.2% were between units responsible for different products, but what is measured is internal boundary crossing, not contributions from an external community (Capraro, M., Dorner, M., Riehle, D. 2018, DOI 10.1145/3196398.3196417).

There are three longitudinal designs in adjacent fields. A report that descriptively followed a single company’s incentive scheme states that the number of projects and the number of participants increased, but the figures appear only in a chart and not in the text, and the authors themselves state that determining causality requires a randomized controlled trial (Dey, T., Jiang, N., Fitzgerald, B. 2022, DOI 10.1109/MS.2022.3192647). A longitudinal log analysis showing that the reaction to governance lowered the retention of new participants has no obtainable period, sample, or statistical method [requires primary verification] (Halfaker, A., Geiger, R. S., Morgan, J. T., Riedl, J. 2013, DOI 10.1177/0002764212469365). A study that measured the gap between a declared decentralization of governance and the actual state of development showed concentration with a weighted Gini of 0.82, the top single contributor at 24%, and roughly half of the core at 98.1% (Destefanis, G., Xu, J., Bartolucci, S. 2026, 6,741 GitHub actions, September 2020 to February 2025, DOI 10.1016/j.is.2026.102695).

The industry adoption ledgers are single points in time or estimates. At IBM Cloud, 116 of 155 services adopted Carbon 10, and the remaining 21 are said not to adopt under approved exceptions (IBM Carbon, as of January 2021, https://v10.carbondesignsystem.com/case-studies/consistency-in-the-cloud/); the NPS improvement of 57% and the preference of 88% on the same page have their definitions, comparison targets, samples, and questions undisclosed [requires primary verification]. The GOV.UK Frontend version tracker is a single-point value in which 6 services have already upgraded to version 5, with the denominator undisclosed (X-GOVUK, 2023-12-20, https://x-govuk.org/posts/govuk-services-frontend-stats/), and GDS’s “more than 1,200” is an estimate with no stated sample, calculation method, or reference date (GDS, https://accessibility.blog.gov.uk/2024/01/11/get-to-wcag-2-2-faster-with-the-gov-uk-design-system/).

Some organizations have measurement infrastructure, but the figures are not published. GitLab only states that it tracks 38 foundational components, and the “88%” and “2256/16728” on the page are format examples, not measured values (GitLab, https://gitlab.com/gitlab-org/gitlab-design/-/issues/2495). Brevo likewise does not put a measured adoption rate in the text, and the denominator is undisclosed (Brevo, https://engineering.brevo.com/how-to-track-design-system-adoption/).

Measurements That Could Not Be Confirmed

On the peer-reviewed side, nothing measuring the adoption rate can be found. There are zero studies validating the soundness of practitioner tools that aggregate adoption rates, zero comparing organizations with different governance strength in parallel, and zero identifying causality. On the industry side as well, no panel survey follows the same organization, and no DS publishing the trend in external contributions can be found. The GOV.UK DS leaves the increase in contributions as “TBD” in its 2025 community objectives, so the target value itself is unset (GOV.UK Design System team, https://team-playbook.design-system.service.gov.uk/community/community-objectives/), and the absence of measurement is pointed out in adjacent fields as well, with a systematic review concluding that measuring the business impact of inner source is unstudied (Buchner, S., Riehle, D. 2023, DOI 10.1145/3611648).

Transfer from Adjacent Fields

Transfer holds when the definition of the boundary and the logs can be matched. The measurement of contributions crossing organizational boundaries holds in the form of fixing the definition of the boundary first and then following the changes that cross it (Capraro, M., Dorner, M., Riehle, D. 2018). For an object where both the declaration document and the contribution log can be obtained, the gap between the declared governance and the actual state can be measured (Destefanis, G., Xu, J., Bartolucci, S. 2026).

Transfer breaks down when the kind of contribution being measured changes. What the boundary-crossing contribution measurement counts is internal boundary crossing, and replacing it with the number of external contributions changes the object itself (Capraro, M., Dorner, M., Riehle, D. 2018). A before-and-after comparison does not constitute identification of causality (Dey, T., Jiang, N., Fitzgerald, B. 2022), in the single-protocol case the HHI and the statistical test have not been obtained (Destefanis, G., Xu, J., Bartolucci, S. 2026), and the longitudinal analysis dealing with the reaction to governance cannot have its mechanism copied as-is because its period, sample, and statistical method have not been obtained [requires primary verification] (Halfaker, A. et al. 2013).

Have Survival Periods and Fix Costs Under Preserved Compatibility Been Tracked?

Adjudication Criterion

Cases tracking, in organizations that chose to preserve compatibility, the survival period of old tokens and the cost of fixes

What Must Be Measured to Fill It

  • P1 survival period of old names: measure the period during which a deprecated name continued to be used in practice.
  • P2 reaction rate to deprecation: measure how many users migrated after the announcement.
  • P3 fix cost of migration: measure the amount of change and the effort required for migration.
  • P4 rate of actual harm when compatibility was broken: measure how much a breaking change affected the side using it.
  • P5 operation of deprecation markers: confirm whether the announcement, the presentation of a replacement, and the removal timing were observed.
  • P6 measurement taking the DS directly as the object: measure P1 through P5 with design tokens.

How Far Measurement Has Reached

The closest thing to the survival period is lagging time in package ecosystems. Android evolved at an average of 115 API updates per month, 28% of client API references were to old versions, the median lagging time was 16 months, and the propagation time was 14 months (McDonnell, Ray, Kim 2013, DOI 10.1109/ICSM.2013.18). The cycle from deprecate to replace to remove has also been observed across revisions, but the number of APIs, the number of apps, and the period have not been obtained outside the body text [requires primary verification] (Li, Gao, Bissyandé, Ma, Xia, Klein 2020, DOI 10.1007/s10664-019-09764-z).

The reaction to deprecation is weak. In a study that followed more than 25,000 clients of 5 Java APIs and 60 JDK clients, clients did not raise their versions, and when they did react they tended to choose removal rather than the recommended replacement (Sawant, Robbes, Bacchelli 2018, DOI 10.1007/s10664-017-9554-9). The observation period and the tracking method have not been obtained [requires primary verification]. For npm, at the time of collection 27% of clients had directly adopted a deprecated release (Cogo, Oliva, Hassan 2022, DOI 10.1109/TSE.2021.3055123).

Fix costs have been quantified as the diff of a migration. In a study that manually classified 335 Python migrations, the change per migration had a median of 8 lines and used 7 APIs, the maximum was 758 lines, and only 16.4% could be handled by a simple replacement of a function call (Islam, Jha, Akhmetov, Nadi 2024, DOI 10.1145/3643731). The rate of actual harm when compatibility is broken has also been measured: in 7 years of Maven Central, about one third of releases contained at least one breaking change, and that proportion was the same for minor and major (Raemaekers, van Deursen, Visser 2014, DOI 10.1109/SCAM.2014.30). In another large-scale analysis, 83.4% conformed to semver, and only 7.9% of all clients were affected (Ochoa, Degueule, Falleri, Vinju 2022, 119,879 library updates and 293,817 clients, DOI 10.1007/s10664-021-10052-y). Impact occurs even with non-major updates: 11.7% of client packages and 13.9% of releases were affected, and manifest breaking changes were 43.75% each in minor and patch (Venturini, Cogo, Polato, Gerosa, Wiese 2023, DOI 10.1145/3576037). The days until recovery (134 days for clients, 7 days for providers) cannot be confirmed in the abstract [requires primary verification].

The operation of deprecation markers is not observed: in Maven Central removals are carried out without tags (Raemaekers, van Deursen, Visser 2014), and for npm, 66% of the packages with a deprecated release deprecated all of their releases, so users could not move to a replacement, and 31% of partial deprecations had no replacement (Cogo, Oliva, Hassan 2022).

Industry has plans but no tracking results: Backbase announces that it will remove deprecated tokens in the 2025.04 release and allow a migration period of 2 years including long-term support (page updated 2025-12-04, https://designsystem.backbase.com/latest/design-tokens/value-and-cost-of-migration-YSNi7jJT). Migration costs are given in forms such as “5 days and 4 hours per custom journey”, but all are estimates carrying the label Estimated and are not measurements.

Measurements That Could Not Be Confirmed

No measurement of renaming and deprecation taking design systems directly as its object was found with 7 search terms. In the context of design and HCI as well, no peer-reviewed study measuring DS change management was found. Searching OpenAlex for research taking design system deprecation as its subject produced 364 hits with no match, and no ex-post measurement of the survival period of old tokens and fix costs can be found either. The first-hand testimony from Microsoft Fluent states, about when v9 will reach parity with v8, “How long this will take I can’t tell you, that’s a question that has a lot of variables”, and is not an answer to a question about the side-by-side period (2022-06-24, https://github.com/microsoft/fluentui/discussions/23183). A statement that the speaker is employed by Microsoft does not appear on the page [requires primary verification]. Among industry surveys as well, State of Design Tokens 2024 publishes only its section structure, and whether there are questions on token retirement or migration periods cannot be determined.

Transfer from Adjacent Fields

Transfer holds when dependencies are managed by version identifiers and references can be taken statically. Android’s lagging time was obtained by matching API versions against client references to produce a median of 16 months (McDonnell, Ray, Kim 2013), and the fix cost of migration can also be counted in the same unit as long as the change appears as a diff (Islam, Jha, Akhmetov, Nadi 2024). If the deprecation cycle can be observed across revisions, the survival period can be measured as a distribution (Li, Gao, Bissyandé, Ma, Xia, Klein 2020).

Transfer breaks down when the identifier does not function as a declaration of compatibility. If the proportion of releases containing breaking changes is the same for minor and major, the premise of reading the survival period from the version identifier collapses (Raemaekers, van Deursen, Visser 2014). When no migration target is defined, the survival period becomes not “the time during which migration did not happen” but “the time during which migration could not happen” (Cogo, Oliva, Hassan 2022). Even with an announcement, if the side reacting chooses removal rather than the recommended replacement, the announcement does not lead to migration (Sawant, Robbes, Bacchelli 2018).

Can Audit Results for DS-Adopting and Non-Adopting Services Be Compared?

Adjudication Criterion

A comparison of accessibility audit results between services that adopted the DS and services that did not

What Must Be Measured to Fill It

  • P1 identification of adoption: identify which service uses which design system.
  • P2 identification of the non-adopting group and control design: place non-adopting services with matched conditions side by side.
  • P3 audit results: apply the same audit to both groups.
  • P4 implementation fidelity: measure whether the adopting service actually used those components.
  • P5 control of confounders: remove differences such as size, age, and technology category.
  • P6 longitudinal measurement before and after adoption: measure the same service before and after adoption.

How Far Measurement Has Reached

There is an evaluation of design system artifacts themselves. A study that evaluated the design systems of four major companies against 46 WCAG 2.2 criteria (selected from 91 criteria for the portion a DS can cover) counted non-conforming components by criterion, giving 7 for 1.4.3, 12 for 2.5.5, and 6 for 4.1.2 (Diniz & Gama 2024, DOI 10.5753/sbes.2024.3566). This is an evaluation at the level of artifacts and is not the audit result of a service that adopted one.

The closest thing at the service level is a large-scale automated audit that takes technology category into account. In the UI frameworks category, adopting pages scored 0.655 and non-adopting pages 0.672 (Martins & Duarte 2024, A3 indicator where lower is better, p<0.001, 1,072,937 pages, DOI 10.1007/s10209-023-01010-0); the study as a whole covers 2,884,498 pages and 166,311 sites, with an average of 30 errors per page and 0.5% of pages with zero errors. However, the technology difference within the same category is at most 0.196, exceeding the category effect of 0.017, and what is being compared is adoption and non-adoption of a technology category, not adoption of a design system.

Results close to an organization of the confounders also exist in public surveys. In an assessment aggregating the self-reports of government agencies, the implementation index was 1.96 out of 5, policy 3.04, acquisition 3.44, and testing and remediation 2.00, and the degree of implementation governs conformance while the size of the agency does not (U.S. GSA, FY2025, 60 of 212 agency responses submitted to the government-wide analysis, https://www.section508.gov/manage/section-508-assessment/).

Measurements That Could Not Be Confirmed

No comparison of accessibility audits between DS adopters and non-adopters was found with 7 search terms, and there is no study measuring before and after adoption or measuring implementation fidelity. On the public monitoring side as well, no audit includes DS adoption as a variable. The term design system itself does not appear in the UK GDS monitoring report (1,203 websites and 21 mobile apps, 29,787 issues, 70% conformance rate on simplified testing, period 2022-01-01 to 2024-09-01, https://www.gov.uk/government/publications/accessibility-monitoring-of-public-sector-websites-and-mobile-apps-from-2022-to-2024/accessibility-monitoring-of-public-sector-websites-and-mobile-apps-from-2022-to-2024), and the Netherlands dashboard has no field for DS or CMS (9,101 sites, statuses A 874, B 4,414, C 537, D 2,359, E 917, https://dashboard.digitoegankelijk.nl/). Japan’s Ministry of Internal Affairs and Communications (MIC) survey also does not have DS adoption as a variable; in fiscal 2023, 48.8% of the 1,208 organizations’ official sites that miChecker could analyze had pages with A/AA problems, and by status of guideline use the figure increases monotonically from 29.2% (in use) to 48.4% (unaware of them) (https://www.soumu.go.jp/info-accessibility-portal/webaccessibility/assets/documents/webaccessibility/chosa_houkoku_gaiyou_r5.pdf). No public register that can join an organization’s DS adoption to audit results has been found either. WebAIM’s aggregation by technology category cannot be cited as an effect of DS adoption. The same report records Bootstrap at 63.3 under Web frameworks and React at 43.5 and Astro at 9.0 under JavaScript frameworks (February 2026, top 1,000,000 home pages, an average of 56.1 errors per page, WCAG 2 failures on 95.9% of pages, https://webaim.org/projects/million/), but the classification is a technology signature detectable from the HTML, not the presence or absence of DS adoption, and services using an in-house design system or GOV.UK Frontend appear in no category; the source itself states that attribution to a technology is not possible. The DS side also acknowledges its own limits, writing that using accessible components does not guarantee an accessible service, and its VPAT covers only its own 44 components and does not compare with adopters (USWDS, source is the existing corpus source/review/design-system-practices/industry.md).

Transfer from Adjacent Fields

Transfer holds when the detection range of automated audits can be determined first. A benchmark of 6 tools measured ceilings of coverage at 50%, completeness at 14-38%, and correctness at 66-71% (Vigo, Brown, Conway 2013, test page set and ground truth not obtained [requires primary verification], DOI 10.1145/2461121.2461124), and with matched categories and a large sample, small differences in technology adoption can be extracted (Martins & Duarte 2024). The finding that the degree of implementation governs conformance while agency size does not can be used in designing the control group (U.S. GSA, FY2025).

Transfer breaks down when the granularity of the category is coarser than DS adoption. If the technology difference within the same category is at most 0.196 and exceeds the category effect of 0.017, the difference within the category overlaps with the effect of the DS (Martins & Duarte 2024). The source itself qualifies that aggregation by technology category cannot be attributed (WebAIM 2026), and an evaluation at the artifact level is not the audit result of a service that adopted it (Diniz & Gama 2024). As long as the coverage of automated audits is at most 50%, failures that are not detected do not appear in the comparison (Vigo, Brown, Conway 2013).

What Is Missing in Common Across All Six

Apart from the individual empty pieces, there are patterns that appear across the six criteria.

The first is that the operational definition of the treatment splits a continuous quantity in two. A quantity that could be moved is cut into two values, such as adopted and not adopted, single and multiple, exhaustive and minimal, and both of the two groups after the cut have an internal range. This form appears in all of C1 (the number of systems in operation), C2 (agreement or disagreement), C3 (vocabulary size), C4 (governance strength), C5 (renaming or preserving compatibility), and C6 (adopted or not adopted). In C6 in particular, what is being compared is adoption and non-adoption of a technology category, not DS adoption (Martins & Duarte 2024).

The second is the absence of a time scale. Longitudinal questions are being answered with cross-sectional data. C1 asks for propagation time, but what is measured is the staleness of a cross-section (Legay, Decan, Mens 2021), and C4 asks for a trend, but the industry adoption ledgers are single points in time or estimates (X-GOVUK 2023, GDS 2024). C5 asks for a survival period, but the large-scale analyses that were the object are snapshots with no description of time measurement (Cogo, Oliva, Hassan 2022), and C6 asks for before and after adoption, but no study matches.

The third is the absence of attribution. They do not separate whether a result comes from the DS or from the implementation on the side using it. C6 is the clearest: the source itself states that attribution to a technology is not possible (WebAIM 2026), and the DS side also writes that it does not guarantee an accessible service (USWDS). In C2, the effect of consistency takes different signs for screen appearance and command language (Satzinger & Olfman 1998), and in C5 the survival period is not separated into what is due to circumstances on the using side and what is due to the design on the providing side (Cogo, Oliva, Hassan 2022).

The fourth is self-selection. The ability to adopt and the difference in results are confounded. C4 has only surveys with self-selected samples (zeroheight’s n=147, Sparkbox’s open-call sample), and in C6 adoption is not randomized and correlates with size, age, and team structure. The industry comparison for C1 is also a sample of organizations operating multiple systems that came forward.

The fifth is that the party doing the measurement is a stakeholder and there is no blinding. In C3, the case that reported a reduction acknowledges that quality was judged by its own team (BILL), and another case is a company’s own evaluation of something it sells (Figma). In C4, indicators other than the breakdown of adoption have unclear definitions (IBM Carbon), and a page describing measurement infrastructure has no measured values (GitLab). In C5, all costs carry an estimate label (Backbase), and in C2 the timing measurement and blinding are either not obtained or not described (Satzinger & Olfman 1998, Ozok & Salvendy 2000). On the other hand, there is an example that explicitly states the judgment is blind to the treatment condition (Open Design System Bench).

The sixth is that effect sizes are buried under the measurement ceiling. In C6, the coverage of automated audits is at most 50% (Vigo, Brown, Conway 2013), the tool used covers only 38% of the 30 criteria of WCAG 2.1 (Martins & Duarte 2024), and the category effect of 0.017 is buried under the technology difference of 0.196 within the same category. In C2, the achievement rate reached 100% in both conditions and no time difference emerged (Opsahl & Sitompul 2025), and in C1 the counting of duplicates depends on the choice of threshold (Yandrapally, Stocco, Mesbah 2020).

The six patterns are not independent. Cutting the treatment into two values collapses the time axis, and when the time axis collapses, neither attribution nor self-selection can be separated. That there are only two measurements taking the DS directly as their object also appears as a consequence of this chain.

References

The access date for all URLs is 2026-09-11.

Academic Literature

Public Bodies and Standards

Industry Primary Sources

Unverified Items

  • The sample size of C1-01 (Lamine & Cheng 2022), and the observation period and number of review participants of C1-02 (Sundelin et al. 2024) [requires primary verification]
  • The task content, the presence of timing measurement, and the presence of blinding in C2-01 (Satzinger & Olfman 1998) [requires primary verification]
  • The sample, independent variables, dependent variables, and controls of C2-05 (Payne & Green 1989) (the publisher does not post the abstract, and the body text is behind a paywall) [requires primary verification]
  • The number of participants and the number of trials in C2-09 (Fitts & Seeger 1953) [requires primary verification]
  • The scoring procedure for the compliance rate, the raters, blinding, the number of trials, and the measured statistics and token values in C3-01 (Cha et al. 2026) [requires primary verification]
  • The maximum of +0.30 on ParEval-OMP in C3-07 (Zi et al. 2025) [requires primary verification]
  • In C3-05 (Cui 2025), an arXiv administrative note points to text overlap with another paper, and the number of models in the conclusion (18) does not match the abstract (19)
  • The figures in C3-10 (Brickify 2025) were not obtained because the retrieved HTML was truncated
  • The period, data sources, and quantitative indicators of C4-02 (Riehle et al. 2016) (all of governance, adoption rate, contribution counts, longitudinal design, and control) [requires primary verification]
  • The period, sample, and statistical method of C4-05 (Halfaker et al. 2013) (body text unreachable) [requires primary verification]
  • The HHI value and the statistical test of C4-06 (Destefanis et al. 2026) [requires primary verification]
  • The interview period, analysis procedure, and adoption rate figures of C4-08 (Richter et al. 2025) [requires primary verification]
  • The observation period, tracking method, and time to reaction in C5-01 (Sawant et al. 2018) [requires primary verification]
  • The number of APIs, number of apps, and specific period in C5-03 (Li et al. 2020) [requires primary verification]
  • The days until recovery in C5-07 (Venturini et al. 2023) (134 days for clients, 7 days for providers) [requires primary verification]
  • For C6-01 (Diniz & Gama 2024), 4 DSs and 46 criteria are the basis for the description in this note, and the other inclusion conditions have not been obtained [requires primary verification]
  • The test page set and ground truth of C6-05 (Vigo et al. 2013) [requires primary verification]
  • Whether the n=2,220 of UX Tools is the population of that section or the value of the whole survey, the collection period, and the question text are undisclosed [requires primary verification]
  • The definition and comparison target of IBM Carbon’s NPS +57%, the question behind 88%, and the population definition of 155 and 116 are not stated on the primary page [requires primary verification]
  • The model name, prompt, and presence of human cross-checking for the Figma Code Connect LLM judge, and the selection criteria for the 27 cases, are undisclosed [requires primary verification]
  • “898 runs” and “+23-26pt” for Open Design System Bench are absent from the README page. Whether they exist in another file in the repository has not been confirmed [requires primary verification]
  • A statement that the Microsoft Fluent speaker is employed by Microsoft does not appear on the page (the thread label is Maintainer) [requires primary verification]
  • The primary URL of Sparkbox’s comparison of two conditions, DS present and absent, using Carbon has not been identified [requires primary verification]
  • WebAIM’s technology identification method (whether it is signature detection) is not stated on the primary page
  • The claim cited by the existing note design-system-practices as position B on C5, that “Fluent cannot answer about the side-by-side period”, exceeds the scope of the source’s statement. This note treats this testimony in a limited way, as a statement about when v9 will reach parity with v8

← All Notes · Home