Competency Measurement Premised on AI Use

How should human competency be measured in environments where the use of AI tools is taken as a given?

Conventional competency measurement targeted the “individual without tools.” In environments where AI is used routinely, the object of measurement changes. On top of pre-existing aptitudes (articulation ability, proactiveness, domain knowledge, aesthetic judgment), abilities for using AI (prompt design, output evaluation, agency allocation, metacognition) are layered. The two are not independent; they interact. People with strong articulation ability and high proactiveness are lifted even further by AI.

This note organizes this structure from both empirical and theoretical perspectives, drawing on 28 academic sources and 16 industry sources.

Related: ai-cognition-skill-gap-debate (the cognitive divide debate) / maker-to-editor-paradigm (from maker to editor) / adoption-approval-paradox (90% adoption x 10% approval) / design-education-ai-adaptation (design education’s adaptation to AI).

Does AI Compress or Amplify Ability Differences?

Evidence for Compression: The Lower the Skill, the Greater the Benefit

Dell’Acqua et al. (2023/2026) showed in an RCT with 758 BCG consultants that on tasks where AI excels, below-average performers achieved a 43% quality improvement while above-average performers gained only 17%. The boundary of AI capability is a jagged frontier, with a counterintuitive asymmetry in which AI excels at seemingly difficult tasks and fails at seemingly easy ones. Inside this frontier, ability differences are compressed.

Brynjolfsson et al. (2023) tracked 5,179 customer service agents and showed that AI adoption produced a 34% productivity gain for low-skill, inexperienced workers while having almost no effect on experienced workers. AI functions as a mechanism that transfers experts’ tacit knowledge to novices.

Noy & Zhang (2023) confirmed in an RCT with 453 college-educated professionals that task time fell by 40%, output quality rose by 18%, and the productivity distribution compressed.

Cruces et al. (2026) showed in an RCT with 1,174 participants that the education-based productivity gap of 0.548 SD without AI shrank to 0.139 SD with AI adoption. AI eliminates three quarters of the gap.

Conditions for Amplification: Task Complexity and Expertise

Compression is not universal.

In the same Dell’Acqua et al. experiment, on tasks outside the frontier (domains where AI is weak), the group using AI saw its accuracy fall by 19 percentage points. When AI’s limits are misjudged, high and low performers lose equally.

An (2025) proposed that task complexity switches the effect of AI. On simple tasks AI narrows the gap between experts and non-experts, but on complex tasks it widens the gap. The determinant of the amplification effect is held to be domain expertise rather than prompting skill¹.

Doshi & Hauser (2024) demonstrated that AI improves individual creativity (novelty +8.1%, usefulness +9.0%) while shrinking collective diversity by more than 10%. Those with lower intrinsic creativity benefit more (22–26% improvement), but everyone converges on similar ideas.

The Concept of “Augmentable Cognition”

Espinal Maya (2026) proposed a framework that decomposes human capital into three components. These are physical-manual capital (H^P), routine cognitive capital (H^C), and augmentable cognitive capital (H^A). AI capital substitutes for H^C and complements H^A. Evaluating 18,796 O*NET tasks with an LLM and linking them to a Colombian household survey of 105,517 individuals, the study demonstrated that the wage returns to augmentable cognition rise in AI-adopting firms.

Within this framework, articulation ability, abstract reasoning, problem structuring, and evaluative judgment are classified as “augmentable cognition.” AI amplifies these rather than substituting for them. Those who already possess these aptitudes at high levels achieve even higher productivity with AI.

Articulation Ability and Proactiveness Determine the Effectiveness of AI Use

As of 2026, no study has been found that directly measures the “articulation ability x AI-use effectiveness” interaction. Indirect evidence, however, converges from multiple directions.

Prompting Skill and Articulation Ability

Gibreel & Arpaci (2025) developed PECS (9 items, α=0.92), the first validated scale for measuring prompt engineering competence. Its measurement dimensions include “differentiated use of contextual constraints,” “format specification,” and “critical evaluation of responses.” These connect directly to articulation ability (the capacity to structure and explicitly express one’s own intentions).

Woo et al. (2024/2026) delivered a 100-minute prompt engineering intervention to 27 undergraduates and demonstrated improvements in AI self-efficacy, AI knowledge, and prompting ability alike. This suggests that cultivating articulation ability may contribute causally to AI-use skills.

However, Meincke et al. (2025, Wharton) showed that the effects of prompting techniques are highly context-dependent. “Polite prompts vs imperative form” produces differences of up to 60 points at the individual-question level, but these even out at the aggregate level. This suggests that prompting skill should be measured not as “mastery of universal techniques” but as “the capacity for iterative trial and evaluation attuned to conditions and goals.”

Proactiveness and Power User Behavior

The Microsoft Work Trend Index 2024 (31,000 respondents, 31 countries) quantified the behavioral differences between AI Power Users (using AI multiple times per week) and everyone else. Power Users experiment 68% more frequently, are 49% more likely to consider using AI before a task, and are 30% more likely to keep trying after failure. This behavioral pattern is itself an operational definition of “proactiveness.”

BCG AI at Work 2025 (10,600 respondents, 11 countries) showed that 79% of those who received more than five hours of training became regular AI users, compared with only 67% of those with less than five hours. Those who proactively invest time in learning receive more of AI’s benefits.

Three Work Styles

Dell’Acqua et al. (2025) extended the classification of work styles with AI into three types in a field study of 244 BCG consultants.

The Centaur maintains a clear division of roles between human and AI with strategic delegation. They handle the domains where they excel and delegate tasks inside the frontier to AI. This contributes to upskilling, which strengthens existing domain expertise.

The Cyborg integrates human and AI fluidly. The two interweave at the sentence level, contributing to newskilling, the acquisition of new AI-related capabilities.

The Self-Automator delegates to AI both what to do and how to do it. They improve neither domain expertise nor AI expertise, bearing the risk of skill stagnation.

This classification can be read as a framework for measuring the quality of “proactiveness.” Centaurs and Cyborgs engage with AI proactively, whereas Self-Automators are passive. Even among those equally “using AI,” the direction of ability development differs by work style.

The Metacognition Paradox

The most troublesome finding for measuring AI-use competence is the distortion of metacognition.

Fernandes et al. (2025) had 246 participants perform logical reasoning tasks using ChatGPT-4o. Performance improved by about 3 points, but self-assessment was overestimated by about 4 points. Overconfidence arises that exceeds the actual performance gain.

Furthermore, the study discovered the paradox that higher AI literacy is associated with lower metacognitive accuracy. The more people know about AI, the more they overestimate their own performance when using it. The Dunning-Kruger effect disappears under AI use, with everyone converging on the same level of overconfidence as high-skill individuals.

This finding demonstrates the limits of self-report measurement of AI-use competence.

Zhang et al. (2026) confirmed through latent profile analysis that the correlation between self-reported and objectively assessed AI literacy among teachers is low. Six profiles exist, mixing patterns of overestimation, underestimation, and alignment.

Self-report AI literacy scales (AILQ, SNAIL, MAILS, etc.) are widely used, but they must be designed on the premise of their divergence from objective competence measurement. Markus et al. (2025) developed AICOS, an objective AI competency scale, in an attempt to address this problem.

Existing Measurement Frameworks

AI Literacy Scales (Academic)

Four major scales have been published through peer review.

Long & Magerko (2020) organized AI literacy into 17 core competencies (CHI 2020 Best Paper Honorable Mention). Ng et al. (2021) proposed the four-part framework of “know, use, evaluate, ethics,” which many subsequent scale-development efforts have referenced. Ng et al. (2024) developed and validated the 32-item AILQ (four dimensions: affective, behavioral, cognitive, and ethical). Laupichler et al. (2023) developed the 31-item SNAIL (three factors: technical understanding, critical appraisal, and practical application) through a Delphi study with 53 experts. Carolus et al. (2023) developed the 34-item MAILS.

All of these are self-report instruments. Given the finding of Zhang et al. (2026) (the mismatch between self-report and objective assessment), designs that additionally employ performance-based objective assessment are required.

Industry Frameworks

Maeda (2026) proposed the E-P-I-A-S framework (Explorer -> Practitioner -> Integrator -> Architect -> Steward) as a self-assessment tool for evaluating designers’ AI skill maturity in stages. It is unaccompanied by validation research, and the author himself acknowledges its provisionality with “something is better than nothing.”

NN/G’s Elman (2026) proposed the Judge-Evaluate-Iterate loop. Define research-grounded objective criteria, evaluate AI output against those criteria, and improve prompts based on failure patterns. It redefines the designer’s role as “the arbiter who defines what good design is.”

The OECD (2025) published the AI Capability Indicators, a framework for measuring and comparing AI capability across nine domains of human ability (language, social interaction, problem solving, creativity, metacognition and critical thinking, knowledge and learning, vision, manipulation, and robotic intelligence). The domains where AI is weak (creativity, metacognition, ethical judgment) emerge as the abilities on which humans should be measured.

The Relationship Between Generative AI Literacy and Job Performance

Liu et al. (2025) developed a five-dimension generative AI literacy scale (foundational technology, prompt optimization, content evaluation, innovative application, and ethical compliance) and demonstrated a significant positive association with job performance (β=0.680). Creative self-efficacy mediates the relationship.

Shi et al. (2025) confirmed with 257 university students that AI literacy and self-regulated learning are both positively associated with writing performance. AI literacy was also the strongest predictor of well-being (β=0.503).

What Should Be Measured: A Proposed Three-Layer Model

Integrating the findings above, competency measurement premised on AI use consists of three layers.

Layer 1: Foundational Aptitudes (Existing Before AI, Amplified by AI)

Articulation: the capacity to structurally express one’s intentions, constraints, and evaluation criteria. It is the foundation of prompt engineering and indirectly determines the effectiveness of AI use.

Proactiveness: the tendency to try new tools, persist after failure, and retry under altered conditions. It corresponds to Microsoft’s Power User behaviors (experimentation +68%, persistence +30%).

Domain knowledge: expertise in the target domain. The higher the task complexity, the more domain knowledge amplifies the effectiveness of AI use (An 2025). It corresponds to Espinal Maya’s “augmentable cognitive capital.”

Aesthetic judgment and taste: the pre-reflective ability to evaluate the quality of AI output and judge what is good. It corresponds to “aesthetic judgment” in Huang & Poon’s (2026) SuperSkillsStack and to NN/G’s “curated taste.” It forms the foundation of the “agency allocation judgment” and “reliability judgment” identified in maker-to-editor-paradigm.

Layer 2: AI-Use Skills (Newly Required in AI Environments)

Prompt design: the technique of communicating intent to AI. However, as Meincke et al. (2025) show, it should be measured not as a universal technique but as a context-dependent capacity for trial and evaluation. PECS (Gibreel & Arpaci 2025) is the first validated scale.

Output evaluation: the ability to judge the accuracy, appropriateness, and bias of AI output. It corresponds to Naik et al.’s (2025) “reliability judgment.”

Agency allocation: the judgment of which tasks to delegate to AI and where to intervene oneself. It corresponds to Dell’Acqua et al.’s Centaur/Cyborg/Self-Automator classification. It is the ability to recognize the shape of the jagged frontier and judge appropriately.

Metacognition: the ability to accurately assess one’s own performance when using AI. As Fernandes et al. (2025) show, metacognitive accuracy declines under AI use. The ability to recognize and correct this distortion is indispensable for sustained improvement in the effectiveness of AI use.

Layer 3: Interaction (The Amplification Structure of Layer 1 x Layer 2)

Layers 1 and 2 are not independent.

Those with strong articulation ability can communicate intent more precisely in prompt design, raising the quality of AI output. Those with high proactiveness accumulate trials without fear of failure, learning AI’s limits and possibilities experientially. Those with deep domain knowledge can detect errors in AI output and wield AI as a tool on complex tasks. Those with sharp aesthetic judgment can choose distinctive directions away from AI’s homogeneous output.

Measuring this interaction requires a design that compares “ability without AI” and “ability with AI” on the same task and gauges which dimensions of foundational aptitude correlate with the difference (the AI boost). The RCT design of Dell’Acqua et al. (AI/no-AI conditions x ability level) provides a template experimental paradigm for capturing this structure.

Measurement Caveats

The Limits of Self-Report

The finding of Zhang et al. (2026) raises fundamental doubts about self-report measurement of AI literacy. With overestimation and underestimation intermixed, self-report alone cannot capture the true distribution of ability. Dual measurement combining objective (performance-based) assessment with self-report is necessary.

Tool Dependence

Measurement results depend on the AI tool used. Someone who excels with GPT-4 will not necessarily achieve the same results with Claude. This is because the shape of the jagged frontier differs from model to model. Competency measurement requires either a cross-tool design or explicit specification of the particular tool as a condition.

Evaluator Bias

Evaluators’ own biases intervene in the quality assessment of AI-generated artifacts. Schilke & Reimann (2025) demonstrated the “transparency dilemma,” in which those who disclose AI use are trusted less than those who do not. In competency measurement as well, evaluations may decline when AI involvement is disclosed.

References

Empirical Studies of Productivity and Ability Differences

• Dell’Acqua, F. et al. (2023/2026). Navigating the Jagged Technological Frontier. Organization Science. https://doi.org/10.1287/orsc.2025.21838
• Dell’Acqua, F. et al. (2025). The Cybernetic Teammate. NBER WP 33641. https://doi.org/10.3386/w33641
• Brynjolfsson, E. et al. (2023). Generative AI at Work. NBER WP 31161. https://doi.org/10.3386/w31161
• Noy, S. & Zhang, W. (2023). Experimental Evidence on Productivity. Science, 381(6654). https://doi.org/10.1126/science.adh2586
• Peng, S. et al. (2023). Impact of AI on Developer Productivity. arXiv:2302.06590. https://arxiv.org/abs/2302.06590
• Cruces, G. et al. (2026). Does Generative AI Narrow Education-Based Productivity Gaps? NBER WP 34851. https://doi.org/10.3386/w34851
• Doshi, A.R. & Hauser, O.P. (2024). Generative AI Enhances Individual Creativity but Reduces Collective Diversity. Science Advances, 10(28). https://doi.org/10.1126/sciadv.adn5290
• Espinal Maya, C. (2026). Augmented Human Capital. arXiv:2604.01066. https://arxiv.org/abs/2604.01066
• An, T. (2025). AI as Equalizer or Amplifier? arXiv:2512.10961. https://arxiv.org/abs/2512.10961

AI Literacy Scales and Measurement

• Long, D. & Magerko, B. (2020). What Is AI Literacy? CHI 2020. https://doi.org/10.1145/3313831.3376727
• Ng, D.T.K. et al. (2021). Conceptualizing AI Literacy. Computers and Education: AI, 2. https://doi.org/10.1016/j.caeai.2021.100041
• Ng, D.T.K. et al. (2024). AI Literacy Questionnaire. BJET, 55(3). https://doi.org/10.1111/bjet.13411
• Laupichler, M.C. et al. (2023). SNAIL Scale. Computers in Human Behavior Reports, 12. https://doi.org/10.1016/j.chbr.2023.100338
• Carolus, A. et al. (2023). MAILS. Computers in Human Behavior: Artificial Humans. https://doi.org/10.1016/j.chbah.2023.100014
• Zhang, S. et al. (2026). Misalignment Between Self-Reported and Objective AI Literacy. arXiv:2601.06101. https://arxiv.org/abs/2601.06101
• Markus, A. et al. (2025). AICOS. arXiv:2503.12921. https://arxiv.org/abs/2503.12921
• Gibreel, O. & Arpaci, I. (2025). Prompt Engineering Competence Scale. Information Development. https://doi.org/10.1177/02666669251336455
• Woo, D.J. et al. (2024/2026). Effects of Prompt Engineering Intervention. BERJ, 52. https://doi.org/10.1002/berj.70087
• Liu, X. et al. (2025). Generative AI Literacy and Job Performance. Behavioral Sciences, 15(6). https://doi.org/10.3390/bs15060811
• Shi, J. et al. (2025). AI Literacy and Self-Regulated Learning. Behavioral Sciences, 15(5). https://doi.org/10.3390/bs15050705

Metacognition and Collaborative Evaluation

• Fernandes et al. (2025). AI Makes You Smarter But None the Wiser. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2025.108645
• Meincke, L. et al. (2025). Prompt Engineering is Complicated and Contingent. Wharton GAIL. https://gail.wharton.upenn.edu/research-and-insights/tech-report-prompt-engineering-is-complicated-and-contingent/
• Fragiadakis, G. et al. (2024). Evaluating Human-AI Collaboration. arXiv:2407.19098. https://arxiv.org/abs/2407.19098

Industry Frameworks

• Microsoft. (2024). Work Trend Index. N=31,000. https://www.microsoft.com/en-us/worklab/work-trend-index/ai-at-work-is-here-now-comes-the-hard-part
• BCG. (2025). AI at Work. N=10,600. https://www.bcg.com/publications/2025/ai-at-work-momentum-builds-but-gaps-remain
• McKinsey. (2025). Superagency in the Workplace. N=3,851. https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/superagency-in-the-workplace-empowering-people-to-unlock-ais-full-potential-at-work
• OECD. (2025). AI Capability Indicators. https://www.oecd.org/en/publications/2025/06/introducing-the-oecd-ai-capability-indicators_7c0731f0.html
• Maeda, J. (2026). AI Upskilling for Product Designers. https://johnmaeda.medium.com/ai-upskilling-for-product-designers-0012eabfdeaf
• NN/G. (2026). The Core Skill of Design in the AI Era: Critique. https://www.nngroup.com/articles/ai-era-critique/
• Mollick, E. (2023). Centaurs and Cyborgs on the Jagged Frontier. https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged

Footnotes

1. However, this is a single-organization observation with N=10–20, and its external validity is limited. ↩