Shuichiro Ogawa
日本語

Notes · updated 2026-07-29

Recent Currents in LLM-as-a-Judge: A Literature Map of 53 Core Studies and a Standalone Chapter on Creativity Evaluation

Scope and Method

This note is a literature map that organizes the state of research on LLM-as-a-Judge (the technique of using an LLM as an evaluator of other generated outputs), drawn from two corpora, academic and industry.

  • Academic (primary axis, mode: academic): 53 items, after merging 5 duplicates from 58 explored. Corpus: source/review/llm-as-a-judge/papers.md.
  • Industry (supplementary, mode: ai-frontier): 18 T1v vendor primary sources, 9 T2 public institutions, 12 T3 practitioner opinions. Corpus: source/review/llm-as-a-judge/ai-frontier.md. By user specification, the scope was limited to “vendor primary sources and evaluation-tool practice”; the analyst lineage (research firm reports) was not collected.

The covered period is 2023-01 to 2026-07 for the academic side (including a small number of precursor anchors from 2021–2022) and 2024-07 to 2026-07 for the industry side. As requested, “research that uses LLM-as-a-Judge to evaluate creative artifacts” is set up as a standalone chapter. The period before the concept’s establishment, that is, the lineages and demands from which LLM-as-a-Judge emerged, is covered in llm-as-a-judge-history.

Academic Cluster 1: Three Generations of Judges (prompted → fine-tuned → reasoning)

The lineage of LLM-as-a-Judge can be read as three generations.

The first generation is the prompted judge, which turns a general-purpose LLM into an evaluator through prompting. As prehistory, BARTScore (Yuan et al. 2021), which evaluates text via generation probability, and UniEval (Zhong et al. 2022), with its Boolean QA format, laid the groundwork for model-based evaluation. The paradigm was established by Zheng et al. (2023) with MT-Bench and Chatbot Arena, showing that a GPT-4 judge agrees with human evaluation at over 80% while simultaneously enumerating in advance the themes of subsequent research: position bias, verbosity bias, and self-enhancement bias. G-Eval (Liu et al. 2023) raised human correlation with CoT and form-filling, and at the same time pointed out a bias toward LLM-generated text.

The second generation is the fine-tuned judge, tuned specifically for evaluation. Representatives are PandaLM (7B), Prometheus (13B, rubric-based evaluation), Auto-J (58 scenarios), JudgeLM (bias reduction via swap augmentation), and Prometheus 2 (supporting both direct assessment and pairwise), aiming at open evaluators freed from dependence on GPT-4. However, Huang et al. (2024) demonstrated that fine-tuned judges, while strong in-domain, behave as task-specific classifiers and do not serve as general substitutes for GPT-4; the reach of this generation has known limits.

The third generation is the reasoning judge with strengthened reasoning, together with the expansion of evaluation targets. JudgeLRM (Chen et al. 2025) and J1 (Whitehouse et al. 2025) train judge reasoning with RL on verifiable rewards and argue for the limits of SFT alone. MCTS-Judge (Wang et al. 2025) brought the scaling of evaluation-time compute (test-time scaling) into code evaluation. On the side of evaluation targets, Agent-as-a-Judge (Zhuge et al. 2024), which evaluates the intermediate processes of agents, and MLLM-as-a-Judge (Chen et al. 2024), which extends to vision-language tasks, appeared in parallel.

Academic Cluster 2: Reliability, Bias, and Meta-Evaluation

Bias research has quantified, one by one, the concerns that the founding paper enumerated.

  • Position bias: Wang et al. (ACL 2024) showed that merely swapping answer order can reverse the verdict, and Shi et al. (2024) measured it systematically across 15 models × roughly 150,000 instances.
  • Length bias: Dubois et al. (2024) controlled it statistically with regression-based Length-Controlled AlpacaEval.
  • Self-preference bias: Panickssery et al. (2024) showed a linear correlation between a judge’s self-recognition capability and bias strength, and Wataoka et al. (2024) supplied a mechanistic hypothesis of perplexity-based familiarity.
  • Overviews: CALM (Ye et al. 2024) automatically quantifies 12 kinds of bias, and three surveys (Gu et al. 2024; Li et al. 2024, From Generation to Judgment; Li et al. 2024, LLMs-as-Judges) organized the field from the perspectives of reliability, taxonomy, and methodology.

Meta-evaluation benchmarks that measure judges themselves are also in place. JudgeBench (Tan et al. 2024) uses hard problem pairs with objective ground truth to show that even GPT-4o-class judges drop to near-random performance. RewardBench (Lambert et al. 2024), as an infrastructure for reward-model evaluation, connected the meta-evaluation of judges and reward models. As the reference point for human preference, Chatbot Arena (Chiang et al. 2024) is the de facto standard, but objections to its reliability have come from the industry side’s T3 (Karpathy’s evaluation-crisis argument, discussed below).

Standalone Chapter: Research Using LLM-as-a-Judge to Evaluate Creative Artifacts

The evaluation of creative artifacts carries difficulties different from reasoning tasks with correct answers (the subjectivity of evaluation dimensions, disagreement among experts, the measurement of originality). This chapter organizes 28 items (B-series specific) into four areas.

Text Creation (Stories, Creative Writing, Humor)

Gómez-Rodríguez & Williams (2023) showed, with detailed-rubric human evaluation, that commercial LLMs are at or above human level on most dimensions of creative writing, while humans remain superior on originality. Ismayilzada et al. (2024) confirmed, for short-story creation, a divergence between LLM self-evaluation and expert evaluation; in creative fields, one cannot say that “strong generation implies trustworthy evaluation.” On the evaluation-infrastructure side, LitBench (Fein et al. 2025), the first standard benchmark for creative-writing evaluation, reported 73% human agreement for off-the-shelf judges (Claude) versus 78% for a trained reward model. Reference-based evaluation using TTCW as references (Li et al. 2025), personalized judges that adapt to individual judgment criteria (Kumar et al. 2025), and Zhou et al. (2025), who reached expert-level humor ranking (82.4%) by fine-tuning on a small amount of human preference data, indicate the approaches to improving creative-writing evaluation.

Automated Scoring of Creativity Tests (Psychometric Lineage)

In psychological creativity measurement, LLM scoring greatly surpasses the older semantic-distance-based methods. Organisciak et al. (2023) reported r=.81 with human ratings for AUT (Alternate Uses Task) scoring (versus r=.12–.26 for semantic distance), and DiStefano et al. (2025) reported r=.72 for metaphor creativity. MuseScorer (Bangash et al. 2025) reached participant-level correlation of r=.89 with frequency-based originality scoring requiring no human annotation. Rabeyah et al. (2024) reported that the creativity evaluations of multiple LLMs agree with one another (Spearman > 0.7) and that no favoritism toward self-generated outputs was observed; this result splits from the general finding on self-preference bias (Panickssery et al.).

Visual Artifacts (Image Generation, Aesthetic Evaluation, Visual Creativity)

The lineage that uses MLLMs as judges has repeatedly confirmed that the quality of agreement depends on task format. MLLM-as-a-Judge (Chen et al. 2024) showed that judges are close to humans in pairwise settings, while divergence remains in scoring and ranking. VIEScore (Ku et al. 2023) reported a human correlation of 0.4 for the GPT-4o version on conditional image generation (human-human 0.45) and also showed the gap with open MLLMs. MJ-Bench (Chen et al. 2024) assembled a dataset that examines text-to-image generation judges along four dimensions: alignment, safety, quality, and bias. In aesthetic evaluation, Jiang & Chen (2025) identified aesthetic-reasoning hallucination, in which interpretations are strung together without evidence, and suppressed it with evidence-based reasoning (ArtCoT). Zero-shot scoring of visual creativity (Orwig & Beaty 2026) reported r=.29–.68 with human evaluation, noting that making reasoning visible provides interpretability but does not raise accuracy.

Design and Educational Evaluation (UI Critique, Design Creativity, Grading)

The design domain proceeds along two tracks, HCI and engineering design. On the HCI side, starting from automatic feedback on UI mockups with GPT-4 (Duan et al., CHI 2024), UICrit (UIST 2024), a dataset of 3,059 designer critiques, improved LLM feedback by 55% with few-shot prompting while stating explicitly that expert quality remains out of reach. Iterative visual prompting (Duan et al. 2024) closed the gap with experts by about 50% on some metrics. On the engineering-design side, Zhang, Han & Ahmed-Kristensen (ICED25 and elsewhere) scored the novelty and usefulness of design concepts with structured evaluation and multi-agent CoT, reporting improved expert agreement. In educational assessment, design assignment grading (Huang et al. 2024, reliability with humans of 0.65–0.78) and essay scoring (Kim 2025’s comparison of prompting strategies; Li & Liu 2024’s comparison of 18 models) are approaching “a level educators could accept,” premised on rubric elaboration and iterative tuning.

Chapter Conclusion: How Far Can It Be Trusted?

Across the four areas, three findings converge.

  1. Dimensions that can be formalized show high human correlation. For dimensions with clear rubrics (fluency, coherence, the frequency-based aspect of AUT originality), r=.7 or higher appears repeatedly.
  2. Alignment with small amounts of human data works. Humor (82.4%), UICrit few-shot (+55%), and the LitBench reward model (78%) all show judges aligned with human data outperforming zero-shot judges.
  3. Originality and expert quality remain out of reach. On the originality dimension of creative writing (B1), the quality of expert design critique (B23), and the judgment of novelty (B17), the divergence between LLM judges and human experts persists.

Meanwhile, how far self-preference bias operates in creativity evaluation remains unsettled: the general-task demonstration (Panickssery et al.) and the non-detection report on AUT (Rabeyah et al.) split.

Gaps (Unaddressed Topics)

  • Connection to validity theory for creativity evaluation. No study in this corpus could be confirmed that examines LLM judges through the validity frameworks of educational or psychological measurement (construct validity).
  • Visual design critique beyond UI. Application of judges to graphic, product, and brand design is sparse compared with UI critique (the Duan et al. line).
  • Verification of preference bias toward AI-generated artifacts in creativity contexts. No study was identified that replicates the mechanism research on self-preference (perplexity familiarity) on creative artifacts.
  • Non-English creativity evaluation. Apart from Japanese AES (Li & Liu 2024), judge research targeting non-English creative artifacts is nearly absent from the corpus.
  • Application of reasoning judges to creativity evaluation. RL-trained judges (J1, JudgeLRM) and test-time scaling (MCTS-Judge) have been validated on tasks with correct answers; results of transfer to subjective quality are unconfirmed.

The Current State of Industry Implementation (ai-frontier Supplement)

The following is a supplement based on the mode: ai-frontier corpus (source/review/llm-as-a-judge/ai-frontier.md), divided into the T1v / T2 / T3 tiers so as not to be conflated with the academic side.

T1v: Two Types of Productization Shown by Vendor Primary Sources

Productization splits into two types.

  • Managed services with general-purpose model judges: OpenAI (the score model grader on the Evals platform; however, read-only from 2026-10-31, shutdown scheduled for 2026-11-30), Google Vertex AI (Gen AI Evaluation Service, pointwise/pairwise, bias reduction via response flipping), AWS Bedrock (Model Evaluation GA 2025-03; agent-oriented AgentCore Evaluations GA 2026-03), Azure AI Foundry (weighted-rubric evaluators), LangSmith (offline/online evaluation).
  • Specialized fine-tuned judge offerings: Atla (Selene / Selene Mini 8B), Patronus AI (GLIDER 3.8B; the RAG-specialized Lynx). These correspond to productized versions of the academic second generation (the Prometheus line).

The cautions in official documents converge with the academic bias findings. OpenAI explicitly notes position bias and verbosity bias and recommends pairwise or pass/fail; Anthropic states the calibration precondition of “not trusting a judge until confirmed to diverge little from human experts” and the separation of generator and evaluator. Azure documents reduced reliability on short texts (under about 20 tokens) and an English-only limitation; Ragas documents judge non-determinism and a failure pattern dominated by false positives; Hugging Face officially records the echo-chamber effect (the circularity of evaluating LLMs with LLMs). Five items of self-favoring assertions (claims of benchmark superiority or workload reduction) were separated out as marketing-claim on the corpus side.

T2: Public Institutions Converge on “Conditional Use”

The position of public institutions is neither recommendation nor prohibition but conditional use.

  • NIST (AI 800-2 initial public draft, 2026-01) defines LLM-as-a-judge as one method for automated benchmark evaluation while recording concerns about egocentric bias and length bias.
  • UK AISI has the deepest accumulation. While implementing model-graded scorers in its evaluation framework Inspect, its autonomous systems evaluation standard (2024-10) advises against use for numeric ratings on the grounds of “known accuracy and bias issues,” published autograder-human agreement (Cohen’s Kappa 0.52, human-human 0.8), and advanced as far as a statistical framework for evaluating judge reliability with Bayesian GLM (2025-07).
  • J-AISI (Guide to Evaluation Perspectives on AI Safety, version 1.10, 2025-03) includes model-based evaluation as one evaluation method while cautioning that fully automated evaluation with tools alone is difficult.
  • Stanford HAI AI Index 2026 points to declining reliability of benchmarks in general (citing an external analysis with error rates of up to 42%) but makes no direct mention of LLM-as-a-Judge.

T3: Convergence Points in Practitioner Opinion

The views of authority-verified practitioners (Hamel Husain, Eugene Yan, Simon Willison, Shreya Shankar, Clémentine Fourrier, Andrej Karpathy), treated as personal opinion, converge on the following points.

  • Recommendation of binary judgments. Pass/Fail is more consistent than a 1–5 Likert scale, and experts’ critiques are reused as few-shot examples for the judge (Husain’s Critique Shadowing).
  • Criteria drift. Evaluation criteria cannot be fixed before seeing outputs and are updated iteratively (Husain / Shankar).
  • Evaluating the evaluator. Do not trust a judge without checking it against human judgment (Fourrier; Yan’s criterion that it “should be comparable to human-human correlation”).
  • Both pessimism about evals and a defense of them. Karpathy points to an evaluation crisis that includes overfitting to Chatbot Arena, while Shankar argues back against anti-eval sentiment.
  • Direct practitioner knowledge on evaluating creative artifacts is thin; Yan’s “recommend pairwise for subjective tasks” is the closest mention.

Recent Major Updates (Chronological)

PeriodEventTier
2024-05UK AISI releases InspectT2
2024-10AISI advises against model grading for numeric ratingsT2
2024-12AWS Bedrock LLM-as-a-Judge preview; Patronus GLIDER announcedT1v
2025-03Bedrock Model Evaluation GA; J-AISI guide version 1.10T1v/T2
2025-07AISI publishes the Bayesian GLM judge-reliability frameworkT2
2026-01NIST AI 800-2 initial public draft (defines LLM-as-a-judge)T2
2026-03AWS AgentCore Evaluations GAT1v
2026-06OpenAI announces the deprecation of the Evals platform (shutdown scheduled for 2026-11)T1v

How to Read the Confidence Levels

T1v is the primary authority on “what is possible, constraints, and pricing,” but for self-favoring claims one refers to the marketing-claims already separated out. T2 is neutral, but drafts (NIST 800-2) are distinguished from finalized documents. T3 is personal opinion, and its figures (such as a 60–80% workload allocation) remain treated as [requires primary verification].

Cross-Checking Academia and Industry

The academic bias demonstrations (position, length, self-preference) flowed, between 2024 and 2026, almost directly into the cautions of vendor official documents (OpenAI, Anthropic, Ragas) and public guidance (NIST, AISI). As a consequence of this flow, the shared premise in practice has become that “a judge is an instrument usable only once calibrated against human evaluation.” Meanwhile, the accumulated research on creativity evaluation seen in this note’s standalone chapter (expert agreement, divergence on originality, the effectiveness of small-scale alignment) has, on the industry side, so far been absorbed as merely one feature of general-purpose rubric evaluators, and no judge product dedicated to creative artifacts has been identified. When introducing judges into the evaluation of design education or creative production, the findings to consult are not in industry documentation but in this chapter’s academic lineages (UICrit, the ICED25 line, the AES line).

Unverified Items

From the academic corpus:

  • The author affiliations of A25 (Self-Preference Bias) [requires primary verification]
  • The NeurIPS 2025 acceptance of B10 (MJ-Bench) [requires primary verification]
  • The full text of B22 (CHI 2024 UI feedback) (ACM DL 403; bibliography confirmed via multiple routes) [requires primary verification]
  • The full text of B29 (HCII 2025 multi-agent) (Springer authentication wall) [requires primary verification]
  • The final venues of A7 (Auto-J) and A24 (JudgeLRM) (currently treated as preprints)

From the industry corpus:

  • The document update date of OpenAI’s evaluation guide, and the publication date of Anthropic’s demystifying-evals [requires primary verification]
  • Direct confirmation, in the body of the official page, of Vertex AI’s default judge (Gemini 2.0 Flash) [requires primary verification]
  • Primary-page confirmation of Atla Selene Mini’s release date and price ($3/1,000 calls), and Patronus’s pricing [requires primary verification]
  • The judge model list and pricing of AWS AgentCore Evaluations [requires primary verification]
  • The judge model lists of LangSmith and Arize Phoenix [requires primary verification]
  • Whether the final version of NIST AI 800-2 has been published, whether NIST AI 600-1 mentions it in the body, and the details of the relevant sections in the body of the J-AISI guide [requires primary verification]
  • The publication month of the Stanford HAI AI Index 2026 and whether it is mentioned across the chapter [requires primary verification]
  • Direct confirmation of Karpathy’s two X posts (including posting dates), and the figures quoted by practitioners (60–80% workload, 30–60% hallucination recall, one-third of the workload in the Tesla era) [requires primary verification]

References

All accessed 2026-07-29.

Academic (Core Studies)

  1. Yuan, W., Neubig, G., Liu, P. (2021). BARTScore: Evaluating Generated Text as Text Generation. NeurIPS 2021. https://proceedings.neurips.cc/paper/2021/hash/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Abstract.html
  2. Zhong, M. et al. (2022). Towards a Unified Multi-Dimensional Evaluator for Text Generation. EMNLP 2022. https://aclanthology.org/2022.emnlp-main.131/
  3. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. https://arxiv.org/abs/2306.05685
  4. Liu, Y. et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. EMNLP 2023. https://aclanthology.org/2023.emnlp-main.153/
  5. Wang, Y. et al. (2023). PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. ICLR 2024. https://arxiv.org/abs/2306.05087
  6. Kim, S. et al. (2023). Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. ICLR 2024. https://arxiv.org/abs/2310.08491
  7. Li, J. et al. (2023). Generative Judge for Evaluating Alignment (Auto-J). arXiv:2310.05470. https://arxiv.org/abs/2310.05470
  8. Zhu, L., Wang, X., Wang, X. (2023). JudgeLM: Fine-tuned Large Language Models are Scalable Judges. ICLR 2025. https://arxiv.org/abs/2310.17631
  9. Wang, P. et al. (2023). Large Language Models are not Fair Evaluators. ACL 2024. https://aclanthology.org/2024.acl-long.511/
  10. Kim, S. et al. (2024). Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. EMNLP 2024. https://arxiv.org/abs/2405.01535
  11. Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML 2024. https://arxiv.org/abs/2403.04132
  12. Dubois, Y. et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. COLM 2024. https://arxiv.org/abs/2404.04475
  13. Panickssery, A., Bowman, S. R., Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024. https://arxiv.org/abs/2404.13076
  14. Lambert, N. et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling. Findings of NAACL 2025. https://arxiv.org/abs/2403.13787
  15. Tan, S. et al. (2024). JudgeBench: A Benchmark for Evaluating LLM-based Judges. ICLR 2025. https://arxiv.org/abs/2410.12784
  16. Gu, J. et al. (2024). A Survey on LLM-as-a-Judge. arXiv:2411.15594. https://arxiv.org/abs/2411.15594
  17. Li, D. et al. (2024). From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge. EMNLP 2025. https://arxiv.org/abs/2411.16594
  18. Li, H. et al. (2024). LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv:2412.05579. https://arxiv.org/abs/2412.05579
  19. Huang, H. et al. (2024). An Empirical Study of LLM-as-a-Judge: Fine-tuned Judge Model is not a General Substitute for GPT-4. Findings of ACL 2025. https://arxiv.org/abs/2403.02839
  20. Wang, Y. et al. (2025). MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation. arXiv:2502.12468. https://arxiv.org/abs/2502.12468
  21. Chen, D. et al. (2024). MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. ICML 2024. https://arxiv.org/abs/2402.04788
  22. Zhuge, M. et al. (2024). Agent-as-a-Judge: Evaluate Agents with Agents. arXiv:2410.10934. https://arxiv.org/abs/2410.10934
  23. Whitehouse, C. et al. (2025). J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning. arXiv:2505.10320. https://arxiv.org/abs/2505.10320
  24. Chen, N. et al. (2025). JudgeLRM: Large Reasoning Models as a Judge. arXiv:2504.00050. https://arxiv.org/abs/2504.00050
  25. Wataoka, K., Takahashi, T., Ri, R. (2024). Self-Preference Bias in LLM-as-a-Judge. arXiv:2410.21819. https://arxiv.org/abs/2410.21819
  26. Shi, L. et al. (2024). Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. AACL-IJCNLP 2025. https://arxiv.org/abs/2406.07791
  27. Ye, J. et al. (2024). Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. arXiv:2410.02736. https://arxiv.org/abs/2410.02736

Academic (Evaluation of Creative Artifacts)

  1. Gómez-Rodríguez, C., Williams, P. (2023). A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing. Findings of EMNLP 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.966
  2. Organisciak, P. et al. (2023). Beyond Semantic Distance: Automated Scoring of Divergent Thinking Greatly Improves with Large Language Models. Thinking Skills and Creativity, 49. https://doi.org/10.1016/j.tsc.2023.101356
  3. Chen, Z. et al. (2024). MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? arXiv:2407.04842. https://arxiv.org/abs/2407.04842
  4. Ku, M. et al. (2023). VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. ACL 2024. https://arxiv.org/abs/2312.14867
  5. Zhang, X. et al. (2023). GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks. arXiv:2311.01361. https://arxiv.org/abs/2311.01361
  6. Jiang, R., Chen, C. (2025). Multimodal LLMs Can Reason about Aesthetics in Zero-Shot. ACM MM 2025. https://arxiv.org/abs/2501.09012
  7. Orwig, W., Beaty, R. E. (2026). How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning. arXiv:2606.29672. https://arxiv.org/abs/2606.29672
  8. Fang, X. et al. (2025). Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMs. ICCV 2025. https://arxiv.org/abs/2503.14478
  9. DiStefano, P. V., Patterson, J. D., Beaty, R. E. (2025). Automatic Scoring of Metaphor Creativity with Large Language Models. Creativity Research Journal, 37(4). https://doi.org/10.1080/10400419.2024.2326343
  10. Ismayilzada, M., Stevenson, C., van der Plas, L. (2024). Evaluating Creative Short Story Generation in Humans and Large Language Models. ICCC 2025. https://arxiv.org/abs/2411.02316
  11. Kumar, H. et al. (2024). Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking. CHI 2025. https://arxiv.org/abs/2410.03703
  12. Rabeyah, A. A. et al. (2024). Do LLMs Agree on the Creativity Evaluation of Alternative Uses? arXiv:2411.15560. https://arxiv.org/abs/2411.15560
  13. Li, R. et al. (2025). Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach. arXiv:2504.15784. https://arxiv.org/abs/2504.15784
  14. Franceschelli, G., Musolesi, M. (2024). Creative Beam Search: LLM-as-a-Judge For Improving Response Generation. ICCC 2024. https://arxiv.org/abs/2405.00099
  15. Duan, P. et al. (2024). Generating Automatic Feedback on UI Mockups with Large Language Models. CHI 2024. https://dl.acm.org/doi/10.1145/3613904.3642782
  16. Duan, P. et al. (2024). UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. UIST 2024. https://arxiv.org/abs/2407.08850
  17. Duan, P. et al. (2024). Visual Prompting with Iterative Refinement for Design Critique Generation. arXiv:2412.16829. https://arxiv.org/abs/2412.16829
  18. Huang, Q., Willems, T., Poon, K. W. (2024). The Application of GPT-4 in Grading Design University Students’ Assignment and Providing Feedback. arXiv:2409.17698. https://arxiv.org/abs/2409.17698
  19. Li, W., Liu, H. (2024). Applying Large Language Models for Automated Essay Scoring for Non-Native Japanese. Humanities and Social Sciences Communications, 11. https://doi.org/10.1057/s41599-024-03209-9
  20. Kim, Y. (2025). Automated Essay Scoring With GPT-4 for a Local Placement Test. TESOL Quarterly, 59(S1). https://doi.org/10.1002/tesq.3405
  21. Zhou, K. L. et al. (2025). Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs. arXiv:2502.20356. https://arxiv.org/abs/2502.20356
  22. Zhang, J., Han, J., Ahmed-Kristensen, S. (2025). Multi-agent LLM with the Chain-of-Thought for Design Creativity Evaluation. HCI International 2025, LNCS 16332. https://doi.org/10.1007/978-3-032-12385-5_21
  23. Zhang, J., Han, J., Ahmed-Kristensen, S. (2025). Exploring the Use of LLMs to Evaluate Design Creativity. Proceedings of the Design Society, 5 (ICED25). https://doi.org/10.1017/pds.2025.10191
  24. Fein, D. et al. (2025). LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing. arXiv:2507.00769. https://arxiv.org/abs/2507.00769
  25. Bangash, A. S. et al. (2025). MuseScorer: Idea Originality Scoring At Scale. EMNLP 2025. https://arxiv.org/abs/2505.16232
  26. Kumar, B. V. et al. (2025). Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment. arXiv:2510.05135. https://arxiv.org/abs/2510.05135

Industry T1v (Vendor Primary Sources)

  1. OpenAI. Graders. https://developers.openai.com/api/docs/guides/graders
  2. OpenAI. Evaluation best practices. https://developers.openai.com/api/docs/guides/evaluation-best-practices
  3. OpenAI. Deprecations. https://developers.openai.com/api/docs/deprecations
  4. Anthropic. Define success criteria and build evaluations. https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
  5. Anthropic. Demystifying evals for AI agents. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
  6. Google Cloud. Configure a judge model (Vertex AI). https://cloud.google.com/vertex-ai/generative-ai/docs/models/configure-judge-model
  7. AWS. Evaluate model performance using another LLM as a judge (Amazon Bedrock). https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-judge.html
  8. AWS. Amazon Bedrock Model Evaluation LLM-as-a-judge is now generally available. https://aws.amazon.com/about-aws/whats-new/2025/03/amazon-bedrock-model-evaluation-llm-as-a-judge/
  9. AWS. Amazon Bedrock AgentCore Evaluations is now generally available. https://aws.amazon.com/about-aws/whats-new/2026/03/agentcore-evaluations-generally-available
  10. Microsoft. Rubric evaluators (Azure AI Foundry). https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rubric-evaluators
  11. Microsoft. General Purpose Evaluators (Azure AI Foundry). https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/general-purpose-evaluators
  12. LangChain. Set up online evaluators (LangSmith). https://docs.langchain.com/langsmith/online-evaluations
  13. Atla. Selene models. https://atla-ai.com/selene-models
  14. Patronus AI. GLIDER. https://docs.patronus.ai/docs/evaluators/advanced_concepts/glider
  15. Patronus AI. Lynx. https://docs.patronus.ai/docs/research_and_differentiators/Lynx/base
  16. Arize. LLM as a Judge (Phoenix). https://arize.com/docs/phoenix/evaluation/concepts-evals/llm-as-a-judge
  17. Ragas. Align an LLM as a Judge. https://docs.ragas.io/en/stable/howtos/applications/align-llm-as-judge/
  18. Hugging Face. Evaluation Guidebook: Model as a judge. https://github.com/huggingface/evaluation-guidebook/blob/main/contents/model-as-a-judge/basics.md

Industry T2 (Public Institutions and Standards)

  1. NIST. AI 800-2 (Initial Public Draft): Practices for Automated Benchmark Evaluations of Language Models. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf
  2. NIST. Towards Best Practices for Automated Benchmark Evaluations. https://www.nist.gov/news-events/news/2026/01/towards-best-practices-automated-benchmark-evaluations
  3. UK AI Security Institute. Inspect. https://inspect.aisi.org.uk/
  4. UK AISI. Autonomous Systems Evaluation Standard. https://ukgovernmentbeis.github.io/as-evaluation-standard/
  5. UK AISI. Early Lessons from Evaluating Frontier AI Systems. https://www.aisi.gov.uk/blog/early-lessons-from-evaluating-frontier-ai-systems
  6. UK AISI. Early Insights from Developing Question-Answer Evaluations for Frontier AI. https://www.aisi.gov.uk/blog/early-insights-from-developing-question-answer-evaluations-for-frontier-ai
  7. UK AISI. LLM Judges on Trial: A New Statistical Framework to Assess Autograders. https://www.aisi.gov.uk/blog/llm-judges-on-trial-a-new-statistical-framework-to-assess-autograders
  8. J-AISI. AIセーフティに関する評価観点ガイド 第1.10版 [Guide to Evaluation Perspectives on AI Safety, Version 1.10]. https://aisi.go.jp/assets/pdf/ai_safety_eval_v1.10_ja.pdf
  9. Stanford HAI. The 2026 AI Index Report: Technical Performance. https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance
  10. NIST. 2024 NIST GenAI Pilot Study: Text-to-Text Evaluation Overview and Results. https://www.nist.gov/publications/2024-nist-genai-pilot-study-text-text-evaluation-overview-and-results

Industry T3 (Practitioner Opinion)

  1. Husain, H. (2024). Creating a LLM-as-a-Judge that drives business results. https://hamel.dev/blog/posts/llm-judge/
  2. Husain, H., Shankar, S. (2025). LLM Evals FAQ. https://hamel.dev/blog/posts/evals-faq/
  3. Yan, E. (2024). Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge). https://eugeneyan.com/writing/llm-evaluators/
  4. Yan, E. (2025). An LLM-as-Judge Won’t Save The Product—Fixing Your Process Will. https://eugeneyan.com/writing/eval-process/
  5. Yan, E. (2025). Product Evals in Three Simple Steps. https://eugeneyan.com/writing/product-evals/
  6. Karpathy, A. (2024). X post (evals workload in the Tesla era). https://x.com/karpathy/status/1795873666481402010
  7. Karpathy, A. (2025). X post (evaluation crisis). https://x.com/karpathy/status/1896266683301659068
  8. Fourrier, C. (2024). Let’s talk about LLM evaluation. https://huggingface.co/blog/clefourrier/llm-evaluation
  9. Fourrier, C. et al. (2024). LLM Evaluation Guidebook. https://github.com/huggingface/evaluation-guidebook
  10. Willison, S. (2024). Creating a LLM-as-a-Judge that drives business results (commentary post). https://simonwillison.net/2024/Oct/30/llm-as-a-judge/
  11. Willison, S. (2025). FAQs about AI Evals (commentary post). https://simonwillison.net/2025/Jul/3/faqs-about-ai-evals/
  12. Shankar, S. (2025). In Defense of AI Evals, for Everyone. https://www.sh-reya.com/blog/in-defense-ai-evals/

← All Notes · Home