Shuichiro Ogawa
日本語

Notes · updated 2026-09-18

What Replaced What in Test Scoring and Item Writing

A focused industry survey of two stages of Japanese mock exams and large-scale testing: scoring and item writing. What digital scoring replaced was not judgement but the movement of paper, the tallying, and the reconciliation.

Contents (14)
  1. Overview
  2. One stage of scoring is now text conversion
  3. Scoring’s workforce left the marking room
  4. The pay structure did not change
  5. What item writing lost was search
  6. What the review checks
  7. Three conditions holding item writing in place
  8. Generative AI has reached the draft and stopped
  9. Why it stops at a draft
  10. The draft is a position the rules set
  11. Item-writing hours are not recorded
  12. Where data use has reached
  13. The asymmetry in market data
  14. Scope and method

Overview

Chiba Prefecture found more than 900 marking errors in its 2023 public high school entrance exams. They spanned 98 schools, and six students were failed by mistake. Teachers at each school had been marking the papers by hand.

From the 2024 exams the prefecture introduced digital scoring. The marking did not disappear. Instead, each answer sheet is scored twice, along two separate paths, and the two results are compared; where they disagree the markers confer and score again.

This note asks which tasks in scoring and item writing were replaced, and how. It draws on public-agency documents, market research, employers’ recruitment terms, and labour-market statistics. Collection ran in mode: industry, and the ledger sits at source/review/mock-exam-scoring-item-writing/industry.md.

Five conclusions.

First, scoring lost transport, tallying, and reconciliation rather than judgement. Saga Prefecture cut marking time by 50 to 75 percent across its prefectural schools, yet its superintendent of education states that “constructed-response questions carrying partial credit cannot be auto-scored.” Saitama Prefecture’s procurement specification accepts “automated first-pass scoring, with visual marking only where the results disagree” as a legitimate design. The machine takes the cases that agree; the cases that disagree come back to a person.

Second, scoring became a two-stage process. A handwritten answer is first converted into text by an image-processing model, and that text is what the scoring model sees. In a joint study by Tokyo University of Agriculture and Technology and the National Center for University Entrance Examinations, the same model scored 78.8 to 81.7 percent accuracy on human-transcribed answers but fell to 51.1 to 80.8 percent on machine-transcribed ones. Transcription error becomes scoring error. A new source of error now sits inside the pipeline, and scoring accuracy is no longer decided by the scoring model alone.

Third, the workforce moved from the marking room to the home, and the pay structure did not change. Kawai Juku recruits home-based contract markers at piece rates of roughly 20 to 125 yen per answer sheet, tax included. Z-Kai also pays by the piece and requires marking to be finished by midnight on an assigned weekday. Work that once meant gathering in a marking room for an hourly wage became work done at home for a rate per sheet. Paying by output rather than by hours survived digitisation unchanged.

Fourth, what item writing lost was search, not judgement. The National Center for University Entrance Examinations built a full-text search system to check whether a newly written item duplicates an existing one. Its stated priority is not semantic matching or tolerance of spelling variation but “completeness of retrieval.” Attempts to generate items themselves, combining textbook data with retrieval-augmented generation, ended in a recorded finding that “building the RAG system alone was not enough to generate the questions the item writers intended.” Because item-writing hours are not recorded, review is layered, and content cannot leak, generation has not reached the point of replacing judgement.

Fifth, generative AI reaches a draft and no further. That position is both where the technology lands and where the rules draw the line. The Ministry of Education’s guidelines treat output as a draft, leave the final decision with staff, and name the use of AI by students in term examinations and the delegation of assessment to AI output as inappropriate examples. The teachers surveyed in the METI demonstration most often put the saving at 10 to 30 minutes per test, and 36 percent reported either no gain or that it took longer. Measurement gives its own reason. The defects in AI-written items concentrate not in duplication with past items but in ambiguity, where more than one option holds, and in the control of difficulty. Generating and verifying with the same model lets its characteristic errors agree on both sides, and flawed items pass. Implementers’ accounts from 2023 and 2026 agree that human review cannot be removed.

One stage of scoring is now text conversion

Automated scoring of constructed responses does not stand on the scoring model alone. Answers in schools are mostly handwritten, so the pipeline runs in two stages: an image-processing model converts the writing to text, and the scoring model reads the text. Handwriting is often messy, and conversion accuracy drives scoring accuracy.

The size of that dependency is measured in the ANLP study cited above. On 25 short constructed-response items from three Japanese-language workbooks for lower secondary students, 1,372 answers, the researchers compared human transcription against transcription produced by a commercial handwriting-recognition system. With GPT-4o, accuracy on human data was 78.8 (zero-shot), 80.7 (few-shot), and 81.7 (fine-tuned). On machine data it fell to 51.1, 64.2, and 80.8. Few-shot learning degraded most; the authors conclude that “for scoring automatically transcribed answers containing noise, few-shot learning is insufficient compared with fine-tuning.” Fine-tuning is more robust, but it means rebuilding the model’s training data every time the scoring target changes.

Variance widens as well. When the researchers varied which items served as few-shot examples and repeated the run twenty times, human data averaged 75.1 percent with a 6.3-point standard deviation across items; machine data averaged 47.7 percent with an 11.3-point spread. The study states plainly that it cannot yet separate how much of that spread comes from per-item differences in recognition accuracy and how much from the scoring model itself. Accuracy drops without anyone knowing where.

Some teams removed the recognition layer instead. One developer deleted an embedding-similarity tier that sat between keyword matching and the scoring model. For an item whose model answer was “extract only the even numbers from the array,” a wrong answer, “extract only the odd numbers,” scored above 0.9 similarity and entered the path that locks in the highest mark. Similarity is structurally helpless against negation, antonyms, and off-by-one answers. The saving from that tier had been seven yen a month. In its place went a database of common wrong answers and a mechanism that makes the scoring model quote its evidence verbatim so the server can check the quote against the answer text. In production data, citations appeared even on 25 answers judged not to match at all, including strings that did not exist in the response.

Verification of the scoring system itself develops holes too. In an open-source marking tool written by a lower-secondary teacher, more than 1,200 self-tests all reported success. Fourteen of them were running against an empty exam containing no questions at all. The cause was a single mistyped word in a fixture, but a safety fallback that routes unknown values into an “other” bucket let the bad data through. Correcting the word broke none of the existing tests, which meant those tests had never checked that behaviour in the first place. The developer added 27 integration tests that verify values, and set three rules: practice data must contain only values that could occur in production, tests must assert that counts are not zero, and a fix must be validated by deliberately breaking the code and confirming the test fails.

Scoring’s workforce left the marking room

Kawai Juku recruits mock-exam markers as home-based contractors. Pay is by output: roughly 20 to 125 yen per answer sheet including tax, plus a fixed printing fee per round. The rate varies with the exam, the subject, and the length of the expected answer.

The procedure runs like this. The marker prints the exam paper, the study guide, and the marking criteria, reads them, and scores answers against the criteria using a web app on a smartphone. No written commentary is required. Kawai Juku nonetheless states that this is not “mere mechanical work of putting circles and crosses where the answer matches”: understanding the criteria and the candidate’s response requires subject expertise. A printer is mandatory, and printing outside the home is not permitted. The second-stage written test and a mock marking exercise are also taken at home and returned by post. A marking period runs ten days to two weeks per exam, containing two to four marking rounds of about four days each.

Z-Kai’s home digital marking has the same structure. Answers converted to image data are scored on a PC through a dedicated application, processing the same question across many candidates together. Images arrive on an assigned weekday and must be completed by midnight that day. Applicants must have at least a junior-college diploma or be a current university or graduate student, and they complete unpaid training at home before starting.

Dedicated outsourcing businesses exist as well. Japan Institute for Educational Measurement lists in its ISMS certification scope “BPO covering the production, operation, scoring, and analysis of test items.” Persol Business Process Design has staff visually mark the constructed-response questions that automated scoring handles poorly, leaving teachers only the essay questions, and also loads teacher-written test items into the system on their behalf. Coaching-sector AI marking takes the same shape. Educational Network has AI begin marking immediately on submission, with a tutor reviewing the result before it reaches the student. The AI does not return the answer.

The pay structure did not change

Moving the work home did not change how pay is determined. Kawai Juku and Z-Kai both pay by the piece. Agaroot Academy, which commissions qualification-exam materials, pays 1,500 yen for writing an original multiple-choice item and 1,000 yen for an explanation of a past item. One recruitment ad for mock-exam item writing quotes an hourly wage of 1,200 to 1,500 yen, but that role bundles proofreading, checking, and editorial assistance into a single job category, so it is not a rate for item writing alone.

The rates cannot simply be compared. Kawai Juku states that the exams it commissions vary with candidate numbers and that there is no quota of sheets to mark, so rate times volume is not income. Tax treatment is not consistent across sources either.

Where scoring sits in employment statistics is unclear. Neither the Japan Standard Occupational Classification nor the Basic Survey on Wage Structure has a category for markers, and the nearest category, “private teachers and cram-school instructors,” cannot be filtered down to markers alone. What is available is the industry-wide figure. In the first half of 2025, according to the Ministry of Health, Labour and Welfare’s Survey on Employment Trends, part-time workers in “education and learning support” had a hiring rate of 19.9 percent and a separation rate of 19.6 percent, the highest of any industry.

The task that machines took over in item writing is finding past and similar items by hand. The research and development division of the National Center for University Entrance Examinations built a full-text search system on an internal Linux server to check whether a newly written item overlaps with existing ones. Pointing the system at a new item file returns related past items, grouped by keywords extracted from the file. Indexing the Center’s exam questions across all subjects takes about 1.5 hours, and updates run daily as incremental registrations.

The design priority is explicit. The requirements state that “terminology in examination questions is standardised, so the semantic matching and tolerance of spelling variation that search systems normally require matter comparatively little. Completeness of retrieval takes priority over them.” General search aims at nearness of meaning; item writing needs coverage. An oversight becomes an examination error, so not missing anything matters more.

What stayed with people sits outside the search itself. Requirements included searching vertical Japanese text, including for the Japanese language paper, and searching inside PDF, Word, and Excel files, which produced a conversion step for each format. LaTeX-format formulas are not supported, and English is handled with wildcards because the index does not account for inflection. The past items being searched are purchased as a commercial dataset and loaded by hand. The requirements also specify that the search mechanism must not be a black box, that its specification be public, and that its source code be open and command-line launchable; the developer published the whole set. Even with the tool in place, shaping the material into a form the tool can read remains human work.

What the review checks

Review in item writing is not an evaluation of how good an item is. It checks whether the item holds up, and what it examines changes with the stage.

Immediately after an item is written, three things are examined: whether it stays within the syllabus and the curriculum guidelines, whether the answer is correct, and whether the scoring arithmetic is right. The Ministry of Education’s notice requires this before, during, and after the examination, checked at least twice and often three times, including by people other than the item’s author. Examination questions and their answers are published as a rule; where a constructed-response item cannot be given a single determinate answer, the intention behind the item or several standard answers are published instead.

Before use, what is examined is whether the item works. From its trial surveys the National Center picked out items with extremely low correct-answer rates, items where high-scoring candidates chose a wrong answer, and items with low discrimination, and had subject-specific working groups review them. Discrimination is defined as “the discrimination of each item, the Pearson correlation between item score and total score.” It measures whether the item separates candidates who can from those who cannot. For each subject, roughly five university faculty members, forty-eight in all, were interviewed about the aim, material, format, and difficulty of the items. No target average correct-answer rate is set.

The results of review are kept for the next examination. The Center asked its item storage and management system to store difficulty and discrimination (item parameters) alongside each item, to manage the relation between source items and derived items, and to attach metadata for frequent replacement and revision. The system assumes the small-question format; large-question formats require design changes. Whether items can be written and reviewed remotely was to be considered cautiously on security grounds.

No trial has yet put this stage in the hands of a machine. The Ministry of Education has budgeted 60 million yen for fiscal 2027 research to trial online and generative-AI-assisted item creation and review.

Three conditions holding item writing in place

Item writing shows almost no AI substitution. Three conditions overlap.

Review is layered. The Ministry of Education, Culture, Sports, Science and Technology requires entrance-exam items to be checked at least twice and often three times, including by people other than the item’s author. The National Center’s item-authoring committee is capped at 800 members; in fiscal 2022 it ran 22 subcommittees with 461 people, covering 6 subjects and 30 subject areas over about two years, with each subcommittee meeting 12 to 19 times a year, totalling 320 sessions and 1,163 days. Separately sit a first review committee of about 160 people, a second of up to 30, and an evaluation and analysis committee of up to 95. Item writing is not one person’s task but an iteration of many committees. Interviews with universities running computer-based testing report that authoring is “more labour-intensive in creating and checking questions than paper tests.” The review burden does not fall when the delivery format changes.

Content cannot leak. The Center operates access-control systems, bans personal computers, smartphones, and recording media, and isolates electronic data on a dedicated server behind biometric authentication and a severed external network. Committee members’ names remain confidential until the end of the fiscal year in which the exam they worked on is administered. When designing its item storage and management system, the Center stated that remote item writing and review would be considered cautiously on security grounds. A system for marking answers at home took hold; a system for writing questions at home did not.

Items are a depleting asset. The Center’s own account notes that a single subject needs thousands to tens of thousands of items, that pretesting exposes a substantial number of simulated candidates to the content, and that memorised leakage cannot be entirely prevented. Reuse shifts item parameters such as discrimination, forcing retirement and replacement. A practice report from Tokyo Gakugei University’s attached Setagaya Elementary School argues that once an item is scored, returned to pupils, and explained, it is effectively public, so reusing it unchanged is unfair. Item writing is not a task that finishes once it is done; it recurs.

Generative AI has reached the draft and stopped

Attempts to bring generative AI into item writing stop at producing drafts. The Ministry of Education’s guidelines list “creating teaching materials and test questions” among school-administration uses, but treat the output as a draft, with the final decision resting with staff. In a Ministry of Economy, Trade and Industry demonstration, Benesse tested automatic draft generation of test items in primary schools. The demonstration ran from November 2023 to February 2024 at Warabi Municipal Kita Elementary School in Saitama and Showa Gakuen Elementary School in Chiba. The final report is public as a ministry commissioned-survey report, so the breakdown behind the headline numbers can be checked against the primary source.

Forty-seven teachers answered. 96 percent said it had advantages for teaching quality and 64 percent that it would cut working time, with the saving put at about 30 minutes per test against writing items by hand. Within that 64 percent, 11 percent expected a saving of an hour or more, 13 percent 30 to 60 minutes, and 40 percent 10 to 30 minutes. The remaining 36 percent split between “hardly any efficiency gain” at 21 percent and “it actually takes longer than writing from scratch” at 15 percent. The largest group sits in the 10 to 30 minute band.

Scoring is not part of this demonstration. The report’s subject is item generation, and the word scoring never appears anywhere in it. The other three companies funded under the same call covered school administration, image analysis, and the automatic generation of teaching records; no demonstration took scoring as its subject. Benesse was also the only one taking item generation as its subject. The vision was split into three phases, and the demonstration covers only the first, “improving the quality of teaching materials.” Individualised learning and learner autonomy, the second and third phases, sit outside its scope.

One of the generated examples printed in the report contains a hallucination. The explanation attached to an item generated without knowledge data makes a false statement about the ownership of Mount Fuji. The land above the eighth station belongs to Sengen Taisha in Fujinomiya, Shizuoka, as settled by a 1974 Supreme Court decision. The report presents the output as an example of questions that “stay generic” and does not mention the error.

Why it has not gone further shows in the failure records. A joint project between a printing company and a textbook publisher prototyped item generation by combining textbook and workbook data with retrieval-augmented generation. The conclusion after repeated testing was that “building the RAG system alone was not enough to generate the questions the item writers intended.” Only after validating prompts for consistent question and passage formats, and validating data splitting aligned to the textbook’s structure, could the system generate items matching specified formats and ranges with kanji and expression suited to the textbook’s grade level. Specifying difficulty from past-item correct-answer rates, generating items similar to past ones, and handling figures and tables through multimodal processing all remain open.

Because generation falls short, the measurement side’s norms thicken instead. The precision of a test score is reliability; whether a test measures what it intends to measure is validity, framed through Messick’s six aspects and Kane’s interpretive argument. What matters there is treating validity not as something to be “verified” or “established” but as “an effort to be increased.” Validity is a process that keeps being renewed rather than a state to be achieved. Faster generation does not remove the need for it.

Why it stops at a draft

The reason is not that generation is unusable. It is that the share usable as written is low, and the shortfall concentrates in particular properties.

A study generating items for adult learners across four models found hallucination at about 5% and items unusable as written at about 8%. The most common defect was not overlap with past items. It was ambiguity, where more than one option holds. In one case four models given the same prompt did not agree on the correct answer, and multiple choice was the format generation handled worst.

In medical education research, 49% of ChatGPT-generated items had construction defects and 22% had factual or conceptual errors. Even so, 91% (59 of 65) were judged reasonable as a starting point for revision. The state is one where most items need work and most are worth the work. The authors concluded it is only feasible under subject-expert supervision. That 91% is the measured value behind the word draft.

Against expert-written items, the gap shows in the control of difficulty. In a primary examination in emergency medicine, 100 ChatGPT-4o items were compared with 100 from an expert panel: factual errors 6% versus 4%, irrelevance 6% versus 0%, duplication 14% versus 7%, all narrow. Inappropriate difficulty was 14% versus 1%. Discrimination was comparable at 0.22 versus 0.26, but the difficulty index was 0.78 versus 0.69, with the AI items falling on the easier side. Item-writing time was 24.5 person-hours versus 96. The shortfall is not speed. It is the share of generated items that can be used as they stand.

That difficulty control is the hard part is confirmed by the implementers. The Benesse staff member who led item generation in the METI demonstration describes the hardest task as teaching the model “work people had previously done implicitly,” naming per-unit content and difficulty setting as the specifics. The reaction of teachers in the demonstration was that if you write prompts this far, you can produce good items. Producing a good item requires converting tacit knowledge into explicit instruction. That conversion is done by a person, not by the model.

The teachers in the demonstration record the same structure from the other side. “The AI isn’t infallible, so in the end a check will be needed, and once you factor in the cost of generating items as text and the work of selecting and typing prompts, I feel the benefit of using generative AI is thin against the effort it takes.” A distribution where the most common saving is 10 to 30 minutes and 36 percent feel no gain is consistent with that comment. Demand for the draft itself, though, is strong. “Building everything from zero means the purchased test is faster, but with the purchased test there is always something you want changed. What I would most like is a state where a base comes out and I can then rework just the questions I want.” The demand for a draft and the complaint about finishing one sit in the same comment box under the same question.

Filtering and revision close the gap. In a field study across 91 classes and 1,686 students, AI-written exams and a past-paper benchmark were compared under item response theory, and discrimination came out equivalent at 1.3 versus 1.2. But what was administered were the top ten items by quality label after a self-correction loop. Equivalence describes not the level AI reached alone but the level of the items that survived selection.

Verification stays with people for the same reason. A developer who had been generating and verifying with the same model found that when the generating side settled on a wrong answer, the same model reproduced the same error and agreed, letting the flawed item through. Tracing errors arise not at random but systematically and in a way specific to the model, so separating sessions does not make the check independent. The fix was to split generation and verification across different models, shuffle the options, and record which model verified each item. Even then, every item that passes verification is still read by a person before publication. “Stack two AIs and the humans don’t go down,” is that developer’s conclusion.

Line up the accounts from 2023 and 2026 and the conclusion has not moved. An implementation that migrated from GPT-3.5 to GPT-4 because the older model passed template strings, and recorded a 7.9 percent misjudgement rate, and a practice that reached κ 0.7 to 0.8 on constructed-response marking, both keep final responsibility and double-checking with people. What changed is the method for finding errors.

On the scoring side, a regulator draws the line at the same place. England’s qualifications and examinations regulator holds that using AI as the sole marking mechanism does not comply with its rules, and that human judgement is required in marking decisions. It cites the difficulty an unexplainable black box creates for appealing a score, and notes that marking performance differs greatly between mathematics and English.

Teachers’ own assessments run along the same line. In a survey of 1,659 primary and secondary teachers in Korea, the highest-rated use of generative AI was generating ideas for lesson themes and assessment plans, at 76.0%. The lowest was assisting marking with a rubric, at 32.6%. Uses on the creating side rate high; uses on the judging side rate low.

Adoption does not so much reduce verification work as change its kind. In a survey of 328 Japanese school staff, the task consuming the most time was learning how to use AI and how to teach with it, at 35.1%, followed by checking submitted work for copy-paste from AI and verifying the accuracy of AI answers, both at 32.4%. Fact-checking and copy-paste checks are stages AI brought in.

The draft is a position the rules set

The Ministry of Education’s guidelines fix the draft position explicitly. Generative AI is only ever a draft, the text reads, and staff must check, revise, and complete the output themselves. It then names, as inappropriate examples, having students use generative AI in term examinations and small tests, and having assessment carried out on AI output without a teacher’s own judgement. The draft is where the technology lands and a line the rules draw at the same time.

Since fiscal 2024, no public demonstration with item creation as its subject can be confirmed. The ministry’s successor projects moved to school administration, centred on writing student reports, analysing questionnaires, and handling non-attendance. The school-administration chat that Benesse supplied under a later project does not list item creation among its key uses. The list of time savings from earlier projects runs through lesson plans, student reports, training reports, and questionnaire analysis; item creation does not appear.

The cost of reaching a draft at all is also on the record. In the fiscal 2024 demonstration, writing the prompts was itself the main obstacle. Many staff gave up because they did not know what to type, or stopped after one exchange when the answer was not good enough. The reference data pulls in the other direction. Spreadsheet forms needed cell merges undone before use, and handwritten notes and observation memos needed digitising first. Because the preparation cost held back the accumulation of data, the model had too little to draw on and answer accuracy did not rise.

The National Center for University Entrance Examinations sorts the concerns about automated item generation into seven. Reliability and validity; limits of control; hallucination; leakage; overlap with copyrighted works and past items; duplication between items and the resulting loss of variety; bias and discriminatory expression. Overlap with past items is one of the seven.

The list rests on a position reached after reviewing prior implementations. The conclusion common to those cases is that automated generation is only ever an assistant to human item writers, and that “as a way of producing an initial draft it is extremely good.” The realistic flow is put as generating a draft by machine, then having people scrutinise and revise it. Five countermeasures are listed, and the ones called the most important are review by human experts and statistical validation through pretesting. Both are stages that human item writing also involves, and neither is dropped by adding automation.

The line sits here because of what assessment is for. Expert commentary attached to pilot-school practice notes that in a Japanese-language unit the AI substituted for “the ability to structure an argument,” which was the point of the unit; that it is hard to read what the learner decided to reject among the AI’s suggestions; and that students accept AI answers as correct and use them without noticing errors. The OECD’s Digital Education Outlook 2026 calls omitting the learning process cognitive offloading, and argues that the future of generative AI in education turns not on how efficiently teachers’ tasks can be automated but on how well teachers can exercise professional judgement. If delegating item writing to a machine conflicts with the purpose of assessment, the range that can be delegated is settled somewhere other than the maturity of the technology. For now, both land in the same place.

Item-writing hours are not recorded

The reason item-writing load cannot be measured lies in the records. In a collective-bargaining document from Tokushima University, the administration explains that the number of committee meetings and the time spent marking can be collected to some degree. But “for individual faculty work such as writing questions at home or in the laboratory and the time spent on it, records often do not exist.” The union inverts this: marking time is plainly measurable, because markers gather in the marking room in the morning and cannot leave until marking is done.

The same asymmetry shows in pay design. Tottori University of Environmental Studies sets 40,000 yen per item-writing round, 60,000 for the lead. Tokushima University uses a point system: 20 points for the first-semester mathematics paper, 10 for the second, plus 8 for the lead, for 38 points. What a university can justify as the basis for payment is limited to hours that leave a record.

Surveys of faculty involved in authoring also fail to put a number on the load. Saori Kubo of Tohoku University divided authoring work into three phases, item writing, setting marks and criteria plus scoring, and post-examination evaluation, and derived 255 data points from semi-structured interviews with six faculty members. Heavy load, rapid turnover of those responsible, and the risk of examination errors appear as obstacles. The sample is six people and does not represent the population.

Where data use has reached

Data use has advanced around scoring rather than inside it, and on the design side.

The National Institute for Educational Policy Research plans to introduce item response theory into the National Assessment of Academic Ability: lower-secondary science in fiscal 2025, lower-secondary English in fiscal 2026, and Japanese and mathematics in fiscal 2027. IRT scores are set to a mean of 500 with a standard deviation of 100; IRT bands run one to five with three as the reference, and a student in band three can answer a difficulty-three item correctly with roughly 80 percent probability. Test design moves from measuring correct-answer rates to estimating item difficulty and discrimination.

The National Center likewise analyses discrimination as “the discrimination of each item, the Pearson correlation between item score and total score.” In the 2017 trial survey, somewhat many items had low correct-answer rates, and subject-specific working groups examined items with low discrimination and items where high-scoring candidates chose wrong answers. The National Institute’s measurement team sets out the trade-off plainly: narrowing the difficulty range raises estimation precision for a particular ability but thins the reported content, while widening it makes characterisation easier and precision worse.

There is also movement toward holding items as an asset. Center documents cite shared examinations requiring roughly 12,000 items for medicine and 8,000 for dentistry. The Center estimates that accumulating 10,000 items over five years costs about 101.25 million yen a year, 506.25 million over five. The Ministry of Education has budgeted 60 million yen for fiscal 2027 research to trial online and generative-AI-assisted item creation and review.

On the scoring side, automated judgement is structured so that it never becomes the final decision. The Ministry’s trial combines machine scoring with two independent human markings and a human check of the results. Saitama Prefecture’s specification puts automated scoring first and sends only disagreements to visual review. Saga Prefecture’s superintendent stating that “constructed-response questions carrying partial credit cannot be auto-scored” is an acknowledgement of that limit. Its deputy director for promotion, meanwhile, notes that reduction is largest for science, social studies, and English, where multiple-choice items dominate, and that “we can now teach based on data analysis rather than on teachers’ impressions.” The value of digital scoring is shifting from time saved to answers becoming data.

In the United States, scoring appears as the area where AI adoption lags. A 2026 Gallup and Walton Family Foundation survey found 58 percent of teachers report no guidance at all on grading and feedback, second only to one-to-one tutoring among ten tasks. Sixty percent of teachers use AI tools, but only 16 percent of them use AI for grading at least monthly, fewer than for lesson preparation or administration. Fifty-seven percent of AI users say grading and feedback quality improved, so those who use it rate it highly. Absent guidance and low use are observed together.

The asymmetry in market data

Scoring is beginning to be counted as a market; item writing is not. In Japan, Fuji Chimera Research Institute places “digital scoring systems” among 38 items in education DX and ICT, and its 2023 edition placed the same item among 39. Neither lists an item corresponding to “question creation.” Neither the 15 markets in Yano Research Institute’s education industry white paper nor the 11 markets in its school-business survey include mock exams, scoring, or item writing.

Overseas surveys do have item-writing categories, but no figures. The Business Research Company places “Test Items And Question Banks” among content subsegments, 360iResearch places “Assessment Authoring Software” among software subsegments, and Stratistics MRC places “Test Creation Platforms.” None discloses revenue for the segment.

Figures appear only in vendor claims. Prometric states that its item-writing AI achieves “up to 18 times the speed” of conventional processes, “up to 85 percent reduction” in review time, and “over 60 percent reduction” in test-development cost. No source, methodology, measurement period, or sample is given, so this survey excluded it from the ledger.

Domestic market size for digital scoring systems likewise cannot be confirmed from published research-house material. It appears to exist only inside Fuji Chimera’s paid report.

Scope and method

Collection ran under the source-collection skill in mode: industry. Four source-researcher profiles ran in parallel: official, analyst, developer-voice, and labour-market. The labour-market profile normally targets design occupations, so the target job category was reinterpreted to cover scoring and item-writing work. The ledger sits at source/review/mock-exam-scoring-item-writing/industry.md.

Where a public agency’s programme page returns HTTP 403, the report itself was still obtained by downloading the PDF directly from the index of commissioned survey reports. Both METI’s final results report and the National Center’s seminar proceedings were read page by page this way. A failed PDF retrieval is not evidence that a document does not exist.

Where the preceding survey The Workload of Mock-Exam Operations and Where AI Actually Lands covered the whole exam-operations pipeline, this note narrows to two stages. Items judged heavy, meaning vendor claims of product effect asserted numerically without methodology, were excluded from the ledger and are not used here. Figures are reported as their sources’ claims, not as findings of this note.

Unverified items

  • Many Ministry of Education and municipal PDFs are compressed with FlateDecode and AES, so text cannot be extracted; descriptions rest on search indexes and summaries. This is not evidence that the documents do not exist.
  • Saitama Prefecture’s specification and Chiba Prefecture’s improvement page return HTTP 404 and were confirmed via cache and index.
  • Saga Prefecture’s 50 to 75 percent reduction figure has no disclosed calculation method or sample.
  • Some National Center documents sit on dncwww.boctac.net, whose DNS does not resolve, and were confirmed through archives.
  • The Ministry of Economy, Trade and Industry’s “Future Classroom” HTML page returns HTTP 403; the final results report itself was obtained as a PDF through the commissioned-survey index (see References). An HTML failure here does not mean the underlying document is unavailable.
  • Prometric’s US patent could not be confirmed at source (HTTP 403).
  • The 2025 Gallup survey does not disclose sample size or field period, and the original report was not obtained.
  • 6Wresearch contradicts itself on the same page, showing 3.81 percent headline CAGR against annual growth of 9.8 to 11.34 percent.
  • Fuji Chimera’s domestic digital-scoring market size sits inside a paid report and the figure was not obtained.
  • Japan Institute for Educational Measurement’s ISMS scope is third-party certification wording; the breakdown of actual contracted work is not disclosed.
  • Part of Tokushima University’s bargaining material (handout documents) is not public.
  • The Ministry of Education’s 60 million yen for fiscal 2027 is a planned procurement amount.
  • Nagase’s job listing was confirmed through a site reproducing a Hello Work posting.
  • Benesse’s item-bank scale (767 items, over 70,000 cumulative respondents, two-parameter logistic model) could not be confirmed: the source PDF fails certificate validation and the introduction page carries no figures.
  • The National Center’s CBT item-authoring working group figures (181 drafts to 32 items registered, about 20 members in 5 teams, 200 to 500 respondents needed per item for accurate parameter estimation against roughly 160 in the trial) were confirmed only via search index; the source returns 404.
  • ETS’s e-rater advisory flags and human-machine discrepancy thresholds were confirmed only in summary; the full text was not obtained.
  • TOPPAN’s 90 percent scoring accuracy and ScanNet’s 10 percent improvement are self-reported with no disclosed sample, period, or definition.
  • One developer’s removal of an embedding-similarity tier and the Education Family News article could not be attributed to a named individual (handle only, or via media).
  • Kawai Juku’s marking-operations page returns HTTP 403; no new primary material beyond the existing ledger was obtained.
  • Technical documents on rd.dnc.ac.jp were unreachable (connection refused).
  • No primary material was found for Kyōiku Dōjin-sha’s “Hanaru AI.”
  • No official statistics aggregate markers or item writers as independent occupations.
  • No primary data was found tracking digital-scoring adoption’s effect on per-sheet rates over time.
  • No primary data was found tracking how generative AI has changed item-writer requirements or rates.
  • The 5 percent hallucination and 8 percent unusable-item figures from the four-model comparison could not be confirmed in the source PDF’s body text.
  • The 49 percent defect rate and 22 percent factual-error rate for ChatGPT-generated items rest on a publisher’s abstract.
  • The emergency-medicine comparison figures were confirmed through an abstract.
  • The field study of AI-written exams (91 classes, 1,686 students) is an arXiv preprint and has not been peer reviewed.
  • Arsaga Partners’ survey does not disclose its population definition or sampling method.
  • The final report of the METI demonstration with Benesse was obtained, and discloses the model (Azure OpenAI Service GPT-4) and the input and output fields; the full prompts, the number of items generated, the error rate, and the hours teachers spent correcting output are not recorded.
  • The Benesse staff interviews carry no publication date, only that they were conducted in 2024.
  • The Zenn and note developer accounts are tied to handles only; real names and affiliations are unverified.
  • Whether the public call for the METI demonstration anticipated scoring is unconfirmed: the contractor’s announcement page now redirects and cannot be read. [requires primary verification]
  • Whether the output examples printed in the final report are the raw generation log or were tidied for publication is not stated. [to confirm]
  • The absence of an item-creation case in the list of past time savings is not proof that none was run.

References

Public agencies (T1)

Research houses and think tanks (T2)

Employers and institutions

Developer and practitioner accounts (T3)

Academic studies (item and scoring quality)

The Workload of Mock-Exam Operations and Where AI Actually Lands / Research Currents in AI, Education, and the Learning Sciences: Evidence on Generative AI and Learning (2024–2026) / AI Adaptation in Design Education — The Current State and Structural Challenges of Curriculum Reform / Competency Measurement Premised on AI Use — The Aptitude x AI Skill Interaction and Measurement Frameworks

Author: Shuichiro Ogawa (Design Researcher / Consultant) About me →