- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces a new benchmark for grading raw examination documents with complete question papers, grading rubrics and handwritten answer scripts. The authors investigate 5 frontier model-hardness configurations to grade this benchmark. They further did some analysis based on the results.
Critical strengths
The benchmark is new and combines several features that are not jointly included in previous benchmarks.
Critical weaknesses
-
limited scientific insights. It mainly built a benchmark, ran frontier LLMs and did some descriptive analyses. There is no new method or deeper mechanistic analysis, making the work more like a benchmark evaluation study.
-
As shown in table2, this benchmark only includes 10 answer scripts for each STEM subject. Such a limited number of student scripts may not be sufficient to evaluate the automatic graders and draw general conclusions.
-
The reliability of the reference scores is not clear. Even human graders can disagree on the same answer, especially when the rubric leaves room for interpretation. The paper does not report human inter-rater agreement for the reference scores, and agreement metrics such as kappa are usually used instead of accuracy alone for automatic grading evaluation.
-
Some parts of the main text are difficult to understand because important definitions are only explained in the appendix without being linked from the main text. For instance, the definitions of “reasoning steps” and “grading units classified as stated standard or missing boundary”.
-
For the OCR/JSON analysis, frontier agents may already perform OCR internally when processing the original PDFs.
-
Cross-run consistency pass rate is used as a metric where higher consistency is considered better. However, agents could also consistently make the same error across runs.
Questions for authors
None
7 comments
Response to Reviewer C7U6 (2/5)
3. Accurate document reading does not by itself yield sound grading. An input-route intervention separates document reading from the remaining stages of grading. Replacing the raw PDFs with human-verified inputs reduces Material reading failures from 8.9–9.9% to 1.5–1.7%, and raises the joint pass rate by 13.1 points for GPT but only 3.7 points for Qwen. Even then, 21.4–25.9% of correctly scored records fail justification quality, and the highest joint pass rate is 52.5% (Section 5.2).
4. Agents can demonstrate high inconsistency in grading. Among units scored correctly in all three runs, 17.7–25.4% fail cross-run consistency because the stated reasons or grading judgments differ between runs (Appendix G.3, Table 32). Over ten GPT/PDF runs, the share of units that are scored correctly and pass justification quality in every run decreases from 53.8% for one run to 15.6% for ten, so failures are not confined to a fixed subset of difficult units (Appendix G.2, Table 30).
5. Agents lack a stable policy for cases that the rubric leaves open. The grading judgment field, analyzed together with reason and judgment consistency, separates two behaviors. On units with scoring discretion, Claude and Qwen declare a grading judgment most frequently (30.2% and 27.2% of records) but maintain the same decision on the same basis in only 53.0% and 46.8% of the units where they write one. MiniMax declares a judgment in only 7.3% of records and has the highest rate of Score explanation problems (28.1%), which indicates that its interpretation of the rubric remains implicit in its reasons (Section 5.3, Figure 6).
W2: Data coverage and generalizability
In our benchmark, each STEM subject includes 10 answer scripts spanning five overall performance levels. Each answer script contains responses to 40–53 leaf questions, yielding 400–530 grading units per subject (Table 2). These units cover diverse answer forms and reasoning requirements, allowing us to examine grading performance across different question types and student performance levels.
| STEM subject | Answer scripts | Leaf questions per script | Grading units |
|---|---|---|---|
| Mathematics | 10 | 42 | 420 |
| Physics | 10 | 48 | 480 |
| Chemistry | 10 | 53 | 530 |
| Biology | 10 | 40 | 400 |
Across the four STEM subjects, this amounts to 1,830 grading units. Thus, it provides sufficient grading cases to support our evaluation of automatic graders. In addition, student-level bootstrap resampling across the five subject tasks (2,000 replicates; Appendix G.2) yields 95% intervals within ±3 percentage points for each agent's joint pass rate, with all pairwise differences between agents excluding zero. As the comparison below shows, its response-level scale exceeds CHECK-MAT and EDU-CIRCUIT-HW and is comparable in scale to the Mohler dataset and SAS-Bench.
| Benchmark | Subject coverage | Grading cases | Counting scope |
|---|---|---|---|
| CHECK-MAT (Khrulev, 2025) | Mathematics | 122 | Handwritten solutions |
| EDU-CIRCUIT-HW (Sun et al., 2026) | Circuit analysis | 1,334 | Handwritten solutions: 513 observation + 821 test |
| Mohler et al. (2011) | Computer science | 2,273 | Text answers to 80 questions from 31 students |
| SAS-Bench (Lai et al., 2026) | Nine subjects | 4,109 | Expert-annotated responses to 1,030 questions |
| ReMarkBench — STEM tasks | Mathematics, Physics, Chemistry, Biology | 1,830 | Student–leaf-question grading units |
Across all six grading tasks, ReMarkBench comprises 3,570 grading units in total, exceeding the evaluation-set sizes of widely used frontier-model benchmarks:
| Benchmark | Area | Evaluation items | Counting scope |
|---|---|---|---|
| HLE-Diamond (Center for AI Safety et al., 2026) | Expert knowledge and reasoning | 1,000 | 500 reasoning and 500 knowledge questions |
| GPQA Diamond (Rein et al., 2023) | Graduate-level scientific reasoning | 198 | Diamond subset of physics, chemistry, and biology questions |
| BrowseComp (Wei et al., 2025) | Web research | 1,266 | Questions requiring web browsing to find hard-to-locate information |
| SWE-bench Verified (OpenAI, 2024) | Software engineering | 500 | Human-validated GitHub issue-resolution tasks |
| Terminal-Bench-Science 0.1 (Terminal-Bench-Science Team, 2026) | Agentic scientific research workflows | 70 | Research tasks across the life, physical, mathematical, engineering, and earth sciences |
| GDP.pdf (Garre et al., 2026) | Professional document understanding and reasoning | 100 | Tasks across ten domains, based on real-world PDFs |
| ReMarkBench — six tasks | Examination grading across six tasks | 3,570 | 3,450 student–leaf-question + 120 writing-dimension grading units |
Response to Reviewer C7U6 (3/5)
To test generalizability on independent materials, we applied our end-to-end evaluation pipeline to 100 handwritten student responses across 10 GCSE mathematics questions from Medly (35 zero-score, 23 partial-credit, and 42 full-credit answers), graded on the PDF route by GPT-5.6 Sol (Codex 0.150.1, medium reasoning effort, K = 3 independent runs = 300 records) and evaluated by the same GPT-5.6 Sol judge (low reasoning effort; Appendix G.10, Table 40):
| Measure | Count | Rate (%) |
|---|---|---|
| Score accuracy (record level) | 291 / 300 | 97.0 |
| Justification quality pass rate (record level) | 208 / 300 | 69.3 |
| Justification failure among correctly scored records | 88 / 291 | 30.2 |
| Cross-run consistency pass rate (response level, K = 3) | 80 / 100 | 80.0 |
| Consistency failure among all-correct responses | 15 / 94 | 16.0 |
| Joint pass rate (response level, K = 3) | 42 / 100 | 42.0 |
Even at 97.0% score accuracy, 30.2% of correctly scored records fail justification quality and 16.0% of responses scored correctly in all three runs fail cross-run reason or judgment consistency, yielding a 42.0% joint pass rate. These external results directly reproduce our main findings outside the examination system used to build ReMarkBench.
W3: Reference-score reliability
Reference-score reliability. Three experts independently scored every grading unit in the five subject tasks using the official rubrics. Agreement was measured before disagreements were resolved through cross-checking and discussion (Appendix B.4, Table 11).
| Subject | Three-expert exact agreement (%) | Mean pairwise quadratic weighted κ |
|---|---|---|
| Mathematics | 93.57 | 0.9825 |
| Physics | 95.42 | 0.9717 |
| Chemistry | 95.85 | 0.9676 |
| Biology | 88.50 | 0.9494 |
| English Reading | 98.09 | 0.9786 |
These results show high expert agreement across all five tasks, supporting the reliability of the reference scores. For disputed or borderline answers, the experts discussed the answer, applicable grading points, and rubric interpretation to agree on the final reference score. Appendix B.4.3 provides three adjudication examples. For English Writing, we use official script-level performance labels, cross-checked against the answer scripts by two experts.
Agreement metrics. We also report quadratic weighted κ between agent and reference scores under the PDF route with three grading runs (summary ranges in Appendix G.2; per-subject breakdown below).
| Agent | Mathematics | Physics | Chemistry | Biology | English Reading | Five-subject mean | English Writing |
|---|---|---|---|---|---|---|---|
| GPT | 0.947 | 0.882 | 0.903 | 0.736 | 0.939 | 0.882 | 0.847 |
| Claude | 0.925 | 0.861 | 0.918 | 0.929 | 0.946 | 0.916 | 0.936 |
| MiniMax | 0.853 | 0.797 | 0.835 | 0.763 | 0.891 | 0.828 | 0.837 |
| Grok | 0.935 | 0.862 | 0.900 | 0.871 | 0.945 | 0.903 | 0.929 |
| Qwen | 0.944 | 0.883 | 0.922 | 0.893 | 0.929 | 0.914 | 0.922 |
Coefficients are computed per run and averaged over the three runs. The five-subject mean weights the five subject tasks equally. Subject-task scores are normalized by each question’s maximum score; English Writing compares converted script-level predictions with official performance levels.
The five-subject mean quadratic weighted κ ranges from 0.828 to 0.916 across agents (and 0.837 to 0.936 on English Writing), providing additional support for the high score agreement observed with exact-match accuracy. However, these same outputs achieve only 55.5%–75.2% justification quality pass rates on the five subjects and 0.0%–2.2% on English Writing (Table 3), showing that high score κ alone can still mask widespread justification and consistency failures. We retain exact-match accuracy because it determines whether the agent assigns the correct score.
W4: Key definitions in the main text
Following your suggestion, beyond defining reasoning steps and scoring-discretion levels inline, we reviewed the full main text to clarify key definitions, subset denominators, and appendix cross-references upon first appearance. Specifically:
Response to Reviewer C7U6 (1/5)
Thank you for recognizing the novelty of our benchmark. We clarify our methodological contribution, key definitions, and the role of cross-run consistency below. We also provide additional evidence on data coverage, reference-score reliability, and input-format effects.
W1: Methodological contribution and empirical insights
We respectfully disagree that the paper offers no new method and only descriptive analysis. We need to clarify that the paper contributes a new evaluation methodology for grading and a comprehensive mechanistic analysis of the agentic grading process that locates where grading fails and tests why.
Methodological contribution. Section 3.2 identifies three challenges of grading real examinations that score-only evaluation on pre-segmented answers does not test. Our method makes each one measurable (Appendix H, Table 43). For every grading unit, the agent returns a grading record with a score, evidence, a reason, and any grading judgment, and we treat the justification as an argument from the student's answer and the rubric to the score.
Reading raw documents. In a real examination, grading starts from the question paper, the rubric, and each student's handwritten answer script. Pairing each answer with its question and rubric entry is itself part of grading: the hierarchy of nested questions must be recovered from the layout, an answer may span several pages, and handwriting, notation, tables, and drawings must be read correctly. To our knowledge, ReMarkBench is the first benchmark to evaluate arbitrary grading agents end to end on examination documents not pre-segmented by question (Table 1).
Connecting answers to grading points. A score is useful to a student or a teacher only if it can be traced to what the student wrote: which part of the answer earned or lost each mark. Student answers rarely follow the rubric point by point, since steps are reordered, merged, skipped, or reworded, so establishing this link is the core of grading, and a correct score does not show that it was established. To our knowledge, our evaluation method is the first to check grading justifications automatically against both the answer and the rubric without a reference rationale.
Resolving what the rubric leaves open. Real rubrics are incomplete: they often do not say how to credit equivalent expressions, partially correct answers, or other borderline cases. More than a third of our grading units in the five subject tasks leave such discretion to the grader. The grader must then decide, and fair grading requires the same decision each time the same answer is graded. A score reveals neither the decision nor whether it is stable. To our knowledge, our evaluation method is the first to compare the grounds behind a score across repeated runs for each grading unit.
The proposed evaluation metrics and method in ReMarkBench are also validated against blind human-expert annotations and across judge models (Section 5.1; Appendix F, Tables 25–29).
ReMarkBench enables a mechanistic analysis of agents' grading failures. Beyond final scores, ReMarkBench records the chain from the student's answer to the evidence, the grading basis, and the score, allowing us to pinpoint where along this chain a failure arises and test why through input-route interventions and task-condition analysis.
1. Agents can give the correct score but cannot yet reliably justify it. On the same cases and under the same criteria, the best agent matches two human experts in score accuracy (95.3% vs. 95.5%), whereas all agents fall well below the experts in justification quality (59.7–77.7% vs. 92.0%) (Appendix G.11). On the full benchmark, 21.5–37.9% of correctly scored records fail justification quality (Appendix G.3, Table 31): the justification misstates the student's answer, cites evidence that its reasoning does not use, or leaves the awarded score unexplained. In Figure 1, one run reads the student's 15 cm as 16 cm and still awards the correct score because the rubric accepts 14–16 cm.
2. The capability that limits grading differs across agents. Justification quality distinguishes failures in reading the answer, extracting evidence, and explaining the score. MiniMax is limited by reading: it misstates the student's answer in 18.2% of records, compared with 8.9–9.9% for the other agents. GPT is limited by score explanation: with score accuracy similar to Grok's (88.3% vs. 89.9%), it leaves scores ungrounded roughly three times as often (10.2% vs. 3.3% of records) (Section 5.2).