acceptodds
Under review as a conference paper at ICLR 2027

ReMarkBench: Benchmarking Agentic Grading of Raw Examination Documents

Abstract

Autonomous grading agents that assess student work with minimal human intervention can reduce teachers' workload and provide timely, informative feedback at scale. However, existing benchmarks mostly focus on pre-segmented student answers and check whether scores are correct, not whether each score rests on the student's answer and the rubric, or whether its grounds stay the same when the answer is regraded. To address these gaps, we introduce ReMarkBench, a benchmark for evaluating end-to-end agentic grading from raw examination documents: complete question papers, grading rubrics, and handwritten answer scripts from a public university-entrance examination. It covers 3,570 grading units in six grading tasks across Mathematics, Physics, Chemistry, Biology, and English. For every grading unit, the grading agent must produce a score and a supporting justification that cites the student's actual answer and explains the score under the rubric. We therefore assess three complementary dimensions: score accuracy, justification quality, and cross-run consistency. Across five frontier grading agents, each a model–harness configuration reading the raw documents directly, even the best reaches a joint pass rate of only 41.0%, averaged equally across the six tasks over three independent runs, and many correctly scored records still fail justification quality. Substantial gaps remain even for an agent grading from human-verified structured inputs instead of the raw documents, so the difficulty goes beyond reading the documents. ReMarkBench provides a testbed for diagnosing these gaps and advancing accurate, interpretable, and reproducible automated grading.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agents, automated grading, LLM-as-a-judge, document understanding, interpretability, cross-run consistency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C7U6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces a new benchmark for grading raw examination documents with complete question papers, grading rubrics and handwritten answer scripts. The authors investigate 5 frontier model-hardness configurations to grade this benchmark. They further did some analysis based on the results.

Critical strengths

The benchmark is new and combines several features that are not jointly included in previous benchmarks.

Critical weaknesses
  1. limited scientific insights. It mainly built a benchmark, ran frontier LLMs and did some descriptive analyses. There is no new method or deeper mechanistic analysis, making the work more like a benchmark evaluation study.

  2. As shown in table2, this benchmark only includes 10 answer scripts for each STEM subject. Such a limited number of student scripts may not be sufficient to evaluate the automatic graders and draw general conclusions.

  3. The reliability of the reference scores is not clear. Even human graders can disagree on the same answer, especially when the rubric leaves room for interpretation. The paper does not report human inter-rater agreement for the reference scores, and agreement metrics such as kappa are usually used instead of accuracy alone for automatic grading evaluation.

  4. Some parts of the main text are difficult to understand because important definitions are only explained in the appendix without being linked from the main text. For instance, the definitions of “reasoning steps” and “grading units classified as stated standard or missing boundary”.

  5. For the OCR/JSON analysis, frontier agents may already perform OCR internally when processing the original PDFs.

  6. Cross-run consistency pass rate is used as a metric where higher consistency is considered better. However, agents could also consistently make the same error across runs.

Questions for authors

None

7 comments
AuthorsOpenReview ↗

Response to Reviewer C7U6 (1/5)

Thank you for recognizing the novelty of our benchmark. We clarify our methodological contribution, key definitions, and the role of cross-run consistency below. We also provide additional evidence on data coverage, reference-score reliability, and input-format effects.

W1: Methodological contribution and empirical insights

We respectfully disagree that the paper offers no new method and only descriptive analysis. We need to clarify that the paper contributes a new evaluation methodology for grading and a comprehensive mechanistic analysis of the agentic grading process that locates where grading fails and tests why.

Methodological contribution. Section 3.2 identifies three challenges of grading real examinations that score-only evaluation on pre-segmented answers does not test. Our method makes each one measurable (Appendix H, Table 43). For every grading unit, the agent returns a grading record with a score, evidence, a reason, and any grading judgment, and we treat the justification as an argument from the student's answer and the rubric to the score.

  1. Reading raw documents. In a real examination, grading starts from the question paper, the rubric, and each student's handwritten answer script. Pairing each answer with its question and rubric entry is itself part of grading: the hierarchy of nested questions must be recovered from the layout, an answer may span several pages, and handwriting, notation, tables, and drawings must be read correctly. To our knowledge, ReMarkBench is the first benchmark to evaluate arbitrary grading agents end to end on examination documents not pre-segmented by question (Table 1).

  2. Connecting answers to grading points. A score is useful to a student or a teacher only if it can be traced to what the student wrote: which part of the answer earned or lost each mark. Student answers rarely follow the rubric point by point, since steps are reordered, merged, skipped, or reworded, so establishing this link is the core of grading, and a correct score does not show that it was established. To our knowledge, our evaluation method is the first to check grading justifications automatically against both the answer and the rubric without a reference rationale.

  3. Resolving what the rubric leaves open. Real rubrics are incomplete: they often do not say how to credit equivalent expressions, partially correct answers, or other borderline cases. More than a third of our grading units in the five subject tasks leave such discretion to the grader. The grader must then decide, and fair grading requires the same decision each time the same answer is graded. A score reveals neither the decision nor whether it is stable. To our knowledge, our evaluation method is the first to compare the grounds behind a score across repeated runs for each grading unit.

The proposed evaluation metrics and method in ReMarkBench are also validated against blind human-expert annotations and across judge models (Section 5.1; Appendix F, Tables 25–29).

ReMarkBench enables a mechanistic analysis of agents' grading failures. Beyond final scores, ReMarkBench records the chain from the student's answer to the evidence, the grading basis, and the score, allowing us to pinpoint where along this chain a failure arises and test why through input-route interventions and task-condition analysis.

1. Agents can give the correct score but cannot yet reliably justify it. On the same cases and under the same criteria, the best agent matches two human experts in score accuracy (95.3% vs. 95.5%), whereas all agents fall well below the experts in justification quality (59.7–77.7% vs. 92.0%) (Appendix G.11). On the full benchmark, 21.5–37.9% of correctly scored records fail justification quality (Appendix G.3, Table 31): the justification misstates the student's answer, cites evidence that its reasoning does not use, or leaves the awarded score unexplained. In Figure 1, one run reads the student's 15 cm as 16 cm and still awards the correct score because the rubric accepts 14–16 cm.

2. The capability that limits grading differs across agents. Justification quality distinguishes failures in reading the answer, extracting evidence, and explaining the score. MiniMax is limited by reading: it misstates the student's answer in 18.2% of records, compared with 8.9–9.9% for the other agents. GPT is limited by score explanation: with score accuracy similar to Grok's (88.3% vs. 89.9%), it leaves scores ungrounded roughly three times as often (10.2% vs. 3.3% of records) (Section 5.2).

AuthorsOpenReview ↗

Response to Reviewer C7U6 (2/5)

3. Accurate document reading does not by itself yield sound grading. An input-route intervention separates document reading from the remaining stages of grading. Replacing the raw PDFs with human-verified inputs reduces Material reading failures from 8.9–9.9% to 1.5–1.7%, and raises the joint pass rate by 13.1 points for GPT but only 3.7 points for Qwen. Even then, 21.4–25.9% of correctly scored records fail justification quality, and the highest joint pass rate is 52.5% (Section 5.2).

4. Agents can demonstrate high inconsistency in grading. Among units scored correctly in all three runs, 17.7–25.4% fail cross-run consistency because the stated reasons or grading judgments differ between runs (Appendix G.3, Table 32). Over ten GPT/PDF runs, the share of units that are scored correctly and pass justification quality in every run decreases from 53.8% for one run to 15.6% for ten, so failures are not confined to a fixed subset of difficult units (Appendix G.2, Table 30).

5. Agents lack a stable policy for cases that the rubric leaves open. The grading judgment field, analyzed together with reason and judgment consistency, separates two behaviors. On units with scoring discretion, Claude and Qwen declare a grading judgment most frequently (30.2% and 27.2% of records) but maintain the same decision on the same basis in only 53.0% and 46.8% of the units where they write one. MiniMax declares a judgment in only 7.3% of records and has the highest rate of Score explanation problems (28.1%), which indicates that its interpretation of the rubric remains implicit in its reasons (Section 5.3, Figure 6).

W2: Data coverage and generalizability

In our benchmark, each STEM subject includes 10 answer scripts spanning five overall performance levels. Each answer script contains responses to 40–53 leaf questions, yielding 400–530 grading units per subject (Table 2). These units cover diverse answer forms and reasoning requirements, allowing us to examine grading performance across different question types and student performance levels.

STEM subjectAnswer scriptsLeaf questions per scriptGrading units
Mathematics1042420
Physics1048480
Chemistry1053530
Biology1040400

Across the four STEM subjects, this amounts to 1,830 grading units. Thus, it provides sufficient grading cases to support our evaluation of automatic graders. In addition, student-level bootstrap resampling across the five subject tasks (2,000 replicates; Appendix G.2) yields 95% intervals within ±3 percentage points for each agent's joint pass rate, with all pairwise differences between agents excluding zero. As the comparison below shows, its response-level scale exceeds CHECK-MAT and EDU-CIRCUIT-HW and is comparable in scale to the Mohler dataset and SAS-Bench.

BenchmarkSubject coverageGrading casesCounting scope
CHECK-MAT (Khrulev, 2025)Mathematics122Handwritten solutions
EDU-CIRCUIT-HW (Sun et al., 2026)Circuit analysis1,334Handwritten solutions: 513 observation + 821 test
Mohler et al. (2011)Computer science2,273Text answers to 80 questions from 31 students
SAS-Bench (Lai et al., 2026)Nine subjects4,109Expert-annotated responses to 1,030 questions
ReMarkBench — STEM tasksMathematics, Physics, Chemistry, Biology1,830Student–leaf-question grading units

Across all six grading tasks, ReMarkBench comprises 3,570 grading units in total, exceeding the evaluation-set sizes of widely used frontier-model benchmarks:

BenchmarkAreaEvaluation itemsCounting scope
HLE-Diamond (Center for AI Safety et al., 2026)Expert knowledge and reasoning1,000500 reasoning and 500 knowledge questions
GPQA Diamond (Rein et al., 2023)Graduate-level scientific reasoning198Diamond subset of physics, chemistry, and biology questions
BrowseComp (Wei et al., 2025)Web research1,266Questions requiring web browsing to find hard-to-locate information
SWE-bench Verified (OpenAI, 2024)Software engineering500Human-validated GitHub issue-resolution tasks
Terminal-Bench-Science 0.1 (Terminal-Bench-Science Team, 2026)Agentic scientific research workflows70Research tasks across the life, physical, mathematical, engineering, and earth sciences
GDP.pdf (Garre et al., 2026)Professional document understanding and reasoning100Tasks across ten domains, based on real-world PDFs
ReMarkBench — six tasksExamination grading across six tasks3,5703,450 student–leaf-question + 120 writing-dimension grading units
AuthorsOpenReview ↗

Response to Reviewer C7U6 (3/5)

To test generalizability on independent materials, we applied our end-to-end evaluation pipeline to 100 handwritten student responses across 10 GCSE mathematics questions from Medly (35 zero-score, 23 partial-credit, and 42 full-credit answers), graded on the PDF route by GPT-5.6 Sol (Codex 0.150.1, medium reasoning effort, K = 3 independent runs = 300 records) and evaluated by the same GPT-5.6 Sol judge (low reasoning effort; Appendix G.10, Table 40):

MeasureCountRate (%)
Score accuracy (record level)291 / 30097.0
Justification quality pass rate (record level)208 / 30069.3
Justification failure among correctly scored records88 / 29130.2
Cross-run consistency pass rate (response level, K = 3)80 / 10080.0
Consistency failure among all-correct responses15 / 9416.0
Joint pass rate (response level, K = 3)42 / 10042.0

Even at 97.0% score accuracy, 30.2% of correctly scored records fail justification quality and 16.0% of responses scored correctly in all three runs fail cross-run reason or judgment consistency, yielding a 42.0% joint pass rate. These external results directly reproduce our main findings outside the examination system used to build ReMarkBench.

W3: Reference-score reliability

Reference-score reliability. Three experts independently scored every grading unit in the five subject tasks using the official rubrics. Agreement was measured before disagreements were resolved through cross-checking and discussion (Appendix B.4, Table 11).

SubjectThree-expert exact agreement (%)Mean pairwise quadratic weighted κ
Mathematics93.570.9825
Physics95.420.9717
Chemistry95.850.9676
Biology88.500.9494
English Reading98.090.9786

These results show high expert agreement across all five tasks, supporting the reliability of the reference scores. For disputed or borderline answers, the experts discussed the answer, applicable grading points, and rubric interpretation to agree on the final reference score. Appendix B.4.3 provides three adjudication examples. For English Writing, we use official script-level performance labels, cross-checked against the answer scripts by two experts.

Agreement metrics. We also report quadratic weighted κ between agent and reference scores under the PDF route with three grading runs (summary ranges in Appendix G.2; per-subject breakdown below).

AgentMathematicsPhysicsChemistryBiologyEnglish ReadingFive-subject meanEnglish Writing
GPT0.9470.8820.9030.7360.9390.8820.847
Claude0.9250.8610.9180.9290.9460.9160.936
MiniMax0.8530.7970.8350.7630.8910.8280.837
Grok0.9350.8620.9000.8710.9450.9030.929
Qwen0.9440.8830.9220.8930.9290.9140.922

Coefficients are computed per run and averaged over the three runs. The five-subject mean weights the five subject tasks equally. Subject-task scores are normalized by each question’s maximum score; English Writing compares converted script-level predictions with official performance levels.

The five-subject mean quadratic weighted κ ranges from 0.828 to 0.916 across agents (and 0.837 to 0.936 on English Writing), providing additional support for the high score agreement observed with exact-match accuracy. However, these same outputs achieve only 55.5%–75.2% justification quality pass rates on the five subjects and 0.0%–2.2% on English Writing (Table 3), showing that high score κ alone can still mask widespread justification and consistency failures. We retain exact-match accuracy because it determines whether the agent assigns the correct score.

W4: Key definitions in the main text

Following your suggestion, beyond defining reasoning steps and scoring-discretion levels inline, we reviewed the full main text to clarify key definitions, subset denominators, and appendix cross-references upon first appearance. Specifically:

Reviewer_qrGEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReMarkBench, a benchmark for evaluating whether grading agents can perform end-to-end assessment using authentic examination papers, marking rubrics, and handwritten student responses. It contains 3,570 grading units across six task categories and evaluates agents along three dimensions: score accuracy, rationale quality, and run-to-run consistency. Five model–agent harness configurations are evaluated over three runs each, with the best configuration achieving a joint pass rate of 41.0%. The authors also compare three input formats: raw PDFs, model-generated OCR transcriptions, and human-verified JSON, to distinguish document-reading failures from downstream grading errors.

Critical strengths
  1. The benchmark better reflects real-world grading workflows. Unlike existing grading benchmarks that rely on pre-segmented inputs, ReMarkBench requires agents to process raw examination PDFs and authentic handwritten responses. This enables evaluation of document reading and question–answer alignment, which are often overlooked in previous benchmarks.

  2. The evaluation extends beyond score accuracy. The paper assesses whether grading rationales faithfully represent student responses, whether the cited evidence supports assigned scores, and whether scores and rationales remain consistent across runs. The results show that many cases fail rationale-quality checks despite receiving correct scores, suggesting that score accuracy alone may overestimate grading reliability.

  3. The input-format analysis provides useful diagnostic insights. For the GPT configuration, the joint pass rate increases from 28.0% with PDF inputs to 29.4% with OCR and 39.4% with human-verified JSON. These results suggest that improving input quality helps but does not eliminate grading errors, as substantial failures remain even with verified structured inputs.

Critical weaknesses
  1. Limited data diversity constrains generalizability. The dataset consists of approximately 80 student scripts from a single examination, selected by performance level. Although it contains 3,570 grading units, many originate from the same scripts and should not be treated as independent samples. The findings may therefore not generalize to other examination years, assessment systems, scanning conditions, or student populations.

  2. The reliability of reference scores is insufficiently established. The paper states that experienced graders assigned and cross-checked reference scores but does not report inter-annotator agreement or provide sufficient details on resolving disagreements and borderline rubric cases. Uncertainty in reference scores could affect the validity of the reported grading accuracy.

  3. LLM-based rationale evaluation introduces measurement uncertainty. Although the authors conduct expert validation, it primarily covers the GPT/PDF configuration. For Evidence extraction, judge–reference agreement is only κ = 0.58, compared with an average expert–expert agreement of κ = 0.81, with similar gaps in overall rationale quality. Furthermore, all 15 manually reviewed English writing samples were judged as failing, making it difficult to assess whether the judge can reliably distinguish acceptable from unacceptable rationales. These limitations weaken confidence in the rationale-quality and joint pass metrics.

  4. The input-format comparison only partially isolates document-reading effects. While all five configurations are evaluated on PDFs, the PDF/OCR/JSON comparison is limited to the GPT configuration. Differences in models and agent harnesses also complicate cross-configuration comparisons. The paper does not sufficiently clarify how PDF rendering, available tools, context handling, and computational budgets are aligned across harnesses.

  5. The interpretation and comparability of aggregate metrics require clarification. The joint pass metric requires all three evaluation dimensions to pass simultaneously, providing a meaningful but strict reliability criterion that obscures individual failure modes. Moreover, English writing uses script-level evaluation rather than question-level reference scores. Equally averaging these tasks may therefore combine metrics with different interpretations.

Questions for authors
  1. Could the authors report inter-annotator agreement for reference scores and clarify how disagreements and borderline rubric cases were resolved?

  2. Given the relatively low judge agreement for Evidence extraction and overall rationale quality, could the authors extend validation across configurations and judges, and include manually annotated English writing samples with acceptable rationales?

  3. Since the approximately 80 scripts come from a single examination and were selected by performance level, which findings do the authors expect to generalize across examination years, scanning conditions, and student populations? Could the results be replicated on independent examination materials?

  4. As the PDF/OCR/JSON comparison is limited to GPT, could the authors clarify differences in PDF processing and computational budgets across agent harnesses, and examine whether other configurations exhibit similar effects from input format?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qrGE (1/5)

Thank you for your constructive feedback. We address your concerns below regarding reference-score agreement, judge reliability, generalizability, input formats and agent harnesses, and the interpretation of aggregate metrics.

W2 & Q1: Reference-score agreement

We recruited three human experts to score every grading unit in the five subject tasks using the official rubrics. The degree of agreement among them is reported below (Appendix B.4, Table 11).

SubjectThree-expert exact agreement (%)Mean pairwise quadratic weighted κ
Mathematics93.570.9825
Physics95.420.9717
Chemistry95.850.9676
Biology88.500.9494
English Reading98.090.9786

From the table, we can find that experts independently assign identical scores to 88.50–98.09% of grading units, with a mean pairwise quadratic weighted κ of 0.9494–0.9825. This high agreement, together with adjudication of disputed cases, supports the reference scores as a sound basis for evaluating agent score accuracy.

For the handling of disagreements and borderline rubric cases. For disputed or borderline answers, the experts discussed the answers, the applicable grading points, and the rubric interpretation to reach an agreement on the final reference score. Appendix B.4.3 provides three adjudication examples.

For reference scores in English Writing, as unit-level reference scores are unavailable, we use the official script-level performance labels, cross-checked against the corresponding answer scripts by two experts.

W3 & Q2: Judge reliability

We evaluate the LLM-judge reliability with three different ablations: (W3.1) extending blind human-expert validation from GPT/PDF to Claude/PDF grading outputs, (W3.2) replacing the judge model itself across all five grading agents, and (W3.3) testing passing and defect-injected English Writing justifications.

W3.1 Cross-agent validation against human experts (fixed judge: GPT-5.6 Sol; graded agents: GPT/PDF vs. Claude/PDF). To check whether our LLM judge (GPT-5.6 Sol) remains accurate when evaluating non-GPT grading agents or favors its own model family, three human experts blindly relabeled sampled grading outputs from Claude/PDF using the same decision rules and sampling design as GPT/PDF (115 justification records, 100 reason-consistency units, and 60 judgment-consistency units per configuration; 3-expert majority vote = Ref.; Appendices F.1–F.3, Tables 25–26):

Evaluated CheckJudge–Ref. Agree (%)Judge–Ref. κInter-Expert κJudge Fail (%)Expert Fail (%)Diff (pp)
GPT/PDF (graded agent)
Subcategory: Misstated answer or unsupported rule99.10.880.858.77.8+0.9
Subcategory: Evidence extraction84.20.580.8129.135.4−6.4
Subcategory: Ungrounded score or ignored answer part88.00.670.6919.116.6+2.5
Subcategory: Weak argument or contradiction83.60.520.5213.812.1+1.7
Overall justification quality79.80.720.8445.444.0+1.4
Cross-run: Reason consistency94.20.820.7722.121.0+1.1
Cross-run: Judgment consistency93.70.830.9357.358.7−1.4
Claude/PDF (graded agent)
Subcategory: Misstated answer or unsupported rule95.70.820.9110.58.0+2.5
Subcategory: Evidence extraction84.90.670.8613.518.9−5.4
Subcategory: Ungrounded score or ignored answer part89.80.690.8011.210.9+0.3
Subcategory: Weak argument or contradiction87.20.550.738.610.5−1.9
Overall justification quality81.60.760.8331.935.7−3.8
Cross-run: Reason consistency92.90.770.8818.417.8+0.6
Cross-run: Judgment consistency89.30.790.8747.149.3−2.2

(Agree (%) and Fail (%): five-subject design-weighted judge–reference agreement rate and failure rate from Table 26; Diff = Judge − Expert, computed before rounding. κ: sample Cohen's κ of judge vs. 3-expert majority reference, alongside mean pairwise κ among the 3 experts from Table 25.)

AuthorsOpenReview ↗

Response to Reviewer qrGE (2/5)

On both configurations, design-weighted judge–expert agreement reaches 83.6%–99.1% across all subcategories and consistency checks (79.8% and 81.6% on overall justification quality), and every check passes the alternative annotator test at ε = 0.20. Design-weighted failure rates stay within 2.5 pp of human experts on every subcategory and consistency check except Evidence extraction (under-reported by 6.4 and 5.4 pp), and overall justification failure rates closely track experts (+1.4 and −3.8 pp). Across all seven checks, the judge agrees with human experts on which configuration has the lower failure rate; in particular, it rates Claude/PDF better than GPT/PDF on overall justification quality (31.9% vs. 45.4%) and both consistency checks. Within the validated samples and configurations, we do not observe a consistent pattern of overall failure-rate overestimation or preference for the judge's own model family.

W3.2 Cross-judge sensitivity (fixed grading outputs across all five agents; replacing the judge: GPT-5.6 Sol vs. Claude Opus 5). Whereas W3.1 varies the graded agent against human experts, we also test varying the judge model itself (Section 5.1; Appendix F.3, Table 27): keeping all checks, prompts, and pass rules unchanged, we re-judged the run-1 records of six students (277 records per configuration on the same grading units across the five subject tasks) using Claude Opus 5 instead of GPT-5.6 Sol.

JudgeQwen/PDFGrok/PDFGPT/PDFClaude/PDFMiniMax/PDF
GPT-5.6 Sol (main judge)74.1 (1)66.9 (2)65.2 (3)63.9 (4)60.3 (5)
Claude Opus 5 (replacement judge)75.5 (1)67.6 (2)59.3 (4)66.5 (3)56.7 (5)
Diff (GPT − Claude, pp)−1.4−0.7+5.9−2.6+3.7

(Cells report five-subject equally weighted justification quality pass rates in %, with within-row configuration ranks in parentheses; Diff = GPT main − Claude replacement, computed before rounding.)

Replacing the judge largely preserves the overall justification quality ranking across configurations (Kendall τ = 0.8): Qwen/PDF, Grok/PDF, and MiniMax/PDF retain first, second, and last place (only third and fourth swap, matching GPT-5.6 Sol's full-benchmark order in Table 3), and no pass rate shifts by more than 5.9 pp, showing that our main configuration comparisons hold across judge models.

W3.3 Acceptable English Writing justifications. We added 15 human-written records verified by experts as passing all applicable checks, plus 45 targeted defective variants.

Validation examplesRate (%)
Expert-verified acceptable records (pass rate)93.3
Misstated answer or unsupported rule (defect detected)100.0
Evidence extraction (defect detected)90.0
Ungrounded score or ignored answer part (defect detected)91.1
Weak argument or contradiction (defect detected)100.0
All targeted defective variants (defect detected)94.8

Rates are averaged over three runs. On these validation examples, the judge recognizes acceptable justifications and detects targeted defects, suggesting that the low agent pass rates are not simply due to indiscriminate rejection. Appendix F.4 (Tables 28–29) reports the validation results, and Appendix F.5 provides a complete expert-verified passing example.

Q3 / W1 — Generalizability and independent materials.

Although our primary materials come from one sitting of a public university-entrance examination, they span all five official performance levels under consistent annual standards and were scanned under standard real-world school examination conditions. Specifically, we expect three qualitative findings to extend across examination sittings and student cohorts. High score accuracy can coexist with justification and consistency failures. Task difficulty varies with answer form and rubric structure. Justification failures show distinct patterns across material reading, evidence extraction, and score explanation.

To test generalizability on independent materials, we applied our end-to-end evaluation pipeline to 100 handwritten student responses across 10 GCSE mathematics questions from Medly (35 zero-score, 23 partial-credit, and 42 full-credit answers), graded on the PDF route by GPT-5.6 Sol (Codex 0.150.1, medium reasoning effort, K = 3 independent runs = 300 records) and evaluated by the same GPT-5.6 Sol judge (low reasoning effort; Appendix G.10, Table 40):

AuthorsOpenReview ↗

Response to Reviewer qrGE (3/5)

MeasureCountRate (%)
Score accuracy (record level)291 / 30097.0
Justification quality pass rate (record level)208 / 30069.3
Justification failure among correctly scored records88 / 29130.2
Cross-run consistency pass rate (response level, K = 3)80 / 10080.0
Consistency failure among all-correct responses15 / 9416.0
Joint pass rate (response level, K = 3)42 / 10042.0

Even at 97.0% score accuracy, 30.2% of correctly scored records fail justification quality and 16.0% of responses scored correctly in all three runs fail cross-run reason or judgment consistency, yielding a 42.0% joint pass rate. These results confirm that high score accuracy can mask justification and consistency failures on external materials.

The four most frequent justification failures, using the taxonomy in Section 5.2, are:

Failure typeRate (%)
Evidence extraction – unused evidence17.0
Evidence extraction – missing evidence6.3
Score explanation – weak argument6.1
Material reading – misstated answer3.7

As in the main experiments, unused evidence is more frequent than missing evidence, while weak arguments and misstated answers also occur.

W4 & Q4 (1): harness differences

Harness differences (PDF processing, tools, context, and compute budgets). All five grading agents receive the exact same original PDFs, grading instructions, and seven-field output schema, run at medium or default reasoning effort without artificial token caps, and read rendered page images directly without external OCR, PDF text extraction, or external retrieval, while retaining each harness's native scheduling and tool execution. Because the five configurations span four different agent harnesses, their runtime behavior can differ along two axes: (1) PDF visual processing (rendering DPI and how often runs zoom into answer regions or crop pages) and (2) execution environment and compute budget (exposed tool counts, share of image- or file-reading calls, per-script subagent isolation, automatic context compactions, runtime, and token usage). Run-log statistics across all five PDF configurations are summarized below (Section 5.1; Appendix G.7, Table 36).

1) PDF rendering and visual reading strategy (vs. five-task mean misstated-answer rate):

ModelAgent HarnessRendered DPIZoom Runs (%)Crop Runs (%)Misstated Answer (%)
GPT-5.6 SolCodex 0.150.1130–3006709.9
Claude Opus 5Claude Code 2.1.26070–120089839.3
MiniMax M3Claude Code 2.1.260100–300112218.2
Grok 4.6Grok Build 1.0.40150–3007209.8
Qwen3.8-MaxQoder CN 1.1.59130–120083228.9

All agents render pages via deterministic tools (such as pdftoppm and pymupdf) following two strategies: Grok renders full pages directly at 150–300 DPI upfront, whereas Claude, Qwen, and GPT first browse at 70–180 DPI and then zoom into answer regions or crop two-page spreads at 200–1200 DPI in 67%–89% of runs. Spanning all four harnesses, these four agents achieve nearly identical misstated-answer rates (8.9%–9.9%). Conversely, MiniMax shares the exact same harness (Claude Code 2.1.260) with Claude, yet renders mostly at 100–150 DPI, rarely zooms (11%) or crops (22%), and reaches 18.2%. Thus, all four harnesses provide sufficient rendering clarity; while model and harness effects interact, harness limits alone cannot explain these reading differences, which still depend largely on the model and how it uses the inspection tools.

2) Available tools, context handling, and compute budget (vs. five-task mean joint pass rate):

ModelAgent HarnessAvail. ToolsRead Calls (%)1:1 Subagent (%)CompactionsMean Time (min)Mean Tokens (Out)Joint Pass (%)
GPT-5.6 SolCodex 0.150.1957100040.225.6M (232K)33.4
Claude Opus 5Claude Code 2.1.2602680100012.624.7M (273K)37.1
MiniMax M3Claude Code 2.1.260287594035.728.0M (369K)28.4
Grok 4.6Grok Build 1.0.402685100015.49.8M (305K)41.6
Qwen3.8-MaxQoder CN 1.1.593182100061.438.6M (420K)48.7

(Zoom, Crop, and 1:1 Subagent (%): share of runs adopting each strategy; Read Calls (%): share of tool calls reading images or files; Compactions: automatic context-compaction events; Mean Tokens (Out): total and parenthesized output tokens per run; Misstated Answer and Joint Pass (%): equally weighted means across the five subject tasks.)

Reviewer_UXG8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReMarkBench to evaluate the ability of agents to perform end-to-end grading directly from raw examination documents: unlike existing evaluations that receive pre-segmented student answers and rubrics, it requires the agent to read the complete question paper, rubric, and handwritten answer scripts, and to perform question identification, answer alignment, scoring, and justification generation on its own. The benchmark contains 3,570 grading units across six tasks: Mathematics, Physics, Chemistry, Biology, English Reading, and English Writing, and evaluates agents along three dimensions: score accuracy, justification quality, and cross-run consistency. The authors test five model configurations; the best configuration achieves a joint pass rate of only 41.0%, averaged equally across the six tasks. Through OCR and human-verified structured-input experiments, the paper further distinguishes document-reading errors from subsequent grading-reasoning errors.

Critical strengths

S1. Real-world examination grading requires not only correct scores but also justifications grounded in evidence and consistency across repeated gradings. By incorporating these requirements into a unified evaluation framework, this paper characterizes grading quality more comprehensively than evaluations that focus solely on final scores.

S2. The benchmark directly uses raw PDF documents, hierarchical question structures, handwritten answers, and authentic rubrics, covering multiple subjects and diverse answer forms, and thus reveals issues that are easily overlooked in evaluation settings where answers have already been pre-segmented.

S3. The authors examine not only final scores but also categorize justification failures into answer misreading, inadequate evidence extraction, and insufficient score explanation, and further analyze the relationship between question type, rubric ambiguity, and performance.

Critical weaknesses

W1. The paper lacks human graders as a performance baseline. Appendix J acknowledges that the study "does not include human graders as a performance baseline." Without such a baseline, readers cannot determine whether the 41.0% joint pass rate indicates that current agents perform poorly or that the benchmark itself is difficult and human graders would not fare much better.

W2. The joint pass rate mixes two different units of analysis. For the five subject tasks, the joint pass rate requires a correct score, justification quality in every run, and cross-run consistency, whereas English Writing is counted per script (correct performance level plus justification quality pass rate), and only score consistency is compared for writing (Section 4, Appendix B.2). Consequently, the six-task equally weighted average of 41.0% aggregates metrics defined at different granularities.

W3. Compatible changes in explanations count as consistency failures (Table 20), while explicitly declared departures from clear rubric requirements do not automatically fail justification quality (Table 16). These choices may penalize harmless variation and, in principle, allow some rubric-inconsistent justifications to pass the quality checks. Human grading baselines and sensitivity analyses would help establish how well the joint pass rate reflects practical grading reliability.

Questions for authors

Q1. Can the authors provide the score accuracy, justification quality pass rate, and cross-run consistency of human graders on the same set of grading units, as an upper bound on performance?

Q2. The six-task equally weighted average joint pass rate mixes two protocols: the subject tasks and the writing task. Can the authors report them separately, or provide a unified protocol?

Q3. What proportion of reason-consistency failures are compatible shifts rather than conflicting claims? Would allowing compatible shifts materially change the joint pass rates or rankings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer UXG8 (1/2)

Thank you for your constructive feedback. We address your concerns below regarding human grading baselines, the comparability of the two evaluation protocols, and the treatment of compatible shifts and departures from the rubric.

W1 & Q1: Human grading baseline

We recruited two human experts for each task to independently grade 20 randomly selected cases from each of the five subject tasks and all 20 English Writing scripts, establishing human grading baselines evaluated against agents on the same cases with the same criteria. Because repeated blind grading of the same answer scripts by the same person doesn’t apply to human graders, we approximate cross-run consistency with between-expert consistency across the two experts: score, reason, and judgment consistency for the five subject tasks, and score consistency for English Writing (Appendix G.11, Tables 41–42).

Five-subject mean (same cases as the human baseline)

GraderScore accuracy (%)Justification quality (%)Consistency (%)Joint pass rate (%)
Human95.592.073.067.0
GPT92.059.767.030.0
Claude95.361.381.033.0
MiniMax88.760.069.035.0
Grok92.767.374.042.0
Qwen93.077.770.048.0

English Writing

GraderScore accuracy (%)Justification quality (%)Consistency (%)Joint pass rate (%)
Human80.065.451.744.2
GPT56.72.243.30.8
Claude76.70.661.70.0
MiniMax55.00.315.00.0
Grok73.30.068.30.0
Qwen75.02.257.51.7

Human graders achieve higher joint pass rates than all five agents: 67.0% versus 30.0%–48.0% for the five-subject mean, and 44.2% versus 0.0%–1.7% for English Writing. While some agents approach human graders in score accuracy, human graders achieve much higher justification quality pass rates across all tasks, supporting justification weaknesses as an important contributor to the low joint pass rates. The differences between human graders and agents are particularly obvious for English Writing.

Agents can have higher consistency than humans. Within this baseline sample, the two human experts assign the exact same score on 95.0% of the 100 five-subject cases and 51.7% (62/120) of the English Writing dimension units. As humans may interpret the rubrics in different ways and contain personal biases, some agents achieve higher consistency pass rates than human graders, including in English Writing. This implies the high potential of applying LLM agents for grading.

W2 & W3 / Q2: Revised evaluation protocols

Following your suggestion, we now report English Writing separately and average only the five subject tasks with equal weights (Table 3). We also count explicitly declared departures from clear rubric requirements as justification quality failures (Appendix D.3.2, Table 18, formerly Table 16).

Five-subject mean

GraderScore accuracy (%)Justification quality (%)Consistency (%)Joint pass rate (%)
GPT88.359.470.233.4
Claude90.366.870.037.1
MiniMax84.955.562.028.4
Grok89.967.473.741.6
Qwen90.675.271.748.7

English Writing

GraderScore accuracy (%)Justification quality (%)Consistency (%)Joint pass rate (%)
GPT56.72.243.30.8
Claude76.70.661.70.0
MiniMax55.00.315.00.0
Grok73.30.068.30.0
Qwen75.02.257.51.7

Although the numerical values have changed, the paper's main conclusions remain unchanged. First, agents score accurately but remain far from reliable graders: even after excluding English Writing from the average, the highest five-task mean joint pass rate is only 48.7%. Second, task difficulty still varies with answer form and rubric structure, with English Writing and Biology remaining particularly challenging. Third, justification failures continue to show distinct patterns across material reading, evidence extraction, and score explanation. English Writing has particularly low justification quality pass rates of 0.0%–2.2%.

AuthorsOpenReview ↗

Response to Reviewer UXG8 (2/2)

Q3: Compatible shifts

Across the five PDF configurations on the five subject tasks (K = 3), 51.9% (1,600 / 3,084; 34.1%–77.1% across agents) of reason-consistency failures are compatible shifts rather than conflicting claims (48.1%, 1,484 / 3,084; Appendix G.9, Table 38). Following your suggestion, we incorporate a new evaluation protocol with a relaxed criterion to accept compatible shifts, and consider the original as the strict criterion. All other evaluation criteria remain unchanged. The results are as follows (Appendix E.3, Table 22, formerly Table 20; Appendix G.9, Table 39).

AgentCross-run consistency: strict (%)Cross-run consistency: relaxed (%)Joint pass: strict (%)Joint pass: relaxed (%)
GPT70.277.833.435.4
Claude70.078.237.139.3
MiniMax62.070.528.429.7
Grok73.780.941.643.3
Qwen71.776.448.750.9

Allowing compatible shifts increases cross-run consistency pass rates by 4.7–8.5 percentage points, showing that this criterion has some effect on measured consistency. However, joint pass rates rise by only 1.3–2.3 points, reaching at most 50.9%, with no change in the joint-pass ranking. Thus, rejecting compatible shifts is not the main reason for the low joint pass rates, as these units may still fail other evaluation requirements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.