acceptodds
Under review as a conference paper at ICLR 2027

ExamInkBench: Benchmarking End-to-End Line-Level Student Answer Reconstruction from Complete Exam Pages

23 views

Abstract

Reliable automated assessment requires reconstructing students' complete responses from full examination pages rather than transcribing isolated handwriting. Existing benchmarks typically evaluate handwriting recognition, document parsing, and answer grounding separately, making coupled failure modes difficult to measure. Therefore, we curate a gold-standard corpus of 4,017 fully annotated samples from approximately 31,769 real mathematics examination papers, with 141 public samples forming ExamInkBench and 3,876 reserved for internal development and self-evolution. Each response is represented as an ordered sequence of answer lines grouped by question item. We design a two-level evaluation protocol that separately measures line-level extraction and ordering, as well as intra-line transcription fidelity. Evaluations of mainstream multimodal models reveal frequent answer-line omissions and incorrect question associations, while accurate local recognition does not guarantee correct page-level reconstruction. Motivated by these structured failures, we propose ExamInk-Evo, a failure-driven skill self-evolution method that mines recurrent errors, induces targeted reconstruction and verification skills, and retains updates only when held-out validation performance improves. Experiments with tool-using agents show that the method consistently improves both extraction and transcription, with larger gains under stricter line-matching thresholds. The evolved skills also transfer positively across agents, indicating that they capture reusable reconstruction procedures rather than agent-specific behaviors.

Primary area:
datasets and benchmarks
Keywords:
Educational AI, Multimodal Large Language Models, Benchmarking, Student Assessment, Vision-Language Models, AI Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9C2VOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reconstruction of handwritten student responses from full mathematics examination pages, including question association, answer-line coverage/order, and transcription. It annotates 4,017 samples from a larger source pool, releasing 141 for evaluation and retaining 3,876 for development. The evaluator performs per-question text-similarity matching and reports line F1, matched-pair order accuracy, and transcription metrics including unmatched-content penalties. Eleven multimodal systems and two tool-using agents are evaluated. ExamInk-Evo uses recurrent failure diagnostics and held-out selection to improve an agent's reconstruction pipeline, with gains reported for both self-evolution and cross-agent transfer. The appendix implements this as deterministic selection/configuration of a fixed skill catalog with cached visual re-extractions.

Critical strengths
  1. An important, concrete evaluation target. Section 3 treats omissions, question association, ordering, and mathematical transcription as parts of one reconstruction problem. The corpus has substantial annotation effort, with an image-grounded agreement audit in Appendix A. This is valuable for educational document processing, where local OCR quality alone is insufficient.
  2. Explicit evaluation mechanics and positive transfer. Appendix B specifies normalization, matching, tie-breaking, unmatched-line penalties, and edge cases. Tables 2–3 report improvements for both agents and bidirectional frozen-skill transfer. These results are promising evidence that reusable reconstruction procedures can help beyond a single backend.
Critical weaknesses
  1. The claimed end-to-end task is broader than the documented evaluation, and the evaluation population is insufficiently characterized. Section 3.1 says the system identifies question items from an image; however, Appendix C.3 supplies candidate question IDs from the task definition, and Appendix D.3 requires the visual re-perception model to reproduce the supplied IDs. This is still a useful reconstruction task, but it does not establish autonomous question discovery from a page. The “none-line” skill likewise fills a supplied candidate set. Please make the provided inputs explicit and separate discovery from association. In addition, only 141 pages are public; page-level disjointness is stated, but the split's student/writer, exam-template, and acquisition-source separation is not documented. Figure 3 describes the entire annotated collection, not the public test subset. Consequently, the breadth and independence of the test distribution cannot be assessed from the corpus size alone.
  2. The distinctive contribution of failure-driven skill evolution is not isolated. Sections 5.1–5.3 describe a Proposer and Builder inducing or revising skills, whereas Appendices D.2–D.4 specify a deterministic failure-to-skill dispatch table, eight predefined operators, and a deterministic realization engine over cached outputs/re-extractions. This is a narrower algorithm: selection and parameter adjustment of supplied transformations, rather than open-ended skill induction. Moreover, line re-perception replaces base predictions with additional high-resolution visual inference. Tables 2–3 do not include a matched extra-perception baseline or a fixed application of the same skill catalog. Thus gains could primarily reflect extra visual inference and normalization rather than the evolution procedure. The deterministic design is legitimate, but the claimed novelty and the causal interpretation of transfer need to match that implementation.
Questions for authors
  1. Clarification: Where do candidate question IDs originate, and are exactly the same IDs supplied to every model and agent? Which reported experiment, if any, measures question discovery without that input?
  2. Clarification: What are the 141 test pages' scan/photo composition, writer/student counts, and exam-template overlap with development data? Was the split grouped by any of these factors?
  3. Clarification: Were the results in Tables 2–3 produced by the deterministic dispatch/realization implementation in Appendix D? What, if anything, is newly generated by an LLM during skill proposal/building?
  4. Existing-result request: What were the actual visual re-perception calls and cached-output provenance, and are results already available for re-perception alone or the fixed full catalog? Please summarize existing logs/controls; no new large experiment is required.
6 comments

Thank you for the thoughtful review. We’re glad you found our full-page reconstruction task well motivated by real educational needs. We also appreciate your recognition of the care we took to annotate and review the exam pages.We have uploaded the revised paper.


The reviewer’s concerns focus primarily on specific details of our work. We address each point below.

Weaknesses

[W1.1&Q1] Candidate question IDs and the scope of question discovery

Response: Thank you for pointing this out. We have revised §3.1 and the related descriptions to clarify that the model receives both the full examination page and its candidate question IDs.

  • Every model and agent receives exactly the same candidate IDs for a given page. These IDs come from the page’s task definition and are included in the prompt (Appendix C.3). Visual re-perception uses the same IDs (Appendix D.3), and the none-line skill can record None when no handwritten response is detected for a supplied ID.

  • We appreciate the reviewer’s recognition of the value of full-page reconstruction with supplied IDs. In all reported experiments, the model must still locate the handwritten answer lines, determine which question each line belongs to, and transcribe them in their original order.

[W1.2&Q2] Test set composition and split independence

Response: We clarify this point below. We have added these details to Sec. 3.3 (Lines 223–228).

  • Scan/photo composition. The 141 test pages comprise 12 scans (8.5%) and 129 photographs (91.5%). We emphasized photographs to reflect how students and teachers submit work for automated grading. The samples were collected by a single education and training company. The 141 public test pages contain 996 answer groups and 4,439 nonempty answer lines. Of these lines, 58.5% contain only mathematics and 35.3% combine text and mathematics.

  • Writer count. The test set contains 141 distinct student writers. Each student contributed only one page to the entire dataset, so no student appears in both the test and development sets.

  • Exam-template overlap. All 141 test pages use distinct exam templates. We verified that none of these templates appears in the development set.

  • Split construction. We assigned pages to the public test set or the internal development pool according to the data-use terms each student agreed to at upload.

[W2.1&Q3] Whether skills are predefined or generated during self-evolution

Response: We clarify below.

  • The skills were developed through self-evolution. Each agent started with an empty skill repository and no predefined mapping from failures to skills. Appendix D.3 gives examples of correspondences observed between failures and proposed changes during development. Appendix D.4 gives examples of the resulting skills. Table 2 evaluates each agent’s selected configuration, and Table 3 transfers it to the other agent without further updates. The realization engine supports replay of recorded outputs during development, while the agents in both test evaluations generated fresh visual outputs using their own backends.

  • The LLM generates the skill instructions. The Proposer recommends a change to address a recurring error. The Builder turns that proposal into instructions specifying when to trigger the skill, what operation to perform, which parameters to use, and when to incorporate the result. These instructions can cover image preprocessing, visual rereading, line segmentation, notation normalization, question assignment, and line ordering.

  • Example: Local strip rereading. An agent repeatedly missed a short handwritten step between two correctly transcribed lines. The Proposer recommended rereading the intervening strip at higher resolution. The Builder generated instructions to use the neighboring lines as anchors, crop and magnify the strip, and insert the recovered step only when both anchors aligned. If alignment failed, the original transcription was retained.

We have revised Appendices D.3 and D.4 to avoid ambiguity (Lines 1040–1133).

Due to space limitations, we will continue our response in the next window.

[W2.2&Q4] Isolating the contribution of skill evolution

Response: Both baselines already use visual re-perception. On the test set, Codex (native) averaged approximately 9 re-perception calls per sample, Codex + ExamInk-Evo approximately 13, and Codex + All-Skills approximately 12. WorkBuddy (native) averaged approximately 7 calls per sample, WorkBuddy + ExamInk-Evo approximately 13, and WorkBuddy + All-Skills approximately 14.

The +ExamInk-Evo rows measure the combined effect of the selected skills. These skills guide where to reread, what to check, and how to incorporate recovered lines. Table 2 compares each evolved configuration with its native agent, including any additional visual calls the skills trigger. In Table 3, the receiving agent applies the frozen skills using its own visual backend. In Tables 2 and 3, the agent decides when and where to reread, and no cached outputs are reused across configurations.

We conducted a small-scale rereading experiment on five samples. As the reviewer noted that no new large-scale experiment was needed, we compared native WorkBuddy’s usual rereading policy with 20 rereads per sample on the same five pages. Both runs used DeepSeek-V4.1-Flash without evolved skills. On these five pages, the additional rereading improved line F1 by only 1.3%.

The fixed All-Skills comparison tests the effectiveness of self-evolution. We evaluated Codex with the full skill catalog enabled in a fixed configuration on the same test pages (see the table below). ExamInk-Evo outperforms All-Skills on all seven metrics at all three thresholds. A likely explanation is that selection on held-out data allows ExamInk-Evo to exclude skills whose corrections do not improve reconstruction overall.

We have added it to Appendix E (Lines 1134–1149, 1165–1178).

Codex (GPT-5.6-sol) configurations across three line-matching thresholds. Bold indicates the best score at each threshold.

Line thresholdCodex configurationLine F1 ↑Precision ↑Recall ↑LOA ↑ROUGE-L ↑CER ↓LaTeX TER ↓
0.4Baseline0.85350.84320.86390.72730.74250.36580.2950
0.4+ All-Skills0.88440.87260.89660.79210.78760.31350.2512
0.4+ ExamInk-Evo0.89660.88410.90940.81460.80620.28570.2275
0.5Baseline0.79890.78940.80870.63930.71020.42810.3612
0.5+ All-Skills0.83840.82720.85000.71530.76020.36660.3074
0.5+ ExamInk-Evo0.85420.84230.86640.74200.78090.33450.2788
0.6Baseline0.70950.70100.71820.51870.65010.53970.4816
0.6+ All-Skills0.75780.74760.76820.60490.70590.46860.4162
0.6+ ExamInk-Evo0.77910.76830.79030.63630.73040.42860.3793

We hope these clarifications and additional analyses address your concerns and better convey the scope and contribution of our work. We would be grateful if you would consider them in your final assessment.

Thank you for the detailed response. My concern has been resolved, and I have raised my score accordingly.

Reviewer_bc16OpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ExamInkBench, a benchmark for reconstructing student answers from complete mathematics examination pages. The task requires extracting handwritten answer lines, associating them with the correct questions, preserving their order, and transcribing textual and mathematical content. The authors curate 4,017 annotated samples, of which 141 form the public evaluation set, and propose a question-aware evaluation protocol covering line extraction, ordering, and transcription fidelity. They also introduce ExamInk-Evo, a failure-driven framework that selects and refines reconstruction skills for tool-using agents. Experiments evaluate 11 multimodal systems and report improvements from skill evolution on two agent systems, together with positive cross-agent skill transfer.

Critical strengths
  1. I think the paper addresses a useful problem in educational AI. Faithfully recovering students’ responses is an important prerequisite for reliable and fair AI feedback, as omissions or transcription errors could lead to feedback that misrepresents a student’s work. I also find the full-page setting practically motivated: question-level extraction may require manual cropping or segmentation, whereas reconstructing answers directly from complete scanned or photographed examination pages could reduce this effort.

  2. Based on the description, I think authors devoted substantial human effort to constructing the ground truth. Having eight independent annotators per sample, three additional reviewers, and a final adjudication stage appears to provide a solid basis for annotation quality. The reported pre-adjudication agreement across several dimensions is also helpful evidence of annotation consistency.

Critical weaknesses
  1. I think the data provenance and split construction need further clarification. The paper does not well specify the examination types or educational levels (e.g., high school, mid school, or mixed) represented in the dataset (at least I didn't see it is clearly defined). Although the overall numbers of scans and photographs are reported, their proportions within the self-evolution, validation, and test sets are unclear. It is also unclear whether these splits are separated by student, examination paper, or underlying question. Providing these details would help assess the benchmark’s representativeness and the independence of the evaluation data.

  2. I have some concerns about whether the evaluation protocol adequately handles reasonable variations in answer representation. For example, splitting one reference line into two predicted lines can incur both extraction and transcription penalties even when the content is fully preserved. Similarly, equivalent LaTeX expressions or symbol conventions may receive different scores despite representing the same formula. I think it would be helpful to clarify how these cases are handled and provide complementary evaluations that distinguish content recognition errors from segmentation and representation differences.

  3. The treatment of student-drawn diagrams is unclear to me. Geometric constructions, function plots, and other graphical responses seem difficult to avoid in realistic mathematics examinations, but the annotation schema and metrics focus on textual and mathematical answer lines. How are such drawings represented, associated with questions, and evaluated? If these cases are excluded or left unassessed, I think this would narrow the benchmark’s practical coverage and should be explicitly discussed and quantified.

  4. I think the execution details of ExamInk-Evo need to be more concrete. In particular, although the skill descriptors are said to contain execution triggers, it remains unclear how these triggers are defined and evaluated on unseen images without access to gold annotations. For example, what signals indicate missing lines or unreliable transcription, and when is visual re-perception invoked? An example skill file and an end-to-end execution trace would help clarify the distinction between offline failure-driven skill selection and runtime skill invocation.

  5. I think the positioning of ExamInk-Evo relative to prior work needs further development. Learning from failures to improve subsequent inference has been explored in [1,2,3]. These works appear to be absent from the current bibliography. I would appreciate a discussion of their connections and differences, particularly what the proposed structured diagnostics and skill evolution add beyond existing approaches. A comparison with a suitably adapted representative method would also help establish the contribution.

[1] Shinn, N., et al. “Reflexion: Language Agents with Verbal Reinforcement Learning.”

[2] Sun, W., et al. “EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions.”

[3] Zhang, J., et al. “AFlow: Automating Agentic Workflow Generation.”

Questions for authors

See weaknesses, I think above weaknesses are not very severe in my view point and I think this paper is useful if all data and labels can be published (both 4000+ examples, seems that you aim to open source the validation part. But I think it would be better to open source the remaining if available, they can be quite valuable and can make your self-evolution part convincing and reproductable), I would appreciate it if they can be well explained.

4 comments

Thank you for the thoughtful review. We’re glad you found our full-page reconstruction task well motivated by real educational needs. We also appreciate your recognition of the care we took to annotate and review the exam pages.We have uploaded the revised paper.


The reviewer’s concerns focus primarily on specific details of our work. We address each point below.

Weaknesses

[W1] Clarification of data provenance and split construction

Response: We clarify below.

  • All samples come from high school mathematics examinations. We chose this level because high school solutions typically involve more complex notation and longer derivations than those in primary and middle school. A single solution may span several lines and combine indexed variables, exponents, and nested fractions, challenging systems to transcribe dense expressions accurately and recover every answer line.

  • The public test set emphasizes photographs to reflect how students and teachers submit work for automated grading. Of the 141 test pages, 129 are photographs (91.5%) and 12 are scans (8.5%). Motion blur, uneven lighting, and perspective distortion in phone photographs can obscure mathematical symbols and line boundaries. The remaining 2,367 scans and 1,509 photographs form the internal development pool. Based on the approximate 8:2 allocation, we estimate that the self-evolution and validation sets contain about 1,894 and 473 scans, respectively, or roughly 61% of each split.

  • The test and development sets are disjoint by page, student, and exam template. The data-use terms students signed at upload determined which pages were eligible for public evaluation or internal development. Each student contributed one page, and we verified that none of the 141 test templates appears in the development set.

We have added these details to Sec. 3.3 (Lines 223–228).

[W2] Scoring sensitivity to line segmentation and LaTeX variation

Response: We clarify both points below.

  1. We added a question-level evaluation that ignores line boundaries. Under the original scoring, if a model splits one reference line into two, at most one fragment can match it. The unmatched fragment counts as an extra predicted line and contributes to transcription error. We therefore rescored the same predictions from all 11 models by concatenating the predicted and reference lines separately for each question, preserving their order. A split that preserves the full text now leaves the score unchanged. All 11 models obtained higher ROUGE-L and lower CER and TER (see the table below). These gains reflect both the removal of line boundaries and the removal of the line-matching threshold.

    We have added this evaluation to Appendix F (Lines 1150–1165, 1178–1192).

    Transcription scores for 11 models under line-level (threshold 0.5) and question-level (no threshold) scoring.

    ModelLine ROUGE-L ↑Line CER ↓Line TER ↓Question ROUGE-L ↑Question CER ↓Question TER ↓ROUGE-L gain (%)CER reduction (%)TER reduction (%)
    GPT-5.6-sol0.49890.81710.72420.59100.49030.398618.540.045.0
    Kimi-K30.48501.00720.86900.59330.59270.458422.341.247.2
    GLM-5.3-Flash0.43591.01940.94790.56890.59220.569530.541.939.9
    Qwen3.8-Omni-Flash0.46461.03850.93400.60820.60710.504230.941.546.0
    Claude-Fable-5-10.42111.10891.03030.53680.62950.536227.543.248.0
    Kimi-K2.60.47271.14661.02280.57410.69930.572421.439.044.0
    Gemini-3.8-Flash0.45091.15170.96140.63410.62810.467240.645.551.4
    Qwen3.5-Omni-Plus0.37511.18350.98570.56730.66690.663551.243.632.7
    GLM-4.6V0.22191.34431.08140.43600.81560.860596.539.320.4
    DeepSeek-Flash0.32161.52351.41940.50740.84010.771157.844.945.7
    GLM-OCR0.08952.63601.77570.16332.00461.340882.524.024.5
    Mean0.38521.27061.06830.52550.77880.649336.438.739.2
  2. Both evaluations treat common equivalent LaTeX forms identically. Appendix B.1 describes how we normalize whitespace and mathematical delimiters. To accommodate more notation variants, the evaluation script uses a predefined mapping table to apply the same conversions to predictions and references before scoring. For example, it maps \leq to \le and \dfrac to \frac.

Due to space limitations, we will continue our response in the next window.

[W3] Treatment of student-drawn diagrams

Response: Student-drawn diagrams fall outside the current evaluation scope. They appear on 9.2% of the public test pages. For mixed responses, we associate written text and equations with their question and score those lines. The drawings remain unscored because their spatial relationships require annotations beyond our line-based schema.

We acknowledge that this limits the benchmark’s coverage of graphical responses and have clarified this scope in the Limitations section (Lines 505–508).

[W4] Runtime trigger conditions and execution examples

Response: We clarify below.

  • Visual uncertainty triggers local rereading. Visible handwriting between two located lines with no corresponding transcription signals a possible omission. A symbol whose shape is too faint to verify against its transcription signals a possible recognition error. In either case, the agent enlarges and rereads the affected region without consulting gold annotations.

  • Recovering a missing step in a two-column answer. We use a real photograph of Question 17 to construct a first pass that omits one line from part (1). The transcription jumps from x40=25DF\frac{x}{40}=\frac{25}{DF} to OF=OD+DF=25+1000xOF=OD+DF=25+\frac{1000}{x}. An ink band lies between these entries, but part (2) also contains handwriting at the same height. The agent locates both transcribed lines in the left column and enlarges only the strip between and including them. The enlarged view shows a separate handwritten line, DF=1000xDF=\frac{1000}{x}. After matching both surrounding lines in order, the agent inserts the recovered step under 17(1). If either line cannot be matched within that column, it keeps the first-pass transcription.

We have added the trigger conditions and execution example to Appendix D.3 (Lines 1040–1108).

[W5] Positioning ExamInk-Evo relative to prior work on learning from failures

Response: We clarify the mechanistic differences below.

  • ExamInk-Evo represents corrections as executable skills. Reflexion stores verbal reflections that the agent interprets on subsequent attempts. In ExamInk-Evo, a skill’s operation and parameters encode the correction directly, allowing another agent to apply it without further adaptation.

  • ExamInk-Evo diagnoses errors in the structure of an answer. EDU-CIRCUIT-HW studies recognition errors in individual solutions and their impact on grading. ExamInk-Evo also compares question assignments and relative line order, so it can flag a line whose text is correct but whose placement changes the reconstructed answer.

  • ExamInk-Evo uses the error type to determine what to revise. AFlow uses execution feedback to guide tree search over workflow code. In ExamInk-Evo, the most frequent error category points to a specific skill, such as visual re-perception for recurring omissions. Held-out performance then determines which configuration to select.

We have revised Sec. 2 to clarify ExamInk-Evo’s contribution (Lines 147–156).

[Question For Author] The reviewer notes that the weaknesses are not severe and hopes for the full release of the data and labels.

Response: We thank you again for recognizing the practical value of full-page answer reconstruction and the quality of our annotations, and for noting that the weaknesses are not severe.

Public release of the data and labels is subject to the policies of the companies involved and the terms of the students’ signed consent forms. We will work with both the companies and the participating students to obtain permission for a broader release. As permissions are secured from both parties, we will publish additional exam pages together with their annotations in batches.

[Ethics Concerns Response] With student and institutional authorization, eight annotators independently labeled the de-identified exam pages, and we identified no additional ethical concerns.


We hope these clarifications and additional analyses address your concerns and better convey the scope and contribution of our work. We would be grateful if you would consider them in your final assessment.

Thanks the author for the detailed response. It make this paper clearer, and I raised my confidence score.

It is a pity that this work doesn't take account of student-drawn diagrams, as I think it would be more important especially when current LLMs (such as GPT-6) has perform pretty well on recognizing them, and it would be more important to add this into recognition pipeline (as in practice, we cannot skip it if we want to recognize student's input end-to-end). But I think it is acceptable not including this becuase the description order shall be tricky to measure.

Nevertheless, this could be a good work and I think the cost and the effort for crafting this work is huge. I'd like to experience this benchmark once it is published.

On behalf of all authors, we sincerely thank you for your careful evaluation of our rebuttal and manuscript. We are glad our response has clarified the paper for you. We are grateful for your recognition of the effort spent on this benchmark and look forward to your trying it upon release. 

We are pleased that our clarifications addressed your main concern, and we appreciate your constructive suggestion for further strengthening the work. We greatly appreciate your expertise and the time you devoted to reviewing our work. Your continued support will help bring greater attention to the importance of automated student answer reconstruction from exam pages and encourage further research on educational AI and multimodal assessment benchmarks.

Reviewer_wVm4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ExamInkBench, a benchmark for end-to-end line-level student answer reconstruction from complete examination pages. It contains 4,017 annotated samples, with 141 used for public evaluation. The benchmark evaluates question discovery, answer-line coverage, question association, line ordering, textual transcription and transcription fidelity. The authors also propose ExamInk-Evo, a failure-driven skill self-evolution framework that improves reconstruction performance through iterative skill updates.

Critical strengths
  1. Meaningful problem formulation. The paper addresses a practical limitation of existing handwriting recognition and document understanding benchmarks. Evaluating complete answer reconstruction, rather than isolated text recognition, provides a useful perspective on document understanding.

  2. Substantial annotation effort. The dataset contains 4,017 fully annotated samples and 124,706 answer lines. The annotation workflow and inter-annotator agreement analysis provide reasonable evidence of data quality.

Critical weaknesses
  1. Limited evidence for the necessity of skill self-evolution. Appendix D shows that ExamInk-Evo relies on a predefined skill catalog and a deterministic pattern-to-skill mapping. It remains unclear whether the improvements come from the evolution process or simply from applying these heuristic skills. Comparisons with a fixed pipeline or an All-Skills baseline would help establish the necessity of the proposed approach.

  2. Limited size of the public evaluation set. Although the authors annotate 4,017 samples, only 141 are included in the public ExamInkBench evaluation set. Given the diversity of handwriting styles and examination layouts, this relatively small evaluation set may limit the robustness and representativeness of the benchmark results.

Questions for authors

Overall, I think it's a meaningful paper. I like this research topic, which is important for realistic educational applications.

1 comment

We thank you for recognizing our paper as a meaningful contribution to document understanding and for emphasizing the importance of full-page answer reconstruction for real-world educational applications. We also appreciate your recognition of the care taken to annotate the dataset and assess its quality. We have uploaded the revised paper.


The reviewer’s concerns focus primarily on a misunderstanding of the presentation of skill evolution in ExamInk-Evo. We address each point below.

Weaknesses

[W1] The contribution of skill evolution beyond predefined heuristics

Response: We clarify below.

  1. The skills were developed through self-evolution. We believe this concern stems from a misunderstanding of the initial setup. Our wording in Appendix D may have suggested that the skills and their mappings were defined in advance. Each agent started with an empty skill repository and no predefined mapping from failures to skills. The Proposer identified a change to address a recurring error, and the Builder generated instructions specifying when and how to apply it. Appendix D.3 gives selected examples of how recurring failures led to proposed changes during evolution. Appendix D.4 presents a subset of the skills ultimately generated through this process.

    We have revised Appendices D.3 and D.4 to avoid ambiguity (Lines 1040–1133).

  2. The All-Skills comparison demonstrates the effectiveness of self-evolution. We evaluated Codex with the full skill catalog enabled in a fixed configuration (see the table below). ExamInk-Evo outperforms All-Skills at all three thresholds. A likely explanation is that self-evolution uses held-out performance to filter out skills whose corrections do not improve reconstruction overall.

    We have added this comparison to Appendix E (Lines 1134–1149, 1165–1178).

Codex (GPT-5.6-sol) configurations across three line-matching thresholds

Line thresholdCodex configurationLine F1 ↑Precision ↑Recall ↑LOA ↑ROUGE-L ↑CER ↓LaTeX TER ↓
0.4Baseline0.85350.84320.86390.72730.74250.36580.2950
0.4+ All-Skills0.88440.87260.89660.79210.78760.31350.2512
0.4+ ExamInk-Evo0.89660.88410.90940.81460.80620.28570.2275
0.5Baseline0.79890.78940.80870.63930.71020.42810.3612
0.5+ All-Skills0.83840.82720.85000.71530.76020.36660.3074
0.5+ ExamInk-Evo0.85420.84230.86640.74200.78090.33450.2788
0.6Baseline0.70950.70100.71820.51870.65010.53970.4816
0.6+ All-Skills0.75780.74760.76820.60490.70590.46860.4162
0.6+ ExamInk-Evo0.77910.76830.79030.63630.73040.42860.3793

[W2] Size and representativeness of the public evaluation set

Response: We clarify the robustness of our benchmark results below.

  1. The 141 test pages cover 141 distinct writers and 141 distinct exam templates. Each student contributed one page, and no test template appears in the development set. The 129 photographs and 12 scans contain 996 answer groups and 4,439 nonempty answer lines. The results therefore cover a range of handwriting styles and layouts within high school mathematics submissions from a single provider.

    We have added these details to Sec. 3.3 (Lines 223–228).

  2. The scope of the public release is governed by company policy and the consent students provided at submission. The remaining 3,876 annotated pages are used for self-evolution and validation. We will work with the company and participating students to secure permission and release additional pages with their annotations in batches as approvals are obtained.

[Questions for Authors] Recognition of a meaningful paper on educational applications

Response: We are grateful for your generous assessment of our paper. It is especially encouraging that you found our work meaningful and share our enthusiasm for a topic you recognize as important for realistic educational applications.


We hope these clarifications and additional analyses address your concerns and better convey the scope and contribution of our work. We would be grateful if you would consider them in your final assessment.

Comments on the paper

1 comment

We thank all reviewers for their detailed and constructive feedback and for the time and effort they devoted to evaluating our work.

The reviewers strongly endorsed the practical value of reconstructing handwritten answers from complete exam pages, where faithful reconstruction requires assigning each line to the correct question and preserving the order of the student’s work. They also recognized the annotation effort behind ExamInkBench and the transfer of learned reconstruction skills across agents.

Alongside these strengths, the reviewers identified three areas where clearer explanations and additional experiments would strengthen the paper. We describe how we addressed each below.

  1. Additional experiments (Reviewer wVm4 W1, Reviewer 9C2V W2.2, Q4).

    • To test the contribution of skill evolution, we compared ExamInk-Evo with a fixed configuration that enabled all available skills. On Codex, ExamInk-Evo outperformed the All-Skills baseline on all seven metrics at all three line-matching thresholds, demonstrating the value of selecting skills through self-evolution.
    • To test whether more visual rereading could explain the gains, we compared WorkBuddy’s usual policy with 20 rereads per page on the same five pages, without evolved skills. Extra rereading improved line F1 by only 1.3%. This modest gain, together with the All-Skills comparison, supports the value of skills that guide where to reread and how to use the recovered content.
  2. Skill evolution and execution (Reviewer wVm4 W1, Reviewer bc16 W4, W5, Reviewer 9C2V W2.1, Q3).

    • We clarified that each agent starts with an empty skill repository, generates executable skills from recurring errors, and selects them on held-out pages. We also added an execution trace showing how a learned skill recovers an omitted handwritten line.

    • We distinguished ExamInk-Evo from prior work in how it diagnoses reconstruction errors, represents corrections, and determines which skills to revise.

  3. Benchmark scope and evaluation

    • Task and scoring (Reviewer bc16 W2, W3, Reviewer 9C2V W1.1, Q1). We clarified the role of the supplied question IDs, how common LaTeX variants are normalized, and why student drawings fall outside the scoring scope. We also added a question-level evaluation for all 11 models to show how transcription scores change when line boundaries and the matching threshold are removed.
    • Test set details (Reviewer wVm4 W2, Reviewer bc16 W1, Q, Reviewer 9C2V W1.2, Q2). We documented the source of the test pages and the proportions of photographs and scans. We also confirmed that the test set shares no students or exam templates with the development set.

We have responded to every concern raised by the reviewers and uploaded the revised manuscript. In follow-up, Reviewer 9C2V confirmed that their concern was resolved and raised their score. Reviewer bc16 reported that the paper was clearer and raised their confidence. We thank the reviewers again for their suggestions and the Area Chair for overseeing the discussion. We respectfully ask that the information provided in our rebuttal be considered in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.