- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies reconstruction of handwritten student responses from full mathematics examination pages, including question association, answer-line coverage/order, and transcription. It annotates 4,017 samples from a larger source pool, releasing 141 for evaluation and retaining 3,876 for development. The evaluator performs per-question text-similarity matching and reports line F1, matched-pair order accuracy, and transcription metrics including unmatched-content penalties. Eleven multimodal systems and two tool-using agents are evaluated. ExamInk-Evo uses recurrent failure diagnostics and held-out selection to improve an agent's reconstruction pipeline, with gains reported for both self-evolution and cross-agent transfer. The appendix implements this as deterministic selection/configuration of a fixed skill catalog with cached visual re-extractions.
Critical strengths
- An important, concrete evaluation target. Section 3 treats omissions, question association, ordering, and mathematical transcription as parts of one reconstruction problem. The corpus has substantial annotation effort, with an image-grounded agreement audit in Appendix A. This is valuable for educational document processing, where local OCR quality alone is insufficient.
- Explicit evaluation mechanics and positive transfer. Appendix B specifies normalization, matching, tie-breaking, unmatched-line penalties, and edge cases. Tables 2–3 report improvements for both agents and bidirectional frozen-skill transfer. These results are promising evidence that reusable reconstruction procedures can help beyond a single backend.
Critical weaknesses
- The claimed end-to-end task is broader than the documented evaluation, and the evaluation population is insufficiently characterized. Section 3.1 says the system identifies question items from an image; however, Appendix C.3 supplies candidate question IDs from the task definition, and Appendix D.3 requires the visual re-perception model to reproduce the supplied IDs. This is still a useful reconstruction task, but it does not establish autonomous question discovery from a page. The “none-line” skill likewise fills a supplied candidate set. Please make the provided inputs explicit and separate discovery from association. In addition, only 141 pages are public; page-level disjointness is stated, but the split's student/writer, exam-template, and acquisition-source separation is not documented. Figure 3 describes the entire annotated collection, not the public test subset. Consequently, the breadth and independence of the test distribution cannot be assessed from the corpus size alone.
- The distinctive contribution of failure-driven skill evolution is not isolated. Sections 5.1–5.3 describe a Proposer and Builder inducing or revising skills, whereas Appendices D.2–D.4 specify a deterministic failure-to-skill dispatch table, eight predefined operators, and a deterministic realization engine over cached outputs/re-extractions. This is a narrower algorithm: selection and parameter adjustment of supplied transformations, rather than open-ended skill induction. Moreover, line re-perception replaces base predictions with additional high-resolution visual inference. Tables 2–3 do not include a matched extra-perception baseline or a fixed application of the same skill catalog. Thus gains could primarily reflect extra visual inference and normalization rather than the evolution procedure. The deterministic design is legitimate, but the claimed novelty and the causal interpretation of transfer need to match that implementation.
Questions for authors
- Clarification: Where do candidate question IDs originate, and are exactly the same IDs supplied to every model and agent? Which reported experiment, if any, measures question discovery without that input?
- Clarification: What are the 141 test pages' scan/photo composition, writer/student counts, and exam-template overlap with development data? Was the split grouped by any of these factors?
- Clarification: Were the results in Tables 2–3 produced by the deterministic dispatch/realization implementation in Appendix D? What, if anything, is newly generated by an LLM during skill proposal/building?
- Existing-result request: What were the actual visual re-perception calls and cached-output provenance, and are results already available for re-perception alone or the fixed full catalog? Please summarize existing logs/controls; no new large experiment is required.
6 comments
[W2.2&Q4] Isolating the contribution of skill evolution
Response: Both baselines already use visual re-perception. On the test set, Codex (native) averaged approximately 9 re-perception calls per sample, Codex + ExamInk-Evo approximately 13, and Codex + All-Skills approximately 12. WorkBuddy (native) averaged approximately 7 calls per sample, WorkBuddy + ExamInk-Evo approximately 13, and WorkBuddy + All-Skills approximately 14.
The +ExamInk-Evo rows measure the combined effect of the selected skills. These skills guide where to reread, what to check, and how to incorporate recovered lines. Table 2 compares each evolved configuration with its native agent, including any additional visual calls the skills trigger. In Table 3, the receiving agent applies the frozen skills using its own visual backend. In Tables 2 and 3, the agent decides when and where to reread, and no cached outputs are reused across configurations.
We conducted a small-scale rereading experiment on five samples. As the reviewer noted that no new large-scale experiment was needed, we compared native WorkBuddy’s usual rereading policy with 20 rereads per sample on the same five pages. Both runs used DeepSeek-V4.1-Flash without evolved skills. On these five pages, the additional rereading improved line F1 by only 1.3%.
The fixed All-Skills comparison tests the effectiveness of self-evolution. We evaluated Codex with the full skill catalog enabled in a fixed configuration on the same test pages (see the table below). ExamInk-Evo outperforms All-Skills on all seven metrics at all three thresholds. A likely explanation is that selection on held-out data allows ExamInk-Evo to exclude skills whose corrections do not improve reconstruction overall.
We have added it to Appendix E (Lines 1134–1149, 1165–1178).
Codex (GPT-5.6-sol) configurations across three line-matching thresholds. Bold indicates the best score at each threshold.
| Line threshold | Codex configuration | Line F1 ↑ | Precision ↑ | Recall ↑ | LOA ↑ | ROUGE-L ↑ | CER ↓ | LaTeX TER ↓ |
|---|---|---|---|---|---|---|---|---|
| 0.4 | Baseline | 0.8535 | 0.8432 | 0.8639 | 0.7273 | 0.7425 | 0.3658 | 0.2950 |
| 0.4 | + All-Skills | 0.8844 | 0.8726 | 0.8966 | 0.7921 | 0.7876 | 0.3135 | 0.2512 |
| 0.4 | + ExamInk-Evo | 0.8966 | 0.8841 | 0.9094 | 0.8146 | 0.8062 | 0.2857 | 0.2275 |
| 0.5 | Baseline | 0.7989 | 0.7894 | 0.8087 | 0.6393 | 0.7102 | 0.4281 | 0.3612 |
| 0.5 | + All-Skills | 0.8384 | 0.8272 | 0.8500 | 0.7153 | 0.7602 | 0.3666 | 0.3074 |
| 0.5 | + ExamInk-Evo | 0.8542 | 0.8423 | 0.8664 | 0.7420 | 0.7809 | 0.3345 | 0.2788 |
| 0.6 | Baseline | 0.7095 | 0.7010 | 0.7182 | 0.5187 | 0.6501 | 0.5397 | 0.4816 |
| 0.6 | + All-Skills | 0.7578 | 0.7476 | 0.7682 | 0.6049 | 0.7059 | 0.4686 | 0.4162 |
| 0.6 | + ExamInk-Evo | 0.7791 | 0.7683 | 0.7903 | 0.6363 | 0.7304 | 0.4286 | 0.3793 |
We hope these clarifications and additional analyses address your concerns and better convey the scope and contribution of our work. We would be grateful if you would consider them in your final assessment.
Thank you for the detailed response. My concern has been resolved, and I have raised my score accordingly.
Thank you for the thoughtful review. We’re glad you found our full-page reconstruction task well motivated by real educational needs. We also appreciate your recognition of the care we took to annotate and review the exam pages.We have uploaded the revised paper.
The reviewer’s concerns focus primarily on specific details of our work. We address each point below.
Weaknesses
[W1.1&Q1] Candidate question IDs and the scope of question discovery
Response: Thank you for pointing this out. We have revised §3.1 and the related descriptions to clarify that the model receives both the full examination page and its candidate question IDs.
Every model and agent receives exactly the same candidate IDs for a given page. These IDs come from the page’s task definition and are included in the prompt (Appendix C.3). Visual re-perception uses the same IDs (Appendix D.3), and the none-line skill can record
Nonewhen no handwritten response is detected for a supplied ID.We appreciate the reviewer’s recognition of the value of full-page reconstruction with supplied IDs. In all reported experiments, the model must still locate the handwritten answer lines, determine which question each line belongs to, and transcribe them in their original order.
[W1.2&Q2] Test set composition and split independence
Response: We clarify this point below. We have added these details to Sec. 3.3 (Lines 223–228).
Scan/photo composition. The 141 test pages comprise 12 scans (8.5%) and 129 photographs (91.5%). We emphasized photographs to reflect how students and teachers submit work for automated grading. The samples were collected by a single education and training company. The 141 public test pages contain 996 answer groups and 4,439 nonempty answer lines. Of these lines, 58.5% contain only mathematics and 35.3% combine text and mathematics.
Writer count. The test set contains 141 distinct student writers. Each student contributed only one page to the entire dataset, so no student appears in both the test and development sets.
Exam-template overlap. All 141 test pages use distinct exam templates. We verified that none of these templates appears in the development set.
Split construction. We assigned pages to the public test set or the internal development pool according to the data-use terms each student agreed to at upload.
[W2.1&Q3] Whether skills are predefined or generated during self-evolution
Response: We clarify below.
The skills were developed through self-evolution. Each agent started with an empty skill repository and no predefined mapping from failures to skills. Appendix D.3 gives examples of correspondences observed between failures and proposed changes during development. Appendix D.4 gives examples of the resulting skills. Table 2 evaluates each agent’s selected configuration, and Table 3 transfers it to the other agent without further updates. The realization engine supports replay of recorded outputs during development, while the agents in both test evaluations generated fresh visual outputs using their own backends.
The LLM generates the skill instructions. The Proposer recommends a change to address a recurring error. The Builder turns that proposal into instructions specifying when to trigger the skill, what operation to perform, which parameters to use, and when to incorporate the result. These instructions can cover image preprocessing, visual rereading, line segmentation, notation normalization, question assignment, and line ordering.
Example: Local strip rereading. An agent repeatedly missed a short handwritten step between two correctly transcribed lines. The Proposer recommended rereading the intervening strip at higher resolution. The Builder generated instructions to use the neighboring lines as anchors, crop and magnify the strip, and insert the recovered step only when both anchors aligned. If alignment failed, the original transcription was retained.
We have revised Appendices D.3 and D.4 to avoid ambiguity (Lines 1040–1133).
Due to space limitations, we will continue our response in the next window.