- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies LLM grading of long-form programming submissions using two university exams. Across 333 configurations, the authors evaluate model choice, prompt components, grading personas, and decoding settings. Several closed models achieve strong agreement with human grades, while a credit-withholding preamble causes severe under-grading in many open models. Ablations attribute this effect primarily to policy instructions rather than persona adjectives. LoRA fine-tuning on approximately 3,900 graded examples substantially improves five small open models and reduces their aggregate error sensitivity to the tested harsh prompts. Cross-exam experiments also show generally beneficial transfer.
Critical strengths
- The study provides a strong empirical validation. It uses two real exams, more than 1,600 students, many model families, and extensive prompt ablations, making the observed failure mode convincing and practically relevant.
Critical weaknesses
-
The proposed method seemed to be limited in fine-tuning generalization. The strongest results evaluate new students on exam questions and rubrics seen during training. Thus, the claimed transferable grading ability and robustness on unseen exams remain uncertain.
-
The evaluation measures agreement with two noisy human graders instead of correctness against an independent gold standard. Grader-pair clustering may also make the reported confidence intervals optimistic.
Questions for authors
-
Can authors report persona sensitivity for the cross-exam adapters, i.e., an adapter trained only on one exam and evaluated under neutral and harsh personas on the other?
-
Do the fine-tuned models remain responsive to legitimate, non-conflicting rubric or grading-policy changes? A controlled test would clarify whether the reduced persona sensitivity is desirable robustness or general instruction insensitivity.
-
How do the central parity confidence intervals and verdicts change under a grader-pair/block bootstrap or another analysis accounting for the grading-assignment structure?
-
What institutional review, student consent, data-use authorization, and re-identification audit supported the public release of complete student notebooks and grades?
Thank you for the review. New text is blue, reworded text is red.
1. New students, not new exams (W1). You are right. The paper now says the human-level results are for new students of an exam seen in training. For an unseen exam we ran the same test on the adapters trained on the other exam only (new Table 21): they improve grading in 9 of 10 cells, but none is shown to be as good as a human grader.
2. Agreement with noisy graders is not correctness (W2). Agreed. Neither exam has a gold standard. This is now stated in Limitations.
3. Grader-pair bootstrap (Q3). Done (new Table 20). We resample whole grader pairs instead of single students.
So some of our intervals were too optimistic, as you suspected. The text is corrected.
4. Persona sensitivity of the cross-exam adapters (Q1). Done (new Table 23). Each single-exam adapter graded the exam it never saw, under all five personas.
So the stability we reported holds for adapters trained on the exam being graded, and only partly for an unseen exam. Section 8.4 now says this.
5. Do fine-tuned models still follow legitimate policy changes (Q2)? No. We ran the controlled test you suggested (new Table 24). One sentence was added to the neutral prompt: deduct 2 marks per question as a late penalty, or add 2 marks as an approved adjustment.
So your second reading is the right one: fine-tuning makes the grader ignore grading instructions in the prompt, legitimate ones included. The abstract, Section 8.4, the Discussion and the Conclusion now say this, and we recommend applying policy changes outside the model.
Both experiments use the paper's own adapters. As a check we also retrained all adapters with the same recipe and repeated both experiments; the retrained adapters match the originals to within 0.14 MAE on average (0.63 at most) and give the same picture.
6. Ethics (Q4). The statement now says the data are pseudonymised, not anonymous, and names the remaining risk. Grading was part of normal assessment and the release has the instructors' permission. Students agreed to this use of their work when they signed up for the course, and the release was approved by the head of research ethics of the department where the courses took place. All students were undergraduates; none was a minor.
Re-identification audit: the release went through several rounds of manual checking, and we also ran an automated scan of every notebook. Neither found any student identifier (names, emails, ID numbers, account names, personal file paths). The Ethics Statement now says this.