- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RODS, a framework for optimizing natural-language grading rubrics when only a small number of scored student responses is available. RODS rewrites real responses toward target score levels, uses an LLM-based generative adversarial quality-control module (GAQC) to filter or refine synthetic responses, and alternates label-supervised and consistency-supervised rubric refinement with beam search. The consistency signal compares the target label used for synthesis with the score produced when the same rubric grades the synthetic response; a semi-supervised variant additionally uses unlabeled responses. The evaluation covers more than 1,700 responses, 12 questions, three datasets, two LLM backbones, six baselines, and component ablations. The reported averages favor RODS over the strongest rubric-optimization baseline, but several methodological and reporting inconsistencies currently prevent me from treating the gains as conclusive.
Critical strengths
- Important and practically relevant problem. Reducing the amount of scored data needed to adapt transparent, teacher-editable rubrics is valuable for real educational deployment. The paper also identifies two genuine risks of synthetic supervision - label noise and distribution shift - and attempts to address both within a coherent optimization pipeline.
- Broad empirical effort and useful component analysis. The study evaluates 12 questions from three subject areas with both a local Qwen model and a Gemini model, includes rubric-specific and general prompt-optimization baselines, and reports several targeted ablations. The reported accuracy gains and the observation that naive synthetic demonstrations can hurt performance are potentially useful findings if the validity issues below are resolved.
Critical weaknesses
The evaluation contains internal inconsistencies and does not establish statistical robustness. For single-label multiclass classification, the micro-F1 formula in Eq. (11) is algebraically equal to accuracy, yet Tables 4-7 report substantially different values for the two metrics. This suggests either a metric implementation error or a mislabeled metric. The central correlation analysis is also weak: it uses only Q1 and Q4, reports r=0.25-0.43 with p=0.219-0.434, and nevertheless presents the result as support for self-consistency as a surrogate objective. In addition, the test sets are small (approximately 11-50 responses per question), per-question hyperparameters are selected on small validation sets, and no paired confidence intervals or significance tests are provided for the claimed improvements. Even the stated Qwen accuracy gain in Section 4.2 (4.4) does not match Table 1 (51.7 - 46.1 = 5.6 percentage points). These issues directly affect the main empirical claim. 2. The GAQC decision rule is contradictory and its claimed fidelity improvement is insufficiently validated. Equation (4) defines d_omega(x) as P(real|x), and the accompanying text says that larger values indicate higher fidelity and should be retained. However, Algorithm 1 and Appendix A.2 retain a sample when d_omega < 0.5, which is the opposite rule under that definition. If d_omega instead means discriminator accuracy on a known synthetic sample, then Eq. (4) and the surrounding interpretation are incorrect. This ambiguity blocks reproduction of a core component. Moreover, GAQC is supported mainly by downstream ablation and one qualitative example; there is no independent human blind evaluation or held-out fidelity measure, while the same model family generates, characterizes, and judges the responses. The semi-supervised variant also uses validation and test response texts during optimization, making it transductive and not directly comparable to baselines that do not receive those inputs.
Questions for authors
GAQC definition. What exactly is d_omega(x) in the implementation: the probability of predicting "real," the probability of correctly detecting a synthetic sample, or a dataset-level discrimination accuracy? Please reconcile Eq. (4), Algorithm 1, and Appendix A.2, state the actual inequality used, and report the fraction of samples accepted, refined, and discarded at each iteration. 2. Metric correctness and uncertainty. Why does the reported micro-F1 differ from accuracy under the single-label multiclass setup and Eq. (11)? Was macro-F1 or another averaging scheme actually computed? Please also clarify how the split and generation repetitions were aggregated, correct the Qwen headline improvement, and provide paired uncertainty or significance estimates for RODS versus GradeOPT. 3. Evidence for the surrogate objective. Given that all four reported correlations in Figure 4 are statistically non-significant and cover only two questions, what evidence shows that higher self-consistency reliably predicts better held-out grading across the 12 questions? An analysis over all candidate rubrics and questions, with uncertainty and a control for shared-model self-confirmation, would materially affect my assessment. 4. Fairness of the semi-supervised and compute comparisons. Can the authors report results for a strictly inductive unlabeled-data setting and/or give competing methods the same unlabeled validation/test inputs? Please also provide exact LLM-call and token budgets per method, because multiplying baseline optimization steps by three does not by itself demonstrate equal compute once RODS synthesis, repeated discrimination, and regeneration are counted.