acceptodds
Under review as a conference paper at ICLR 2027

Rubric Optimization for LLM-Based Automated Grading with Data Synthesis

Abstract

Large language models (LLMs) have recently emerged as a promising approach for automatic grading (AG) due to their strong instruction-following and reasoning abilities. Recent work has explored automatic rubric optimization using labeled student responses to address rubric quality issues. However, its effectiveness is often limited by the scarcity of annotations. In this work, we propose a new optimization paradigm, RODS, which leverages the generative capabilities of LLMs to synthesize responses as supplementary data, enabling existing rubric optimization methods to perform robustly under limited-label settings. To mitigate noise in synthetic labels, we design a consistency-based optimization mechanism. We further introduce a generative adversarial-based quality control (GAQC) module to improve the fidelity of synthetic samples, reducing distribution shift and enhancing alignment with real student responses. In addition, we integrate both synthetic and scarce human-labeled data within an iterative optimization framework, ensuring that the optimization process remains grounded in improving grading performance on real responses. Experiments on multiple AG datasets demonstrate that RODS consistently improves rubric quality and grading accuracy, highlighting its effectiveness and practical potential for real-world LLM-based automatic grading.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Rubric Optimization, Automatic Grading System, Large Language Model, Data Synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2UMcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RODS, a framework for optimizing natural-language grading rubrics when only a small number of scored student responses is available. RODS rewrites real responses toward target score levels, uses an LLM-based generative adversarial quality-control module (GAQC) to filter or refine synthetic responses, and alternates label-supervised and consistency-supervised rubric refinement with beam search. The consistency signal compares the target label used for synthesis with the score produced when the same rubric grades the synthetic response; a semi-supervised variant additionally uses unlabeled responses. The evaluation covers more than 1,700 responses, 12 questions, three datasets, two LLM backbones, six baselines, and component ablations. The reported averages favor RODS over the strongest rubric-optimization baseline, but several methodological and reporting inconsistencies currently prevent me from treating the gains as conclusive.

Critical strengths
  1. Important and practically relevant problem. Reducing the amount of scored data needed to adapt transparent, teacher-editable rubrics is valuable for real educational deployment. The paper also identifies two genuine risks of synthetic supervision - label noise and distribution shift - and attempts to address both within a coherent optimization pipeline.
  2. Broad empirical effort and useful component analysis. The study evaluates 12 questions from three subject areas with both a local Qwen model and a Gemini model, includes rubric-specific and general prompt-optimization baselines, and reports several targeted ablations. The reported accuracy gains and the observation that naive synthetic demonstrations can hurt performance are potentially useful findings if the validity issues below are resolved.
Critical weaknesses

The evaluation contains internal inconsistencies and does not establish statistical robustness. For single-label multiclass classification, the micro-F1 formula in Eq. (11) is algebraically equal to accuracy, yet Tables 4-7 report substantially different values for the two metrics. This suggests either a metric implementation error or a mislabeled metric. The central correlation analysis is also weak: it uses only Q1 and Q4, reports r=0.25-0.43 with p=0.219-0.434, and nevertheless presents the result as support for self-consistency as a surrogate objective. In addition, the test sets are small (approximately 11-50 responses per question), per-question hyperparameters are selected on small validation sets, and no paired confidence intervals or significance tests are provided for the claimed improvements. Even the stated Qwen accuracy gain in Section 4.2 (4.4) does not match Table 1 (51.7 - 46.1 = 5.6 percentage points). These issues directly affect the main empirical claim. 2. The GAQC decision rule is contradictory and its claimed fidelity improvement is insufficiently validated. Equation (4) defines d_omega(x) as P(real|x), and the accompanying text says that larger values indicate higher fidelity and should be retained. However, Algorithm 1 and Appendix A.2 retain a sample when d_omega < 0.5, which is the opposite rule under that definition. If d_omega instead means discriminator accuracy on a known synthetic sample, then Eq. (4) and the surrounding interpretation are incorrect. This ambiguity blocks reproduction of a core component. Moreover, GAQC is supported mainly by downstream ablation and one qualitative example; there is no independent human blind evaluation or held-out fidelity measure, while the same model family generates, characterizes, and judges the responses. The semi-supervised variant also uses validation and test response texts during optimization, making it transductive and not directly comparable to baselines that do not receive those inputs.

Questions for authors

GAQC definition. What exactly is d_omega(x) in the implementation: the probability of predicting "real," the probability of correctly detecting a synthetic sample, or a dataset-level discrimination accuracy? Please reconcile Eq. (4), Algorithm 1, and Appendix A.2, state the actual inequality used, and report the fraction of samples accepted, refined, and discarded at each iteration. 2. Metric correctness and uncertainty. Why does the reported micro-F1 differ from accuracy under the single-label multiclass setup and Eq. (11)? Was macro-F1 or another averaging scheme actually computed? Please also clarify how the split and generation repetitions were aggregated, correct the Qwen headline improvement, and provide paired uncertainty or significance estimates for RODS versus GradeOPT. 3. Evidence for the surrogate objective. Given that all four reported correlations in Figure 4 are statistically non-significant and cover only two questions, what evidence shows that higher self-consistency reliably predicts better held-out grading across the 12 questions? An analysis over all candidate rubrics and questions, with uncertainty and a control for shared-model self-confirmation, would materially affect my assessment. 4. Fairness of the semi-supervised and compute comparisons. Can the authors report results for a strictly inductive unlabeled-data setting and/or give competing methods the same unlabeled validation/test inputs? Please also provide exact LLM-call and token budgets per method, because multiplying baseline optimization steps by three does not by itself demonstrate equal compute once RODS synthesis, repeated discrimination, and regeneration are counted.

Reviewer_w8RxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RODS, a framework to optimize LLM grading rubrics when human-scored responses are scarce. RODS generates synthetic responses by rewriting ground-truth responses toward target scores, uses an LLM discriminator and feedback loop to improve their similarity to student responses, and alternates rubric refinement using human labels and generation-grading consistency. Candidate rubrics are selected through beam search using both signals. Experiments cover 12 grading tasks from three datasets and two backbone models, with 10 training labels and 10-20 validation labels per task. Table 1 reports average accuracy improvements over GradeOpt of 5.6 percentage points for Qwen and 2.3 for Gemini. Ablations support the framework’s components. A transductive variant additionally uses unlabeled validation/test response texts.

Critical strengths
  • The framework addresses a practical constraint. Obtaining scored responses for every assessment question is expensive. Grounding both synthesis and rubric selection is a useful design choice.
Critical weaknesses
  • The evidence for the central consistency-based interpretation remains limited. Generation-grading agreement does not establish agreement with human grading. The same model can generate and grade responses consistently under an incorrect or shifted interpretation of the rubric. Figure 4 examines only two questions and reports modest correlations, approximately (r=0.25-0.43), with all reported (p)-values above 0.2. These results are insufficient to support the claimed relationship. Although removing consistency supervision reduces performance, that ablation does not isolate the benefit of posterior-guided refinement over simply trusting synthetic target labels.

  • It seems to be some inconsistency in evaluation and algorithm specifications. The reported micro-F1 differs from accuracy despite the stated single-label setup, and Algorithm 1 reverses Equation 4’s filtering direction. These discrepancies leave the implemented procedure unclear and reduce confidence in reproducibility.

Questions for authors
  • What F1 averaging scheme was actually implemented? Please reconcile Tables 4–5 with Equation 11 and clarify whether correcting the discrepancy changes any conclusions.

  • Does the implementation retain high (P(real∣x))(P(\mathrm{real}\mid x)), or low synthetic-classification accuracy? Which definition and threshold direction produced the reported experiments?

  • Under the same synthesis and search budget, does bidirectional guidance outperform prior-only guidance that trusts synthetic target labels? Can the consistency-accuracy analysis be reported across all 12 tasks, with uncertainty estimates?

Reviewer_Aq3GOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes RODS, an automatic rubric optimization framework for automated grading. The framework first synthesizes new responses by rewriting real student answers toward target scores. It then uses a generative quality-control procedure to reduce the gap between synthetic and real responses, and iteratively updates rubric candidates using label-based and consistency-based supervision. The framework is evaluated on 12 questions from three educational datasets with Qwen3-4B-Instruct and Gemini-3.1-Flash-Lite. Experimental results demonstrate the superiority of RODS over various rubric optimization baselines, and the ablation study supports the value of combining real labels, controlled synthesis, quality control, and iterative search.

Critical strengths
  • The research problem addressed in this paper is practically important, as current automatic grading methods often rely on many human labels for rubric calibration, which is costly and difficult to scale up.
  • The rubric optimization framework seems novel and well-designed. Grounding synthetic supervision in real student responses gives the paper a credible argument that its data may reflect real grading cases better than idealized generated answers. The consistency signal also helps the synthesis and grading models reach a shared understanding of the rubric.
  • The experiments are comprehensive and thoroughly demonstrate the contribution of each module in RODS.
Critical weaknesses
  • The evaluation covers only a small set of 12 questions across three datasets, which is insufficient to demonstrate the generalizability of the proposed method across different domains. It is recommended that the authors conduct experiments on more commonly used meta-evaluation datasets, such as SummEval, Topical-Chat, and HANNA.
  • As the authors claim existing rubric optimization methods are often limited by the scarcity of annotations, the paper should compare different baselines under varying sizes of the training set (labeled data), which is needed to demonstrate the unique advantage of RODS when labeled samples are scarce.
  • The paper lacks comparisons with other prompt optimization methods for LLM evaluators, such as AutoCalibrate, HD-Eval, and HPSS.
Questions for authors
  • For Figure 4, why do the authors report results only for Q1 and Q4? Does the conclusion hold consistently across other questions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.