- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to infer a reward for an unseen task from a small support set. It meta-trains a rubric generator and a rubric-conditioned judge across tasks, using oracle rewards from training tasks to optimize candidate rankings. At test time, the inferred rubric and judge are used for best-of-N candidate selection or short policy optimization. The method is evaluated on personalized Amazon review generation and data curation for downstream language-model fine-tuning.
Critical strengths
-
Learning a reward for a new task from limited examples is a useful problem, and expressing that reward as a rubric offers an interpretable route to adaptation.
-
The data-curation experiment provides a meaningful downstream test: at 32 candidates, the normalized score is 0.92 versus 0.79 for the strongest reported non-oracle baseline (Table 2). The paper also tests whether the learned reward can guide candidate selection and short policy optimization.
Critical weaknesses
-
The full design is not sufficiently grounded. The proposed reward properties and broad problem setup are plausible, but the paper does not sufficiently motivate or isolate choices such as branching, distillation, and candidate augmentation. This makes it difficult to tell which parts are essential to the result.
-
The claimed advantage of explicit rubrics remains difficult to separate from general task-conditioned reward learning. A comparably trained judge without generated rubrics is missing. Moreover, augmentation raises the review score from 0.43 to 0.55 while also raising the candidate pool's oracle ceiling from 0.62 to 0.80 (Table 1), so that gain cannot be attributed to better scoring alone.
-
The evaluation supports a narrower conclusion than the broader self-improvement framing. The review task relies on an LLM-based similarity proxy, and key comparisons lack paired uncertainty estimates. The policy-optimization result is encouraging but covers only 20 steps in the review domain: oracle reward rises from 0.31 to 0.38, versus 0.35 for the next-best learned-reward baseline (Figure 4). This does not yet establish sustained improvement.
Questions for authors
-
Under comparable supervision and compute, how does a support-conditioned judge without explicit rubrics perform? Which major components are essential?
-
Can you provide paired uncertainty estimates for the main gains and an independent check of the review-domain evaluation proxy?
-
On the same augmented candidate pool, how much of the gain comes from scoring rather than generation, and what is the total computational cost?
4 comments
Response to Reviewer sZjP [2/N]
Existing evidence supports the task-specific role of rubrics [...] the claimed advantage of explicit rubrics remains difficult to separate from general task-conditioned reward learning.
The paper already includes task conditioned fine-tuned reward-model baselines with no rubric. On unaugmented review candidates, Meta-Rubrics achieves 0.43, compared with 0.38 for the CE reward model and 0.37 for the BT reward model; in data curation, it achieves 0.92 versus 0.71 for the support-conditioned fine-tuned reward model. **For both domains, this is trained on the identical set of dataset samples as the rubric generator and writer. **
We also show in Appendix I a complementary intervention within the trained system: replacing the task’s rubric with a rubric from another task reduces review selection performance from 0.43 to 0.37, while retaining the trained judge and candidate pool. This shows that correct task-specific rubric information materially affects selection; the rubric is not merely incidental text accompanying a stronger judge.
We therefore distinguish two claims. The existing evidence supports that the learned rubric carries useful task-specific information and that the complete approach outperforms the evaluated alternatives given access to an identical budget of training samples.
Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.
References
- GPT 4 model Card (https://openai.com/index/gpt-4-research/)
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (https://arxiv.org/abs/2402.03300)
- VinePPO: Refining Credit Assignment in RL Training of LLMs (https://arxiv.org/abs/2410.01679)
- RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems (https://arxiv.org/abs/2510.02263)
- Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024)
- Teaching Language Models to Critique via Reinforcement Learning (ICML 2025)
- Learning to Discover at Test-Time (https://arxiv.org/pdf/2601.16175)
Thank you for the rebuttal and clarifications. Your response clarifies some of my concerns regarding the baselines and the unaugmented data-curation results. I now better appreciate the paper’s empirical contribution and consider raising my Informativeness score from 1 to 2. However, the specific contribution of explicit rubrics relative to comparably trained task-conditioned reward models remains insufficiently isolated, and the requested paired uncertainty estimates have not been provided. These remain my main concerns, so I maintain my overall rating.
Response to Reviewer sZjP [1/1]
We thank reviewer sZjP for their continued engagement in the rebuttal process and for raising their informativeness score. We now address the concerns that the reviewer raises.
paired uncertainty estimates + trained task-conditioned reward models
We now add an evaluation on the litbench domain[1], which showcases the benefit of a rubric writer and judge in comparison to directly predicting the reward based on the task or the task + support information. We use the inkling 2 small model with the Tinker fine-tuning API for this experiment with LoRA for all methods/baselines on the same set of 512 prompts with 8 samples per prompt for the support set. This is a domain also used in the RLTF-FM paper[2] for their experiments, and we match the setup and judge prompt from the original litbench benchmark. For our evaluations, we use gpt 6 luna, which with the prompt has 71.24% agreement with human reviewers, using labels from the litbench paper. We also include the performance gap to oracle with 95% paired CI.
| Method | Mean selected score ↑ | Gap to oracle ↓ [95% paired CI] |
|---|---|---|
| Random selection | 6.459 | 2.194 [1.954, 2.544] |
| Direct, task only | 7.977 | 0.676 [0.532, 0.832] |
| Direct, task + support | 7.769 | 0.885 [0.564, 1.429] |
| Rubric writer/judge (support) | 8.191 | 0.462 [0.305, 0.616] |
| Original Luna oracle | 8.653 | --- |
We also measure the Spearman ρ = 0.840 for the rubric writer with the oracle reward function, with 95% CI [0.793, 0.873].
We would like to reiterate that we report standard errors for all methods and baselines in our main results (Tables 1, 2, 5, and 6; Figures 4 and 7), consistent with common practice for quantifying uncertainty in prior work [3,4]. Given our academic compute constraints and the limited time remaining in the rebuttal period, we are unable to rerun the full experimental suite to obtain bootstrap estimates for these results. We respectfully ask the AC to consider these constraints, together with the uncertainty estimates already provided, when assessing this aspect of our evaluation.
Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.
References:
- LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (https://arxiv.org/abs/2507.00769)
- Expanding the Capabilities of Reinforcement Learning via Text Feedback (https://arxiv.org/abs/2602.02482)
- Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024)
- Teaching Language Models to Critique via Reinforcement Learning (ICML 2025)
Response to Reviewer sZjP [1/N]
We thank Reviewer sZjP for recognizing the method’s contribution to learning a reward function from limited examples via a rubric and the significance of our evidence in the user review and data curation settings. Below, we address your questions regarding the branching/distillation, task specific reward learning, and discussion about uncertainty.
In the data-curation experiment, given only three reward-labeled support examples for a new task, Meta-Rubrics selects training data whose utility is measured by actually fine-tuning a target model (Smol-LM2) and evaluating its held-out negative log-likelihood. Evaluation comprises 100 tasks from 21 suites held out during training (Appendix G) and the outcome measures downstream training utility rather than agreement with an LLM judge on unseen test samples from well established benchmarks such as MMLU, which are used for frontier model evaluations [1].
At 32 candidates, Meta-Rubrics without candidate augmentation achieves a normalized score of 0.92, compared with 0.79 for the strongest reported non-oracle baseline and 0.68 for a single support-conditioned sample. Relative to that starting point, Meta-Rubrics obtains 0.24 of the available 0.32 oracle headroom, or 75%; the strongest baseline obtains 0.11, or approximately 34%. This substantiates that inferring an evaluator from sparse task-specific evidence enables substantially more effective use of additional candidates.
Here, neither the candidate augmentation nor the review-domain similarity proxy confound this result and directly supports the paper’s central contribution: learning to infer a useful reward for an unseen task from limited support information. Finally, in Table 1, we report the performance of Meta-Rubrics without candidate augmentation (0.43 pass@32), which also outperforms the best performing baseline (0.39 pass@32).
Both branching and distillation are practical implementation details rather than key choices for our method. Branching is a common variance reduction technique as seen in GRPO[2], VinePPO [3], RLAD [4] and other prior methods. Furthermore, learning a list/group of rewards is also a practical implementation detail, where if we branch + have a set of target candidates, the amount of optimization would be done over M rubrics X N candidates per rubric, which makes optimization inefficient and potentially harder to balance during optimization. Therefore, we opt to learn a list of rewards for candidates to reduce the calls to two per given prompt. If we do choose this design choice, however, during inference time, it is quite likely that we would not evaluate with the same fixed length of candidates. Therefore, distillation is an extraction technique taking a set of n-candidates and learning the individual scores. Our main contribution is understanding the ability to use interaction to infer a useful reward representation that can adapt to new tasks at deployment time, and branching and distillation are practical implementation details to instantiate such a system. We can move these choices to the appendix and/or clearly label them in the manuscript as practical design choices.
We are happy to refine our framing to avoid overstating sustained self-improvement. As noted in our limitations section, the inferred reward function can become unreliable as policy optimization shifts the response distribution away from the support data. That said, our 20-step update horizon is aligned with prior work such as TTT-Discover, which evaluates RL over 50 steps. Crucially, our results demonstrate that inferring a targeted reward model yields clear optimization gains for the base model within limited interaction budgets across both the review generation and data curation tasks. Future work can explore joint adaptation of rubric generation and policy training to maintain alignment throughout longer optimization runs.
Note that we already report standard errors for each method and baseline approach in our main results ( Table 1/2/5/6 and Figure 4/7), which is a common practice to measure uncertainty in several prior works [5,6].