acceptodds
Under review as a conference paper at ICLR 2027

Meta-Rubrics: Learning to Discover Rubrics at Deployment for Efficient Task Adaptation

Abstract

Large language model (LLM) agents are increasingly deployed in open-ended environments where task objectives are under-specified and no task-specific evaluator is available. Reinforcement learning (RL) and inference-time search have driven major gains in verifiable domains such as math and coding, but these gains rely on a verification–generation (VG) gap: checking a solution is easier than producing one. In open-ended tasks, however, this gap cannot be assumed, because a verifier must first infer what constitutes success. This raises a central question: can an agent can construct or widen such a gap at deployment time from a small amount of environment interaction and use this gap to aid policy adaptation? We formulate this as a reward discovery setting and propose a framework, Meta-Rubrics, which infers a reward function, represented as a natural-language rubric, which can accurately assign credit to guide policy behavior at deployment time. In this framework, a rubric generator infers task-specific evaluation criteria, and an LLM judge applies these criteria to produce scalar scores and criterion-level feedback for candidate trajectories. The generator and judge are jointly optimized with RL over branched rollouts so that induced scores recover oracle reward orderings on held-out trajectories. At deployment time, this inferred reward can rank candidate responses or supply a signal for lightweight policy optimization with RL. We evaluate Meta-Rubrics on user simulation in Human-LM and curating data for a language model for an unseen task with a fixed fine-tuning budget. In both settings, Meta-Rubrics outperforms the highest non-oracle adaptation approaches such as in-context learning, evolutionary search, and rubric-based fine-tuning baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, RL, Meta-Learning, Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sZjPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to infer a reward for an unseen task from a small support set. It meta-trains a rubric generator and a rubric-conditioned judge across tasks, using oracle rewards from training tasks to optimize candidate rankings. At test time, the inferred rubric and judge are used for best-of-N candidate selection or short policy optimization. The method is evaluated on personalized Amazon review generation and data curation for downstream language-model fine-tuning.

Critical strengths
  1. Learning a reward for a new task from limited examples is a useful problem, and expressing that reward as a rubric offers an interpretable route to adaptation.

  2. The data-curation experiment provides a meaningful downstream test: at 32 candidates, the normalized score is 0.92 versus 0.79 for the strongest reported non-oracle baseline (Table 2). The paper also tests whether the learned reward can guide candidate selection and short policy optimization.

Critical weaknesses
  1. The full design is not sufficiently grounded. The proposed reward properties and broad problem setup are plausible, but the paper does not sufficiently motivate or isolate choices such as branching, distillation, and candidate augmentation. This makes it difficult to tell which parts are essential to the result.

  2. The claimed advantage of explicit rubrics remains difficult to separate from general task-conditioned reward learning. A comparably trained judge without generated rubrics is missing. Moreover, augmentation raises the review score from 0.43 to 0.55 while also raising the candidate pool's oracle ceiling from 0.62 to 0.80 (Table 1), so that gain cannot be attributed to better scoring alone.

  3. The evaluation supports a narrower conclusion than the broader self-improvement framing. The review task relies on an LLM-based similarity proxy, and key comparisons lack paired uncertainty estimates. The policy-optimization result is encouraging but covers only 20 steps in the review domain: oracle reward rises from 0.31 to 0.38, versus 0.35 for the next-best learned-reward baseline (Figure 4). This does not yet establish sustained improvement.

Questions for authors
  1. Under comparable supervision and compute, how does a support-conditioned judge without explicit rubrics perform? Which major components are essential?

  2. Can you provide paired uncertainty estimates for the main gains and an independent check of the review-domain evaluation proxy?

  3. On the same augmented candidate pool, how much of the gain comes from scoring rather than generation, and what is the total computational cost?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sZjP [1/N]

We thank Reviewer sZjP for recognizing the method’s contribution to learning a reward function from limited examples via a rubric and the significance of our evidence in the user review and data curation settings. Below, we address your questions regarding the branching/distillation, task specific reward learning, and discussion about uncertainty.

The central result holds without candidate augmentation and without an LLM-based evaluation proxy.

In the data-curation experiment, given only three reward-labeled support examples for a new task, Meta-Rubrics selects training data whose utility is measured by actually fine-tuning a target model (Smol-LM2) and evaluating its held-out negative log-likelihood. Evaluation comprises 100 tasks from 21 suites held out during training (Appendix G) and the outcome measures downstream training utility rather than agreement with an LLM judge on unseen test samples from well established benchmarks such as MMLU, which are used for frontier model evaluations [1].

At 32 candidates, Meta-Rubrics without candidate augmentation achieves a normalized score of 0.92, compared with 0.79 for the strongest reported non-oracle baseline and 0.68 for a single support-conditioned sample. Relative to that starting point, Meta-Rubrics obtains 0.24 of the available 0.32 oracle headroom, or 75%; the strongest baseline obtains 0.11, or approximately 34%. This substantiates that inferring an evaluator from sparse task-specific evidence enables substantially more effective use of additional candidates.

Here, neither the candidate augmentation nor the review-domain similarity proxy confound this result and directly supports the paper’s central contribution: learning to infer a useful reward for an unseen task from limited support information. Finally, in Table 1, we report the performance of Meta-Rubrics without candidate augmentation (0.43 pass@32), which also outperforms the best performing baseline (0.39 pass@32).

The paper does not sufficiently motivate or isolate choices such as branching and distillation

Both branching and distillation are practical implementation details rather than key choices for our method. Branching is a common variance reduction technique as seen in GRPO[2], VinePPO [3], RLAD [4] and other prior methods. Furthermore, learning a list/group of rewards is also a practical implementation detail, where if we branch + have a set of target candidates, the amount of optimization would be done over M rubrics X N candidates per rubric, which makes optimization inefficient and potentially harder to balance during optimization. Therefore, we opt to learn a list of rewards for candidates to reduce the calls to two per given prompt. If we do choose this design choice, however, during inference time, it is quite likely that we would not evaluate with the same fixed length of candidates. Therefore, distillation is an extraction technique taking a set of n-candidates and learning the individual scores. Our main contribution is understanding the ability to use interaction to infer a useful reward representation that can adapt to new tasks at deployment time, and branching and distillation are practical implementation details to instantiate such a system. We can move these choices to the appendix and/or clearly label them in the manuscript as practical design choices.

The evaluation supports a narrower conclusion than the broader self-improvement framing [...] the policy-optimization result is encouraging but covers only 20 steps in the review domain.

We are happy to refine our framing to avoid overstating sustained self-improvement. As noted in our limitations section, the inferred reward function can become unreliable as policy optimization shifts the response distribution away from the support data. That said, our 20-step update horizon is aligned with prior work such as TTT-Discover, which evaluates RL over 50 steps. Crucially, our results demonstrate that inferring a targeted reward model yields clear optimization gains for the base model within limited interaction budgets across both the review generation and data curation tasks. Future work can explore joint adaptation of rubric generation and policy training to maintain alignment throughout longer optimization runs.

Uncertainty estimates

Note that we already report standard errors for each method and baseline approach in our main results ( Table 1/2/5/6 and Figure 4/7), which is a common practice to measure uncertainty in several prior works [5,6].

AuthorsOpenReview ↗

Response to Reviewer sZjP [2/N]

Existing evidence supports the task-specific role of rubrics [...] the claimed advantage of explicit rubrics remains difficult to separate from general task-conditioned reward learning.

The paper already includes task conditioned fine-tuned reward-model baselines with no rubric. On unaugmented review candidates, Meta-Rubrics achieves 0.43, compared with 0.38 for the CE reward model and 0.37 for the BT reward model; in data curation, it achieves 0.92 versus 0.71 for the support-conditioned fine-tuned reward model. **For both domains, this is trained on the identical set of dataset samples as the rubric generator and writer. **

We also show in Appendix I a complementary intervention within the trained system: replacing the task’s rubric with a rubric from another task reduces review selection performance from 0.43 to 0.37, while retaining the trained judge and candidate pool. This shows that correct task-specific rubric information materially affects selection; the rubric is not merely incidental text accompanying a stronger judge.

We therefore distinguish two claims. The existing evidence supports that the learned rubric carries useful task-specific information and that the complete approach outperforms the evaluated alternatives given access to an identical budget of training samples.

Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.

References

  1. GPT 4 model Card (https://openai.com/index/gpt-4-research/)
  2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (https://arxiv.org/abs/2402.03300)
  3. VinePPO: Refining Credit Assignment in RL Training of LLMs (https://arxiv.org/abs/2410.01679)
  4. RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems (https://arxiv.org/abs/2510.02263)
  5. Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024)
  6. Teaching Language Models to Critique via Reinforcement Learning (ICML 2025)
  7. Learning to Discover at Test-Time (https://arxiv.org/pdf/2601.16175)

Thank you for the rebuttal and clarifications. Your response clarifies some of my concerns regarding the baselines and the unaugmented data-curation results. I now better appreciate the paper’s empirical contribution and consider raising my Informativeness score from 1 to 2. However, the specific contribution of explicit rubrics relative to comparably trained task-conditioned reward models remains insufficiently isolated, and the requested paired uncertainty estimates have not been provided. These remain my main concerns, so I maintain my overall rating.

AuthorsOpenReview ↗

Response to Reviewer sZjP [1/1]

We thank reviewer sZjP for their continued engagement in the rebuttal process and for raising their informativeness score. We now address the concerns that the reviewer raises.

paired uncertainty estimates + trained task-conditioned reward models

We now add an evaluation on the litbench domain[1], which showcases the benefit of a rubric writer and judge in comparison to directly predicting the reward based on the task or the task + support information. We use the inkling 2 small model with the Tinker fine-tuning API for this experiment with LoRA for all methods/baselines on the same set of 512 prompts with 8 samples per prompt for the support set. This is a domain also used in the RLTF-FM paper[2] for their experiments, and we match the setup and judge prompt from the original litbench benchmark. For our evaluations, we use gpt 6 luna, which with the prompt has 71.24% agreement with human reviewers, using labels from the litbench paper. We also include the performance gap to oracle with 95% paired CI.

MethodMean selected score ↑Gap to oracle ↓ [95% paired CI]
Random selection6.4592.194 [1.954, 2.544]
Direct, task only7.9770.676 [0.532, 0.832]
Direct, task + support7.7690.885 [0.564, 1.429]
Rubric writer/judge (support)8.1910.462 [0.305, 0.616]
Original Luna oracle8.653---

We also measure the Spearman ρ = 0.840 for the rubric writer with the oracle reward function, with 95% CI [0.793, 0.873].

We would like to reiterate that we report standard errors for all methods and baselines in our main results (Tables 1, 2, 5, and 6; Figures 4 and 7), consistent with common practice for quantifying uncertainty in prior work [3,4]. Given our academic compute constraints and the limited time remaining in the rebuttal period, we are unable to rerun the full experimental suite to obtain bootstrap estimates for these results. We respectfully ask the AC to consider these constraints, together with the uncertainty estimates already provided, when assessing this aspect of our evaluation.

Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.

References:

  1. LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (https://arxiv.org/abs/2507.00769)
  2. Expanding the Capabilities of Reinforcement Learning via Text Feedback (https://arxiv.org/abs/2602.02482)
  3. Confronting Reward Model Overoptimization with Constrained RLHF (ICLR 2024)
  4. Teaching Language Models to Critique via Reinforcement Learning (ICML 2025)
Reviewer_mhSHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes meta-rubrics that guides LLM learn the evaluation rubrics for the coming task, and use the rubrics to filter data/conduct RL.

The core mechanism might be: the meta-rubrics is a more efficient indicator to the true reward shaping.

In experiments with user simulation in human-LM and data curation, meta-rubrics outperformed baselines such as non-oracle adaptation approaches/search/finetune rubrics/LLM-judges.

Critical strengths
  1. This work provides an insight into a low-resource/fixed budget setting where quick adaption to implicit (not documentated or observed) preference/reward is needed. From my view meta-rubrics provide sufficient evidences on the amazon user comment generation and the data selection/filter/curation setting.
Critical weaknesses
  1. The experiment setting is quite attypical. As far as I learn from the method, the authors shall experiment with complex evaluation tasks, such as creative writing, image generation. The amazon user comment generation is not sufficient/typical for validation. Also, for data curation task, there lack baseline comparisons.
  2. Current analysis shows limited findings/knowledge with the proposed meta-reward. To what extent shall we measure meta-reward ease the calibration with the true preference? And to what is the ceiling if we apply the meta-rubrics method to the full set? How can the meta-rubrics extend to other tasks? And for what tasks meta-rubrics would fail, and can we define/measure this boundary?
Questions for authors

See weakness 2. After going through this submission, I am still confused with these questions. Perhaps the current experiments did not exhibit the advantages of meta-rubrics in its way.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mhSH [1/N]

We thank Reviewer mhSH for recognizing the method’s contribution to low-resource adaptation to new environments where the target reward is unknown and the significance of our evidence in the user review and data curation settings. Below, we address your questions regarding the experimental setting and calibration of the reward function.

The experiment setting is quite atypical

For our evaluation, we match the experimental setting of prior works and papers that explore adaptation at test-time. For example, for reviews, we directly use the book evaluation from the HumanLM paper [1], which has also been used directly in prior work such as Socio-Foundation[2] and OdysSim[3] for training and evaluation. The data curation setting is a newly proposed environment but still shares close similarity to RSI style evals such as AutoResearch [4], where the goal is to improve a model’s pre-training or post-training ability. Here, we use a commonly used open-source model, SmolLM2, used in prior fine-tuning papers [5,6] and hope to improve its mid-training/post-training behavior by curating useful samples from a larger corpus of samples for a target benchmark, matching the intuitions of prior work [7,8] that curating a well defined corpus for alignment/post-training can lead to high quality outputs. Similar to autoresearch style methods, we allow for environment interaction but we study this in a constrained setting (limited evaluations), since environment interaction isn’t free and is often a bottleneck for adaptation.

Data Curation lacks baseline comparisons

In table 2, copied below for your convenience, we show gains of our approach on Data Curation across several baseline methods, matching the same set of evaluations as the Review setting.

Test-time inference results on Data Curation across inference samples.

Method12481632
Self-Refine0.55 (0.052)0.68 (0.066)0.61 (0.077)0.42 (0.078)0.62 (0.078)0.50 (0.076)
Self-Refine (Support)0.68 (0.033)0.76 (0.061)0.43 (0.076)0.49 (0.083)0.40 (0.083)0.44 (0.085)
RSA (Support)0.68 (0.033)0.50 (0.071)0.53 (0.071)0.56 (0.073)0.59 (0.064)0.62 (0.070)
Feedback Descent (Support)0.68 (0.033)0.45 (0.100)0.54 (0.099)0.53 (0.095)0.60 (0.095)0.77 (0.088)
LLM Rubric Judge (Base)0.55 (0.052)0.52 (0.052)0.56 (0.057)0.60 (0.058)0.54 (0.066)0.55 (0.055)
LLM Rubric Judge (Support)0.68 (0.033)0.69 (0.058)0.75 (0.038)0.78 (0.040)0.81 (0.040)0.79 (0.041)
Fine-Tuned RM (Base)0.55 (0.052)0.55 (0.034)0.56 (0.035)0.57 (0.037)0.58 (0.039)0.57 (0.043)
Fine-Tuned RM (Support)0.68 (0.033)0.70 (0.035)0.71 (0.037)0.71 (0.040)0.70 (0.042)0.71 (0.047)
Semantic Maj@k (Base)0.55 (0.052)0.55 (0.052)0.55 (0.052)0.55 (0.052)0.55 (0.052)0.55 (0.052)
Semantic Maj@k (Support)0.68 (0.033)0.75 (0.044)0.75 (0.044)0.75 (0.044)0.75 (0.044)0.75 (0.044)
Ours (Support)0.68 (0.033)0.78 (0.031)0.84 (0.028)0.87 (0.025)0.90 (0.022)0.92 (0.029)
Oracle (Base)0.55 (0.052)0.65 (0.043)0.74 (0.042)0.79 (0.041)0.87 (0.038)1.00 (0.000)
Oracle (Support)0.68 (0.033)0.79 (0.042)0.86 (0.032)0.91 (0.024)0.94 (0.020)1.00 (0.000)
AuthorsOpenReview ↗

Response to Reviewer mhSH [2/N]

To what extent shall we measure meta-reward ease the calibration with the true preference

meta-rubrics extend to other tasks

We explicitly measure the calibration/alignment of the reward with the true reward as seen in RQ3 in section 5. For example, in Figure 5, we show a monotonically increasing relationship between the predicted reward and the true reward distribution for the data curation task, with a spearman correlation of 0.67, whereas the prompted LLM has a spearman correlation of 0.05, indicating that training for this enables better alignment. Furthermore, we track the spearman correlation coefficient during training as seen in Table 6 in Appendix A.14, showcasing that our method has better alignment to the true reward distribution with a pearson/spearman correlation of 0.41/0.36 in comparison to the best adaptation method of 0.19/0.15, even as the policy distribution is evolving:

MethodLLM JudgeLLM RubricRubric DropoutRLEROursOracle Rewards
Pearson r-0.030.150.190.160.411.00
Spearman ρ-0.060.120.150.130.361.00

Here, we measure relative alignment instead of absolute reward because we care about credit assignment and distinguishing between different candidate samples for both inference time sampling and RL rather than matching the reward function directly. It is for this reason that our alignment function that we choose is the harmonic best of n function, which directly measures the contribution of the rubric and judgement to scoring according to the target reward function. That being said, we now also measure the mean absolute error (MAE) between the learnt reward function and the prompted reward function on the reviews task and see that this too decreases from 0.36 to 0.28, indicating some improvement in absolute reward reconstruction after optimization.

Finally, we showcase that the rubrics generated are specific to a new deployment task, where in Appendix A.13, where when shuffling rubrics from a different task (user, product) pair, we see that performance degrades.

Method12481632
Ours (Support)0.35 (0.022)0.38 (0.025)0.39 (0.027)0.40 (0.028)0.41 (0.028)0.43 (0.021)
Ours (Support) - Shuffled0.35 (0.022)0.37 (0.024)0.37 (0.025)0.37 (0.026)0.37 (0.026)0.37 (0.026)
Oracle (Support)0.35 (0.022)0.43 (0.025)0.49 (0.027)0.54 (0.027)0.58 (0.026)0.62 (0.025)

This indicates that there is a task level adaptation that is occurring, using the interaction experience, to learn some structure about the reward function.

Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.

References

  1. HumanLM: Simulating Users with State Alignment Beats Response Imitation (https://arxiv.org/abs/2603.03303)
  2. Socio-Foundation: A Model for Generalizable Individual Behavior Simulation via Hierarchical Capability Distillation (https://arxiv.org/abs/2610.08967)
  3. OdysSim: Building Foundation Models for Human Behavior Simulation (https://arxiv.org/abs/2606.14199)
  4. AutoResearch (https://github.com/karpathy/autoresearch)
  5. SmolVLA (https://arxiv.org/pdf/2506.01844)
  6. Scaling Latent Reasoning via Looped Language Models (https://arxiv.org/abs/2510.25741)
  7. Less is More for Alignment (https://arxiv.org/pdf/2305.11206)
  8. s1: Simple test-time scaling (https://arxiv.org/abs/2501.19393)

Thanks for the clarification. I got to further concerns:

  1. By correlation to the oracle reward, can the authors explain why meta-rubrics achieves high correlation to oracle rewards, while LLM Judge/LLM Rubric all holds only around unrelated? Also, is the LLM-judge a Qwen-32B? can you add one GPT-5 (or after) or Gemini/Claude of same capability level to validate the SOTA standard on this task?

  2. How do the authors view the experiment results, that a ~0.4 correlated to oracle reward signal is able to uplift the performance? Also can you address what scene that meta-rubrics would exert unique advantage to the oracle?

Thank you!

Reviewer_TcfcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper solves the problem of finding rubrics for unseen tasks. It does so by doing rubric+judge writing during training and best of N selection of the trained model during the test time.

Critical strengths
  • The paper solves a difficult problem which is finding rubrics for unseen tasks
  • The novelty of the paper is great by doing joint optimization of rubric and judge + best of N during test time
  • The paper is easy to follow
Critical weaknesses
  • This paper needs better figures, I read a printed version and I could hardly read figure 1. The text in figures also should be readable. Same goes for joint optimization process that could benefit from a more detailed figure or algorithm rather than just "Optimization Procedure" written as text.
  • Only one dataset (HumanLM) is used for evaluation.
Questions for authors
  • What other datasets you can easily add to your method? Even a neutral results on other datasets is more informative than having results on only one dataset. Because method might be overfitted on HumanLM.
1 comment

We thank Reviewer Tcfc for the positive assessment of our work and for recognizing the contribution of finding rubrics for unseen tasks with a novel joint objective for rubric writing and judging to aid test-time inference/policy optimization. Below, we address your questions regarding the figures and additional domains for evaluation.

This paper needs better figures.

Thank you for the feedback. We have refactored both figure 1 (teaser) + figure 2 (branched rollouts) to reflect the changes you have requested in an anonymous url:

If you have any further feedback, happy to clarify and further make edits to the manuscript.

Only one dataset (HumanLM) is used for evaluation.

We do evaluate on a second dataset for Data Curation. The task setup for this is to fine-tune an LLM (Smol-LM 2) for a target downstream benchmark with a fixed fine-tuning token budget. The agent selects training examples from a candidate source pool, and the selected examples are used to fine-tune a small target model,HuggingFaceTB/SmolLM2-135M (Allal et al., 2025). Gold rewards are computed from the downstream improvement in held-out negative log-likelihood or perplexity after fine-tuning. In table 2, copied below for your convenience, we shows gains of our approach on Data Curation.

Test-time inference results on Data Curation across inference samples.

Method12481632
Self-Refine0.55 (0.052)0.68 (0.066)0.61 (0.077)0.42 (0.078)0.62 (0.078)0.50 (0.076)
Self-Refine (Support)0.68 (0.033)0.76 (0.061)0.43 (0.076)0.49 (0.083)0.40 (0.083)0.44 (0.085)
RSA (Support)0.68 (0.033)0.50 (0.071)0.53 (0.071)0.56 (0.073)0.59 (0.064)0.62 (0.070)
Feedback Descent (Support)0.68 (0.033)0.45 (0.100)0.54 (0.099)0.53 (0.095)0.60 (0.095)0.77 (0.088)
LLM Rubric Judge (Base)0.55 (0.052)0.52 (0.052)0.56 (0.057)0.60 (0.058)0.54 (0.066)0.55 (0.055)
LLM Rubric Judge (Support)0.68 (0.033)0.69 (0.058)0.75 (0.038)0.78 (0.040)0.81 (0.040)0.79 (0.041)
Fine-Tuned RM (Base)0.55 (0.052)0.55 (0.034)0.56 (0.035)0.57 (0.037)0.58 (0.039)0.57 (0.043)
Fine-Tuned RM (Support)0.68 (0.033)0.70 (0.035)0.71 (0.037)0.71 (0.040)0.70 (0.042)0.71 (0.047)
Semantic Maj@k (Base)0.55 (0.052)0.55 (0.052)0.55 (0.052)0.55 (0.052)0.55 (0.052)0.55 (0.052)
Semantic Maj@k (Support)0.68 (0.033)0.75 (0.044)0.75 (0.044)0.75 (0.044)0.75 (0.044)0.75 (0.044)
Ours (Support)0.68 (0.033)0.78 (0.031)0.84 (0.028)0.87 (0.025)0.90 (0.022)0.92 (0.029)
Oracle (Base)0.55 (0.052)0.65 (0.043)0.74 (0.042)0.79 (0.041)0.87 (0.038)1.00 (0.000)
Oracle (Support)0.68 (0.033)0.79 (0.042)0.86 (0.032)0.91 (0.024)0.94 (0.020)1.00 (0.000)

This showcases that the method doesn’t overfit on the domain of HumanLM.

Furthermore, we show results in a synthetic bandit domain (Appendix A.9) as additional justification that amplifying the verification generation gap is helpful and leads to lower simple regret.

Please let us know if you find your concerns addressed, and if so, we would be grateful if you are willing to raise your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal-Period Summary – Key Clarifications & New Results

We thank the reviewers and the AC for their thoughtful and constructive feedback. The discussion has helped us clarify the contribution of Meta-Rubrics, strengthen its empirical evaluation, and better define the scope of our claims. For the AC’s convenience, we summarize the key additions and clarifications below:

  • Evaluation Beyond HumanLM + Beyond LLM Judge We clarify that the paper evaluates on the data curation domain, where an agent selects examples from a mid-training corpus to fine-tune SmolLM2, where the reward function measures actual improvement in held-out negative log-likelihood, without relying on an LLM judge. This evaluation comprises 100 tasks from 21 suites held out during training, with only three reward-labeled support examples per new task. At 32 candidates, Meta-Rubrics achieves a normalized score of 0.92, compared with 0.79 for the strongest reported non-oracle baseline. The paper additionally includes a synthetic bandit evaluation examining the benefit of improved verification, that further supports this claim. Finally, we add a creative-writing evaluation on LitBench, using the same backbone and training data for rubric-based and direct reward predictors. Meta-Rubrics achieves a mean selected score of 8.19 versus 7.97 for the task-only predictor and 7.77 for the task-plus-support predictor. We added paired confidence intervals for each method’s gap to the oracle, measured 71.24% agreement between the evaluation judge and human labels, and found a Spearman correlation of 0.840 between rubric-based scores and oracle rewards.

  • Comparisons Against Task-Conditioned Reward Models and the Role of Explicit Rubrics: We clarify that the paper includes reward-model baselines without explicit rubrics trained on the same dataset of samples, with the same base model. On unaugmented review candidates, Meta-Rubrics achieves 0.43, compared with 0.38 and 0.37 for the CE and BT reward models; on data curation, it achieves 0.92 versus 0.71 for the support-conditioned fine-tuned reward model. A complementary intervention replaces the correct rubric with one from another task while retaining the trained judge and candidate pool, reducing review performance from 0.43 to 0.37. These results support the contribution of task-specific rubric information.

  • Separating Selection Gains from Candidate Augmentation: Reviewer sZjP noted that augmentation improves the candidate pool’s oracle ceiling, making it difficult to attribute the augmented result to scoring alone. We therefore emphasized the results without augmentation: Meta-Rubrics achieves 0.43 versus 0.39 for the strongest baseline on review generation, and 0.92 versus 0.79 on data curation. In data curation, starting from a single-sample score of 0.68, our method captures 75% of the available oracle headroom, compared with approximately 34% for the strongest baseline. These comparisons demonstrate useful selection gains independently of candidate augmentation.

  • Alignment with the Underlying Reward: Reviewer mhSH requested quantitative evidence of how closely inferred rewards track the target reward. We highlighted the data-curation analysis, where Meta-Rubrics achieves a Spearman correlation of 0.67 with oracle rewards, compared with 0.05 for the prompted baseline. During policy optimization, our method also exhibits stronger reward alignment than the evaluated alternatives, with Pearson/Spearman correlations of 0.41/0.36. We clarified that our objective prioritizes ranking candidates effectively for selection and credit assignment, rather than exactly reconstructing oracle reward values.

  • Clarifying Design Choices, Scope, and Presentation: In response to reviewer sZjP, we distinguished the central contribution of inferring useful task-specific rewards from limited interaction from the practical implementation choices such as branching and distillation. We also scope the self-improvement framing to the demonstrated candidate selection and test-time RL optimization, where our limitations already acknowledge that inferred rewards may become unreliable as policies move beyond the support distribution. We also revised the overview and branched-rollout figures in response to reviewer Tcfc.

  • Clarifying When a Proxy Reward Is Useful: We emphasized that Meta-Rubrics targets settings where oracle access is expensive or time-consuming; unrestricted access to a reliable oracle would favor using it directly. As an exploratory extension, a single-task KernelBench experiment examines inferred rubrics for curated subtasks. Under the same budget of 30 target-environment evaluations, correctness-gated speedup increases from 2.03× without curated tasks to 2.62× with dynamic interleaving and 2.67× with fixed interleaving. We treat this as preliminary evidence for future work.

We respectfully ask the AC to consider these clarifications and additional results in the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.