acceptodds
Under review as a conference paper at ICLR 2027

On Predicting the Post-Training Potential of Pre-Trained LLMs

Abstract

The performance of Large Language Models (LLMs) on downstream tasks is fundamentally constrained by the capabilities acquired during pre-training. However, traditional benchmarks like MMLU often fail to reflect a base model's plasticity in complex open-ended scenarios, leading to inefficient model selection. We address this by introducing a new task of predicting post-training potential—forecasting a base model's performance before post-training. We propose RuDE (Rubric-based Discriminative Evaluation), a unified framework that bypasses the generation gap of base models by leveraging response discrimination. Guided by our systematic 4C Taxonomy, RuDE constructs controlled contrastive pairs across diverse domains by fine-grained rubric violations. Extensive experiments demonstrate a correlation greater than 90% with post-training performance. Crucially, validation via Reinforcement Learning (RL) confirms that RuDE effectively identifies high-potential smaller models that outperform larger counterparts, offering a compute-efficient mechanism for foundation model development.

Primary area:
interpretability and explainable AI
Keywords:
large language models, pre-training, predicting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sjBJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

RuDE predicts post-training performance through rubric-based response discrimination. Across 16 base/instruct model pairs and four domains, Pearson correlations range from 0.62 to 0.91. A three-model HealthBench RL experiment provides additional ranking evidence.

Critical strengths
  1. The model-selection problem is useful, and the 28,683-example construction pipeline is substantial.

  2. Cross-domain correlations, Spearman analysis, score confidence intervals, and controlled RL provide complementary evidence.

  3. Construction diagnostics include failure rates and a 100-example human audit.

Critical weaknesses
  1. Equation (2) specifies response likelihoods, while the implementation describes two-shot A/B selection. Table 11 shows an 8.6-point reversal effect for Qwen3-14B; the small signed average does not establish ranking robustness.

  2. Released instruct models use heterogeneous training pipelines. Their correlations support retrospective prediction, but not algorithm-independent trainability. Controlled RL covers only three Qwen models and lacks sufficient recipe, train/test-split, and seed details.

  3. Only one domain exceeds 0.90 Pearson correlation. HealthBench reports p < 0.001 in Figure 3/Table 15 but p = 0.005 in Figure 10. Correlation uncertainty and actual model-selection errors would be more informative than significance alone.

  4. PPL reaches 0.93 on AdvancedIF using a smaller model subset. Compare predictors on identical models. Construction ablations should compare post-training predictive accuracy, not merely discrimination difficulty.

Questions for authors
  1. What exact scoring implementation produced the results, and are rankings stable after averaging option orders?

  2. What are the RL recipe, disjoint training/test sets, and seed uncertainty? Disclose RuDE/evaluation prompt overlap; task-specific evaluation does not itself require separate prompts.

  3. Can you reconcile the statistics and compare predictors on identical model sets, including held-out families?

Reviewer_4xipOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tries to predict which base LLM will perform better after post-training, before spending the compute to post-train every candidate. Instead of asking the base model to generate a good answer, the authors check whether it can distinguish a better answer from a worse one. The discrimination score is then used to rank the model’s expected post-training potential. The motivation draws on generation evaluation consistency and the intuition that a base model already favoring preferred responses may require a smaller distributional shift during post-training.

Critical strengths
  1. Choosing a base model without post-training every candidate is a real practical challenge. Recognizing a good answer and generating one are different abilities, so testing recognition is a sensible approach. The underlying intuition builds on the generation–evaluation consistency work cited by the authors; the contribution I find useful is applying it to model selection before post-training and constructing a dataset for that purpose. A reliable evaluation of this kind could help researchers allocate training resources more effectively.
  2. The authors try to construct negative answers with specific defects rather than obviously broken alternatives. The generator–verifier loop checks both whether the selected requirements are violated and whether the remaining requirements are preserved. This seems to be one of the most reusable parts of the work. I also appreciate the human verification, actual RL experiment, rank correlations, bootstrap confidence intervals, difficulty levels, and option-order analysis. These provide a broader validation effort than reporting a single correlation, even though I have concerns about how some results are interpreted.
Critical weaknesses

The headline claim of greater-than-90% correlation applies to AdvancedIF, which reaches 0.91. The other reported correlations are 0.67 for HealthBench, 0.62 for WritingBench, and 0.80 for PRBench. These are still positive results, but the abstract makes the evidence sound more uniformly strong than it is. The cross-model comparison is also based on only 16 models, not 28,000 independent model-selection decisions. The paper already includes rank correlations and bootstrap score intervals, so those are not missing. However, I remain uncertain about the reliability of the predictive relationship, particularly when choosing between similar candidates rather than observing an overall trend across very different families and sizes. I also found the comparison with traditional benchmarks difficult to follow. The main motivating comparison concerns HealthBench, where SuperGPQA has a correlation of 0.56, compared with RuDE’s 0.67. This does not, on its own, justify the broad statement that traditional benchmarks cannot predict post-training potential. The connection between the later appendix comparisons and this central claim should be made much clearer. I would like a direct explanation of how consistently RuDE improves on the baselines and how uncertain those improvements are. Finally, the main comparison uses released instruction-tuned models that did not receive the same post-training data, methods, or resources. I therefore remain uncertain about how much of the relationship reflects base-model potential and how much reflects the post-training effort invested by each organization. The controlled RL experiment helps, but it involves only three Qwen models on HealthBench over 200 steps. I could not determine the training/evaluation split or the number of independent runs from the description. This is not an allegation of leakage; it prevents me from assessing the comparison adequately. Changing the base model also changes many properties at once, so the experiment does not establish that discrimination itself causes better learning.

I do not fully understand which scoring implementation produced the reported results. The definition compares the conditional log-likelihoods of the complete preferred and negative answers. Later, the implementation describes two-shot A/B prompting and swapped option positions. These are not automatically the same procedure, and the paper should explicitly explain whether the results come from complete-answer likelihoods, probabilities of answer labels, or generated selections. This is central to understanding and reproducing the method.

Questions for authors
  1. Scoring implementation: Which procedure produced the numbers in the result tables: complete-answer log-likelihood comparisons, A/B-label probabilities, or generated selections? How does this procedure correspond to Equation 2, and how are answer-length differences and invalid selections handled?
  2. RL validation: What are the HealthBench training/evaluation splits, exact training settings, and number of independent runs? Are evaluation questions separated from training questions and their derived variants? How stable is the reported model ordering across runs?
  3. Predictive evidence and baselines: How should the greater-than-90% headline be interpreted given the lower correlations in three domains? Please clarify the cross-domain comparison with traditional benchmarks and the uncertainty in the claimed improvements. How reliable is RuDE for selecting between similar candidates rather than broadly different models?
  4. Label validation: What caused the disagreements in the human audit, and how were domain-specific correctness questions assessed? Is there independent evidence that preferred answers satisfy the requirements and negative answers violate only the intended criteria? A small domain-expert audit would be a useful additional validation.
Reviewer_Him9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a discriminative proxy that, when applied to base models, correlates more strongly with the performance of the corresponding instruction-tuned models on HealthBench, PRBench, AdvancedIF, and WritingBench than any of C-Eval, MMLU, or SuperGPQA. The proxy dataset is constructed from the rubrics of each of the four benchmarks using a generator–verifier loop with Gemini-3-Pro and GPT-4.1. The paper also includes an RL experiment with three models, in which the ranking of the models' final performance matches the ranking of their proxy scores.

Critical strengths

Forecasting a base model’s performance before post-training in an interesting and important research problem.

Critical weaknesses

The abstract states the core problem as “forecasting a base model performance before post-training”. However, Equation 1 weakens this to a ranking problem, and no experiment actually makes or scores forecasts.

The paper aims to study the post-training potential of base models. However, the authors do not control for the instruction-tuning applied to the models. Consequently, the main results test whether RuDE correlates with the quality of the released instruction models, not whether it predicts post-training potential of base models under a fixed PostTrain(ω, T) as defined in Eq. 1. In Appendix Q the authors argue that this heterogeneity in PostTrain indicates that the correlations found must be robust; however, this only holds if post-training quality PostTrain_i is independent of base model quality_i. This is very unlikely to be the case: labs with better pre-training pipelines likely also have better post-training pipelines. In summary, the correlations demonstrated can’t be credited to properties of the base models, since each models’ post-training confounds them.

The only controlled experiments are those in Section 5.2. However, authors don’t show that looking at general benchmarks would have led to a different conclusion than Qwen/Qwen2.5-14B > Qwen/Qwen3-4B-Base > Qwen/Qwen2.5-7B. Indeed, taking the benchmark results from the Qwen3 technical report tables 6 and 7 under “general tasks” (all multiple-choice benchmarks), 4 out of the 5 reported benchmarks lead to the same ordering:

Base modelMMLUMMLU-ReduxMMLU-ProSuperGPQABBHMeanOrder after RL (§5.2)
Qwen/Qwen2.5-14B79.6676.6451.1630.6878.1863.261st
Qwen/Qwen3-4B-Base72.9972.7950.5828.4372.5959.482nd
Qwen/Qwen2.5-7B74.1671.0645.0026.3470.4057.393rd

Lastly, the authors don’t ablate RuDE against a much simpler approach: just use as positive samples the responses of a strong model (i.e., Opus or Sol) and as negative examples those of a weak model (i.e., Haiku or Luna), correcting for length etc. Also, I don't see what is the point of the taxonomy. The authors claim “Guided by our 4C Taxonomy, RuDE constructs controlled contrastive pairs.", but as far as I can see, the taxonomy doesn't enter construction at all (none of the prompt templates in Appendix F reference it), with each benchmark’s rubrics being used instead.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.