- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the limited generalization of reward models used to evaluate proposed AI system modifications: training on familiar optimization scenarios can improve in-distribution performance while degrading performance on unseen scenarios. To address this, LitRM incorporates scientific literature through continual pretraining and teacher reasoning supervision, then uses student-generated reasoning to guide evidence reading and reward prediction. Experiments across three benchmarks report improved OOD AUC while maintaining strong in-distribution performance, without literature retrieval at inference time.
Critical strengths
- The paper investigates an important generalization problem in AI optimization reward modeling. Using scientific literature to enrich reasoning supervision is a promising direction for addressing this problem
- The reported results show substantial OOD AUC gains over the evaluated baselines across three benchmarks while maintaining strong Seen performance. Additional backbone and distribution-shift experiments broaden the empirical support for these gains.
Critical weaknesses
- My main concern is the lack of evaluation in an actual AI optimization workflow. The reported AUC and within-BASE ranking results measure reward-model performance, but do not establish whether using LitRM to guide experiments improves the final system or reduces experimentation costs. GenRM (Zhang et al., 2025, Sec. 4) and CLoud (Ankner et al., 2024, Sec. 3.1 and Fig. 4) evaluate downstream performance through Best-of-N selection. Could the authors integrate LitRM into a propose–select–execute workflow and compare final system performance and total computational cost against baseline-guided and unguided search under matched budgets?
- The rationale for the staged training pipeline could be clearer. Why use separate literature pretraining, teacher warm-up, student rollout, and joint training stages? Although the progressive ablation shows gains from adding components, it does not explain why this particular training arrangement is preferable to a simpler alternative.
- The case studies in Appendix C are limited and do not systematically explain the model's selection behavior. Which modifications does LitRM evaluate better than the baselines, and where does it fail? Does its preference change appropriately with the BASE configuration? Analyzing these patterns and testing whether removing or replacing reasoning changes rankings would better support the claim of learning transferable optimization mechanisms.
- The figures need clearer labeling. For example, Figure 1 lacks a legend identifying which methods, model sizes, and benchmarks the points represent. Although Appendix B.1 provides additional details, these mappings should be clear from the figure and caption themselves.
Questions for authors
Seek weakeness section.
2 comments
Response [2/2]
W3. Selection behavior and reasoning analysis.
We have added modification-category analysis, reasoning intervention experiments, and a BASE-dependent case study to better understand LitRM's prediction and ranking behavior. The results are included in the revised manuscript (Sec. 6 and Tables 23-25).
- Modification Category Analysis. We evaluate LitRM across different types of system modifications, as shown in Table IV. LitRM consistently outperforms GenRM-Direct across all categories, with the largest improvement in model architecture modifications (+18.63 pp) and a smaller gain in optimization strategies (+9.89 pp).
Table IV. OOD AUC (%) across modification categories on RecOptBench.
| Modification Category | Samples / Positives | GenRM-Direct | Ours |
|---|---|---|---|
| Model Architecture | 2,635 / 368 | 50.82 | 69.45 |
| Training Engineering | 2,097 / 295 | 52.65 | 66.65 |
| Loss Design | 1,255 / 325 | 48.28 | 64.58 |
| Optimization Strategy | 935 / 63 | 51.41 | 61.30 |
| Data Processing | 645 / 153 | 50.55 | 63.85 |
- Reasoning Intervention. To examine whether reasoning influences LitRM's predictions and candidate preferences, we compare original student reasoning with empty and length-matched shuffled reasoning. As shown in Table V, removing or shuffling reasoning reduces OOD AUC by 9.77–16.04 pp and reverses 25.31%–29.89% of candidate-pair rankings. These results indicate that LitRM's predictions and rankings depend on the reasoning content rather than merely its presence.
Table V. Effect of reasoning interventions on RecOptBench-OOD. Ranking Flip Rate measures the proportion of candidate pairs whose relative ordering changes compared with the original reasoning.
| Reasoning Condition | OOD AUC (%) ↑ | Δ AUC (pp) | Ranking Flip Rate |
|---|---|---|---|
| Original Student CoT | 70.56 | — | — |
| Empty CoT | 60.79 | −9.77 | 25.31% |
| Length-matched Shuffled CoT | 54.52 | −16.04 | 29.89% |
| GenRM-Direct (Baseline) | 51.40 | — | — |
- BASE-dependent Case Study. We further include a case study in Appendix C examining similar training-duration modifications under different BASE configurations. For a Keras 1D CNN, increasing training from 3 to 5 epochs yields a 10.08% improvement, with LitRM assigning a score of 0.717. In contrast, increasing training from 5 to 10 epochs for an EfficientNet-based classifier results in a 2.11% degradation, with a lower score of 0.382. The corresponding reasoning reflects different considerations about underfitting and potential training limitations, illustrating LitRM's context-sensitive evaluation of candidate modifications.
W4. Figure clarity and labelling.
Thank you for this suggestion. We have revised Figure 1 by adding clearer labels and a legend to improve readability.
Response [1/2]
Thank you for your careful review and constructive suggestions. We have conducted additional experiments and analyses to address your concerns and further clarify our methodology and findings. Below, we provide point-by-point responses to your comments.
We have added two execution-grounded evaluations using RecOptBench-OOD for candidate selection and a separate set of 1,020 real-world online execution records for score-based filtering to assess LitRM's practical utility in AI optimization, with results included in the revised manuscript (Sec. 5.1 and Appendix B.11).
Table I. Downstream candidate selection performance on RecOptBench-OOD. Success@1: success rate (≥10% relative gain); Realized Gain: actual performance change; Spearman: within-BASE rank correlation; Executions: attempts to first success.
Table II. Candidate filtering based on LitRM scores and actual execution outcomes. Success Rate: proportion of retained candidates achieving ≥10% relative gain.
LitRM's training stages serve complementary purposes: Literature CPT incorporates optimization knowledge, Teacher Warm-up initializes structured reasoning, Student Rollout captures student-generated reasoning, and Joint Training optimizes reasoning generation and reward prediction. Beyond the progressive ablation in the original manuscript, we further compare LitRM with simplified training pipelines, as shown in Table III.
Direct joint training without warm-up and rollout substantially degrades performance, while replacing student-generated rationales with teacher rationales reduces OOD AUC from 66.86% to 55.54%. This is consistent with our design rationale: Teacher Warm-up first establishes structured reasoning capabilities, while Student Rollout exposes reward learning to the model's own, potentially imperfect reasoning, mitigating the training–inference mismatch associated with relying solely on teacher rationales. Joint Training then combines teacher-supervised reasoning generation with reward learning from student-generated traces.
Table III. Comparison with simplified training pipelines on TraceML (AUC, %) and training cost (GPU·h).