- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose MARTINI, which composes cheaper surrogate rewards using limited offline value or preference feedback to steer a pre-trained model toward an oracle reward-tilted distribution. The oracle is unavailable at inference time. The paper provides theoretical analysis and experiments on reward recovery and steering fidelity.
Critical strengths
- The theoretical investigation looks interesting to me, particularly the relationships between surrogate information, learned compositions, and oracle-tilt approximation.
- The authors consider both value and preference feedback, and the compositions can be used with existing steering methods without retraining the generator.
Critical weaknesses
- My main concern is the motivation for using multiple surrogates. Specifically, there are two questions based on your problem setup
- (a) why do we need multiple surrogates instead of one surrogate by estimating the oracle reward from the feedback dataset directly? The motivation, benefits, and necessity of multiple surrogates and with are unclear to me. Please compare this with a reward predictor trained directly on the same feedback dataset, and clarify how you train .
- (b) It is unclear to me how to train 𝑔 using Eqs. (3) & (4) from your collected feedback dataset. Do you combine the two loss functions directly?
- Following 1., this work aims to address calibration and multi-objective alignment. Figure 2 and Appendix K show that surrogate Pareto optimality need not agree with the oracle objective, but do not establish an advantage over single-surrogate calibration. The authors could compare against the best calibrated single-surrogate predictor , including nonlinear calibration, under comparable fitting and tuning procedures.
- Following 2., it is unclear what complementary information the surrogates provide. Which information is missing from one surrogate but recoverable from another? Surrogate-subset ablations would help show whether adding surrogates improves oracle prediction and steering beyond calibration alone.
- The choice of oracle and surrogates in the language experiment is not sufficiently justified to me. For example:
- Why is negative generative perplexity under Qwen2.5-1.5B chosen as the oracle, and which aspects of this reward do BERT, DistilBERT, CoLA, and unigram entropy capture? This implementation advantage does not establish which aspects of the oracle these scores capture or why their combination is preferable. In addition, if you want to approximate Qwen2.5-1.5B, why not use the older (Qwen1.5) and smaller (0.5B) Qwen model or other language models?
- The other tasks raise similar concerns about defining the oracle reward model and surrogates. Please justify the oracle and surrogate choices and compare against a calibrated smaller language model.
Questions for authors
Questions For Authors:
- What evidence distinguishes the benefit of multiple surrogates from the benefit of learning a calibration from oracle feedback?
- For presentation in Sec 2.1, why do you mention the Flow maps. after Steering.? The current paragraph structure makes the text difficult to follow.
4 comments
A Brief Follow-up Ahead of the Rebuttal Deadline - Reviewer q4Pr
Dear Reviewer q4Pr,
We sincerely appreciate your detailed feedback, which motivated several important additions to our revised submission.
With the rebuttal deadline fast approaching, we wanted to ask whether our responses have sufficiently addressed your central concerns regarding the necessity of multiple surrogates and their benefits beyond single-surrogate calibration, and presentation.
In particular, we have introduced direct reward prediction baselines, nonlinearly calibrated single-surrogate comparisons under matched feedback budgets, and extensive subset and leave-one-out ablations demonstrating complementary surrogate information. We have also clarified the separate training objectives, expanded the justification for our oracle and surrogate choices, and updated the manuscript text to improve presentation and clarity.
We hope these new experiments and edits have helped address your questions and would welcome any further discussion before the deadline. We would also be very grateful if you would consider an updated assessment of our work, should you feel these additions merit it.
Thank you again for your time and constructive suggestions!
Response to the Rebuttal (Submission 49102)
Thank the authors for their careful response.
MARTINI is motivated by the scarce feedback setting, but pretrained reward models already provide some useful task-specific representations. We thus learn a low-dimensional composition of these signals rather than learning an oracle reward directly from high-dimensional inputs.
My main concern about the motivation for composing surrogates has been resolved by the updated main text and the response. The training procedure is now clear, and the paper reads better than the previous version.
I have raised my rating to 3 accordingly.
One remaining question. The calibrated single-surrogate baseline is reported for the language and Qwen3.8 experiments, but not for the Skywork oracle, which is the main image setting in Table 1. Since the gap under Qwen3.8 is small (KL_g 0.322 vs. 0.314), could the authors also report under Skywork, so that the benefit of composing several surrogates over calibrating a single one can be assessed there as well?
A minor suggestion for the final version (no response needed). Tables 4–5 report means over subsets of each size. Adding the row for the best single surrogate would make the comparison with the full composition directly visible.
Author response - q4Pr
Dear Reviewer q4Pr,
Thank you very much for your prompt reply and for engaging so constructively with our rebuttal. We are very pleased to hear that we were able to address your main concern, and that you have increased your rating.
We appreciate your additional suggestion to include the calibrated single-surrogate baseline for the original Skywork, and we have updated Table 1 accordingly.
Response to q4Pr
We thank the reviewer for their time and detailed review. We appreciate the positive comments regarding the theory and general applicability of our method. Please find below detailed answers to the concerns raised.
Direct reward prediction
We thank the reviewer for the insightful comment and note that MARTINI is motivated by the scarce feedback setting, but pretrained reward models already provide some useful task-specific representations. We thus learn a low-dimensional composition of these signals rather than learning an oracle reward directly from high-dimensional inputs.
As suggested, we added a comparison between surrogate composition and direct reward prediction from frozen DINOv2 features at matched feedback budgets (Appendix D.3). Under both the Skywork and Qwen3.8 image oracles, surrogate composition has the largest advantages at small feedback budgets (see n=32 table below). This supports the data-efficiency benefit of pretrained surrogate information.
Nonlinear single-surrogate fitting
We thank the reviewer for this suggestion. We added a nonlinearly calibrated single-surrogate method, trained using the same oracle feedback and fitting objective as MARTINI (see table below). We have reported results under both language and the stronger Qwen3.8 ImageNet oracle experiments. Under Qwen3.8 (updated Table 1), MARTINI reduces energy distance from 0.0410 to 0.037 relative to this calibrated single while improving ideal-tilt KL from 0.322 to 0.314. In language generation, MARTINI also improves ideal-tilt KL relative to the calibrated single (Figure 5) and energy distance (Figure 3), demonstrating benefits beyond simply learning a nonlinear fit of a single surrogate.
Surrogate complementarity
We have now introduced exhaustive subset and leave-one-out recovery ablations across language and ImageNet (Appendix D.4). Mean held-out PCC increases from 0.226 using individual surrogates to 0.431 using all four in language and from 0.527 to 0.705 under Qwen3.8 (see below table). Removing unigram entropy or CoLA substantially reduces language recovery, while removing ImageReward under Qwen3.8 increases the MSE from 0.526 to 0.703 (Table 5).
We additionally examine complementarity in protein–ligand cofolding (Appendix F.1), where we show SMINA and Boltz-2 Affinity individually correlate with experimental binding affinity, but exhibit moderate mutual correlation. These findings provide evidence of the complementary information in the surrogate models.
Training objectives
We appreciate the reviewer’s comments regarding the clarity of our methods. Equations (3) and (4) describe two separate feedback regimes, not a combined training objective, and are optimized independently. With numerical oracle values, we fit g by minimizing MSE between the composed surrogate score and observed oracle reward. With preference feedback, we instead fit g using the PL negative log-likelihood of the observed candidate rankings. We now clarify this distinction in §3.2.
Questions
Q1. Oracle and surrogate selection
We have expanded our experimental sections with additional motivation regarding model choices. Generative perplexity provides an established evaluation signal for non-autoregressive language generation. The surrogates capture complementary aspects of this objective: BERT and DistilBERT provide contextual token-reconstruction scores, CoLA measures grammatical acceptability, and unigram entropy captures token diversity and repetition. Their individual contributions are now examined empirically in Appendix D.4. We have similarly expanded the ImageNet and protein–ligand experimental descriptions to clarify the exposition surrounding surrogate model choices.
The pretrained FMLM generator uses the BERT vocabulary, allowing differentiable evaluation of these surrogates on continuous token representations. Smaller Qwen models are incompatible with this setting, as they use a different tokenizer, so their scores cannot be directly differentiated through the FMLM representation.
Q2. Presentation
We thank the reviewer for identifying the unclear ordering in Section 2.1. The revised manuscript now introduces flow maps before steering to help clarify the presentation.
Closing remarks
We appreciate the reviewer’s time and valuable feedback helping strengthen the manuscript. We hope that our rebuttal addresses the points raised by the reviewer. We are happy to engage further and would appreciate reconsideration of the paper in light of these revisions.