acceptodds
Under review as a conference paper at ICLR 2027

Steered Not Stirred: Effective Reward Tilting using Surrogate Reward Mixing

Abstract

Continuous-time generative models, from diffusion models, flow matching, and now flow maps, owe their practical value to high-fidelity generation and steerability. In many applications, a ground-truth latent “oracle” reward is unavailable or expensive to evaluate, e.g., human judgments or experimental measurements. Although the oracle reward is not directly observable, it can often be approximated through multiple cheaper surrogates that, individually, provide incomplete characterizations but collectively reveal substantial information about the downstream task. In this paper, we present a method for composing such surrogates to steer a pretrained model toward the distribution tilted by an unavailable . Given a small feedback set of oracle rankings or numerical rewards, we introduce Mixing And Reward Tilting for INference-time Improvement (MARTINI), which learns a surrogate composition for plug-and-play use with standard inference-time steering methods. Our theory characterizes surrogate information limits and relates population recovery error to ideal-target KL at small tilt strength. For preferences, the bound additionally accounts for an irreducible mismatch between the Plackett–Luce population target and the conditional mean. Experiments demonstrate improved oracle-tilt fidelity in language generation and ImageNet alignment, and improved joint pose accuracy and validity in protein–ligand cofolding, without inference-time oracle queries.

Primary area:
generative models
Keywords:
Flow models, Flow Map, Steering, Reward-alignment

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_q4PrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose MARTINI, which composes cheaper surrogate rewards using limited offline value or preference feedback to steer a pre-trained model toward an oracle reward-tilted distribution. The oracle is unavailable at inference time. The paper provides theoretical analysis and experiments on reward recovery and steering fidelity.

Critical strengths
  1. The theoretical investigation looks interesting to me, particularly the relationships between surrogate information, learned compositions, and oracle-tilt approximation.
  2. The authors consider both value and preference feedback, and the compositions can be used with existing steering methods without retraining the generator.
Critical weaknesses
  1. My main concern is the motivation for using multiple surrogates. Specifically, there are two questions based on your problem setup
    1. (a) why do we need multiple surrogates instead of one surrogate by estimating the oracle reward from the feedback dataset directly? The motivation, benefits, and necessity of multiple surrogates g(R)g(\mathbf{R}) and R=Rk\mathbf{R} = R_k with k>1k>1 are unclear to me. Please compare this with a reward predictor trained directly on the same feedback dataset, and clarify how you train gg.
    2. (b) It is unclear to me how to train 𝑔 using Eqs. (3) & (4) from your collected feedback dataset. Do you combine the two loss functions directly?
  2. Following 1., this work aims to address calibration and multi-objective alignment. Figure 2 and Appendix K show that surrogate Pareto optimality need not agree with the oracle objective, but do not establish an advantage over single-surrogate calibration. The authors could compare against the best calibrated single-surrogate predictor gj(Rj(x))g_j(R_j(x)), including nonlinear calibration, under comparable fitting and tuning procedures.
  3. Following 2., it is unclear what complementary information the surrogates provide. Which information is missing from one surrogate but recoverable from another? Surrogate-subset ablations would help show whether adding surrogates improves oracle prediction and steering beyond calibration alone.
  4. The choice of oracle and surrogates in the language experiment is not sufficiently justified to me. For example:
    1. Why is negative generative perplexity under Qwen2.5-1.5B chosen as the oracle, and which aspects of this reward do BERT, DistilBERT, CoLA, and unigram entropy capture? This implementation advantage does not establish which aspects of the oracle these scores capture or why their combination is preferable. In addition, if you want to approximate Qwen2.5-1.5B, why not use the older (Qwen1.5) and smaller (0.5B) Qwen model or other language models?
    2. The other tasks raise similar concerns about defining the oracle reward model and surrogates. Please justify the oracle and surrogate choices and compare against a calibrated smaller language model.
Questions for authors

Questions For Authors:

  1. What evidence distinguishes the benefit of multiple surrogates from the benefit of learning a calibration from oracle feedback?
  2. For presentation in Sec 2.1, why do you mention the Flow maps. after Steering.? The current paragraph structure makes the text difficult to follow.
4 comments
AuthorsOpenReview ↗

Response to q4Pr

We thank the reviewer for their time and detailed review. We appreciate the positive comments regarding the theory and general applicability of our method. Please find below detailed answers to the concerns raised.

Direct reward prediction

We thank the reviewer for the insightful comment and note that MARTINI is motivated by the scarce feedback setting, but pretrained reward models already provide some useful task-specific representations. We thus learn a low-dimensional composition of these signals rather than learning an oracle reward directly from high-dimensional inputs.

As suggested, we added a comparison between surrogate composition and direct reward prediction from frozen DINOv2 features at matched feedback budgets (Appendix D.3). Under both the Skywork and Qwen3.8 image oracles, surrogate composition has the largest advantages at small feedback budgets (see n=32 table below). This supports the data-efficiency benefit of pretrained surrogate information.

MethodSkywork PCC ↑Qwen3.8 PCC ↑
Direct DINOv20.210.34
g(R)0.640.49

Nonlinear single-surrogate fitting

We thank the reviewer for this suggestion. We added a nonlinearly calibrated single-surrogate method, trained using the same oracle feedback and fitting objective as MARTINI (see table below). We have reported results under both language and the stronger Qwen3.8 ImageNet oracle experiments. Under Qwen3.8 (updated Table 1), MARTINI reduces energy distance from 0.0410 to 0.037 relative to this calibrated single while improving ideal-tilt KL from 0.322 to 0.314. In language generation, MARTINI also improves ideal-tilt KL relative to the calibrated single (Figure 5) and energy distance (Figure 3), demonstrating benefits beyond simply learning a nonlinear fit of a single surrogate.

MethodLang. KL (val.)↓Lang. KL (pref.)↓Qwen Energy↓Qwen KL↓
Calibrated single0.5230.5920.04100.322
MARTINI0.4380.4530.0370.314

Surrogate complementarity

We have now introduced exhaustive subset and leave-one-out recovery ablations across language and ImageNet (Appendix D.4). Mean held-out PCC increases from 0.226 using individual surrogates to 0.431 using all four in language and from 0.527 to 0.705 under Qwen3.8 (see below table). Removing unigram entropy or CoLA substantially reduces language recovery, while removing ImageReward under Qwen3.8 increases the MSE from 0.526 to 0.703 (Table 5).

SurrogatesLanguage PCC ↑Qwen3.8 PCC ↑
Single (mean)0.2260.527
Two (mean)0.3040.608
Three (mean)0.3760.667
All four0.4310.705

We additionally examine complementarity in protein–ligand cofolding (Appendix F.1), where we show SMINA and Boltz-2 Affinity individually correlate with experimental binding affinity, but exhibit moderate mutual correlation. These findings provide evidence of the complementary information in the surrogate models.

Training objectives

We appreciate the reviewer’s comments regarding the clarity of our methods. Equations (3) and (4) describe two separate feedback regimes, not a combined training objective, and are optimized independently. With numerical oracle values, we fit g by minimizing MSE between the composed surrogate score and observed oracle reward. With preference feedback, we instead fit g using the PL negative log-likelihood of the observed candidate rankings. We now clarify this distinction in §3.2.

Questions

Q1. Oracle and surrogate selection

We have expanded our experimental sections with additional motivation regarding model choices. Generative perplexity provides an established evaluation signal for non-autoregressive language generation. The surrogates capture complementary aspects of this objective: BERT and DistilBERT provide contextual token-reconstruction scores, CoLA measures grammatical acceptability, and unigram entropy captures token diversity and repetition. Their individual contributions are now examined empirically in Appendix D.4. We have similarly expanded the ImageNet and protein–ligand experimental descriptions to clarify the exposition surrounding surrogate model choices.

The pretrained FMLM generator uses the BERT vocabulary, allowing differentiable evaluation of these surrogates on continuous token representations. Smaller Qwen models are incompatible with this setting, as they use a different tokenizer, so their scores cannot be directly differentiated through the FMLM representation.

Q2. Presentation

We thank the reviewer for identifying the unclear ordering in Section 2.1. The revised manuscript now introduces flow maps before steering to help clarify the presentation.

Closing remarks

We appreciate the reviewer’s time and valuable feedback helping strengthen the manuscript. We hope that our rebuttal addresses the points raised by the reviewer. We are happy to engage further and would appreciate reconsideration of the paper in light of these revisions.

AuthorsOpenReview ↗

A Brief Follow-up Ahead of the Rebuttal Deadline - Reviewer q4Pr

Dear Reviewer q4Pr,

We sincerely appreciate your detailed feedback, which motivated several important additions to our revised submission.

With the rebuttal deadline fast approaching, we wanted to ask whether our responses have sufficiently addressed your central concerns regarding the necessity of multiple surrogates and their benefits beyond single-surrogate calibration, and presentation.

In particular, we have introduced direct reward prediction baselines, nonlinearly calibrated single-surrogate comparisons under matched feedback budgets, and extensive subset and leave-one-out ablations demonstrating complementary surrogate information. We have also clarified the separate training objectives, expanded the justification for our oracle and surrogate choices, and updated the manuscript text to improve presentation and clarity.

We hope these new experiments and edits have helped address your questions and would welcome any further discussion before the deadline. We would also be very grateful if you would consider an updated assessment of our work, should you feel these additions merit it.

Thank you again for your time and constructive suggestions!

Response to the Rebuttal (Submission 49102)

Thank the authors for their careful response.

MARTINI is motivated by the scarce feedback setting, but pretrained reward models already provide some useful task-specific representations. We thus learn a low-dimensional composition of these signals rather than learning an oracle reward directly from high-dimensional inputs.

My main concern about the motivation for composing surrogates has been resolved by the updated main text and the response. The training procedure is now clear, and the paper reads better than the previous version.

I have raised my rating to 3 accordingly.

One remaining question. The calibrated single-surrogate baseline fj(Rj)f_j(R_j) is reported for the language and Qwen3.8 experiments, but not for the Skywork oracle, which is the main image setting in Table 1. Since the gap under Qwen3.8 is small (KL_g 0.322 vs. 0.314), could the authors also report fj(Rj)f_j(R_j) under Skywork, so that the benefit of composing several surrogates over calibrating a single one can be assessed there as well?

A minor suggestion for the final version (no response needed). Tables 4–5 report means over subsets of each size. Adding the row for the best single surrogate would make the comparison with the full composition directly visible.

AuthorsOpenReview ↗

Author response - q4Pr

Dear Reviewer q4Pr,

Thank you very much for your prompt reply and for engaging so constructively with our rebuttal. We are very pleased to hear that we were able to address your main concern, and that you have increased your rating.

We appreciate your additional suggestion to include the calibrated single-surrogate baseline for the original Skywork, and we have updated Table 1 accordingly.

Reviewer_V3zeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MARTINI, a framework for inference-time steering of pretrained generative models using multiple surrogate rewards. The method learns a reward composition from limited oracle scores or preference rankings and uses it with existing steering algorithms. The authors provide theoretical connections between reward estimation and reward-tilted distributions. Experiments on language generation, image generation, and protein–ligand cofolding show improvements in several distribution-matching and task-specific metrics.

Critical strengths

The paper addresses a practical problem in generative alignment, where reliable oracle rewards are expensive or unavailable during inference. Combining multiple cheaper surrogate rewards is well motivated.

The theoretical analysis is relatively comprehensive. It connects reward prediction error to the KL divergence between reward-tilted distributions and characterizes the information limits of surrogate rewards.

The experiments cover three different domains and several steering methods. The evaluation considers both reward quality and distribution matching, providing a broader assessment than reward improvement alone.

Critical weaknesses
  1. The contribution to generative modeling is limited. MARTINI mainly learns a combined reward through standard regression or preference fitting. The generation process still relies on existing steering methods. Similarly, the theory focuses on reward estimation and ideal reward tilting, without directly analyzing errors from the actual sampling process. The connection between the proposed method and generative modeling could be better justified.

  2. The benefits of combining complementary rewards are not sufficiently demonstrated. The main motivation is that different surrogate rewards provide complementary information about the oracle. However, the real-world experiments mostly favor linear or correlation-based combinations. The need for nonlinear composition is mainly demonstrated in synthetic experiments. It remains unclear how much additional information each surrogate contributes and whether the improvements come primarily from simple reward calibration.

  3. The evaluation relies partly on oracle information. In the ImageNet experiments, steering strengths are selected using the distance to an oracle-tilted reference distribution. This information would not be available in the intended inference setting. It is unclear whether the reported improvements can be maintained with an oracle-free selection strategy.

Questions for authors
  1. Beyond learning a more accurate reward function, what is the specific contribution of MARTINI to generative steering? Could the proposed reward composition be used equally well with other reward-based optimization methods?

  2. Can the authors provide stronger evidence that the surrogate rewards contain complementary information? In particular, how does performance change when individual surrogates are added or removed?

  3. How would the steering strength be selected without access to the oracle-tilted reference distribution? Would MARTINI still outperform the baselines under such a setting?

5 comments
AuthorsOpenReview ↗

Response to V3ze (1)

We greatly appreciate the reviewer's time, effort, and dedication to the review process. We are grateful that the reviewer agrees that MARTINI aims to solve a “practical problem” of inference using unavailable oracle rewards, motivating the need to use cheaper surrogates. We also appreciate that the reviewer found our theoretical analysis to be “comprehensive” and that our experimental evaluation provides “a broader assessment than reward alone”. We now answer the main questions grouped by theme and also note that the global response summarizes the main changes to the paper.

Contribution

We appreciate the reviewer's concern regarding the contribution of MARTINI. We have updated the manuscript to clarify this contribution and to further strengthen the motivation for our problem setting.

Our contributions to generative modelling are threefold:

  1. The principal contribution to the generative modeling literature is the introduction of a new setting (Contribution 1) where we wish to sample from a tilted measure q∗q^* defined by an oracle reward that is inaccessible at inference. We note that this setting is highly practical, as in many real-world tasks, e.g., wet lab experiments in protein design, the real reward is too expensive or time-consuming to evaluate.
  2. To the best of our knowledge, MARTINI is the first framework to learn a surrogate-reward composition from listwise Plackett–Luce preferences for inference-time steering.
  3. We show that learning a composition function via a simple MSE regression objective allows us to target the tilted measure, and this differs from multi-objective/Pareto steering, since our surrogates provide information about one oracle rather than independent optimization goals (see Appendix K)

In addition, we provide theoretical contributions on how the KL objective, up to leading order, can be approximated with an MSE regression. In this rebuttal, we have further strengthened this theory to include bounds in the 0<β≤10<\beta\leq 1 Appendix G.3, and experiments that highlight that the theory matches practice in Appendix E.

The generation process still relies on existing steering methods.

We agree and argue this is the strength of our method. It plugs into existing steering methods with minimal modification. This allows us to test MARTINI across three very different domains in images, language, and proteins. The goal of this paper is not to derive a new steering method, but rather a new setting that is representative of high-impact applications and how existing methods can be used in this setting cheaply.

Similarly, the theory focuses on reward estimation and ideal reward tilting, without directly analyzing errors from the actual sampling process.

Inference-time steering has two separate sources of error: the target can be wrong, and the sampler can fail to reach its target. Many existing steering analyses take the reward as given and focus on the resulting sampling procedure. Our setting is challenging precisely because the oracle is unavailable at inference, so the first is an additional source of error: even an exact sampler for qgq_g returns the wrong distribution when g∘Rg\circ\mathbf R is a poor approximation for the oracle. Our theory quantifies this target-mismatch term, including the part that no composition of the surrogates can remove (Theorems 3.1 and 3.2; Proposition G.1).

Our theory characterizes the exact KL floor, a conditional finite-β\beta upper bound, and the surrogate-information ceiling. In our experiments, we estimate composition KL by importance reweighting unsteered samples. Specifically, Figure 5 reports ideal composition fidelity independently of the sampler, while Figure 3 evaluates realized steering against the oracle-tilted reference. We believe this highlights the practical impact of the final samples and encompasses errors during the process—which, by definition, have to be accounted for in the final samples. We hope this clarifies our contributions as well as how the theory informs practice through our evaluation metrics.

AuthorsOpenReview ↗

Response to V3ze (2)

Benefits of combining complementary rewards

We thank the reviewer for their suggestion, and agree that the original manuscript lacks this analysis. We have included exhaustive subset and leave-one-out recovery experiments across both language and ImageNet in App. D.4.

SurrogatesLanguage PCC ↑Qwen3.8 PCC ↑
Single (mean)0.2260.527
Two (mean)0.3040.608
Three (mean)0.3760.667
All four0.4310.705

We observe consistent improvements in average recovery as more surrogates are included across both experimental settings. Leave-one-out ablations further identify informative contributions from unigram entropy and CoLA in language, and ImageReward in images. We observe similar evidence of complementary information in protein–ligand cofolding (Appendix F.1): SMINA and Boltz-2 Affinity have oracle correlations of 0.49 and 0.40 respectively, but a mutual correlation of only 0.31.

These findings support Corollary G.5, which establishes that incorporating additional surrogate information can reduce the irreducible recovery and KL floors.

Need for nonlinear composition

We agree that nonlinear composition is not universally necessary. Indeed, the preferred composition depends on the relationship between the surrogates and the oracle. Held-out selection favours an MLP in language generation under value feedback, and we demonstrate effective steering over baselines under this model.

To distinguish composition from simple calibration, we additionally compare against a nonlinearly fitted single-surrogate baseline, using the same oracle-feedback budget. Under Qwen3.8, MARTINI reduces the embedding energy of this baseline from 0.0410 to approximately 0.037, while improving ideal KL-tilt from 0.322 to 0.314. These results provide evidence that combining complementary signals can improve upon calibrating an individual surrogate.

Reliance on oracle information

We thank the reviewer for highlighting the distinction between oracle-free inference and oracle-dependent selection. We clarify that MARTINI does not evaluate the oracle during the generation process, and samples are steered exclusively with the fitted surrogate composition. However, evaluating fidelity to the oracle-tilted distribution requires an oracle-scored reference. Consequently, we apply the same guidance-strength sweep and selection criterion to all comparable steering methods and baselines, providing a shared retrospective selection criterion across comparable methods, while still relying on the oracle reference.

To address this, we have added oracle-free guidance-strength selection experiments to Appendix I.3. Rather than selection against the oracle-tilted reference, we minimize energy distance to the distribution induced by each fitted steering reward, requiring no additional feedback. Across independent fits, MARTINI achieves the lowest oracle-target energy distance, Wasserstein distance, and KDE KL among oracle-free-selected baselines in both nonlinear and GMM experiments (Tables 17–18). Notably, MARTINI reduces the energy distance of the unsteered prior from 0.1587 to 0.0511 in the GMM setting, a reduction of 67.8%. These results establish the feasibility of effective oracle-free selection in the studied settings. Extending its reliability to higher-dimensional generation remains an important direction for future work.

MethodNonlinear Energy ↓GMM Energy ↓
Prior0.22180.1587
MARTINI0.04150.0511

Q1

We acknowledge this question. Please note that we have answered this above under the heading “Contribution”.

Could the proposed reward composition be used equally well with other reward-based optimization methods?

The generality of MARTINI is that it acts as a plug-and-play method complementary to reward-based optimization frameworks that accept a composed scalar reward as their steering or optimization objective.

Q2

This question is addressed above under “Benefits of combining complementary rewards”, where we include new analysis on image and language surrogate models. Our results motivate the use of multiple surrogates when learning a composition g, and provide evidence of their complementarity.

Q3

We thank the reviewer for this question. We have addressed this above under “Reliance on oracle information”. We clarify that the oracle is not queried during generation, but the main image results use oracle-dependent retrospective strength selection. We have also added experiments across two controlled settings demonstrating that guidance strength can be selected without oracle access, with MARTINI outperforming the baselines on key distributional metrics.

Closing remarks

We value the reviewer's time and energy in providing feedback. We believe we have answered the main points raised by the reviewer. We would be happy to engage on any remaining clarification and would be grateful if the reviewer considered an updated evaluation in light of our revisions.

AuthorsOpenReview ↗

A Gentle Follow-up on Any Remaining Questions - Reviewer V3ze

Dear Reviewer V3ze,

Thank you again for your thoughtful review and suggestions. As the rebuttal draws to a close, we wanted to follow up on whether our responses have addressed your main concerns, particularly regarding surrogate complementarity and oracle-free guidance-strength selection.

In direct response to your suggestions, we introduced extensive surrogate-subset and leave-one-out ablations demonstrating complementary information across surrogates, comparisons against nonlinearly calibrated single-surrogate baselines, and new experiments demonstrating effective oracle-free selection in controlled settings. We have also clarified the contribution of our theoretical framework to generative steering.

We hope these additions have helped resolve your concerns and would be happy to discuss any outstanding questions. If you feel our responses and new evidence warrant it, we would sincerely appreciate reconsideration of your initial evaluation.

Thank you again for your valuable feedback!

Reviewer_JWciOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new method (MARTINI) for steering diffusion based models during generation towards a reward function. Because ground truth reward functions are often intractable or expensive to compute, the method uses an ensemble of cheaper and inaccurate surrogate models for steering. The main contribution of this paper is to learn a composition function for combining the surrogate models based on a small set of samples from the expensive oracle rewards. Besides the training set of oracle samples, the composition function is trained without access to the reward oracle. The method is evaluated on language and image generation, as well as on protein-ligand generation.

Critical strengths

Learning the combination of surrogate models instead of manually selecting them is a novel idea and it improves the results compared to uniform weighting, while only needing a small set trainable parameters and excessive numbers of expensive oracle samples. There exist a plethora of pre-trained models and the paper demonstrates with experiments in 3 different fields that the method allows to steer them cheaply towards the application of interest.

Critical weaknesses

(1) Steering pre-trained models with an ensemble of surrogates is not new and the method is a minor contribution that only adds a learned weighting function to find a better composition of the surrogates. (2) The experiments on the different domains use pre-trained models that are behind the current state-of-the-art in their respective fields. Although the results show improvement for steering, it remains unclear how useful this is in real applications where better models are already available. Furtherore, it is unclear if the cheap surrogate models could also improve the results for stronger oracles like e.g. Qwen3.8 instead of Qwen2.5. (3) The experimental results only show a clear improvement on text-to-image results, but results for text and protein-ligand generation seem weak. In text generation, the energy distance to the target distribution does not improve over the unsteered priors and even gets worse with an increasing steering strength. The main improvement is only visible in the distribution of oracle scores KL-KDE, but only with larger steering strength, where energy distance already gets significantly worse. In protein-ligand generation, the method improves the RMSD metric slightly by about 3 %, but the PoseBusters validity drops by 9 %. (4) Some of the evaluations are hard to follow and the authors provide a large number of metrics, but an interpretation is missing. For example in text generation, the best single surrogate (BSS) is worse than the prior on all metrics. Furthermore, the energy distance needs another encoder model and it is unclear how the effect of the selected encoder influences the results.

Questions for authors

(1) To access the surrogate scores, the model needs to forward and backward pass through all surrogate models in every inference step. How much is this computational overhead? (2) How would a trained/fine-tuned single surrogate model compare to it (training cost, inference cost, sample quality)? (3) MARTINI is evaluated on cheaper models. How does it compare to modern SOTA models in the respective fields (e.g. QWEN3.8)? (4) Why does the validity decrease for steered ligand samples? I fear that MARTINI finds a better orientation, but sacrifices the quality of intra-molecular positions.

5 comments
AuthorsOpenReview ↗

Response to JWci (1)

We value the reviewer’s time and constructive feedback on our paper. We are grateful to the reviewers acknowledgement of the novelty and importance of this contribution. We also appreciate that the reviewer found our experimental setup to be diverse in three different fields and that MARTINI allows cheaper steering using surrogates towards harder applications of interest. We next address the main outstanding questions raised in the review.

Contribution

We acknowledge the reviewer’s healthy skepticism regarding the breadth of our contribution. We would like to very politely disagree with the reviewer and apologize for the initial lack of clarity in the presentation. Indeed, the setting of our work is both highly novel and representative of practical downstream applications where the true oracle reward is too expensive to access during inference. Furthermore, we consider both value and k-shot preference feedback. To the best of our knowledge, ours is the first framework to learn a composition of pretrained surrogate rewards from listwise Plackett–Luce preferences for inference-time steering.

Unlike fixed reward ensembles or multi-objective steering, MARTINI learns a composition to approximate one oracle-defined tilt q∗q^*. Appendix K illustrates why surrogate Pareto efficiency need not agree with the oracle objective. Comparisons with uniform weighting and ReNO demonstrate improvements in key distributional metrics.

Our analysis connects surrogate fitting to the target distribution: Proposition G.1 establishes an exact finite-β\beta KL decomposition and information floor, while Theorem 3.1 identifies MSE as the leading small-β\beta steering-divergence term. We estimate composition KL empirically, rather than claiming to measure the theoretical information floor. We additionally provide a conditional bound for 0<β≤10<\beta\leq 1 (Lemma G.4) and numerical evidence for the leading-order approximation with Qwen3.8 (Appendix E.4).

We hope that this satisfactorily addresses the reviewer's initial point that our setting and corresponding theory are both novel and depart from existing literature, as the setting is motivated by actual tasks like protein design where there is truly an access gap to an oracle reward.

Qwen3.8-27B oracle

We thank the reviewer for this great suggestion. We have now extended the ImageNet experiments to use Qwen3.8-27B as an oracle in the value regime, while retaining the same cheaper surrogates (see updated Table 1). We additionally include an extra baseline, steering with the best nonlinear fitting of a single surrogate. MARTINI obtains the lowest mean embedding-space energy (0.037), sliced Wasserstein (0.006), and both Gaussian and KDE reward-marginal KL losses (0.227 and 0.252). These results demonstrate that compositions of inexpensive surrogates remain useful when approximating a substantially stronger oracle.

For the generator choice, we use a released ImageNet flow map, the FMLM language model, and DeCAF for protein–ligand cofolding. DeCAF is a recent cofolding flow-map model, and our other generators were selected in part for their available flow-map implementations. MARTINI is designed to transfer to compatible pretrained generators without retraining, although performance gains on newer, untested generators remain to be established.

Protein and language results

Proteins

We understand the reviewer’s concern about our experiments. We first highlight that in this rebuttal we include updated protein results (Table 2) which significantly strengthen the impact of using MARTINI. Effectively, the two main changes are (1) we revised the experimental design to match Boltz-1x/DeCAF practice to guide and steer with unnormalized rewards; (2) we also revised evaluation to select poses based on each method's reward used at steering, rather than ground-truth RMSD, which is unavailable at inference (Appendix F.4 contains further details). Reward-based pose selection is more representative of deployment because ground-truth RMSD is not available at inference. In short, our new protein results now demonstrate a convincing 13.88%13.88\% relative improvement in joint success over the best single surrogate and 5.39%5.39\% over DeCAF (FKS potentials). Importantly, PoseBusters validity also improves over both guided baselines, reaching 94.18%, alongside improvements in RMSD accuracy and lDDT-PLI. We retain the normalized-reward experiments and the 10-versus-15-particle comparison as additional ablations (Appendix F.4, Table 14); retrospective best-pose selection with ground-truth structural criteria is reported separately in Table 13.

AuthorsOpenReview ↗

Response to JWci (2)

Language

In our language experiments, we believe there may be a potential cause for confusion. We very respectfully clarify that in the language generation experiment, MARTINI does not only improve over the prior in reward-marginal KL. Specifically, at the common guidance strength λ=0.03\lambda=0.03, MARTINI achieves an energy distance of 4.66×10−44.66\times 10^{-4} compared with 5.84×10−45.84\times10^{-4}, a 20%20\% reduction, while under the preference regime, MARTINI achieves a 14%14\% reduction. These improvements occur alongside reduced reward-marginal KL (as noted by the reviewer), and improved generative perplexity in both value and preference regimes (from 94.5 to 90.2 and 94.5 to 91.5).

The reviewer correctly observes that stronger guidance can worsen energy distance even while reducing reward-marginal KL. This is consistent with oversteering: intuitively, matching a reward distribution does not imply matching the full generated distribution. We therefore assess both, alongside perplexity and diversity, and do not interpret improvements in reward KL alone as evidence of better distributional fidelity, as demonstrated by our qualitative samples in Appendix J.1.

We have updated the manuscript to provide an extended discussion surrounding metrics, evaluations, and interpretations of results as suggested by the reviewer, to better explain surrogate choices, metrics motivations, and quantitative results.

Evaluation metrics

We agree that our embedding-space energy distance depends on the evaluation representation. Evaluating generative models is inherently multifaceted, and no single metric fully captures distributional fidelity, reward alignment, and sample quality. We therefore use established pretrained representations - DINOv2 for images, and MPNet for languages. We do not solely rely on these metrics, and thus also evaluate Gaussian and KDE approximations to KL on oracle reward marginals, absolute oracle-reward gap, and domain-specific quality and diversity measures, including perplexity and Self-BLEU for language, which all capture different aspects of generation. We interpret these results jointly rather than relying on a single metric, while acknowledging robustness to alternative encoders has not been established.

Questions

Q1. Inference-time cost

MARTINI differentiates the composed scalar reward using one reverse-mode call per guidance update. Gradients propagate through all active surrogate networks and the look-ahead generator, but never through the unavailable oracle; no separate generator backward pass is needed per surrogate. Appendix B.3 provides forward/backward accounting and measured inference times.

Q2. Fine-tuning a single surrogate

This is a great question. At matched feedback budgets, our nonlinear single baseline is fitted and evaluated using the same objective and selection protocol. Specifically, on an NVIDIA H100, generating 64 ImageNet images takes 67.4s with MARTINI, compared with 68.1s for uniform weighting, 67.5s for the calibrated single, and 74.2s for ReNO (Appendix B.3, Table 3). Finally, we note that the goal of this paper is steering rather than finetuning, and none of our experimental protocol performs fine-tuning and as such this computational cost is not considered.

Q3. Using SOTA oracle rewards

We thank the reviewer for this question. We have addressed it above under the heading “Qwen3.8-27B”, where we included a new experiment on images with Qwen3.8 as the oracle reward. Our results in the value regime suggest that MARTINI is very performant, and our conclusions still hold over baselines. Under Qwen3.8 (value feedback, updated Table 1):

MethodEnergy ↓KL (Gaussian) ↓KL (KDE) ↓
Prior0.0720.7240.611
Calibrated single0.0410.2340.256
MARTINI0.0370.2270.252

Q4

Appendix F.4 provides additional protein steering and selection analyses. Our revised main evaluation instead selects poses using available in-silico rewards, while measuring success through both RMSD accuracy and PoseBusters validity.

We find PB validity to be sensitive to the reward scale. When steering with unnormalized rewards, which matches the Boltz/DeCAF steering practice, MARTINI improves accuracy, PoseBusters validity, and joint success relative to the guided DeCAF baselines under both retrospective ground-truth-structure-based selection (Appendix F.4, Table 13) and reward-based selection (Table 2).

Closing remarks

We thank the reviewer again for their feedback that enabled us to increase the quality of our submission through new rebuttal results. We hope the reviewer finds our rebuttal convincingly addresses all the salient points of their original review. We are, of course, more than happy to engage further, but if the reviewer is satisfied we would be encouraged if the reviewer may consider upgrading their score if merited.

AuthorsOpenReview ↗

Kindly Inviting Any Further Comments - Reviewer JWci

Dear Reviewer JWci,

We greatly appreciate your constructive feedback, which has helped us substantially strengthen our experimental evaluation.

With the rebuttal period nearing its end, we wanted to ask whether our responses and new experiments have satisfactorily addressed your principal concerns regarding stronger oracle models, language and protein performance, and computational overhead, as well as better contextualising and clarifying our contributions. Motivated by your suggestions, we introduced new Qwen3.8-27B experiments demonstrating improvements over the baselines, revised protein experiments showing improvements in both joint success and PoseBusters validity, and clarified the language results and computational comparisons.

We hope these additions have strengthened your assessment of our work. We would welcome any further questions and would be very grateful if you might consider revisiting your score in light of these results, should you feel it is merited.

Thank you again for your time and engagement!

Reviewer_7qkmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors propose MARTINI, which fits a composition of cheap surrogate rewards (correlation weights, ridge, or an MLP) to a small offline set of oracle scores or Plackett–Luce rankings, then plugs that composition into existing inference-time steerers (FKS, FMRG, MFM, Itô) for diffusion and flow-map models.

Critical strengths

The idea of using cheap surrogate scores and tilting for steering the diffusion and flow-matching processes is reasonable and seems to provide some experimental improvements on a spectrum of different tasks.

Critical weaknesses
  1. Essentially the method boils down to regressing the oracle on a handful of surrogates.
  2. The theoretical guarantees apply to small beta (beta<<1), yet in experiements we have beta=1 and large guidance strength. So the theory does not really apply to the experiments.
  3. The empirical improvements are modest and oracle-dependent. Guidance strength is chosen per pair using the oracle-scored reference, which undercuts the “no oracle queries at inference” claim. Protein success rate rises only 2.7 points while PoseBusters validity drops 6.5 points.
Questions for authors

Is the method much more than regressing the oracle on a handful of surrogates? Do the theoretical guarantees actually apply to the regime used in the experiments (where beta is not really small)?

3 comments
AuthorsOpenReview ↗

Response to 7qkm (1)

We thank the reviewer for their time and effort reviewing our paper. We greatly value that the reviewer recognised the value in our new contribution and noted the positive empirical findings on "on a spectrum of different tasks”. We are grateful for the constructive feedback regarding presentation and have now updated the abstract figure, highlighted the research question and improved wording in the introduction to enhance clarity. We now turn to the main questions grouped by theme.

Regression of the oracle using surrogates

We acknowledge the reviewer’s point that MARTINI's algorithmic contribution effectively boils down to regression using surrogates on a small feedback set. However, we would like to politely disagree with the characterisation as a weakness. Indeed, the effective steering under limited information is precisely the challenge that our method seeks to overcome. To clarify this, we motivate our contributions again.

We introduce a novel problem statement (Contribution 1) which is motivated directly by how diffusion and flow map models are used in high-impact downstream applications. For example, obtaining experimental binding affinity data requires wet lab experiments, and thus cheaper reward surrogates are crucial for effective sample generation. We argue that this formulation is both representative, highly distinct from Pareto-optimal steering (see Appendix K), and has eluded the generative modelling literature, as comparatively little work learns a composition of pretrained surrogates from limited oracle feedback for inference-time steering. Our additional calibrated-single baselines help isolate the benefit of combining surrogate information.

We also note that we already consider both numerical values and k-shot ranking feedback. To the best of our knowledge, MARTINI is the first framework to learn a composition of pretrained surrogate rewards from listwise Plackett–Luce preferences for inference-time steering. The simplicity of the fitting objectives is a result of rigorous theoretical analysis of their relationship to sampling from the oracle tilt $ q ^ * $. In particular, we establish in Proposition G.1 the exact decomposition $ \mathrm{KL}(q^* \Vert q_g)=K_{\min}+\mathrm{KL}(q^*(\mathbf R)\Vert q_g(\mathbf R)),and,inTheorem3.1,relatestheleadingsmall−, and, in Theorem 3.1, relates the leading small- \beta $ KL error to MSE regression.

We have further expanded our theoretical contributions in Appendix G.3, which now gives a conditional nonasymptotic bound for 0<β≤10 < \beta \leq 1 (Lemma G.4), while Appendix E.4 tests the leading-order KL approximation under Qwen3.8, including $ \beta = 1 $.

We hope this clarifies the contribution as more than a series of simple regression objectives, as the setting, theory, and demonstration across three different domains are all novel.

Modest and oracle-dependent results

We thank the reviewer for raising this distinction. We highlight the contribution of our empirical results (evaluating MARTINI across three different domains) and have strengthened the empirical evidence with a more powerful Qwen3.8-27B image-reward oracle and an additional fitted single surrogate baseline across both language and Qwen3.8 image experiments. Under Qwen3.8, MARTINI reduces embedding-space energy by approximately 11% relative to this new baseline, whilst also improving sliced Wasserstein and reward marginal KL (see updated Table 1).

MethodEnergy ↓Sliced W₁ ↓KL (KDE) ↓
Calibrated single0.04100.00660.256
MARTINI0.03650.00640.252

We clarify that MARTINI never queries R* when generating samples, and inference-time steering uses only the learned composition of pretrained surrogates. However, assessing fidelity to the oracle-tilted distribution necessarily requires oracle-scored reference samples. In our main experiments, we therefore evaluate the same guidance-strength grid for all comparable steering methods and baselines, selecting each method’s strength using the same criterion. This ensures a consistent evaluation protocol and does not confer a selection advantage specifically on MARTINI.

Motivated by the reviewer’s observation, we have additionally introduced oracle-free guidance-strength selection experiments in Appendix I.3. Here, strength is selected by minimising energy distance to the distribution induced by the fitted surrogate reward, without accessing R*. We show MARTINI achieves the lowest oracle-target energy, Wasserstein, and KDE KL among the compared oracle-free selected methods in both nonlinear and GMM experiments. In particular, oracle-target energy decreases from 0.1587 to 0.0511 in the GMM experiment. These results demonstrate the feasibility of effective oracle-free selection, while extending its reliability to higher-dimensional generation remains future work. We have updated the manuscript’s discussion (Conclusion and Appendix I.3) to highlight this.

AuthorsOpenReview ↗

Response to 7qkm (2)

Proteins

We thank the reviewer for inquiring about the empirical gains stated in our original protein experiments which has motivated us to take a more rigorous approach to experimental design and revised the setup to more closely match Boltz-1x/DeCAF. Concisely, this means that steering is performed with unnormalized surrogate rewards.

We also revised evaluation to select poses using in-silico rewards available for steering, rather than ground-truth RMSD, which is unavailable at inference (Appendix F.4 contains further details). This matches practice more accurately in other domains and does not select using the ground-truth-based performance metric (RMSD) unavailable in real drug discovery campaigns. Specifically, generated poses are selected without consulting experimental affinity measurements or ground-truth crystal poses. Under this more realistic protocol (updated Table 2), MARTINI improves joint success by 5.39% relative to DeCAF and 13.88% relative to BSS, while also improving PoseBusters validity (updated Table 2)

MethodJoint success rate (%) ↑PB-valid (%) ↑
DeCAF (FKS)52.4893.33
BSS48.5788.78
MARTINI55.3194.18

Under best-pose selection based on ground-truth structural criteria, MARTINI also attains higher PB validity (99.90% versus 99.49%; Table 13). In both the previous and current results, MARTINI achieves a higher success rate, which is based on accurate poses that are also physically valid. The isolated PB-validity score only captures the pose validity distribution rather than overall pose success. These results address the previously observed validity trade-off while demonstrating improvements in both pose accuracy and physical validity relative to the guided baselines.

Small-beta theoretical guarantees

We value the reviewer's insightful observation. We first note that our work already describes the theoretical contribution as “a distributional justification for MSE fitting in this regime” rather than a concrete upper bound guaranteeing the performance of our method. Thus, what is interesting is how well the theory is a guide to actual practice. We have updated the text to highlight Proposition G.1, which characterizes the exact KL-optimal composition and irreducible KL floor at finite-β\beta, and have added Lemma G.4, which bounds KL by recovery-error variance for 0<β≤10<\beta\leq 1 under a covariance-stability condition. In addition, Appendix E.4 (Figure 9) now tests the expansion directly in the regime the reviewer asks about. On the Qwen3.8 ImageNet value-feedback experiment, we estimate the full KL by Monte Carlo and show that, at the tilt strength we use, the geometric mean of the exact-to-leading order KL ratio is approximately 1.06, supporting our notion that the link between MSE fitting and steering fidelity is not confined to the small-β\beta limit.

We thank the reviewer for their insightful question. We believe we have addressed this point above in the response to the first and second weaknesses, which now include new theoretical results and experiments for the beta regimes that match practice.

Closing remarks

We thank the reviewer again for their time and effort in reviewing our paper. We believe the rebuttal answers and updates to the paper, as requested by the reviewer, have strengthened and clarified our submission and have addressed the main clarification points. We would be thrilled if the reviewer would consider a fresh evaluation of our paper with this rebuttal in context. We are also happy to engage in further discussion. Please do let us know!

AuthorsOpenReview ↗

Kindly awaiting more feedback - Reviewer 7qkm

Dear Reviewer 7qkm,

We thank you again for your time and thoughtful feedback. As the rebuttal period nears the end, we wanted to ask whether our responses have sufficiently addressed your main concerns regarding the novelty of MARTINI, the relevance of our theoretical results, and the empirical improvements.

In particular, we highlight our new finite-β\beta theoretical results and empirical validation, alongside experiments with a stronger Qwen3.8 oracle, revised protein results demonstrating improvements in both joint success and validity, and additional oracle-free guidance-strength selection experiments.

We hope these additions have helped address your concerns and would be happy to provide any further clarification before the deadline. We would also greatly appreciate your consideration of an updated evaluation should you feel our responses and new results warrant it.

Thank you again for your time and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response to All Reviewers

We are grateful to all reviewers for the time and detailed feedback in evaluating our submission. We are encouraged by the recognition of MARTINI’s motivation and theoretical contributions. We appreciate Reviewer JWci’s assessment that learning surrogate combinations is a “novel idea” and Reviewer 7qkm’s observation that our approach is “reasonable”, demonstrating improvements across different tasks, as well as Reviewer V3ze for noting our theoretical analysis as “relatively comprehensive”, and Reviewer q4Pr who finds the theoretical investigation “interesting”. We recognise that several reviewers found aspects of the presentation difficult to follow and have revised the manuscript to improve the presentation and clarity of our setting, methodology, and empirical evaluation.

We value all the constructive feedback, and our revision has focused on addressing them directly. We summarize the main shared concerns and post concise answers here in this global comment and defer more detailed responses to the individual comments to each reviewer.

Stronger empirical results: (Reviewers JWci, q4Pr, V3ze)

In response to JWci's suggestion, we extended ImageNet experiments to a stronger Qwen3.8-27B oracle, retaining the same surrogates (updated Table 1). We also introduced a nonlinearly calibrated single-surrogate baseline across language and image experiments, addressing concerns from q4Pr and V3ze about whether composition offers benefits. Under Qwen3.8, MARTINI reduces energy distance by approximately 11% relative to this baseline. We also revised our protein–ligand evaluation to use reward-based instead of RMSD-based pose selection, obtaining 55.31% joint success versus 52.48% for DeCAF (FKS potentials), alongside improved PoseBusters validity (94.18% versus 93.33%, reward-selected; updated Table 2). For protein–ligand cofolding, we now fit g using unnormalized surrogates matching Boltz-1x/DeCAF practice, with a linear composition used in Table 2. Additional ablations are reported in Appendix F.4 Tables 13, 14. We further compare surrogate composition against direct oracle-reward prediction from frozen image representations at matched feedback budgets (Appendix D.3), demonstrating advantages in reward recovery when feedback is scarce.

Surrogate complementarity: (Reviewers V3ze, q4Pr)

Following suggestions from V3ze and q4Pr, we added exhaustive surrogate-subset and leave-one-out ablations across language and ImageNet (Appendix D.4). For example, Pearson correlation improves from 0.226 to 0.431 in language and 0.527 to 0.705 under Qwen3.8 when comparing average individual-surrogate recovery with the full composition. In protein–ligand cofolding, SMINA and Boltz-2 Affinity also exhibit distinct signals, correlating with the affinity oracle at 0.49 and 0.40, respectively, but only 0.31 with one another (Appendix F.1). These findings complement our theoretical characterization of the information gain of additional surrogates (Corollary G.5).

Theoretical analysis: (Reviewer 7qkm)

In response to 7qkm's concerns regarding the small-β\beta regime, we expanded the text to highlight the exact finite-β\beta characterization of the KL-optimal composition (Proposition G.1) and added a nonasymptotic KL upper bound for 0<β≤10<\beta\leq 1 under covariance stability (Lemma G.4). We additionally introduced a numerical study under Qwen3.8 (Appendix E.4, Figure 9), evaluating the accuracy of the leading-order KL approximation across tilt strengths, including the experimental setting β=1\beta=1.

Oracle-free inference: (Reviewers 7qkm, V3ze)

Following concerns from 7qkm and V3ze, we clarified the distinction between oracle-free steering and oracle-based evaluation. We additionally introduced an oracle-free guidance-strength selection mechanism and evaluated it against calibrated-single and composition baselines in two controlled settings (Appendix I.3, Tables 17–18). MARTINI achieves the lowest oracle-target energy distance, Wasserstein distance, and KDE KL among the compared oracle-free-selected methods in both settings, providing encouraging evidence for practical guidance selection without additional oracle feedback.

Computational cost and improved exposition: (Reviewer JWci)

In response to JWci, we added explicit forward/backward computational accounting and measured inference-time comparisons (Appendix B.3, Table 3). We also expanded the experimental descriptions, clarified the distinct value and Plackett–Luce preference-fitting objectives, and reorganized the background and evaluation discussions to improve clarity.

Closing Remarks

We believe these additions substantially strengthen both the empirical and theoretical foundations of MARTINI, while clarifying its scope and limitations. We thank the reviewers for their constructive suggestions and hope that the revised manuscript and individual responses address the principal concerns.

AuthorsOpenReview ↗

Summary of Revisions and Rebuttal Discussion

Dear Chairs and Reviewers,

We sincerely thank the Area Chair for overseeing the review process and for the time devoted to considering our submission. We are also grateful to the reviewers for their feedback, which has helped us substantially strengthen the paper. As the discussion period concludes, we hope this brief retrospective of our rebuttal and subsequent exchanges will be helpful in assessing the revised submission.

  • Benefits of surrogate composition: We added nonlinearly calibrated single-surrogate baselines, direct reward-prediction comparisons at matched feedback budgets, and exhaustive subset and leave-one-out ablations, providing evidence of complementary surrogate information and benefits beyond calibration.
  • Empirical performance and practical evaluation: We added ImageNet experiments with the stronger Qwen3.8-27B oracle and revised protein experiments using reward-based pose selection. The latter improve both joint success and PoseBusters validity over DeCAF, addressing the original validity trade-off. We also clarified the language results and added measured computational comparisons.
  • Theory and presentation: We clarified the oracle-tilt setting and separate value and preference feedback regimes, highlighted the exact finite-tilt characterisation, and added a conditional nonasymptotic KL bound and numerical validation at the experimental tilt.

Two reviewers subsequently engaged, and we appreciate their further feedback.

Reviewer q4Pr confirmed that their main concern about the motivation for composing surrogates was resolved, that the training procedure was clear, and that the presentation had improved, raising their rating to 3 (weak acceptance). Following their additional request, we also added the calibrated single-surrogate baseline under the original Skywork oracle to Table 1.

Reviewer V3ze acknowledged that the subset analysis and calibrated-single baseline addressed part of their concern, and requested oracle-free guidance-strength selection on the main tasks. We thank them for this suggestion, which prompted us to extend the initial controlled experiments to ImageNet under Qwen3.8 on 17 November. MARTINI retains its advantage across the reported distributional metrics with guidance strengths selected without oracle access. These results have been incorporated into the revised manuscript.

We have not yet received follow-up from the remaining two reviewers or further feedback on the additions posted on 17 November. We believe that our detailed responses and revisions provide substantive answers to the concerns raised, and would be grateful for the Area Chair’s consideration of this updated evidence and the progress made during discussion alongside the original reviews.

Thank you again to the Area Chair, senior chairs, and reviewers for their time and careful consideration of our work.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.