acceptodds
Under review as a conference paper at ICLR 2027

Tilted Schrödinger Bridge Matching

Abstract

Schrödinger bridges provide an entropy-regularized framework and a principled solution for unpaired domain translation. In practice, a pretrained bridge may need to be adapted to human preferences or physical constraints through a reward a problem closely related to reward tilting in diffusion models but underexplored for Schrödinger bridges. We introduce Tilted Schrödinger Bridge Matching (TSBM), a post-training method for fine-tuning a learned bridge between source and target toward a reward-tilted target , while preserving source . We formulate this adaptation as alternating optimization initialized from , provide theoretical justification, and derive a practical algorithm based on Adjoint Matching. We evaluate TSBM on unpaired image-to-image translation targeting digit properties in MNIST and facial attributes in CelebA.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Unpaired domain translation, Diffusion Models, Schrodinger Bridges, Reward fine-tuning, Stochastic Optimal Control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AQcjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Tilted Schrödinger Bridge Matching (TSBM), a method for reward adaptation of pretrained Schrödinger Bridges. The goal is to adjust the target marginal from p1p_1 to p1r(x)∝p1(x)er(x)p_1^r(x)\propto p_1(x)e^{r(x)} while preserving the source marginal p0p_0. The authors use the pretrained bridge as the reference process and derive a relative formulation based on the new and original Schrödinger potentials, yielding a control objective that does not require explicitly evaluating the target marginal density or score during post-training. They further develop an algorithm that alternates between controller and corrector updates. Experiments on Colored MNIST and CelebA demonstrate the effectiveness of TSBM in reward steering while preserving source attributes.

Critical strengths
  1. A key strength of this work is its novel perspective on global marginal reward tilting for pretrained Schrödinger Bridges with non-memoryless source distributions. Built on Adjoint Matching and alternating controller–corrector updates, the formulation avoids explicitly evaluating the original target density or its score, providing a distinctive and well-motivated approach to this problem.
Critical weaknesses
  1. Insufficient training-based baselines. TSBM requires additional post-training, while the main experiments primarily compare it with training-free methods, making it difficult to assess its advantages over other post-training approaches. Although the authors note that suitable post-training methods for SBs are currently lacking, comparisons with Flow/Diffusion post-training methods on the same benchmarks are still needed to adequately evaluate the incremental benefits and practical value of the proposed method.
Questions for authors
  1. In Figure 3(c), DPS (γ=2)(\gamma=2) achieves a higher reward than TSBM. However, the qualitative results in Figure 5 suggest that TSBM produces digits with more consistently red colors. Does this discrepancy indicate that the current Red-Chroma reward may not fully capture the intended color transformation?

  2. Given the similarities between their Controller–Corrector Matching frameworks, could the authors clarify the key innovations of TSBM?

  3. For general reinforcement learning or reward optimization tasks, what advantages does TSBM offer over flow-based post-training methods?

3 comments

We thank Reviewer AQcj for the thoughtful review and constructive feedback. Below, we address each of your comments and questions in detail. We have also uploaded a revised version of the manuscript with all changes highlighted in blue.

1. Comparison with Flow/Diffusion post-training

Reviewer

Insufficient training-based baselines. TSBM requires additional post-training, while the main experiments primarily compare it with training-free methods, making it difficult to assess its advantages over other post-training approaches. Although the authors note that suitable post-training methods for SBs are currently lacking, comparisons with Flow/Diffusion post-training methods on the same benchmarks are still needed to adequately evaluate the incremental benefits and practical value of the proposed method.

For general reinforcement learning or reward optimization tasks, what advantages does TSBM offer over flow-based post-training methods?

Answer

We thank the reviewer for raising this point. Existing flow and diffusion post-training methods do not directly apply to reward-tilting pretrained Schrödinger bridges applied for unpaired trasnfer:

  1. Non-memoryless marginalization bias: General flow and diffusion post-training methods (such as Adjoint Matching [1] or DRaFT [2]) rely on memoryless noise-to-data priors where optimizing the terminal reward alone recovers the desired tilted target distribution (Section 2, Eq. 5). In contrast, when the pretrained process is non-memoryless, applying standard reward post-training without a corrector introduces a source-induced marginalization bias (Section 2, Eq. 6; Appendix B.1 of the original submission).
  2. Inapplicability to unpaired translation via Schrödinger bridges: General flow and diffusion post-training methods are designed for noise-to-data generation rather than unpaired domain transfer between two data distributions via Schrödinger bridges. Because they neither preserve an arbitrary coupled source marginal nor learn a terminal corrector to cancel the non-memoryless coupling bias, they are not directly applicable to fine-tuning Schrödinger bridges for unpaired translation.
  3. Comparison against unadapted Adjoint Matching: In the original submission we've tested against regular flow/diffusion reward fine-tuning methods and whether accounting for non-memorylessness matters in practice. In particular, we evaluate a static-corrector baseline on Colored MNIST (Red-Chroma) under the same controller training budget and evaluation protocol, see Appendix G.3 of original submission or Section 6.1 of the revised submission. Freezing the corrector removes the corrector update, reducing TSBM to regular Adjoint Matching applied to the pretrained bridge without non-memoryless adaptation. As shown in Section 6.1 of the revised submission, full TSBM achieves substantially higher reward and lower red-target hue error across all stages than this unadapted baseline.

To our knowledge, there are currently no fine-tuning methods, other than TSBM, for Schrödinger bridges that account for non-memorylessness, so we do not have other suitable post-training baselines beyond this static-corrector (regular Adjoint Matching) variant.

2. Red-Chroma reward

Reviewer

In Figure 3(c), DPS (γ = 2) achieves a higher reward than TSBM. However, the qualitative results in Figure 5 suggest that TSBM produces digits with more consistently red colors. Does this discrepancy indicate that the current Red-Chroma reward may not fully capture the intended color transformation?

Answer

Indeed, in In Figure 3(c), DPS (γ = 2) achieves a higher reward than TSBM and produces digits with less red color. As one can see in the Appendix D.2 of the original submission, the red-chroma reward favors red color per each pixel, while blue and green channels are penalized, also per pixel. This reward doesn't provide any normalization per digit size, so full image consisting only of red pixel would maximize the reward. In that light, the DPS is maximizing the reward by rather raising the digit size than making it red, which can be see as a reward hacking. In the Figure3(c) we also plot the "thickness error" and one can see that DPS (γ = 2) has significantly thicker digits than TSBM, which allows it to make up for the noted red color loss.

As a result, one can see that TSBM provides better reward maximization-faithfullness trade off, since almost all the digits are red, while their thickness is closer to the original thickness.

Although, as the reviewer suggested this also can be seen as reward capturing not only the color transformation, but also the size of a digit.

3. Key innovations in Controller–Corrector Matching

Reviewer

Given the similarities between their Controller–Corrector Matching frameworks, could the authors clarify the key innovations of TSBM?

Answer

To our knowledge the prior Controller-Corrector Matching and related energy-based Schrödinger bridge frameworks [3, 4, 5] were primarily designed for sampling from Boltzmann distributions or bridging to an explicitly known energy prior. If the reviewer had some other work in mind, we kindly as him to share it. While TSBM shares the high-level alternating Controller-Corrector Matching template, adapting it to reward post-training introduces three key innovations in formulation, algorithm, and problem setting:

  1. Relative formulation for arbitrary marginals without known densities: Our core methodological innovation is the relative formulation for adapting a pretrained bridge between arbitrary endpoint marginals without known densities (Propositions 1-3). Taking the difference between the new and pretrained Schrödinger potentials cancels the unknown target density p1p_1 (whereas prior Controller-Corrector Matching frameworks assume an explicit unnormalized density or energy) and yields the relative terminal corrector h(k)−hold−∇rh^{(k)} - h_{\mathrm{old}} - \nabla r, while preserving the arbitrary source marginal p0p_0.
  2. General reference processes and full adjoint ODE computation: Prior Controller-Corrector Matching frameworks are restricted to simple analytical reference priors (such as memoryless or Brownian priors where the adjoint ODE simplifies analytically). By contrast, TSBM operates relative to a general learned reference bridge PP with nonlinear neural drift, requiring integration of the full backward lean adjoint ODE −da~t/dt=∇xbt(Xt)⊤a~t-\mathrm{d}\tilde{a}_t/\mathrm{d}t = \nabla_x b_t(X_t)^\top \tilde{a}_t via vector-Jacobian products through the frozen drift network. In this respect, our algorithmic framework is strictly more general.
  3. Post-training for unpaired domain translation: Practically, prior Controller-Corrector Matching frameworks train a model from scratch against an explicitly evaluable energy function (e.g., Boltzmann sampling), whereas TSBM post-trains a pretrained Schrödinger bridge for unpaired domain translation where p1p_1 is only implicitly represented by the pretrained forward and backward drifts, resulting in a substantially different training pipeline.

In summary, TSBM's key innovations over prior Controller-Corrector Matching frameworks are (i) the potential-difference relative formulation that eliminates dependence on explicit endpoint densities, (ii) generalization of the adjoint matching step to arbitrary learned reference drifts via full backward ODE integration, and (iii) warm-started post-training of pretrained bridges for unpaired translation.


We hope that our clarifications and supporting experiments address your questions and reservations, and if so, we kindly ask you to consider updating your score accordingly. Please let us know if any questions remain.

References

[1] Domingo-Enrich et al., Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control, ICLR 2025.

[2] Clark et al., Directly Fine-Tuning Diffusion Models on Differentiable Rewards, ICLR 2024.

[3] Liu et al., Adjoint Schrödinger Bridge Sampler, NeurIPS, 2026.

[4] Tamogashev & Malkin, Data-to-Energy Stochastic Dynamics, ICLR 2026.

[5] Shin et al., Efficient Generative Modeling beyond Memoryless Diffusion via Adjoint Schrödinger Bridge Matching, ICML 2026.

Reviewer_AQcjOpenReview ↗

Response

Thank you for the detailed response.

The authors' explanation regarding the Red-Chroma reward sounds reasonable, and reward-tilting also preserves the thickness of the original numbers.

Although the main framework is built upon ASBS, developing a reward post-training method for pretrained Schrödinger bridges is still a meaningful contribution.

However, my main concern regarding the lack of training-based baselines remains:

  • I could not find other training-based studies using the same experimental settings, making it difficult to assess how much TSBM improves over existing post-training methods. Comparisons with training-free methods alone are insufficient to answer this question.

  • I believe various flow-based post-training methods could also be applied to MNIST image translation and CelebA semantic attribute editing, since flow models can also be trained as data-to-data models. However, the authors do not provide sufficient comparisons, making it difficult to determine whether flow-based methods or TSBM are preferable for general reinforcement learning or reward optimization tasks.

Overall, I believe TSBM is an effective approach for Schrödinger bridges. However, the current experiments are insufficient to fully assess its advantages and practical applicability. I therefore still lean toward acceptance, but cannot give a stronger recommendation at this stage.

Reviewer_T8aLOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on reward-based fine-tuning of pretrained Schrödinger Bridges (SBs), which is relatively underexplored compared to fine-tuning diffusion and flow-based models. The authors propose Tilted Schrödinger Bridge Matching (TSBM), which builds on the Adjoint Schrödinger Bridge Sampler (ASBS) framework. The proposed method uses a pretrained SB as the reference process for fine-tuning and alternates controller and corrector matching to learn a reward-tilted target distribution while preserving the source marginal.

The authors provide theoretical justification for the proposed formulation and borrowed convergence of the alternating updates from ASBS. They evaluate TSBM on unpaired image translation tasks, including controlling digit thickness and color on Colored MNIST and modifying facial attributes on CelebA. The results demonstrate effective reward fine-tuning and favorable performance to baseline methods.

Critical strengths
  • The paper provides a clear mathematical formulation of reward-based fine-tuning for pretrained Schrödinger Bridges. The theoretical connection between the original SB problem and the relative stochastic optimal control problem is well established.

  • The derivation using the difference between the new and old correctors is particularly interesting. It allows the authors to use the pretrained SB as the reference process while accounting for its non-memoryless nature. This is an important technical contribution of the paper.

  • The experiments generally support the effectiveness of the proposed method. The Colored MNIST and CelebA experiments show that TSBM can effectively steer generated attributes while largely preserving other image properties. The comparison with DPS and particle-based methods also demonstrates favorable reward alignment and inference efficiency.

Critical weaknesses

1. Computational cost of fine-tuning. As acknowledged by the authors, TSBM can be computationally expensive because it requires repeated forward SDE simulations and backward adjoint ODE computations. Although the paper reports the fine-tuning wall-clock time in Table 3, the computational cost relative to the original SB pretraining is not clearly quantified. This comparison would be helpful for understanding the practical efficiency of the proposed method.

2. Sensitivity to strong reward tilting. The convergence analysis assumes exact optimization, while practical training relies on approximate neural network updates. Although TSBM is initialized from a pretrained SB, a strong reward may shift the target distribution significantly away from the original distribution. I am concerned whether such large distributional shifts affect the convergence speed or training stability of the alternating controller-corrector updates.

3. Main components borrowed from ASBS. The core components such as iterative adjoint matching, corrector matching and convergence analysis is borrowed from ASBS. I can also see the smart modification of the math and corresponding algorithm, however, authors need to clarify additional novelty over modifying ASBS.

Questions for authors

1. Computational cost relative to pretraining. Could the authors compare the computational cost of TSBM fine-tuning with the original SB pretraining (e.g., DSBM) or other baselines? Although the fine-tuning time is reported in Table 3, a direct comparison on computing time would help clarify the practical efficiency of the proposed approach.

2. Convergence under stronger reward tilting. Could the authors provide an empirical convergence analysis across different reward strengths? I am concerned that a large shift between the original and reward-tilted target distributions may lead to slower convergence or unstable training.

3. Robustness to reward hacking. The authors suggest that DPS suffer from reward hacking, supported by the artifacts shown in Figure 4. However, it is unclear why TSBM should be inherently more robust to reward hacking. Could the authors explain whether there is any theoretical justification for this behavior, or whether the observed robustness is mainly empirical?

4. Presentation of the main technical insights. I believe that addressing the non-memoryless nature of the pretrained SB is one of the most important aspects of this work. The discussion in Appendix B.1 and the corrector ablation in Appendix G.3 are particularly helpful for understanding this point. I suggest incorporating some of these explanations and findings into the main sections to better highlight the motivation and importance of the proposed corrector matching procedure.

4 comments

We thank Reviewer T8aL for the thoughtful review and constructive feedback. Below, we address each of your comments and questions in detail. We have also uploaded a revised version of the manuscript with all changes highlighted in blue.

1. Fine-tuning cost versus pretraining cost

Reviewer

Computational cost of fine-tuning. As acknowledged by the authors, TSBM can be computationally expensive because it requires repeated forward SDE simulations and backward adjoint ODE computations. Although the paper reports the fine-tuning wall-clock time in Table 3, the computational cost relative to the original SB pretraining is not clearly quantified. This comparison would be helpful for understanding the practical efficiency of the proposed method.

Computational cost relative to pretraining. Could the authors compare the computational cost of TSBM fine-tuning with the original SB pretraining (e.g., DSBM) or other baselines? Although the fine-tuning time is reported in Table 3, a direct comparison on computing time would help clarify the practical efficiency of the proposed approach.

Answer

We thank the reviewer for this suggestion. In the revised manuscript (Appendix E, Table 3), we added a training-time comparison covering both DSBM pretraining and TSBM fine-tuning. For convenience, we reproduce the image-experiment results (in A100 GPU-hours) below:

Training phaseColored MNIST (A100 GPU-h)CelebA (A100 GPU-h)
DSBM pretraining25170
TSBM fine-tuning226
TSBM share of combined training time7.4%13.3%

As shown above, TSBM fine-tuning accounts for reasonable 7.4% (Colored MNIST) to 13.3% (CelebA) of the combined training budget. We also note that we did not tune TSBM hyperparameters (such as the number of stages, replay-buffer settings, or gradient steps per stage) for training speed, and we expect that similar performance could be achieved with less compute.

2. Limits of exact-update convergence

Reviewer

The convergence analysis assumes exact optimization, while practical training relies on approximate neural network updates.

Answer

We agree with the reviewer and note this in the Limitations section of both the original and revised manuscripts.

3. Stronger reward tilts

Reviewer

Sensitivity to strong reward tilting. The convergence analysis assumes exact optimization, while practical training relies on approximate neural network updates. Although TSBM is initialized from a pretrained SB, a strong reward may shift the target distribution significantly away from the original distribution. I am concerned whether such large distributional shifts affect the convergence speed or training stability of the alternating controller-corrector updates.

Convergence under stronger reward tilting. Could the authors provide an empirical convergence analysis across different reward strengths? I am concerned that a large shift between the original and reward-tilted target distributions may lead to slower convergence or unstable training.

Answer

Following the reviewer's suggestion, we evaluated TSBM with different reward strengths (λ∈{10,30,50,100}\lambda \in \{10, 30, 50, 100\}) on the CelebA Natural Toothy Smile reward and included this ablation in Appendix G.9 (Table 12) of the revised manuscript. In addition to the regular metrics (ImageReward, PickScore, and CLIP-IQA), we report LPIPS diversity [1] and DreamSim diversity [2] over all generated samples (DPS rows exclude raw outputs containing infinite values from all reported metrics). For convenience, we reproduce the evaluation results below:

MethodImageReward (train) ↑\uparrowPickScore (eval) ↑\uparrowCLIP-IQA ↑\uparrowLPIPS diversityDreamSim diversity
Pretrained DSBM-1.234817.61090.59500.43820.4094
TSBM λ=10\lambda=10-0.452817.90420.61970.43200.3804
TSBM λ=30\lambda=300.139818.23450.65950.41830.3520
TSBM λ=50\lambda=500.458518.42390.67490.40810.3297
TSBM λ=100\lambda=1000.921718.57190.72660.38450.2858
DPS* γ=1\gamma=10.163417.80870.62860.43100.3838
DPS* γ=2\gamma=20.676217.72690.63730.42630.3657
DPS* γ=4\gamma=41.049917.49970.62110.41760.3433
CondSMC (K=4K=4)-0.572817.83750.61070.43410.3972

One can see that while higher reward strength TSBM achieves higher reward terms on both ImageReward and PickScore (as well as higher CLIP-IQA), diversity lowers.

No numerical difficulties or convergence problems were encountered, and all reward strengths were evaluated under the exact same hyperparameter settings for training and evaluation. However, as the reviewer noted, higher reward strengths induce a stronger shift and stronger reward signals, which we suppose leads to the decreasing diversity.

4. Novelty and ASBS

Reviewer

Main components borrowed from ASBS. The core components such as iterative adjoint matching, corrector matching and convergence analysis is borrowed from ASBS. I can also see the smart modification of the math and corresponding algorithm, however, authors need to clarify additional novelty over modifying ASBS.

Answer

We agree that TSBM borrows the high-level alternating Controller-Corrector Matching template and proof strategy from ASBS. However, adapting a pretrained bridge requires a substantially different formulation, algorithm, and implementation:

  1. Relative formulation for arbitrary marginals without known densities: Our methodological contribution is the relative formulation for adapting a pretrained bridge between arbitrary endpoint marginals without known densities (Propositions 1-3). Taking the difference between the new and pretrained Schrödinger potentials cancels the unknown target density p1p_1 (which ASBS assumes is known up to a normalizing constant) and gives the relative terminal corrector h(k)−hold−∇rh^{(k)} - h_{\mathrm{old}} - \nabla r while preserving the arbitrary source marginal p0p_0.
  2. General reference processes and full adjoint ODE computation: ASBS is restricted to simple analytical reference priors (such as memoryless or Brownian priors where the adjoint ODE simplifies analytically). By contrast, TSBM operates relative to a general learned reference bridge PP with nonlinear neural drift bt(Xt)b_t(X_t), which requires integrating the full backward lean adjoint ODE −da~t/dt=∇xbt(Xt)⊤a~t-\mathrm{d}\tilde{a}_t/\mathrm{d}t = \nabla_x b_t(X_t)^\top \tilde{a}_t via vector-Jacobian products through the frozen drift network (Eq. 22). In this respect, our algorithmic framework is more general than ASBS.
  3. Different practical problem and implementation: Practically, ASBS trains a network from scratch to sample from Boltzmann distributions with a known energy function, whereas TSBM post-trains a pretrained Schrödinger bridge for unpaired domain translation where p1p_1 is only implicitly represented by the pretrained forward and backward drifts, resulting in a completely different training pipeline.

Thus, rather than a direct modification of ASBS, TSBM addresses a different setting where neither the reference drift is analytical nor the target density is known, resulting in a substantially different algorithm and implementation.

5. Reward hacking with DPS and TSBM

Reviewer

Robustness to reward hacking. The authors suggest that DPS suffer from reward hacking, supported by the artifacts shown in Figure 4. However, it is unclear why TSBM should be inherently more robust to reward hacking. Could the authors explain whether there is any theoretical justification for this behavior, or whether the observed robustness is mainly empirical?

Answer

The observed difference in robustness is empirical rather than a formal theoretical guarantee: under exact optimization, TSBM targets p1r∝p1erp_1^r \propto p_1 e^r and is not inherently immune to a misspecified reward.

In practice, the following two factors may explain why DPS exhibits stronger artifacts and reward over-optimization than TSBM in our experiments. First, DPS adds γσ2∇xr(x^1(x,ti))\gamma \sigma^2 \nabla_x r(\widehat{x}_1(x, t_i)) at every inference step, applying gradient ascent to a differentiable proxy reward known to exploit proxy flaws [3] directly along each trajectory. By contrast, TSBM uses reward gradients only during training under an explicit KL penalty to the pretrained bridge PP [4] and samples without evaluating rr or ∇r\nabla r. Second, DPS [5] approximates log⁡E[er(X1)∣Xt]\log \mathbb{E}[e^{r(X_1)} \mid X_t] via r(x^1(Xt,t))r(\widehat{x}_1(X_t, t)), which yields a bias, which leads to inference process that does not match valid bridge SDE, possibly pushing trajectories off-manifold and producing visual artifacts.

6. Corrector motivation and ablation placement

Reviewer

Presentation of the main technical insights. I believe that addressing the non-memoryless nature of the pretrained SB is one of the most important aspects of this work. The discussion in Appendix B.1 and the corrector ablation in Appendix G.3 are particularly helpful for understanding this point. I suggest incorporating some of these explanations and findings into the main sections to better highlight the motivation and importance of the proposed corrector matching procedure.

Answer

We thank the reviewer for noting the difficulties of adressing non-memoryless nature, which are the cornerstone of our work. Following your suggestion, in the revised paper version we have moved part if the Appendix B.1 discussion to the Background Section 2 and moved the static corrector ablation to the Experiments Section 6.1.


We hope that our new experiments, expanded main-text explanations, and clarifications address your concerns, and if so, we kindly ask you to consider updating your score accordingly. Please let us know if any questions remain.

References

[1] Zhang et al., The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, CVPR 2018.

[2] Domingo-Enrich et al., Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control, ICLR 2025.

[3] Clark et al., Directly Fine-Tuning Diffusion Models on Differentiable Rewards, ICLR 2024.

[4] Uehara et al., Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control, arXiv preprint arXiv:2402.15194, 2024.

[5] Chung et al., Diffusion Posterior Sampling for General Noisy Inverse Problems, ICLR 2023.

I appreciate all the response including additional experiments. All of my concerns are resolved. I personally think this paper is a enough contribution as a poster publication. I raised my score.

Reviewer_MxkhOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper defines the method of fine-tuning bridge models from the distribution perspective. It identifies the distinction between conditional reward tilting and marginal distribution tilting, and introduces a corrector and an alternating training procedure to address this distinction. What’s more, the main idea in this paper is supported by extensive theoretical analysis and experiments. Although the method is a targeted extension of existing theoretical frameworks rather than an entirely new paradigm, and its theoretically motivated objective may not fully align with typical practical requirements, the paper offers a distinctive theoretical perspective and comprehensive mathematical proofs. In particular, eliminating the unknown probability density from the terminal cost enables fine-tuning of pretrained bridges with implicit target distributions.

Critical strengths
  1. The paper identifies a key challenge in fine-tuning bridge models arising from the dependence between inputs and outputs. It explains the resulting bias and develops a corresponding algorithm, supported by theoretical analysis and experimental validation.

  2. Reward guidance is incorporated into the model parameters, eliminating the need for additional reward evaluations or reward gradient computations during inference and keeping inference costs low.

Critical weaknesses
  1. The convergence guarantee relies on exact population level updates and strong regularity and integrability assumptions. It is a theoretical result that does not automatically extend to practical experimental settings.

  2. The high dimensional image experiments do not directly establish whether the outputs accurately follow the target distribution. Moreover, the particle methods use only a small number of particles in the main experiments, and variability across multiple independent training runs is not reported.

Questions for authors

None

3 comments

We thank Reviewer Mxkh for the thoughtful review and constructive feedback. Below, we address each of your comments and questions in detail. We have also uploaded a revised version of the manuscript with all changes highlighted in blue.

1. Exact-update convergence vs practice

Reviewer

The convergence guarantee relies on exact population level updates and strong regularity and integrability assumptions. It is a theoretical result that does not automatically extend to practical experimental settings.

Answer

We agree with the reviewer. We already noted this in the Limitations section ("We assume exact optimization and validity of all theoretical assumptions, which may fail in practice"), and in the revised manuscript we have updated Limitations to state explicitly that our theoretical results rely on exact population-level updates, an exact pretrained bridge, and strong regularity and integrability conditions that may fail under practical neural optimization.

2. High-dimensional exact target-distribution fidelity

Reviewer

The high dimensional image experiments do not directly establish whether the outputs accurately follow the target distribution.

Answer

We agree that our image experiments do not directly measure distributional distance to the exact tilted target p1r(x)∝p1(x)er(x)p_1^r(x) \propto p_1(x)e^{r(x)}. However, assessing exact distributional fidelity in high-dimensional image spaces is rarely feasible in practice for two reasons:

  1. No ground-truth target samples: Sample-based distributional metrics require samples from p1rp_1^r, which are unavailable and generating them is the task itself.
  2. Intractable densities and normalization: Density-based comparisons require evaluating the implicit data density p1(x)p_1(x), the global normalizing constant Zr=∫p1(x)er(x) dxZ_r = \int p_1(x)e^{r(x)}\,\mathrm{d}x, and the output density of the fine-tuned model. In general, computing this combination of quantities is intractable in image dimensions.

In our 2D experiment (Appendix G.1), both the target density p1rp_1^r and ground-truth samples from p1rp_1^r are known analytically, which directly addresses whether TSBM outputs accurately follow the target distribution. On that benchmark, TSBM closely recovers the exact tilted target.

In high-dimensional image experiments (Colored MNIST and CelebA), we still evaluate all tractable aspects of alignment, image quality, and source faithfulness. Following standard practice in reward-aligned generative modeling, we jointly track the optimized reward, held-out preference and naturalness metrics, and input-preservation metrics.

If the reviewer has a specific tractable diagnostic for high-dimensional p1rp_1^r fidelity in mind, we would appreciate any suggestion.

3. Particle-budget for SNIS/SMC

Reviewer

Moreover, the particle methods use only a small number of particles in the main experiments, and variability across multiple independent training runs is not reported.

Answer

Following the reviewer's concern, we evaluated CondSNIS and CondSMC under larger particle budgets (K∈{4,8,16,32}K \in \{4, 8, 16, 32\}) on the CelebA reward-tilting benchmark. We report the full ablation and comparison against TSBM and DPS across all three prompts in Appendix G.7 (Table 10) of the revised manuscript.

For convenience, we also reproduce the Natural Toothy Smile results below (the remaining prompts are in Appendix G.7):

MethodParticlesImageReward (train) ↑\uparrowPickScore (eval) ↑\uparrowCLIP-IQA ↑\uparrowEval time (h)
Pretrained-1.234817.61090.5950$\approx$0.1
TSBM (ours)0.139818.23450.6595$\approx$0.1
DPS* (γ=1\gamma=1)0.062217.73470.6192$\approx$1.0
DPS* (γ=2\gamma=2)0.599317.68310.6312$\approx$1.0
DPS* (γ=4\gamma=4)0.978317.46880.6164$\approx$1.0
CondSNIS4-1.067017.67720.6042$\approx$0.5
CondSNIS8-1.015517.71090.6109$\approx$0.8
CondSNIS16-0.963417.72410.6082$\approx$1.5
CondSNIS32-0.918117.72970.6164$\approx$3
CondSMC4-0.572817.83750.61071.5+
CondSMC8-0.320717.90850.62183+
CondSMC16-0.173917.92630.62976+
CondSMC32-0.046017.97030.638212+

Increasing KK from 4 to 32 monotonically improves both ImageReward and PickScore for CondSNIS and CondSMC. However, the gains exhibit diminishing returns and still fall short of TSBM across all metrics (and remain below DPS on ImageReward), while sampling time grows substantially with KK, e.g., from 1.5+h at K=4K=4 to 12+h at K=32K=32 for CondSMC, compared to ≈\approx 0.1h for TSBM.

Taken together, the linear scaling in computational complerxity, diminishing returns, and weaker overall performance still leave CondSMC and CondSNIS impractical at high particle budgets in this setting.

4.Independent training runs

Reviewer

.... and variability across multiple independent training runs is not reported.

Answer

Following the reviewer suggestion we've conducted the study of stability across multiple independent training runs for the ColoredMNIST data with Thin Stroke reward. We've run 5 independent training runs with exactly the same hyperparameters as in Section 6.1 experiment. We've added the results into the Appendix G.8 in the revised version. For completeness here we double the Table with results:

MethodReward ↑\uparrowHue error (∘^\circ) ↓\downarrow
TSBM (ours), seed1-0.016015.67
TSBM (ours), seed2-0.012415.14
TSBM (ours), seed3-0.009316.76
TSBM (ours), seed4-0.015715.79
TSBM (ours), seed5-0.015013.49
TSBM (ours), mean ±\pm std-0.0137 ±\pm 0.002815.37 ±\pm 1.20
DPS (γ=0.5\gamma=0.5)-0.128520.07
DPS (γ=1\gamma=1)-0.578223.56
DPS (γ=2\gamma=2)-2.335032.72
DPS (γ=4\gamma=4)-7.418946.81
CondSMC-0.684913.65
CondSNIS-0.953313.17

One can see that different training seeds do not affect the result significantly, TSBM still provides the best reward-hue error tradeoff.


We hope that our new ablations and clarifications address your concerns, and if so, we kindly ask you to consider updating your score accordingly. Please let us know if any questions remain.

Thank you for your reply and supplementary experiments which resolved my previous doubts. I believe this paper addresses a specific and meaningful problem: how to fine-tune a pretrained Schrödinger bridge toward a prescribed reward-tilted target distribution when the original target distribution is only implicitly represented and its density or score is not explicitly available. Due to my limited experience in this specific research direction and referring to the opinions raised by other reviewers, I decide to maintain the initial score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal Response

We sincerely thank all the reviewers for their valuable feedback and for recognizing the strengths of our work:

  • Clear formulation for marginal reward tilting: Identifying the non-memoryless bias in bridge fine-tuning and connecting it to relative stochastic optimal control (T8aL, Mxkh, AQcj)
  • Implicit-target relative corrector: Cancelling unknown target densities via potential differences so a pretrained bridge can serve as the reference (T8aL, Mxkh, AQcj)
  • Effective alignment and fast inference: Steering attributes on Colored MNIST and CelebA while preserving unrelated properties and avoiding sampling-time reward evaluations (T8aL, Mxkh)

In response to the reviewers' concerns and suggestions, we have made the following changes, highlighted in blue in the uploaded revised manuscript:

  • Expanded empirical ablations:
    • Added the particle-budget ablation for CondSNIS and CondSMC across larger particle budgets on CelebA tasks in Appendix G.7, Table 10 (Mxkh)
    • Added the independent training runs variability study on Colored MNIST thin-stroke reward in Appendix G.8, Table 11 (Mxkh)
    • Added the reward-strength ablation on CelebA Natural Toothy Smile reward in Appendix G.9, Table 12 (T8aL)
  • Training-time comparison: Added Table 3 in Appendix E directly comparing DSBM pretraining and TSBM fine-tuning compute across Toy 2D, Colored MNIST, and CelebA (T8aL)
  • Manuscript improvements:
    • Moved parts of the non-memoryless bridges tilted Appendix B.1 into the main text Section 2 (T8aL)
    • Moved the quantitative static-corrector ablation on Colored MNIST from Appendix G.3 to Section 6.1 in the main text (T8aL).
    • Updated the Limitations paragraph in Section 7 explicitly mention the stated regularity and integrability conditions (T8aL, Mxkh)

We believe that these additions and clarifications have thoroughly addressed the reviewers concerns and strengthened our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.