- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In multi-objective test-time alignment using multiple Autoregressive Reward Models (ARMs), the predictions of a base LLM and a combined ARM are integrated using a fixed coefficient . This means that the same guidance strength is applied to all tokens. However, at tokens where the disagreement among ARM predictive distributions is large, different objectives tend to lead to larger differences in downstream rewards. This paper refers to such reward-relevant branching points as Cross-Reward Forks and uses Jensen–Shannon (JS) divergence as a measure to identify them. Based on this observation, this paper proposes Disagreement-Informed Routing (DIR). DIR transforms JS divergence into a token-specific coefficient using an empirical cumulative distribution function and replaces the fixed coefficient with . Consequently, DIR dynamically adjusts the guidance strength, placing more weight on the base LLM when JS divergence is small and more weight on the combined ARM when JS divergence is large. Experimental results demonstrate that DIR consistently outperforms existing methods.
Critical strengths
The idea of adaptively adjusting the guidance strength at the token level is interesting, and the proposed method consistently outperforms existing baselines across various experimental settings.
Critical weaknesses
-
DIR appears to require additional ARM forward passes to compute JS divergence at each decoding step, particularly when using preference-conditioned ARMs. This could become a computational bottleneck as the number of objectives increases. Are there any possible solutions to mitigate this overhead?
-
Why should the percentile of JS divergence be considered an appropriate guidance strength? Is there any theoretical guarantee supporting this choice?
-
The experiments consider at most three objectives. It would be interesting to investigate whether DIR remains effective and computationally efficient as the number of objectives increases.
-
The fixed guidance coefficient does not appear to be sufficiently tuned in the baselines. Comparing DIR against baselines with optimized on a validation set would help clarify whether the performance improvements come from token-wise adaptive guidance rather than the choice of guidance strength.
Questions for authors
Please see Critical Weaknesses.
5 comments
Response to Reviewer QomW (2/3)
W3: More than three objectives.
Multi-objective settings typically involve 2–3 objectives, as studied in this work, while settings with 4 or more objectives are generally considered many-objective optimization.
We additionally evaluate four-objective alignment on PKU 7B by adding Truthfulness to Helpfulness, Harmlessness, and Humor. We use the complete 1,500-prompt held-out test set and the same 17 preferences (including the four single-objective endpoints and the uniform four-objective preference) for all methods.
Following the matched comparisons in Appendix G, we compare PARM with DIR using the same four-objective PARM adapter, and GenARM with DIR using the same four separately trained single-objective GenARM adapters. Within each comparison, the guidance checkpoints and frozen Alpaca-7B policy are fixed, and only the decoding rule changes; no separate router is trained.
The table reports unnormalized four-dimensional HV and MIP. For each guided method, we compute HV from its own 17 preference-level mean reward vectors, using a common reference point fixed from training-set reward minima. MIP averages the preference-weighted raw rewards over the same 17 preferences. We compute Base’s HV separately from its single mean reward vector and evaluate its MIP over the same 17 preferences. Both blocks show the same Base results for reference.
| PARM ARM | GenARM ARMs | ||||
|---|---|---|---|---|---|
| Method | HV ↑ | MIP ↑ | Method | HV ↑ | MIP ↑ |
| Base | 79,538.39 | 13.3729 | Base | 79,538.39 | 13.3729 |
| PARM | 292,456.93 | 19.6910 | GenARM | 314,001.53 | 18.8951 |
| DIR | 383,605.51 | 21.2767 | DIR | 415,128.35 | 21.3833 |
After adding Truthfulness as the fourth objective, DIR retains its advantage in both ARM families: +31.17% HV and +8.05% MIP over PARM, and +32.21% HV and +13.17% MIP over GenARM.
We use 2,000 paired bootstrap resamples of the 1,500 test rows to estimate 95% confidence intervals. In each resample, we use the same sampled rows across all methods and all 17 preferences and recompute HV and MIP. The table below reports unnormalized differences (DIR minus the corresponding baseline).
| PARM ARM | GenARM ARMs | ||||
|---|---|---|---|---|---|
| Metric | DIR − PARM | Paired 95% CI | Metric | DIR − GenARM | Paired 95% CI |
| HV | +91,148.59 | [86,236.77, 95,944.09] | HV | +101,126.82 | [96,561.68, 105,992.71] |
| MIP | +1.5858 | [1.4769, 1.7021] | MIP | +2.4882 | [2.3415, 2.6542] |
All four intervals lie entirely above zero, supporting DIR's advantage beyond three objectives. These intervals quantify uncertainty under test-row resampling for the fixed checkpoints.
We also measured inference cost on 64 randomly sampled prompts from the four-objective PKU 7B test set at the uniform preference. All methods used BF16 greedy decoding with batch size 1 and generated exactly 512 tokens per prompt. Within each comparison, we measured each method three times per prompt on the same A100-SXM4-40GB GPU. The table reports the mean and sample standard deviation of the per-prompt average times and the mean peak GPU memory allocated by PyTorch across runs.
| PARM ARM | GenARM ARMs | ||||
|---|---|---|---|---|---|
| Method | Time (s, mean ± SD) | GPU memory (GiB) | Method | Time (s, mean ± SD) | GPU memory (GiB) |
| PARM | 26.21 ± 0.36 | 13.16 | GenARM | 64.39 ± 0.81 | 14.03 |
| DIR | 29.68 ± 0.38 | 14.27 | DIR | 64.52 ± 0.82 | 14.03 |
At the uniform four-objective preference, DIR increased generation time by 13.3% over PARM and by 0.2% over GenARM. Mean peak allocated GPU memory increased by 1.11 GiB in the PARM comparison and was essentially unchanged in the GenARM comparison. At this preference, GenARM already computes all four endpoint distributions, which DIR reuses to measure disagreement.
These results show that DIR retains its HV and MIP gains with four objectives for both ARM families, with modest decoding overhead.
Response to Reviewer QomW (3/3)
W4: Tuning the fixed coefficient.
The used guidance strength followed the published PARM evaluation protocol, which sets the strength under for both PARM and its GenARM baseline.
To distinguish the benefit of token-wise adaptation from the choice of overall guidance strength, we conducted two additional controlled experiments.
-
Fixed-coefficient ablation. In our paper, we use an equivalent form , so setting is equivalent to setting . We tuned PARM on PKU 7B using the same Base and PARM checkpoint as DIR. We swept five fixed ARM weights on the 500-prompt validation set and also checked the PARM-only endpoint.
Fixed ARM weight Validation MIP 0.10 -1.511 0.25 0.131 0.50 2.906 0.75 4.649 0.90 4.894 1.00 (PARM only, additional validation check) 4.584 We selected by validation MIP over the five sweep weights and used it at all 11 preferences on the original 1,500-prompt held-out test set.
Notably, although larger generally improves validation MIP, overly strong ARM weighting substantially degrades generation quality: 11.84% and 28.24% of validation responses exhibit severe repetition loops at and , respectively.
-
Mean-coefficient-matched control. For each PKU 7B test preference, we computed the token-weighted mean of DIR’s realized over its test trajectories and used that value as PARM’s fixed coefficient for all prompts and decoding steps at that preference. The coefficients were derived from the recorded trajectories without using test rewards.
| PKU 7B method | HV | MIP |
|---|---|---|
| Original PARM () | 111.36 | 2.620 |
| PARM matched to DIR’s mean at each preference | 117.84 | 3.154 |
| Validation-selected PARM () | 136.88 | 4.682 |
| Original DIR | 159.78 | 5.831 |
Validation tuning improved PARM on both metrics. DIR retained its advantage over both PARM controls on the same held-out test set.
Response to Reviewer QomW (1/3)
Thank you for your review and feedback.
W1: Additional ARM computation and possible mitigation.
We analyzed the computational cost in Appendix G.2, Table 10 in the original submission (Table 13 in the revised manuscript). For convenience, we reproduce the results below. All methods generate exactly 512 tokens using the same 64 prompts on one A100-SXM4-40GB, with a 7B Base and 1.1B guidance models.
Our DIR achieves substantially lower inference time than all baselines except PARM. Our implementation builds on PARM, the most efficient baseline in this benchmark, which obtains guidance by evaluating a single preference-conditioned ARM at the requested preference. DIR additionally requires the one-hot endpoint distributions to compute disagreement. These evaluations can be batched with the requested-preference evaluation to mitigate sequential execution overhead, as described in Sec. 4.3.
The measured overhead over PARM is 10.2%, while DIR improves HV and MIP by 17.0% and 21.2%, respectively, demonstrating a favorable quality–efficiency trade-off in the evaluated three-objective setting.
W2: Why use the JS percentile as guidance strength? Is there a guarantee?
We use the percentile as a simple, scale-calibrated allocation rule. As explained in Sec. 4.1, the absolute magnitude of JS can vary across ARM families and objective sets. The mapping ut=Fcal(dt) preserves the ordering of disagreement while expressing it on a common [0,1] scale: relatively high-disagreement positions receive stronger guidance, and relatively low-disagreement positions retain more of the Base policy. This choice is motivated by the intervention studies in Sec. 3.2 and Appendices B.6–B.7, which show that higher disagreement identifies positions with greater downstream reward sensitivity.
What is theoretically guaranteed?
Two properties support this construction. First, Eq.(3) establishes that normalized generalized JS equals I(C;V∣ht)/logK, providing an information-theoretic interpretation of the routing signal. Second, under a continuous reference distribution, the probability integral transform makes its CDF values uniform, with mean 0.5. Thus, rank calibration approximately centers guidance around the equal-strength baseline on the calibration distribution while redistributing it across tokens. Finite-sample calibration and test-distribution shifts make this property approximate in practice.
These properties motivate percentile-based routing as an empirical design choice, whose effectiveness is supported by the ablation results in Appendix I.