acceptodds
Under review as a conference paper at ICLR 2027

The Router Is Already in the ARMs: Disagreement-Guided Multi-Objective Test-Time Alignment for LLMs

Abstract

Multi-objective test-time alignment uses autoregressive reward models (ARMs) to steer a frozen language model toward user-specified preferences. But when should reward guidance matter, and how strongly should it override the base policy? We show that the answer is already in the ARMs: disagreement among objective-conditioned next-token distributions reveals cross-reward forks, where different local reward directions can lead to meaningfully different downstream reward outcomes. This disagreement has a precise information-theoretic interpretation—generalized Jensen–Shannon divergence equals the mutual information between objective identity and the next token—and is strongly correlated with downstream cross-reward separation (Spearman 0.766). Building on this insight, we introduce Disagreement-Informed Routing (DIR), which uses ARM disagreement to modulate the token-wise step size from the base policy along the requested preference direction. DIR requires no learned router, auxiliary model, or policy fine-tuning. Experiments across safety alignment, cross-domain transfer, and model scales demonstrate effective multi-objective alignment using the routing signal already present in the ARMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-objective alignment, test-time alignment, autoregressive reward models, large language model alignment, controlled decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_QomWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In multi-objective test-time alignment using multiple Autoregressive Reward Models (ARMs), the predictions of a base LLM p0p_0 and a combined ARM pαp_\alpha are integrated using a fixed coefficient μ\mu. This means that the same guidance strength is applied to all tokens. However, at tokens where the disagreement among ARM predictive distributions is large, different objectives tend to lead to larger differences in downstream rewards. This paper refers to such reward-relevant branching points as Cross-Reward Forks and uses Jensen–Shannon (JS) divergence as a measure to identify them. Based on this observation, this paper proposes Disagreement-Informed Routing (DIR). DIR transforms JS divergence into a token-specific coefficient utu_t using an empirical cumulative distribution function and replaces the fixed coefficient μ\mu with utu_t. Consequently, DIR dynamically adjusts the guidance strength, placing more weight on the base LLM when JS divergence is small and more weight on the combined ARM when JS divergence is large. Experimental results demonstrate that DIR consistently outperforms existing methods.

Critical strengths

The idea of adaptively adjusting the guidance strength at the token level is interesting, and the proposed method consistently outperforms existing baselines across various experimental settings.

Critical weaknesses
  1. DIR appears to require additional ARM forward passes to compute JS divergence at each decoding step, particularly when using preference-conditioned ARMs. This could become a computational bottleneck as the number of objectives increases. Are there any possible solutions to mitigate this overhead?

  2. Why should the percentile of JS divergence be considered an appropriate guidance strength? Is there any theoretical guarantee supporting this choice?

  3. The experiments consider at most three objectives. It would be interesting to investigate whether DIR remains effective and computationally efficient as the number of objectives increases.

  4. The fixed guidance coefficient μ\mu does not appear to be sufficiently tuned in the baselines. Comparing DIR against baselines with μ\mu optimized on a validation set would help clarify whether the performance improvements come from token-wise adaptive guidance rather than the choice of guidance strength.

Questions for authors

Please see Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer QomW (1/3)

Thank you for your review and feedback.

W1: Additional ARM computation and possible mitigation.

We analyzed the computational cost in Appendix G.2, Table 10 in the original submission (Table 13 in the revised manuscript). For convenience, we reproduce the results below. All methods generate exactly 512 tokens using the same 64 prompts on one A100-SXM4-40GB, with a 7B Base and 1.1B guidance models.

MethodGeneration time (s) ↓Mean peak allocated GPU memory (GiB)
PARM35.36 ± 0.3015.376
GenARM67.25 ± 0.6315.407
MOD-w2s81.84 ± 0.7115.422
CAGE139.80 ± 21.5615.422
DIR (ours)38.97 ± 0.4115.428

Our DIR achieves substantially lower inference time than all baselines except PARM. Our implementation builds on PARM, the most efficient baseline in this benchmark, which obtains guidance by evaluating a single preference-conditioned ARM at the requested preference. DIR additionally requires the one-hot endpoint distributions to compute disagreement. These evaluations can be batched with the requested-preference evaluation to mitigate sequential execution overhead, as described in Sec. 4.3.

The measured overhead over PARM is 10.2%, while DIR improves HV and MIP by 17.0% and 21.2%, respectively, demonstrating a favorable quality–efficiency trade-off in the evaluated three-objective setting.

W2: Why use the JS percentile as guidance strength? Is there a guarantee?

We use the percentile as a simple, scale-calibrated allocation rule. As explained in Sec. 4.1, the absolute magnitude of JS can vary across ARM families and objective sets. The mapping ut=Fcal(dt)u_t = F_{\mathrm{cal}}(d_t) preserves the ordering of disagreement while expressing it on a common [0,1][0,1] scale: relatively high-disagreement positions receive stronger guidance, and relatively low-disagreement positions retain more of the Base policy. This choice is motivated by the intervention studies in Sec. 3.2 and Appendices B.6–B.7, which show that higher disagreement identifies positions with greater downstream reward sensitivity.

What is theoretically guaranteed?

Two properties support this construction. First, Eq.(3) establishes that normalized generalized JS equals I(C;V∣ht)/log⁡KI(C;V\mid h_t)/\log K, providing an information-theoretic interpretation of the routing signal. Second, under a continuous reference distribution, the probability integral transform makes its CDF values uniform, with mean 0.50.5. Thus, rank calibration approximately centers guidance around the equal-strength baseline on the calibration distribution while redistributing it across tokens. Finite-sample calibration and test-distribution shifts make this property approximate in practice.

These properties motivate percentile-based routing as an empirical design choice, whose effectiveness is supported by the ablation results in Appendix I.

AuthorsOpenReview ↗

Response to Reviewer QomW (2/3)

W3: More than three objectives.

Multi-objective settings typically involve 2–3 objectives, as studied in this work, while settings with 4 or more objectives are generally considered many-objective optimization.

We additionally evaluate four-objective alignment on PKU 7B by adding Truthfulness to Helpfulness, Harmlessness, and Humor. We use the complete 1,500-prompt held-out test set and the same 17 preferences (including the four single-objective endpoints and the uniform four-objective preference) for all methods.

Following the matched comparisons in Appendix G, we compare PARM with DIR using the same four-objective PARM adapter, and GenARM with DIR using the same four separately trained single-objective GenARM adapters. Within each comparison, the guidance checkpoints and frozen Alpaca-7B policy are fixed, and only the decoding rule changes; no separate router is trained.

The table reports unnormalized four-dimensional HV and MIP. For each guided method, we compute HV from its own 17 preference-level mean reward vectors, using a common reference point fixed from training-set reward minima. MIP averages the preference-weighted raw rewards over the same 17 preferences. We compute Base’s HV separately from its single mean reward vector and evaluate its MIP over the same 17 preferences. Both blocks show the same Base results for reference.

PARM ARMGenARM ARMs
MethodHV ↑MIP ↑MethodHV ↑MIP ↑
Base79,538.3913.3729Base79,538.3913.3729
PARM292,456.9319.6910GenARM314,001.5318.8951
DIR383,605.5121.2767DIR415,128.3521.3833

After adding Truthfulness as the fourth objective, DIR retains its advantage in both ARM families: +31.17% HV and +8.05% MIP over PARM, and +32.21% HV and +13.17% MIP over GenARM.

We use 2,000 paired bootstrap resamples of the 1,500 test rows to estimate 95% confidence intervals. In each resample, we use the same sampled rows across all methods and all 17 preferences and recompute HV and MIP. The table below reports unnormalized differences (DIR minus the corresponding baseline).

PARM ARMGenARM ARMs
MetricDIR − PARMPaired 95% CIMetricDIR − GenARMPaired 95% CI
HV+91,148.59[86,236.77, 95,944.09]HV+101,126.82[96,561.68, 105,992.71]
MIP+1.5858[1.4769, 1.7021]MIP+2.4882[2.3415, 2.6542]

All four intervals lie entirely above zero, supporting DIR's advantage beyond three objectives. These intervals quantify uncertainty under test-row resampling for the fixed checkpoints.

We also measured inference cost on 64 randomly sampled prompts from the four-objective PKU 7B test set at the uniform preference. All methods used BF16 greedy decoding with batch size 1 and generated exactly 512 tokens per prompt. Within each comparison, we measured each method three times per prompt on the same A100-SXM4-40GB GPU. The table reports the mean and sample standard deviation of the per-prompt average times and the mean peak GPU memory allocated by PyTorch across runs.

PARM ARMGenARM ARMs
MethodTime (s, mean ± SD)GPU memory (GiB)MethodTime (s, mean ± SD)GPU memory (GiB)
PARM26.21 ± 0.3613.16GenARM64.39 ± 0.8114.03
DIR29.68 ± 0.3814.27DIR64.52 ± 0.8214.03

At the uniform four-objective preference, DIR increased generation time by 13.3% over PARM and by 0.2% over GenARM. Mean peak allocated GPU memory increased by 1.11 GiB in the PARM comparison and was essentially unchanged in the GenARM comparison. At this preference, GenARM already computes all four endpoint distributions, which DIR reuses to measure disagreement.

These results show that DIR retains its HV and MIP gains with four objectives for both ARM families, with modest decoding overhead.

AuthorsOpenReview ↗

Response to Reviewer QomW (3/3)

W4: Tuning the fixed coefficient.

The used guidance strength followed the published PARM evaluation protocol, which sets the strength λ=1\lambda=1 under log⁡p0+λlog⁡pα\log p_0+\lambda\log p_{\boldsymbol{\alpha}} for both PARM and its GenARM baseline.

To distinguish the benefit of token-wise adaptation from the choice of overall guidance strength, we conducted two additional controlled experiments.

  • Fixed-coefficient ablation. In our paper, we use an equivalent form (1−μ)log⁡p0+μlog⁡pα(1-\mu)\log p_0+\mu\log p_{\boldsymbol{\alpha}}, so setting λ=1\lambda=1 is equivalent to setting μ=0.5\mu=0.5. We tuned PARM on PKU 7B using the same Base and PARM checkpoint as DIR. We swept five fixed ARM weights on the 500-prompt validation set and also checked the PARM-only endpoint.

    Fixed ARM weight μ\muValidation MIP
    0.10-1.511
    0.250.131
    0.502.906
    0.754.649
    0.904.894
    1.00 (PARM only, additional validation check)4.584

    We selected μ=0.90\mu=0.90 by validation MIP over the five sweep weights and used it at all 11 preferences on the original 1,500-prompt held-out test set.

    Notably, although larger μ\mu generally improves validation MIP, overly strong ARM weighting substantially degrades generation quality: 11.84% and 28.24% of validation responses exhibit severe repetition loops at μ=0.90\mu=0.90 and μ=1.00\mu=1.00, respectively.

  • Mean-coefficient-matched control. For each PKU 7B test preference, we computed the token-weighted mean of DIR’s realized utu_t over its test trajectories and used that value as PARM’s fixed coefficient for all prompts and decoding steps at that preference. The coefficients were derived from the recorded trajectories without using test rewards.

PKU 7B methodHVMIP
Original PARM (μ=0.5\mu=0.5)111.362.620
PARM matched to DIR’s mean at each preference117.843.154
Validation-selected PARM (μ=0.90\mu=0.90)136.884.682
Original DIR159.785.831

Validation tuning improved PARM on both metrics. DIR retained its advantage over both PARM controls on the same held-out test set.

Reviewer_ojgiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes token-level multi-objective ARM guidance, unlike methods that use a fixed guidance strength throughout the sequence. They measure disagreement between objective-specific ARM next-token distributions using JS divergence. So when disagreement is high, apply stronger guidance toward the user’s requested objectives and when low rely more on the base model.

Critical strengths

S1. They show that disagreement between reward models is an easy method for identifying token positions which need intervention via reward models.

S2. The improvements extend to experiments with both two and three objectives, showing the usefulness of disagreement-based adaptive guidance.

Critical weaknesses

W1. DIR might perform better because it applies stronger guidance on average, not just because it guides the right tokens. PARM uses a fixed μ=0.5\mu = 0.5, while DIR’s average μt\mu_t is around 0.5 on the calibration data. The paper mentions this average can change at test time. If it increases, stronger guidance could explain part of the improvement. It would be helpful to report the average μt\mu_t for each preference and compare against PARM with different fixed guidance strengths λ\lambda, including one matching DIR’s average.

Questions for authors

How strong is DIR’s guidance on average for each preference and setting? How much does it vary across tokens, and how does it change as the response goes on?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ojgi (1/2)

Thank you for your review and feedback.

W1: Fixed guidance strength for PARM. Is DIR simply using stronger guidance on average?

The used guidance strength followed the published PARM evaluation protocol, which sets the strength λ=1\lambda=1 under log⁡p0+λlog⁡pα\log p_0+\lambda\log p_{\boldsymbol{\alpha}} for both PARM and its GenARM baseline.

To distinguish the benefit of token-wise adaptation from the choice of overall guidance strength, we conducted two additional controlled experiments.

  • Fixed-coefficient ablation. In our paper, we use an equivalent form (1−μ)log⁡p0+μlog⁡pα(1-\mu)\log p_0+\mu\log p_{\boldsymbol{\alpha}}, so setting λ=1\lambda=1 is equivalent to setting μ=0.5\mu=0.5. On PKU 7B, we swept five fixed ARM weights for PARM across all 11 preferences on the 500-prompt validation set. We also checked the PARM-only endpoint on the same validation set.

    Fixed ARM weight μ\muValidation MIP
    0.10-1.511
    0.250.131
    0.502.906
    0.754.649
    0.904.894
    1.00 (PARM only, additional validation check)4.584

    We selected μ=0.90\mu=0.90 by validation MIP over the five sweep weights and used it at all 11 preferences on the original 1,500-prompt held-out test set.

    Notably, although larger μ\mu generally improves validation MIP, overly strong ARM weighting substantially degrades generation quality: 11.84% and 28.24% of validation responses exhibit severe repetition loops at μ=0.90\mu=0.90 and μ=1.00\mu=1.00, respectively.

  • Mean-coefficient-matched control. For each of the 11 PKU 7B test preferences, we computed the token-weighted mean of utu_t across the original DIR test trajectories at that preference. We used this mean as the fixed guidance coefficient for PARM across all prompts and decoding steps at the same preference. The coefficients in this post-hoc control were determined from the recorded DIR trajectories without using test rewards.

    We also ran a mean-matched control on the 18 three-objective HH preferences. At each preference, we set PARM’s fixed coefficient to the token-weighted mean of DIR’s ARM weight. Mean-matched PARM achieved 72.56 HV and 1.011 MIP, while DIR achieved 94.02 HV and 1.381 MIP. The paired differences were +21.46 HV (95% CI [19.87, 23.04]) and +0.370 MIP (95% CI [0.346, 0.393]).

PKU 7B methodHVMIP
Original PARM (μ=0.5\mu=0.5)111.362.620
PARM matched to DIR’s mean at each preference117.843.154
Validation-selected PARM (μ=0.90\mu=0.90)136.884.682
Original DIR159.785.831

DIR remains ahead of both PARM controls on the held-out PKU 7B test set. The validation-tuned constant addresses whether the original baseline was under-tuned, while the mean-matched constant tests justify that DIR’s improvement is not from a larger average guidance coefficient.

AuthorsOpenReview ↗

Response to Reviewer ojgi (2/2)

Q1: How strong is DIR’s guidance on average for each preference and setting?

For each test preference, we report the token-weighted mean of DIR’s ARM weight, defined as the coefficient on log⁡pα\log p_{\boldsymbol{\alpha}} in Eq. (11). We provide per-preference values for five settings and summarize their range, mean, and population variance across preferences, giving each preference equal weight in the summary.

SettingPreferencesRangeMeanVariance
PKU 7B110.489–0.5750.5288.20e−4
PKU 65B110.477–0.5650.5129.10e−4
HH transfer110.489–0.5010.4931.50e−5
PKU three-objective (PARM ARM)180.444–0.5270.4824.21e−4
HH three-objective180.317–0.5140.3952.73e−3

Two-objective settings. The helpfulness weight αhelp\alpha_{\mathrm{help}} indexes each preference. Each column reports the mean ARM weight.

αhelp\alpha_{\mathrm{help}}PKU 7BPKU 65BHH transfer
0.00.57460.56550.4927
0.10.55980.55310.4902
0.20.54930.54230.4888
0.30.55320.52090.4893
0.40.51960.48700.4885
0.50.50900.49540.4903
0.60.50550.48500.5012
0.70.49670.47690.4965
0.80.48890.47980.4942
0.90.49980.49540.4971
1.00.55090.52880.4936

Three-objective settings. Preference weights are ordered as (helpfulness, harmlessness, humor). Each column reports the mean ARM weight.

PreferencePKU (PARM ARM)HH
(0.0, 0.0, 1.0)0.45030.3191
(0.0, 0.2, 0.8)0.44360.3761
(0.0, 0.4, 0.6)0.47530.4377
(0.0, 0.6, 0.4)0.48300.5138
(0.0, 0.8, 0.2)0.49360.4518
(0.0, 1.0, 0.0)0.52740.4003
(0.2, 0.0, 0.8)0.46160.3174
(0.2, 0.4, 0.4)0.48120.3898
(0.2, 0.8, 0.0)0.50730.4254
(0.4, 0.0, 0.6)0.47410.3638
(0.4, 0.2, 0.4)0.46450.3998
(0.4, 0.4, 0.2)0.46950.4419
(0.4, 0.6, 0.0)0.50290.4609
(0.6, 0.0, 0.4)0.48360.3498
(0.6, 0.4, 0.0)0.48820.4182
(0.8, 0.0, 0.2)0.50160.3484
(0.8, 0.2, 0.0)0.47410.3616
(1.0, 0.0, 0.0)0.50040.3366

On PKU 7B, DIR’s mean ARM weight was below PARM’s fixed weight of 0.5 at helpfulness weights of 0.7, 0.8, and 0.9. At these preferences, DIR improved the preference-weighted reward over PARM by 1.550, 2.089, and 2.467, respectively. On HH transfer, DIR’s mean ARM weight across preferences was 0.493. DIR achieved 7.127 HV and 0.949 MIP, compared with 6.063 HV and 0.757 MIP for PARM. On three-objective HH, DIR’s mean ARM weight was below PARM’s default weight of 0.5 at 17 of the 18 preferences.

Q2. How much does it vary across tokens?

We reran DIR on PKU 7B with token-level logging, generating 1,500 responses at each of the 11 test preferences. We computed the standard deviation of utu_t within each response and took the median across responses at each preference. These medians ranged from 0.233 to 0.242.

Q3. How does it change as the response goes on?

Using the same runs, we grouped tokens by their relative position within each response. The table reports the mean utu_t in each decile, pooling tokens across responses and all 11 preferences.

Relative position in the responseMean utu_t
0–10%0.700
10–20%0.634
20–30%0.572
30–40%0.527
40–50%0.496
50–60%0.476
60–70%0.461
70–80%0.447
80–90%0.435
90–100%0.415

The first-decile mean was higher than the last-decile mean at all 11 preferences.

Thank you for the additional experiments. The tuned PARM comparisons answer my question. The methodological novelty remains limited, and the stronger guidance early in the response is consistent with prior safety work showing the importance of initial tokens. That said, DIR still outperforms the new added tuned PARM baseline, which strengthens the empirical support for the method. I will increase my score to 3 and keep the same confidence.

Thank you for your reassessment and for acknowledging that the tuned PARM comparisons address your concern.

Methodological contribution. To our knowledge, this is the first work to validate disagreement among existing objective-conditioned ARMs as a token-wise guidance signal through controlled token-level interventions. Beyond dynamically adjusting token-level guidance, we also want to highlight our introduction and characterization of cross-reward forks—token-level decision points at which different objective-conditioned local choices lead to meaningfully different downstream reward outcomes. This concept makes where multi-objective guidance matters within a response a concrete research question, which provides a basis for future work on multi-objective guidance.

Controlling for token position. Following your observation about early tokens, we extended the analysis in Table 1 by controlling for token position in the PKU 7B Help–Safety setting (64 prompts and 1,241 intervention histories). Within the same prompt and relative-position band (ten bands per response), ARM JS remains associated with downstream fork strength, with a position-controlled rank correlation of 0.726 and a 95% CI of [0.688, 0.759], compared with 0.479 for Base entropy and 0.564 for mean ARM entropy. This supports its predictive value beyond the overall early-to-late positional trend.

If any concern remains, please let us know, and we will be glad to address it before the discussion closes.

Reviewer_Cb2KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when objective-specific autoregressive reward models (ARMs) should influence a frozen language model during multi-objective test-time alignment. It proposes using normalized generalized Jensen–Shannon disagreement among ARM distributions as a token-level routing signal, separately from the user's preference vector. Disagreement has a mutual-information interpretation and is mapped through an empirical calibration CDF to determine the strength of base–ARM interpolation. An additional correction handles objectives already represented by the base model. The evidence combines counterfactual token interventions, budgeted online interventions, and response-level alignment experiments. The appendices extend the diagnostics to distributional interventions and full responses, and compare JS routing with entropy and random routing. Evaluations cover two- and three-objective settings, 7B-to-65B guidance, cross-domain transfer without recalibration, an external LLM quality judge, and a controlled inference-time benchmark.

Critical strengths
  1. The paper gives useful evidence on where multi-objective guidance matters. It draws a clear, operational distinction between within-model uncertainty and disagreement across objective-conditioned models. The evidence for the contribution goes beyond the illustrative token examples. Beyond the initial greedy-token diagnostics, Appendix B.6 tests distributional interventions and includes argmax disagreement as a control. Appendix B.7 shows that the preference separation from early budgeted interventions persists when the base completes the response. Appendix I further tests the signal inside the actual soft-routing decoder. Taken together, these analyses make disagreement a credible empirical routing signal, and the mutual-information identity supplies its local interpretation.
  2. The decoding rule is simple and reusable, and it holds up across several controlled settings. DIR trains no additional router and reuses the baseline guidance checkpoints. The matched three-objective comparisons in Appendix G are especially useful: they hold the base and guidance models fixed for both PARM-backed and GenARM-backed DIR. The transfer and 65B experiments add breadth, and the LLM-judged evaluation offers a check beyond the reward models used to build training preferences. The measured inference overhead is also reported transparently: Table 10 shows that in the three-objective benchmark, DIR takes 38.97 seconds for 512 tokens versus PARM's 35.36 seconds.
Critical weaknesses
  1. The advantage over a well-chosen constant guidance strength is not yet sufficiently isolated. Appendices F.1 and F.2 fix the relevant baseline guidance coefficients at λ = 1. Rank calibration gives a nominal mean-strength match near 0.5 on calibration histories, but Appendix F.4 explicitly notes that its mean can shift during on-policy test generation. The manuscript does not report a validation-selected constant-strength sweep or the realized routing means. Appendix I is an important control and does support JS over entropy and random routing, but it does not establish how much of the improvement over PARM or GenARM survives optimization of their constant strength. This point matters: the central algorithmic claim concerns the benefit of allocating guidance across tokens, and the reported gain over an optimized constant decoder could be smaller.
  2. Uncertainty is incomplete for the decisive routing-signal comparisons. The paper correctly notes that histories are nested within prompts, and Appendix B.1 explains this nesting. It also uses prompt-separated prediction folds and provides bootstrap intervals for the LLM-judge results. However, the full-response budgeted comparisons in Table 7 and the JS-versus-entropy routing comparison in Table 12 have no prompt-level uncertainty estimates. The random-routing standard deviations in Table 12 describe variation across three routing seeds; they do not describe uncertainty in the paired advantage over the prompt population. These comparisons are central to the proposed signal. The point estimates are encouraging, but without these intervals the precision of the advantages remains unclear.
Questions for authors
  1. Clarification and a limited comparison corresponding to weakness 1. Was λ = 1 selected using validation performance or inherited as a fixed protocol? If no strength sweep was performed, could you compare DIR with a small validation-selected sweep of constant PARM strengths on the PKU 7B task, using the existing checkpoints and the same test preferences? Please also report the realized mean routing coefficient by preference. A single validation-selected constant used across preferences would already be informative; no new model training is needed. A persistent advantage would strengthen the attribution to token-wise routing.

  2. Uncertainty analysis corresponding to weakness 2. Could you report paired prompt-level bootstrap intervals for the budget-4 JS-minus-control differences in Table 7 and the JS-minus-base-entropy HV/MIP differences in Table 12? For Table 7, the uniform and argmax-disagreement controls are especially informative. For Table 12, each resampled prompt should retain its responses under all five preferences so that HV and MIP are recomputed on the same resample. This would clarify whether the signal advantage remains persuasive under prompt sampling, using the existing outputs.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Cb2K (1/2)

Thank you for the detailed assessment.

W1&Q1-1: Constant guidance strength and attribution to token-wise routing.

The used guidance strength followed the published PARM evaluation protocol (Lin et al., 2025, §§5.1–5.2 [1]), which sets the strength λ=1\lambda=1 under log⁡p0+λlog⁡pα\log p_0+\lambda\log p_{\boldsymbol{\alpha}} for both PARM and its GenARM baseline. We adopted this setting for default comparison.

Fixed-coefficient ablation. In our paper, we use (1−μ)log⁡p0+μlog⁡pα(1-\mu)\log p_0+\mu\log p_{\boldsymbol{\alpha}}, so setting λ=1\lambda=1 is equivalent to setting μ=0.5\mu=0.5. On PKU 7B, we swept five fixed ARM weights for PARM on the 500-prompt validation set, keeping the Base, PARM checkpoint, decoding procedure, and 11 preference weights fixed across the sweep. We selected one coefficient shared across all preferences by maximizing validation MIP over those five weights.

Fixed ARM weight μ\muValidation MIP
0.10-1.511
0.250.131
0.502.906
0.754.649
0.904.894
1.00 (PARM only, additional validation check)4.584

We selected μ=0.90\mu=0.90 by validation MIP over the five sweep weights and used it at all 11 preferences on the original 1,500-prompt held-out test set.

Notably, although larger μ\mu generally improves validation MIP, overly strong ARM weighting substantially degrades generation quality: 11.84% and 28.24% of validation responses exhibit severe repetition loops at μ=0.90\mu=0.90 and μ=1.00\mu=1.00, respectively.

Mean-coefficient-matched control. For each of the 11 PKU 7B preferences, we computed the token-weighted mean utu_t across the original DIR test trajectories at that preference. We then used that mean as the fixed coefficient for PARM across all prompts and decoding steps at the same preference. This post-hoc control uses the recorded routing coefficients to set the fixed strength and retains PARM decoding.

PKU 7B methodHVMIP
Original PARM (μ=0.5\mu=0.5)111.362.620
PARM matched to DIR’s mean at each preference117.843.154
Validation-selected PARM (μ=0.90\mu=0.90)136.884.682
Original DIR159.785.831

Both controls improve on the original PARM baseline. DIR remains ahead of validation-selected PARM by 22.90 HV and 1.148 MIP, and of mean-matched PARM by 41.94 HV and 2.676 MIP.

The validation-tuned constant addresses whether the original baseline was under-tuned, while the mean-matched constant tests justify that DIR’s improvement is not from a larger average guidance coefficient.

Q1-2: Report the realized mean routing coefficient by preference.

For each test preference, we report the token-weighted mean of DIR’s realized routing coefficient utu_t before the coefficient normalization in Eq. (11). The summary below gives the range, mean, and population variance across preferences, weighting each preference equally.

SettingPreferencesRangeMeanVariance
PKU 7B110.489–0.5750.5288.20e−4
PKU 65B110.477–0.5650.5129.10e−4
HH transfer110.489–0.5010.4931.50e−5
PKU three-objective (PARM ARM)180.444–0.5270.4824.21e−4
HH three-objective180.319–0.5970.4806.54e−3

Two-objective PKU. The helpfulness weight αhelp\alpha_{\mathrm{help}} indexes each preference.

αhelp\alpha_{\mathrm{help}}7B mean utu_t65B mean utu_t
0.00.57460.5655
0.10.55980.5531
0.20.54930.5423
0.30.55320.5209
0.40.51960.4870
0.50.50900.4954
0.60.50550.4850
0.70.49670.4769
0.80.48890.4798
0.90.49980.4954
1.00.55090.5288

Three-objective HH. Preference weights are ordered as (helpfulness, harmlessness, humor).

PreferenceMean routing coefficient utu_t
(0.0, 0.0, 1.0)0.3191
(0.0, 0.2, 0.8)0.3761
(0.0, 0.4, 0.6)0.4377
(0.0, 0.6, 0.4)0.5138
(0.0, 0.8, 0.2)0.4518
(0.0, 1.0, 0.0)0.4003
(0.2, 0.0, 0.8)0.3514
(0.2, 0.4, 0.4)0.4377
(0.2, 0.8, 0.0)0.4788
(0.4, 0.0, 0.6)0.4561
(0.4, 0.2, 0.4)0.5050
(0.4, 0.4, 0.2)0.5645
(0.4, 0.6, 0.0)0.5900
(0.6, 0.0, 0.4)0.4864
(0.6, 0.4, 0.0)0.5966
(0.8, 0.0, 0.2)0.5386
(0.8, 0.2, 0.0)0.5625
(1.0, 0.0, 0.0)0.5711
AuthorsOpenReview ↗

Response to Reviewer Cb2K (2/2)

W2 & Q2 — Paired prompt-level bootstrap intervals

For Table 7 in the original submission (Table 9 in the revised manuscript), we used 10,000 paired prompt-bootstrap draws over the 128 prompts at the full-response intervention budget of four. Each draw used the same sampled prompts for JS and all controls. We report the JS-minus-control differences in directed preference separation.

Intervention comparatorJS − comparatorPaired 95% CI
Uniform+0.219[0.026, 0.425]
Argmax disagreement+0.295[0.118, 0.487]
Random+0.548[0.351, 0.758]
Base entropy+0.512[0.280, 0.758]
Mean ARM entropy+0.705[0.472, 0.949]

For Table 12 in the original submission (Table 15 in the revised manuscript), we used 10,000 paired prompt-bootstrap draws, keeping repeated prompts together. Each draw used the same sampled prompts for both methods at all five preferences. We recomputed each preference’s mean reward vector, followed by HV and MIP.

MetricJSBase entropyJS − Base entropyPaired 95% CI
HV130.58110.13+20.45[16.39, 24.50]
MIP7.0155.432+1.583[1.408, 1.762]

Reference [1] Lin, B., Jiang, W., Xu, Y., Chen, H., & Chen, Y. C. (2025). PARM: Multi-objective test-time alignment via preference-aware autoregressive reward model. ICML.

Thank you for the additional experiments, which substantially address my original concerns about guidance strength and statistical uncertainty.

One comparison remains incomplete. The validation-selected PARM baseline is evaluated only through HV/MIP. Given the repetition issues noted in your response, comparable test-set repetition rates or an LLM-quality evaluation would help establish whether DIR’s advantage over this stronger baseline extends beyond reward scores. These results are not provided in the response.

I acknowledge the remaining reward improvements. However, I still view the use of calibrated ARM disagreement to control token-wise guidance strength as a relatively modest methodological extension. The additional evidence strengthens the effectiveness claim, but does not sufficiently change my assessment of the contribution’s significance. I therefore maintain my rating of 2.

Reviewer_azrhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Disagreement-Informed Routing (DIR), a training-free decoding rule for multi-objective test-time alignment. DIR measures disagreement among objective-conditioned autoregressive reward model (ARM) next-token distributions via normalized generalized Jensen–Shannon divergence, interprets this as a token-level reward-critical signal, and uses a calibrated rank of this signal to modulate how strongly the base policy is moved along a user-specified preference direction. The method is evaluated on PKU safety alignment (7B and 65B), zero-shot HH-RLHF transfer, three-objective helpful-assistant control, LLM-judged overall quality, and inference efficiency, with reported gains over PARM, GenARM, MOD, RS, and CAGE.

Critical strengths
  1. The paper distinguishes reward-specific sensitivity from ordinary predictive uncertainty and operationalizes cross-reward forks at the token level.
  2. The paper focus on the normalized generalized JS divergence has a clean information-theoretic interpretation and is independent of the requested preference vector, which is conceptually appealing.
  3. The experiments reports strong gains over strong baselines: e.g., large HV/MIP improvements over PARM/GenARM and higher overall win rates at both 7B and 65B.
Critical weaknesses
  1. Novelty relative to existing manifold methods QPA largely applies MGDA on a fixed-rank manifold with truncated-SVD retraction; SPA also uses standard geometric tools. The paper should clarify what is new beyond these constructions. A simpler baseline using balanced factors, raw MGDA, and SGD would help. AdamW + BaLoRA does not replace this comparison because its adaptive state introduces additional differences.
  2. The source of practical gains remains unclear The conflict-flip and trajectory experiments establish representation sensitivity, but sensitivity alone does not imply poor task performance. QPA and SPA change the metric, execution, and state handling together, so Table 2 does not isolate the benefit of representation correctness. The impact of plausible factor scales and condition numbers on final performance also remains unclear. Since conflict judgments depend on the metric, representation invariance alone does not establish a preferred trade-off.
  3. Limited scalarization baseline and cost analysis According to Appendix F.1, Weighted DPO uses different warm-start weights but equal objective weights throughout formal training. Its seven endpoints therefore do not represent a full weight sweep. Retaining different weights during formal training would provide a stronger comparison of hypervolume and trade-off coverage under matched budgets. Training time, peak memory, and rejection or backtracking rates would also help assess practical cost.
Questions for authors
  1. What does QPA add beyond standard Riemannian MGDA on the fixed-rank manifold, particularly in gradient recovery, state handling, and implementation?
  2. Could you compare against MGDA + SGD with balanced factors and rebalancing after each step?
  3. How much do equivalent initial representations affect final helpfulness, safety, and hypervolume under plausible factor scales and condition numbers? Can this be separated from the effect of changing geometry?
  4. Could you include a Weighted DPO weight sweep during formal training and report each method's hyperparameter search budget and selection rule?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

We thank the reviewers for their helpful comments. The table summarizes the six main questions, our responses, and where to find the corresponding changes. Additions to the revised manuscript are marked in red.

Main questionRaised byShort answerManuscript revision
Does DIR still outperform PARM after tuning or matching guidance strength?Cb2K W1/Q1, ojgi W1, QomW W4Yes. On PKU 7B, DIR achieves 159.78 HV / 5.831 MIP versus 136.88 / 4.682 for validation-tuned PARM. DIR also outperforms mean-matched PARM on PKU 7B and three-objective Helpful Assistant.Section 5.4 and Table 6 add the fixed-strength comparisons. Appendix J gives the validation sweep.
How strong is DIR’s guidance at test time, and how does it vary?Cb2K Q1, ojgi Q1–Q3DIR’s guidance varies substantially across tokens and over generation. On two-objective PKU 7B, DIR’s mean ARM weight is 0.528 across the 11 test preferences. Median within-response SDs across preferences are 0.233–0.242. The pooled mean drops from 0.700 to 0.415 between the first and last deciles.Appendix J adds the PKU 7B token-wise analysis.
Are the routing-signal gains robust to prompt resampling?Cb2K W2/Q2Yes. Paired 95% bootstrap intervals are above zero for all five intervention comparisons and for both HV and MIP gains over Base entropy.We added paired 95% bootstrap confidence intervals for the full-response intervention in Table 10 and for the routing-signal ablation in Appendix I.
Why use JS percentiles?QomW W2Percentiles provide a scale-free routing strength while preserving the ordering of disagreement. Under a continuous calibration distribution, they are uniform with mean 0.5.The W2 reply expands on the rationale in Section 4.1.
What is the inference overhead, and how can it be reduced?QomW W1The shared PARM ARM batches endpoint and target-preference queries. In three-objective Helpful Assistant, DIR adds 10.2% generation time and 0.052 GiB mean peak allocated memory over PARM.Table 13 adds the GPU memory measurements.
Does DIR remain effective and efficient with four objectives?QomW W3Yes. On four-objective PKU 7B, DIR improves HV/MIP by 31.17%/8.05% over PARM and 32.21%/13.17% over GenARM. At the uniform preference, time increases by 13.3% and 0.2%, respectively.Section 5.5 and Table 7 add the four-objective results. Appendix K gives the protocol, confidence intervals, Pareto plots, and inference costs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.