acceptodds
Under review as a conference paper at ICLR 2027

Ciallo: Correlation-Aware Multi-Objective Alignment for Large Language Models

Abstract

Aligning large language models (LLMs) with diverse human preferences requires balancing multiple objectives that may be compatible or conflicting in different scenarios. However, existing multi-objective alignment methods often optimize different objectives independently, ignoring their correlated preference patterns and conflicting constraints. This oversight prevents the model from reusing shared information across compatible objectives and leveraging contrastive signals from conflicting ones, thereby reducing training efficiency. Motivated by this consideration, we propose Ciallo, a framework that first conducts separated training to learn an initial soft prompt representation for each alignment objective, and then performs joint training under mixed preference weights, where a dynamically tracked Fisher-gradient affinity matrix constrains objective relationships and guides Group Relative Policy Optimization (GRPO) through a Fisher Gram-matching regularizer. Experiments on Reddit Summary with Qwen3-0.6B and Helpful Assistant with Qwen3-4B show that Ciallo consistently achieves macro-averaged gains of and reward points over Reward Soups.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM alignment, Multi-objective alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RqyuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose CIALLO, a two-stage framework for multi-objective LLM alignment that first learns objective-specific soft-prompt controllers and then jointly optimizes the prompts under mixed preference weights, especially for tackling correlational objectives, e.g., harmlessness-helpfulness that may be conflicted with each other. The new method part is Fisher-preconditioned gradient affinities, which regularizes the controller geometry through Fisher Gram matching, with the goal of exploiting compatible objectives while separating conflicting ones. Experiments on Reddit Summary and Helpful dataset with Qwen3 models report improved empirical pareto frontiers over existing baselines.

Critical strengths
  • The method is tackling a practical and important problem, especially for the objectives that are conflicting with each other.
  • There are some ablations towards specific module, parameters to demonstrate the effectiveness.
  • The implementation details are transparent.
Critical weaknesses
  • Novelty and Contribution

The author positioned their paper for solving the correlational relationship between multi-objectives, especially for potentially conflicting objectives, but all the baseline compared to are non-conflicting aware. Standard conflicting-aware MOA methods [1,2] are fully neglected without any comparisons or discussions. What's the advantage of your correlation-aware method for conflicting objectives comparing to those correlation-processing methods remain uncertain.


  • Soundness

First, the method lacks theoretical justification for the Pareto optimality, which is the most important guarantee for multi-objective learning methods. Even though you admit that it only gives local and conditional results, but people from multi-objective learning communities would definitely treat this a big gap for lacking the explanation to your empirical performance that overweights the other method.

The author also didn't a include a systematic ablation to show those component's contribution to the final Pareto frontier. Also, maybe I missed something, but what is "Offset Value"? I didn't find it appeared or defined anywhere in the manuscript. There are also many parameters introduced, are they robust? I only find the ablations for Gram-matching coefficient in the appendix, what about the others, e.g., αC\alpha_C.

Another ablation missing is the core Fisher part, what if you just replace it with some simple baseline, e.g., cosine? So I guess you didn't convincingly establish that Fisher geometry is necessary.

The experiment also needs to be strengthened, as there's only one model family evaluated, whether the results can be extended to further model families (e.g., Llama, gemma) remains unjustified. The scalability to higher-dimensional objective setups remain uncertain. As all the experiments presented are just 2-objective case.


  • Presentation

I think the authors need to spend more time on improving the quality of manuscript. There exists several very obvious mistakes, e.g., line 135-136 with missing citations as "?" and Fig 3 with all the numbers overlapped with each other. The math notation is not professional, especially for the lack of proper usage of \mathrm of the specialized notations, e.g., "KL".


Reference

[1] Reward-free Alignment for Conflicting Objectives. ICML 2026, https://openreview.net/forum?id=vSzRJyg6k0

[2] MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment. AFAA 2026, https://openreview.net/forum?id=yNlF777Uh9

Questions for authors

See weakness section

  1. Resolve the concern on novelty and contribution.

  2. Resolve the concerns and questions over theoretical gap, ablation details, model-family setup and scalability.

  3. Please go over the manuscript again to fix all the presentation issues.


My assessment for the current manuscript is between clear reject and weakly reject; if there's a score of 1.5 this year, I will definitely put it, but now, I will just put it as 2 since the manuscript should be above 1.

4 comments

We thank the reviewer for identifying the missing conflict-aware comparisons and for the detailed questions on theory and evaluation. We address contribution and theoretical scope here, and report the ablations and additional evaluation in the accompanying response.

1. Contribution relative to conflict-aware alignment.

We agree that RACO and MGDA-Decoupled are relevant prior work. RACO applies clipped conflict-averse updates to objective-specific preference losses; MGDA-Decoupled computes mixing coefficients from loss-normalized gradients and applies them to raw gradients. Both explicitly handle objective geometry. We have added a discussion of both methods in Sec. 2 and clarified their distinction from Ciallo.

Ciallo's contribution is the complete relationship-guided controller-learning framework: (i) objective-specific soft-prompt initialization with a frozen backbone; (ii) mixed-preference training of the controller bank and policy; and (iii) Fisher Gram matching, which makes local signed objective-gradient relationships an explicit learning target for controller geometry.

The practical distinction is preference adjustment within one trained checkpoint. RACO's Algorithm 1 takes user weights as inputs to policy optimization; changing those optimization weights does not itself condition an already trained checkpoint. MGDA-Decoupled learns an objective-balancing policy through gradient combination. Ciallo instead learns controllers mixed as s(w)=∑iwisis(w)=\sum_i w_i s_i, so a deployed chatbot can change its helpfulness/harmlessness/humor trade-off as the user's preferences change, without another optimization run. The same checkpoint serves these preferences. This capability motivates learning the controller geometry in addition to resolving training-time conflicts.

2. Theoretical scope and empirical evidence.

We agree that a convergence or Pareto-optimality guarantee would strengthen the paper. Our theoretical claim concerns local cross-objective transfer and controller geometry; our performance claim concerns improved empirical preference-conditioned trade-offs. The former motivates the construction, while the latter is assessed directly through controlled ablations and evaluation across preferences.

For differentiable local losses with gradients gig_i and a fixed positive-definite metric DD, define Hij=giTD−1gjH_{ij}=g_i^T D^{-1}g_j. Under the candidate step p=−D−1∑iaigip=-D^{-1}\sum_i a_i g_i, the first-order loss changes are gjTp=−(Ha)jg_j^T p=-(Ha)_j. Thus Ha>0Ha>0 componentwise certifies a common local descent direction. Fisher affinity records the normalized entries of this transfer matrix; Appendix F.1 (Proposition 1) gives the local-transfer identity, and Appendix F.2 (Proposition 3) establishes Gram consistency under ideal natural-gradient controllers. The common-descent diagnostic above is a clarification in this response; Appendix F.5 states the scope of the paper's analysis.

The empirical evidence also bears directly on the method's contribution. Figures 4–6 in Sec. 5.3 examine the training stages, Gram regularizer, and dynamic affinity target: removing or replacing these components degrades the observed trade-offs. The Fisher/cosine ablation further isolates the metric, with a mean preference-weighted reward gain of 0.0700 over the complete 11-preference sweep. The Llama-3.1-8B-Instruct experiment uses LoRA for Ciallo's stage-two policy adaptation and extends the evaluation to another model family and three simultaneous objectives, with mean gain 1.1717 over PARM across 13 measured preferences. The accompanying response provides the numerical results; the complete reward-space plots are in Appendix A.3–A.4 (Figs. 9–10).

Together, these studies support both the proposed design and its practical benefit in the evaluated settings. We believe this constitutes a substantive methodological and empirical contribution, even though a general convergence guarantee remains open. We therefore frame the results as improved empirical trade-offs, without claiming globally Pareto-optimal policies or treating a first-order stationarity condition as sufficient for optimality.

3. Component ablations, Fisher geometry, and "Offset Value."

The ablations in Secs. 5.3.1–5.3.3 isolate different components: Fig. 4 compares stage-one controllers alone, a frozen backbone, and frozen controllers against joint training; Fig. 5 removes Gram matching; Fig. 6 replaces the dynamic affinity target with fixed random/all-ones targets. These test the training stages, regularizer, and relationship target, respectively. The additional Fisher/cosine comparison tests the metric.

In Figs. 4–6, "Offset Value" means ΔRi(w)=Rivariant(w)−Rifull(w)\Delta R_i(w)=R_i^{\mathrm{variant}}(w)-R_i^{\mathrm{full}}(w): the reward difference at the same preference, in reward units. Negative values indicate degradation; zero is the full-framework reference. We provide this definition here to clarify the plotted differences.

Fisher versus ordinary cosine (D=ID=I), Helpful Assistant. Scores are mean rewards; u(w)=wTR(w)u(w)=w^T R(w) and Δu is Fisher minus cosine. Tuples follow (harmless, helpful).

wFisher rewardsCosine rewardsΔu
(0.0, 1.0)(1.1066, -0.1291)(1.0410, -0.1540)+0.0249
(0.2, 0.8)(1.1428, -0.1669)(1.0530, -0.1700)+0.0204
(0.4, 0.6)(1.1750, -0.2343)(1.1510, -0.2110)-0.0044
(0.6, 0.4)(1.3789, -0.2688)(1.2710, -0.3010)+0.0776
(0.8, 0.2)(1.5848, -0.3291)(1.4290, -0.3060)+0.1200
(1.0, 0.0)(1.6396, -0.3292)(1.4560, -0.3130)+0.1836

Fisher gives a more favorable overall empirical trade-off curve, with mean Δu=0.0700 across all 11 evaluated preferences. This supports its contribution in the tested setting, rather than universal necessity. See Appendix A.3 and Fig. 9 for the complete curves.

4. Hyperparameter sensitivity.

Beyond the Gram-weight sweep in Appendix A.1 (Fig. 7), λ∈{0.1,0.5,1,5}\lambda\in\{0.1,0.5,1,5\}, we vary the affinity EMA rate αC∈{0,0.05,0.5,1}\alpha_C\in\{0,0.05,0.5,1\} on both original tasks (Appendix A.2, Fig. 8). Zero disables subsequent target updates; one removes temporal smoothing. On Helpful Assistant, the alternatives reduce harmlessness by 0.038–0.189 at the six displayed preferences, with mixed helpfulness changes. On Summary, they redistribute summary/faithfulness rewards rather than uniformly improving both. These results support retaining the smoothed default 0.05 while showing that this rate affects the trade-off.

Additional group-size and stage-two mini-batch sweeps on Summary show similar overall trade-off curves: mean u is 1.5008/1.5220/1.5378 for G=4/8/16 and 1.5220/1.5735 for mini-batch sizes 1/2. We claim limited final-performance sensitivity within these tested ranges. Appendix A.7 (Figs. 13–14) reports the group/batch curves. These results do not imply robustness of every untested hyperparameter.

5. Another model family and simultaneous three-objective control.

We additionally evaluate Ciallo and PARM with the same Llama-3.1-8B-Instruct backbone on Helpful Assistant, jointly controlling harmlessness, helpfulness, and humor. Ciallo uses LoRA for stage-two policy adaptation (Appendix A.4). There are 13 measured configurations, including 10 with all three weights positive. The table shows the nine measured configurations on the 0.2 grid; no missing configurations are interpolated.

w (harmless, helpful, humor)Ciallo uPARM uΔu
(1.0, 0.0, 0.0)3.30411.4233+1.8808
(0.6, 0.2, 0.2)1.92610.4620+1.4641
(0.4, 0.4, 0.2)0.3625-0.0802+0.4427
(0.4, 0.2, 0.4)0.82980.2913+0.5385
(0.2, 0.6, 0.2)0.7223-0.4370+1.1593
(0.2, 0.4, 0.4)0.6558-0.1801+0.8359
(0.2, 0.2, 0.6)0.85940.1918+0.6676
(0.0, 1.0, 0.0)1.5170-0.9253+2.4423
(0.0, 0.0, 1.0)1.56990.5591+1.0108

Mean Δu over all 13 preferences is 1.1717; at equal weights, u is 0.3284 versus -0.0163. See Appendix A.4 and Fig. 10 for the reward-space plot. This extends the evidence to another family, an 8B model, and three simultaneous objectives; larger objective counts remain untested.

6. Presentation fixes. The unresolved citations in Sec. 2 have been corrected to Liu et al. (2025), Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models, and Anonymous (2027), Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment. We have resolved the overlapping numerical labels in Fig. 3 by reducing their font size and removing selected overlapping labels. We have also standardized the mathematical notation and formatting.

I try to read your updated manuscript, and I found myself pretty hard to follow it. Since there's no pdf diff, I can't tell which part is newly added and which is the original one.

I will just raise several examples here:

Regarding your 3-objective experiments,

  • "it does not establish many-objective or arbitrary-scale generalization" -- what do you mean by this? do you mean that the formulation cannot be generalized to more general objective case?

  • "There are 13 measured configurations" and "nine on the 0.2 simplex grid. -- What is 13 measured configurations, could you elaborated it?

I also found it a bit hard to follow the result in Appendix F; there are too many notations jump out, and the flow of the result is relatively hard to interpret.

  • Is eq 24 an assumption? what is cic_i?

  • "Proposition 3 is a conditional consistency result, not a recovery or convergence theorem"

  • "but does not imply that its minimum is attained or that minimizing it alone improves the policy objective" -- I am even more confused with this.

I think it would be better to write the manuscript yourself to ensure the content is understandable and with better quality. I believe given the current quality of manuscript, I don't have a clear motivation to recommend acceptance. But if there's any reviewer who has better understanding of the result would like champion this work, I have no objection for that.

Thank you for identifying these unclear passages. We have revised them in the manuscript and prepared a marked-up version with the revised passages highlighted in blue, making the changes easier to identify.

Three-objective experiment. The weights are ordered as (harmlessness, helpfulness, humor). The 13 measured vectors comprise the single-objective vectors (1, 0, 0), (0, 1, 0), and (0, 0, 1); six mixed vectors on the 0.2 simplex grid—(0.6, 0.2, 0.2), (0.4, 0.4, 0.2), (0.4, 0.2, 0.4), (0.2, 0.6, 0.2), (0.2, 0.4, 0.4), and (0.2, 0.2, 0.6)—and four additional vectors: (0.8, 0.1, 0.1), (0.1, 0.8, 0.1), (0.1, 0.1, 0.8), and (1/3, 1/3, 1/3). We have replaced the broad statement about “many-objective or arbitrary-scale generalization” with: “This experiment evaluates three-objective control with Llama-3.1-8B-Instruct; larger objective sets remain to be evaluated.”

Appendix F. Equation (24) is an assumption of Proposition 3, not a condition imposed by the training algorithm. It considers an idealized controller direction di∗d_i^* that follows the natural-gradient direction −M−1gi-M^{-1}g_i at a shared local point and under a shared positive-definite metric MM. Here, gig_i is the local gradient of objective ii, and ci>0c_i>0 is an arbitrary scale that cancels when the Gram matrices are normalized. Under this assumption, Proposition 3 shows that the normalized controller Gram equals the normalized Fisher-gradient affinity.

We have replaced the qualifications around Proposition 3 with a direct explanation: the identity motivates matching the two geometries, and the component ablations in Sec. 5.3 evaluate the regularizer's effect on policy rewards. We also removed an auxiliary paragraph with additional normalized-direction notation to make the result easier to follow.

Reviewer_d8PEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Ciallo, a multi-objective alignment algorithm which uses soft prompt vectors to condition an LLM on objective weights and awareness of agreement or interference between per-objective gradients during fine-tuning to optimize the soft prompt vectors around the model's local sensitivity. The algorithm runs in two stages: first, a separate soft prompt vector for each objective is optimized independently under a DPO objective with the LLM weights frozen; second, GRPO is applied with objective weights sampled from a Dirichlet distribution to jointly fine-tune the model and all of the soft prompt vectors. The soft prompt vectors are updated via a combination of the GRPO loss (backpropagated through the LLM weights) and a loss based on the local sensitivity approximated using the Fisher-aware affinity matrix. The key contributions are the algorithm itself, the theoretical analysis of Fisher-aware relationship modeling, and the empirical results validating the superiority of Ciallo over representative baselines and showing the value of each algorithmic component. Experiments on summarization and assistant tasks show that Ciallo achieves a superior pareto front to the baselines on any combination of two objectives.

Critical strengths

Ciallo outperforms the baselines by a significant margin across reward combinations, showing that Ciallo helps fill the large gap between what is possible and what previous methods achieve. In some cases there is a single point on the Ciallo pareto front which pareto-dominates all the baselines; this is particularly impressive for the helpful/harmless combination because the kind of responses that naturally lead to high rewards for those objectives look very different (i.e. short refusals for harmlessness and long, detailed answers for helpfulness). This is supported by the LLM-as-a-Judge comparisons which suggest that Ciallo's improvements are mainly driven by proper alignment that would satisfy real users, not merely exploiting the weaknesses of the reward models. At the same time, the training costs for Ciallo are reasonable; they are not substantially higher than that of the baselines, and they are justified by the performance gains.

Critical weaknesses

There is a relevant detail about the PARM baseline which seems to be missing from the paper (see questions), and the benchmark datasets used are both fairly old (much older than the LLM models used in the experiments), though they are standard in the multi-objective RLHF literature.

If the authors address some of my questions, I would be open to increasing my score.

Questions for authors

What model does the PARM baseline use for its ARM? How much does its performance improve if a larger ARM is used?

How well does Ciallo perform on a more recent benchmark like the PKU-safeRLHF dataset [1], compared with RIC or PARM?

What is the sensitivity to group size and batch size in phase 2? If these are not sufficiently large, it seems like a poor estimate of D could lead to a spike in the off-diagonal matching loss, threatening the stability of training.

[1] Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513, 2024

8 comments

We thank the reviewer for recognizing the performance gains, LLM-judge validation, and reasonable training cost. We clarify PARM's configuration and provide new ARM-scaling, PKU-SafeRLHF, and phase-two sensitivity results in this response and the accompanying reply.

1. PARM's ARM backbone and the effect of a larger ARM.

In our standard PARM comparisons, its ARM uses the same model as its generation backbone: Qwen3-0.6B on Reddit Summary and Qwen3-4B on Helpful Assistant. Appendix D specifies these standard settings. The additional PKU experiment also uses Qwen3-4B for both components, as detailed in Appendix A.5.

To examine ARM scaling, we keep the Reddit Summary generation backbone at Qwen3-0.6B and replace only the 0.6B ARM with a Qwen3-4B ARM. This is the sole comparison with different ARM and generation-backbone sizes. Both PARM settings are evaluated on 2,000 samples per preference over 11 weights, spaced by 0.1.

Table 1. ARM scaling on summary-faithfulness. All generation backbones are Qwen3-0.6B. Entries are mean reward tuples (summary, faithful); six weights spaced by 0.2 are shown. Define preference-weighted reward u(w)=wTR(w)u(w)=w^T R(w) from the mean reward vector.

w (summary, faithful)PARM: 0.6B ARMPARM: 4B ARMCiallo
(0.0, 1.0)(0.4998, -0.8633)(-0.5251, 0.9964)(1.7742, 1.0804)
(0.2, 0.8)(0.6038, -0.8728)(0.0183, 0.9466)(1.8019, 1.0408)
(0.4, 0.6)(0.6759, -0.8794)(0.3537, 0.8788)(1.8098, 0.8337)
(0.6, 0.4)(0.7319, -0.8898)(0.5859, 0.7627)(2.0419, 0.7404)
(0.8, 0.2)(0.7780, -0.8921)(0.7742, 0.6372)(2.2438, 0.6602)
(1.0, 0.0)(0.7993, -0.8984)(0.8087, 0.5343)(2.2685, 0.5915)

Increasing ARM size substantially improves PARM's empirical trade-off curve, particularly toward higher faithfulness. Ciallo retains a more favorable overall trade-off curve in this stronger comparison. As a supplementary summary over the full sweep, PARM's mean u rises from -0.0652 to 0.7574 (gain 0.8226), compared with 1.5220 for Ciallo. See Appendix A.5 and Fig. 11 for the full ARM-scaling curves.

2. Evaluation on PKU-SafeRLHF.

We additionally compare Ciallo with PARM on PKU-SafeRLHF using Qwen3-4B and the harmlessness-helpfulness objectives, covering 11 preferences spaced by 0.1.

Table 2. PKU-SafeRLHF results. Entries are mean reward tuples (harmless, helpful) in the raw reward space; Δu is Ciallo minus PARM.

w (harmless, helpful)CialloPARMΔu
(0.0, 1.0)(1.1940, 0.8930)(0.2003, 1.1456)-0.2526
(0.2, 0.8)(1.2160, 0.8650)(0.3815, 1.1122)-0.0309
(0.4, 0.6)(1.2930, 0.8770)(0.6539, 1.0469)+0.1537
(0.6, 0.4)(1.3390, 0.7860)(0.9988, 0.9304)+0.1464
(0.8, 0.2)(1.3790, 0.7660)(1.2365, 0.7697)+0.1133
(1.0, 0.0)(1.4230, 0.7130)(1.3980, 0.5700)+0.0250

The complete empirical curves show a more favorable safety-helpfulness trade-off for Ciallo in the high-harmlessness region, while PARM extends further toward the helpfulness extreme. As a supplementary summary over the full sweep, mean u is 1.1043 versus 1.0660 (gain 0.0383). This extends the evaluation to the requested newer benchmark. See Appendix A.6 and Fig. 12 for the complete curves and overall trade-offs.

3. Phase-two group-size and batch-size sensitivity.

We agree that small groups or batches can make the local Fisher and affinity estimates noisy. We evaluate group sizes G=4, 8, 16 and phase-two mini-batch sizes 1, 2 on Reddit Summary with Qwen3-0.6B. The original setting is G=8 and mini-batch size 1 (Appendix B, Table 1). Each setting covers 11 preferences spaced by 0.1, with 2,000 evaluation samples per preference.

Table 3. Group-size sensitivity. Entries are mean reward tuples (summary, faithful); six weights spaced by 0.2 are shown.

w (summary, faithful)G=4G=8 (original)G=16
(0.0, 1.0)(1.7925, 1.0076)(1.7742, 1.0804)(1.7554, 1.0819)
(0.2, 0.8)(1.7823, 0.9648)(1.8019, 1.0408)(1.7328, 1.0751)
(0.4, 0.6)(1.7785, 0.8089)(1.8098, 0.8337)(1.7370, 0.8662)
(0.6, 0.4)(1.9813, 0.6544)(2.0419, 0.7404)(1.9457, 0.6883)
(0.8, 0.2)(2.3070, 0.5233)(2.2438, 0.6602)(2.3279, 0.6259)
(1.0, 0.0)(2.3625, 0.5016)(2.2685, 0.5915)(2.3987, 0.6362)

Table 4. Mini-batch-size sensitivity. Same reward and preference conventions.

w (summary, faithful)Mini-batch=1 (original)Mini-batch=2
(0.0, 1.0)(1.7742, 1.0804)(1.8247, 1.0963)
(0.2, 0.8)(1.8019, 1.0408)(1.8141, 1.0711)
(0.4, 0.6)(1.8098, 0.8337)(1.8005, 0.8920)
(0.6, 0.4)(2.0419, 0.7404)(2.0217, 0.6525)
(0.8, 0.2)(2.2438, 0.6602)(2.3977, 0.5941)
(1.0, 0.0)(2.2685, 0.5915)(2.4587, 0.5781)

The complete curves retain similar overall summary-faithfulness trade-offs, supporting limited sensitivity of final performance over this tested range. As a supplementary summary over the full sweep, the mean preference-weighted reward u(w)=wTR(w)u(w)=w^T R(w) is 1.5008, 1.5220, 1.5378 for G=4, 8, 16, respectively, and 1.5220, 1.5735 for mini-batch sizes 1, 2. The changes relative to the original setting are -0.0213, +0.0158, and +0.0515, respectively. See Appendix A.7 and Figs. 13–14 for the complete group-size and mini-batch-size curves.

Why a noisy D does not cause an unbounded matching-loss value. With positive diagonal damping (Eq. 9), P in Eq. 14 is a normalized inner product, so its entries lie in [-1,1]. The affinity state C is also clipped to [-1,1] after each update (Sec. 4.2.4). Therefore every squared off-diagonal discrepancy is at most 4; Eq. 15 averages these discrepancies, giving 0≤LGram≤40 \leq L_{\mathrm{Gram}} \leq 4. We derive this bound here from the paper's definitions. With the default matching weight 0.5 (Appendix B, Table 1), its contribution to the total loss in Eq. 16 is at most 2. A noisy D can alter the estimated geometry, but cannot make this normalized loss arbitrarily large.

Appendix B (Table 1) lists Fisher damping 1e-6, normalization stabilizers 1e-8, an affinity EMA update rate 0.05, and relation updates every 16 rollout phases. Sec. 4.2.4 specifies that the relation gradients, D, and C are detached before the optimizer step, avoiding second-order differentiation through relation estimation. These address small denominators and abrupt affinity-target changes. The loss-value bound does not itself bound parameter gradients; the sensitivity results provide complementary evidence about final performance, rather than a guarantee that transient loss or gradient spikes cannot occur.

Citation correction. The two unresolved references in Sec. 2 have been corrected to Liu et al. (2025), Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models, and Anonymous (2027), Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment.

Reviewer_d8PEOpenReview ↗

Reply to Author Rebuttal

My questions about PARM's autoregressive reward model and the sensitivity to group/batch size have been addressed to my satisfaction. However, the results of the PKU-safeRLHF evaluation are different enough from the results on the original benchmarks that I believe additional experiments are warranted. Specifically, in the original experiments PARM's pareto curve tended to be clustered together more tightly than that of Ciallo, but in the SafeRLHF experiment we see the opposite. This suggests that we may see significantly different behavior from the other baselines as well. If it is possible within the remaining author response time, I would like to see an evaluation of all the algorithms shown in Fig. 3a on the PKU-safeRLHF benchmark, with their pareto curves plotted together.

One other note: Even though it is mentioned upfront in the abstract, I recommend repeating which model is used for which experiment in section 5.1 for better readability.

Reviewer_pY2uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CIALLO, a framework for multi-objective LLM alignment that considers the relationships between different preference objectives. The method consists of two stages. First, it trains a separate soft prompt for each objective using Online DPO while keeping the backbone model frozen. It then jointly optimizes the soft prompts and the backbone model using GRPO with different preference weights. During joint training, the authors estimate correlations between objectives using Fisher-preconditioned gradients and introduce a Gram-matching regularizer to guide the optimization. The method is evaluated on Reddit Summary and Helpful Assistant using Qwen3-0.6B and Qwen3-4B, respectively. The results show improvements over existing multi-objective alignment methods, including Reward Soups, RIC, and PARM. The paper also includes ablation studies to examine the effects of joint training and correlation modeling.

Critical strengths
  1. The paper addresses a relevant problem in multi-objective LLM alignment. Considering the relationships between different preference objectives is a reasonable direction, as independently optimizing these objectives may overlook useful interactions.
  2. The proposed framework is clearly structured, with separate single-objective training followed by joint optimization. The use of Fisher-gradient affinity and Gram matching provides an explicit way to incorporate objective relationships into the training process.
  3. The experiments cover two established benchmarks and compare against several representative baselines. The authors also provide ablation studies on joint training, correlation modeling, and the affinity matrix, which help demonstrate the contribution of the main components.
Critical weaknesses
  1. The methodological novelty is somewhat limited. The proposed framework mainly combines existing techniques, including soft prompt tuning, Online DPO, GRPO, and gradient-based correlation modeling. Although Fisher Gram matching introduces a new regularization mechanism, the overall approach appears to be an incremental extension of existing methods rather than a fundamentally new solution.
  2. The motivation for the proposed method is not sufficiently convincing. Multi-objective conflicts and gradient interference have already been widely studied in multi-task learning and multi-objective optimization. The paper does not clearly explain why existing approaches are insufficient for this setting or why matching the geometry of soft prompts with Fisher-gradient correlations is particularly necessary.
  3. The experimental scope is relatively limited. The evaluation only covers two benchmarks using Qwen3-0.6B and Qwen3-4B, with the main results focusing on pairwise objective trade-offs. It remains unclear whether the method generalizes to larger language models, more objectives, or more complex preference configurations.
Questions for authors

In Section 2 (Related Work), the authors mention two closely related studies, but both citations appear as "?". Could the authors clarify which works are being referred to and provide the correct references? It would also be helpful to clarify how CIALLO differs from these studies. Other questions and concerns are discussed in the Weaknesses section.

4 comments

We thank the reviewer for the constructive feedback. We clarify the contribution and motivation below, provide the Fisher-versus-cosine ablation, and report the new three-objective experiment in the accompanying response.

1. Methodological contribution.

Online DPO and GRPO provide the optimization backbone. Our contribution is to use objective-gradient relationships as an explicit geometric constraint on the soft-prompt controllers used for continuous preference control. Ciallo computes objective-specific gradients at the same mixed-prompt activation, estimates their signed Fisher affinity, and matches this target with the Fisher-metric Gram matrix of the anchor-centered controller directions (Sec. 4.2, Eqs. 8–17). The affinity tracks the evolving policy while the controllers and backbone are jointly optimized.

We have added a discussion of RACO and MGDA-Decoupled in Sec. 2. RACO applies clipped conflict-averse updates to objective-specific preference losses, with user weights entering policy optimization; MGDA-Decoupled computes mixing coefficients from loss-normalized gradients and applies them to raw gradients. Ciallo makes signed Fisher affinities a learning target for controller geometry. After training, changing the controller mixture adjusts preferences within the same checkpoint, without further optimization. This distinction concerns both the use of objective geometry and preference control at inference time; our experiments do not establish comparative performance against RACO or MGDA-Decoupled.

2. Why objective relationships and the Fisher metric?

We agree that gradient interference is established in multi-objective optimization. The additional question here is how to make those relationships available when learning a mixed controller. A scalarized update can obscure them: at a fixed policy and controller activation, g1=−g2g_1=-g_2 with equal weights yields a zero aggregate gradient despite maximal local conflict. Separate objective-gradient evaluation retains this information (Appendix F.3, Proposition 4 and the following example).

The Fisher metric accounts for local policy-distribution sensitivity. Under a small step −aD−1gi-aD^{-1}g_i, the first-order change in objective j's loss is −agjTD−1gi-a g_j^T D^{-1}g_i. If ideal controller directions follow these preconditioned descent directions, their normalized Fisher Gram equals the gradient-affinity Gram. This motivates matching controller products diTDdjd_i^T Dd_j to gradient products giTD−1gjg_i^T D^{-1}g_j after normalization (Appendix F.1, Proposition 1; Appendix F.2, Proposition 3). The argument is local; the implementation uses a damped diagonal approximation.

The ablations examine the regularizer, its target, and its metric. Removing Gram matching reduces performance (Sec. 5.3.2, Fig. 5); fixed random/all-ones affinity targets also degrade the overall trade-offs (Sec. 5.3.3, Fig. 6). We additionally compare Fisher geometry with the D=ID=I ordinary-cosine variant on Helpful Assistant.

Table 1. Fisher versus cosine at weights spaced by 0.2. Scores are mean rewards. We define preference-weighted reward u(w)=wTR(w)u(w)=w^T R(w), where R(w)R(w) is the vector of mean rewards; Δu is Fisher minus cosine.

w (harmless, helpful)Fisher: (harmless, helpful)Cosine: (harmless, helpful)Δu
(0.0, 1.0)(1.1066, -0.1291)(1.0410, -0.1540)+0.0249
(0.2, 0.8)(1.1428, -0.1669)(1.0530, -0.1700)+0.0204
(0.4, 0.6)(1.1750, -0.2343)(1.1510, -0.2110)-0.0044
(0.6, 0.4)(1.3789, -0.2688)(1.2710, -0.3010)+0.0776
(0.8, 0.2)(1.5848, -0.3291)(1.4290, -0.3060)+0.1200
(1.0, 0.0)(1.6396, -0.3292)(1.4560, -0.3130)+0.1836

Overall, Fisher geometry yields a more favorable empirical trade-off curve than ordinary cosine similarity, supporting its contribution to the controller-matching mechanism. The mean preference-weighted reward gain is 0.0700 across the full sweep of 11 evaluated weights and 0.0704 over the six displayed weights. The complete empirical Pareto curves are provided in Appendix A.3 (Fig. 9) of the revised paper for a direct visual comparison across the full preference sweep.

3. Larger-model and simultaneous three-objective evaluation.

We agree that the original pairwise trade-offs do not fully characterize simultaneous three-objective control. We have now completed an additional Helpful Assistant evaluation using Llama-3.1-8B-Instruct, comparing Ciallo with PARM across harmlessness, helpfulness, and humor. Both methods use this backbone; Ciallo uses LoRA for stage-two policy adaptation (Appendix A.4). Following the RIC-based evaluation protocol in Sec. 5.1, we evaluate 2,000 samples per method and preference. The evaluation covers 13 preference configurations, including 10 with all three weights positive.

Table 2. Three-objective results at the nine measured configurations on the 0.2 grid. Weight and reward tuples follow (harmless, helpful, humor). Scores are mean rewards. We define u(w)=wTR(w)u(w)=w^T R(w) from the mean reward vector; Δu is Ciallo minus PARM. Unmeasured grid configurations are not interpolated.

w (harmless, helpful, humor)Ciallo: (harmless, helpful, humor)PARM: (harmless, helpful, humor)Δu
(1.0, 0.0, 0.0)(3.3041, -2.1719, 1.1572)(1.4233, -1.2179, 0.3486)+1.8808
(0.6, 0.2, 0.2)(3.4669, -2.1678, 1.3978)(1.0064, -1.0675, 0.3583)+1.4641
(0.4, 0.4, 0.2)(0.3610, 0.4532, 0.1845)(0.7611, -1.0764, 0.2298)+0.4427
(0.4, 0.2, 0.4)(1.1405, -1.1332, 1.5006)(0.8652, -1.0995, 0.4127)+0.5385
(0.2, 0.6, 0.2)(-0.5074, 1.3930, -0.0599)(0.5263, -0.9702, 0.1995)+1.1593
(0.2, 0.4, 0.4)(0.0447, 0.5413, 1.0759)(0.6824, -1.1089, 0.3175)+0.8359
(0.2, 0.2, 0.6)(0.2377, -0.6559, 1.5717)(0.7739, -1.1763, 0.4539)+0.6676
(0.0, 1.0, 0.0)(-0.5694, 1.5170, -0.3600)(0.4741, -0.9253, -0.2439)+2.4423
(0.0, 0.0, 1.0)(0.3534, -0.9018, 1.5699)(0.8421, -1.2798, 0.5591)+1.0108

Ciallo achieves improved overall preference-conditioned performance across the three objectives, with a mean preference-weighted reward gain of 1.1717 over all 13 evaluated preferences and 1.1602 over the nine displayed points. At the additional equal-weight configuration (1/3, 1/3, 1/3), u is 0.3284 versus -0.0163 (gain 0.3447). See Appendix A.4 (Fig. 10) of the revised paper for the reward-space plot and an overview of the measured three-objective trade-offs.

This extends the evaluation along three dimensions raised by the reviewer: an 8B backbone, a model family beyond Qwen3, and direct three-objective preference configurations. The Qwen3 ablations examine the proposed mechanism, while this comparison examines performance in the additional setting. The experiment supports this extension; broader model-scale and many-objective generalization remain to be established.

Citation correction. The two unresolved references in Sec. 2 have been corrected to Liu et al. (2025), Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models, and Anonymous (2027), Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment. These studies focus on prompt-aware preference weighting and dynamic preference-policy adaptation, respectively; Ciallo explicitly constrains controller geometry using local objective-gradient affinities.

We thank the reviewer for the valuable comments. Should you have any further concerns, we would be pleased to address them and provide any clarification needed.

I have read the author's response, which addresses some of the questions I raised. The revised manuscript is, however, somewhat difficult to follow. Therefore, I am maintaining my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.