- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose CIALLO, a two-stage framework for multi-objective LLM alignment that first learns objective-specific soft-prompt controllers and then jointly optimizes the prompts under mixed preference weights, especially for tackling correlational objectives, e.g., harmlessness-helpfulness that may be conflicted with each other. The new method part is Fisher-preconditioned gradient affinities, which regularizes the controller geometry through Fisher Gram matching, with the goal of exploiting compatible objectives while separating conflicting ones. Experiments on Reddit Summary and Helpful dataset with Qwen3 models report improved empirical pareto frontiers over existing baselines.
Critical strengths
- The method is tackling a practical and important problem, especially for the objectives that are conflicting with each other.
- There are some ablations towards specific module, parameters to demonstrate the effectiveness.
- The implementation details are transparent.
Critical weaknesses
- Novelty and Contribution
The author positioned their paper for solving the correlational relationship between multi-objectives, especially for potentially conflicting objectives, but all the baseline compared to are non-conflicting aware. Standard conflicting-aware MOA methods [1,2] are fully neglected without any comparisons or discussions. What's the advantage of your correlation-aware method for conflicting objectives comparing to those correlation-processing methods remain uncertain.
- Soundness
First, the method lacks theoretical justification for the Pareto optimality, which is the most important guarantee for multi-objective learning methods. Even though you admit that it only gives local and conditional results, but people from multi-objective learning communities would definitely treat this a big gap for lacking the explanation to your empirical performance that overweights the other method.
The author also didn't a include a systematic ablation to show those component's contribution to the final Pareto frontier. Also, maybe I missed something, but what is "Offset Value"? I didn't find it appeared or defined anywhere in the manuscript. There are also many parameters introduced, are they robust? I only find the ablations for Gram-matching coefficient in the appendix, what about the others, e.g., .
Another ablation missing is the core Fisher part, what if you just replace it with some simple baseline, e.g., cosine? So I guess you didn't convincingly establish that Fisher geometry is necessary.
The experiment also needs to be strengthened, as there's only one model family evaluated, whether the results can be extended to further model families (e.g., Llama, gemma) remains unjustified. The scalability to higher-dimensional objective setups remain uncertain. As all the experiments presented are just 2-objective case.
- Presentation
I think the authors need to spend more time on improving the quality of manuscript. There exists several very obvious mistakes, e.g., line 135-136 with missing citations as "?" and Fig 3 with all the numbers overlapped with each other. The math notation is not professional, especially for the lack of proper usage of \mathrm of the specialized notations, e.g., "KL".
Reference
[1] Reward-free Alignment for Conflicting Objectives. ICML 2026, https://openreview.net/forum?id=vSzRJyg6k0
[2] MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment. AFAA 2026, https://openreview.net/forum?id=yNlF777Uh9
Questions for authors
See weakness section
-
Resolve the concern on novelty and contribution.
-
Resolve the concerns and questions over theoretical gap, ablation details, model-family setup and scalability.
-
Please go over the manuscript again to fix all the presentation issues.
My assessment for the current manuscript is between clear reject and weakly reject; if there's a score of 1.5 this year, I will definitely put it, but now, I will just put it as 2 since the manuscript should be above 1.
4 comments
3. Component ablations, Fisher geometry, and "Offset Value."
The ablations in Secs. 5.3.1–5.3.3 isolate different components: Fig. 4 compares stage-one controllers alone, a frozen backbone, and frozen controllers against joint training; Fig. 5 removes Gram matching; Fig. 6 replaces the dynamic affinity target with fixed random/all-ones targets. These test the training stages, regularizer, and relationship target, respectively. The additional Fisher/cosine comparison tests the metric.
In Figs. 4–6, "Offset Value" means : the reward difference at the same preference, in reward units. Negative values indicate degradation; zero is the full-framework reference. We provide this definition here to clarify the plotted differences.
Fisher versus ordinary cosine (), Helpful Assistant. Scores are mean rewards; and Δu is Fisher minus cosine. Tuples follow (harmless, helpful).
| w | Fisher rewards | Cosine rewards | Δu |
|---|---|---|---|
| (0.0, 1.0) | (1.1066, -0.1291) | (1.0410, -0.1540) | +0.0249 |
| (0.2, 0.8) | (1.1428, -0.1669) | (1.0530, -0.1700) | +0.0204 |
| (0.4, 0.6) | (1.1750, -0.2343) | (1.1510, -0.2110) | -0.0044 |
| (0.6, 0.4) | (1.3789, -0.2688) | (1.2710, -0.3010) | +0.0776 |
| (0.8, 0.2) | (1.5848, -0.3291) | (1.4290, -0.3060) | +0.1200 |
| (1.0, 0.0) | (1.6396, -0.3292) | (1.4560, -0.3130) | +0.1836 |
Fisher gives a more favorable overall empirical trade-off curve, with mean Δu=0.0700 across all 11 evaluated preferences. This supports its contribution in the tested setting, rather than universal necessity. See Appendix A.3 and Fig. 9 for the complete curves.
4. Hyperparameter sensitivity.
Beyond the Gram-weight sweep in Appendix A.1 (Fig. 7), , we vary the affinity EMA rate on both original tasks (Appendix A.2, Fig. 8). Zero disables subsequent target updates; one removes temporal smoothing. On Helpful Assistant, the alternatives reduce harmlessness by 0.038–0.189 at the six displayed preferences, with mixed helpfulness changes. On Summary, they redistribute summary/faithfulness rewards rather than uniformly improving both. These results support retaining the smoothed default 0.05 while showing that this rate affects the trade-off.
Additional group-size and stage-two mini-batch sweeps on Summary show similar overall trade-off curves: mean u is 1.5008/1.5220/1.5378 for G=4/8/16 and 1.5220/1.5735 for mini-batch sizes 1/2. We claim limited final-performance sensitivity within these tested ranges. Appendix A.7 (Figs. 13–14) reports the group/batch curves. These results do not imply robustness of every untested hyperparameter.
5. Another model family and simultaneous three-objective control.
We additionally evaluate Ciallo and PARM with the same Llama-3.1-8B-Instruct backbone on Helpful Assistant, jointly controlling harmlessness, helpfulness, and humor. Ciallo uses LoRA for stage-two policy adaptation (Appendix A.4). There are 13 measured configurations, including 10 with all three weights positive. The table shows the nine measured configurations on the 0.2 grid; no missing configurations are interpolated.
| w (harmless, helpful, humor) | Ciallo u | PARM u | Δu |
|---|---|---|---|
| (1.0, 0.0, 0.0) | 3.3041 | 1.4233 | +1.8808 |
| (0.6, 0.2, 0.2) | 1.9261 | 0.4620 | +1.4641 |
| (0.4, 0.4, 0.2) | 0.3625 | -0.0802 | +0.4427 |
| (0.4, 0.2, 0.4) | 0.8298 | 0.2913 | +0.5385 |
| (0.2, 0.6, 0.2) | 0.7223 | -0.4370 | +1.1593 |
| (0.2, 0.4, 0.4) | 0.6558 | -0.1801 | +0.8359 |
| (0.2, 0.2, 0.6) | 0.8594 | 0.1918 | +0.6676 |
| (0.0, 1.0, 0.0) | 1.5170 | -0.9253 | +2.4423 |
| (0.0, 0.0, 1.0) | 1.5699 | 0.5591 | +1.0108 |
Mean Δu over all 13 preferences is 1.1717; at equal weights, u is 0.3284 versus -0.0163. See Appendix A.4 and Fig. 10 for the reward-space plot. This extends the evidence to another family, an 8B model, and three simultaneous objectives; larger objective counts remain untested.
6. Presentation fixes. The unresolved citations in Sec. 2 have been corrected to Liu et al. (2025), Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models, and Anonymous (2027), Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment. We have resolved the overlapping numerical labels in Fig. 3 by reducing their font size and removing selected overlapping labels. We have also standardized the mathematical notation and formatting.
I try to read your updated manuscript, and I found myself pretty hard to follow it. Since there's no pdf diff, I can't tell which part is newly added and which is the original one.
I will just raise several examples here:
Regarding your 3-objective experiments,
-
"it does not establish many-objective or arbitrary-scale generalization" -- what do you mean by this? do you mean that the formulation cannot be generalized to more general objective case?
-
"There are 13 measured configurations" and "nine on the 0.2 simplex grid. -- What is 13 measured configurations, could you elaborated it?
I also found it a bit hard to follow the result in Appendix F; there are too many notations jump out, and the flow of the result is relatively hard to interpret.
-
Is eq 24 an assumption? what is ?
-
"Proposition 3 is a conditional consistency result, not a recovery or convergence theorem"
-
"but does not imply that its minimum is attained or that minimizing it alone improves the policy objective" -- I am even more confused with this.
I think it would be better to write the manuscript yourself to ensure the content is understandable and with better quality. I believe given the current quality of manuscript, I don't have a clear motivation to recommend acceptance. But if there's any reviewer who has better understanding of the result would like champion this work, I have no objection for that.
Thank you for identifying these unclear passages. We have revised them in the manuscript and prepared a marked-up version with the revised passages highlighted in blue, making the changes easier to identify.
Three-objective experiment. The weights are ordered as (harmlessness, helpfulness, humor). The 13 measured vectors comprise the single-objective vectors (1, 0, 0), (0, 1, 0), and (0, 0, 1); six mixed vectors on the 0.2 simplex grid—(0.6, 0.2, 0.2), (0.4, 0.4, 0.2), (0.4, 0.2, 0.4), (0.2, 0.6, 0.2), (0.2, 0.4, 0.4), and (0.2, 0.2, 0.6)—and four additional vectors: (0.8, 0.1, 0.1), (0.1, 0.8, 0.1), (0.1, 0.1, 0.8), and (1/3, 1/3, 1/3). We have replaced the broad statement about “many-objective or arbitrary-scale generalization” with: “This experiment evaluates three-objective control with Llama-3.1-8B-Instruct; larger objective sets remain to be evaluated.”
Appendix F. Equation (24) is an assumption of Proposition 3, not a condition imposed by the training algorithm. It considers an idealized controller direction that follows the natural-gradient direction at a shared local point and under a shared positive-definite metric . Here, is the local gradient of objective , and is an arbitrary scale that cancels when the Gram matrices are normalized. Under this assumption, Proposition 3 shows that the normalized controller Gram equals the normalized Fisher-gradient affinity.
We have replaced the qualifications around Proposition 3 with a direct explanation: the identity motivates matching the two geometries, and the component ablations in Sec. 5.3 evaluate the regularizer's effect on policy rewards. We also removed an auxiliary paragraph with additional normalized-direction notation to make the result easier to follow.
We thank the reviewer for identifying the missing conflict-aware comparisons and for the detailed questions on theory and evaluation. We address contribution and theoretical scope here, and report the ablations and additional evaluation in the accompanying response.
1. Contribution relative to conflict-aware alignment.
We agree that RACO and MGDA-Decoupled are relevant prior work. RACO applies clipped conflict-averse updates to objective-specific preference losses; MGDA-Decoupled computes mixing coefficients from loss-normalized gradients and applies them to raw gradients. Both explicitly handle objective geometry. We have added a discussion of both methods in Sec. 2 and clarified their distinction from Ciallo.
Ciallo's contribution is the complete relationship-guided controller-learning framework: (i) objective-specific soft-prompt initialization with a frozen backbone; (ii) mixed-preference training of the controller bank and policy; and (iii) Fisher Gram matching, which makes local signed objective-gradient relationships an explicit learning target for controller geometry.
The practical distinction is preference adjustment within one trained checkpoint. RACO's Algorithm 1 takes user weights as inputs to policy optimization; changing those optimization weights does not itself condition an already trained checkpoint. MGDA-Decoupled learns an objective-balancing policy through gradient combination. Ciallo instead learns controllers mixed as s(w)=∑iwisi, so a deployed chatbot can change its helpfulness/harmlessness/humor trade-off as the user's preferences change, without another optimization run. The same checkpoint serves these preferences. This capability motivates learning the controller geometry in addition to resolving training-time conflicts.
2. Theoretical scope and empirical evidence.
We agree that a convergence or Pareto-optimality guarantee would strengthen the paper. Our theoretical claim concerns local cross-objective transfer and controller geometry; our performance claim concerns improved empirical preference-conditioned trade-offs. The former motivates the construction, while the latter is assessed directly through controlled ablations and evaluation across preferences.
For differentiable local losses with gradients gi and a fixed positive-definite metric D, define Hij=giTD−1gj. Under the candidate step p=−D−1∑iaigi, the first-order loss changes are gjTp=−(Ha)j. Thus Ha>0 componentwise certifies a common local descent direction. Fisher affinity records the normalized entries of this transfer matrix; Appendix F.1 (Proposition 1) gives the local-transfer identity, and Appendix F.2 (Proposition 3) establishes Gram consistency under ideal natural-gradient controllers. The common-descent diagnostic above is a clarification in this response; Appendix F.5 states the scope of the paper's analysis.
The empirical evidence also bears directly on the method's contribution. Figures 4–6 in Sec. 5.3 examine the training stages, Gram regularizer, and dynamic affinity target: removing or replacing these components degrades the observed trade-offs. The Fisher/cosine ablation further isolates the metric, with a mean preference-weighted reward gain of 0.0700 over the complete 11-preference sweep. The Llama-3.1-8B-Instruct experiment uses LoRA for Ciallo's stage-two policy adaptation and extends the evaluation to another model family and three simultaneous objectives, with mean gain 1.1717 over PARM across 13 measured preferences. The accompanying response provides the numerical results; the complete reward-space plots are in Appendix A.3–A.4 (Figs. 9–10).
Together, these studies support both the proposed design and its practical benefit in the evaluated settings. We believe this constitutes a substantive methodological and empirical contribution, even though a general convergence guarantee remains open. We therefore frame the results as improved empirical trade-offs, without claiming globally Pareto-optimal policies or treating a first-order stationarity condition as sufficient for optimality.