acceptodds
Under review as a conference paper at ICLR 2027

Language2Utility: Transferable Natural-Language Preference Grounding for Multi-Objective Reinforcement Learning

Abstract

Multi-objective reinforcement learning (MORL) addresses sequential decision making with competing objectives, but preference-conditioned MORL typically takes numerical preference specifications as given. In practice, however, users express desired trade-offs in natural language, creating a semantic gap between linguistic preferences and the numerical preference specifications underlying multi-objective utility. Motivated by a relational view of preference semantics over objective sets, we study transferable grounding of natural-language preferences across heterogeneous objective spaces and introduce Language2Utility. Language2Utility encodes preference-objective relations and contextualizes them with a permutation-equivariant set model over variable-cardinality objective sets. To accommodate practical preference expressions that may not uniquely specify a numerical trade-off, learning combines ordinal constraints with supervision over sets of semantically compatible numerical preferences. We further decouple shared semantic grounding from downstream domain-specific behavioral realization. A semantic-to-behavior bound formalizes the separation by relating language-conditioned utility regret to semantic misalignment and realization error. Across four MO-Gymnasium benchmarks, Language2Utility improves both semantic grounding and behavioral relevance in-domain. Under leave-one-domain-out transfer, where both the target task and its objective cardinality are unseen during training, Language2Utility achieves better zero-shot transfer than the compared baselines. Human-authored evaluations across three practical application scenarios further support semantic transfer to new task contexts and objective sets.

Primary area:
reinforcement learning
Keywords:
multi-objective reinforcement learning, preference-conditioned reinforcement learning, natural-language preference grounding, preference learning, cross-domain transfer, permutation-equivariant set models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FMKpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on mapping natural-language preferences to numerical weights for multi-objective reinforcement learning across different tasks and objective sets. Language2Utility combines frozen preference–objective embeddings, set attention, and ordinal and admissible-set supervision. It outperforms the baselines on four MO-Gymnasium tasks, with semantic transfer further evaluated using 75 human-written preferences. The paper also introduces a controlled benchmark and a theoretical bound relating weight-prediction error to downstream utility regret.

Critical strengths
  • This paper focuses on mapping natural-language preferences to numerical weights for multi-objective reinforcement learning across different tasks and objective sets.

  • The paper is clearly organized, and the appendix provides detailed descriptions of the architecture, data construction, and evaluation metrics.

Critical weaknesses
  • Table 3 includes only 8B direct LLM baselines. Larger open-weight models, such as 70B model, should be evaluated under the same transfer protocol to check whether they already provide comparable zero-shot transfer. Testing additional embedding backbones would also clarify whether the reported gains generalize beyond the chosen encoder.

  • Training accepts any weight vector in the admissible set, while Centroid-L1 measures distance to its centroid and the utility metrics average uniformly over the set. A valid prediction can therefore still receive a worse score. The paper should justify why this uniform weighting represents the uncertainty in users’ preferences.

  • Table 11 contains only 38 test specifications, including one for DST and four for Minecart. The 174 test sentences are different expressions of these specifications. Results across five training seeds do not address the uncertainty caused by such a small set of test preferences.

  • In Table 16, DeepSets w/ Joint has better mean results on all six Minecart metrics, including Set-L1 of 0.1539 vs. 0.1930 for the proposed method. This weakens the claim of consistent transfer gains.

  • Proposition 1 applies to any weight predictor and does not depend on the relational embeddings, set attention, or training objective. It connects prediction error to utility regret, but does not show why the proposed design should reduce prediction error on unseen tasks or objective sets.

Questions for authors

Please see the weakness part.

6 comments
AuthorsOpenReview ↗

Response to W1: Larger Open-Weight LLMs and Embedding Backbones

W1. Table 3 includes only 8B direct LLM baselines. Larger open-weight models, such as 70B model, should be evaluated under the same transfer protocol to check whether they already provide comparable zero-shot transfer. Testing additional embedding backbones would also clarify whether the reported gains generalize beyond the chosen encoder.

Response to W1. We thank the reviewer for suggesting larger open-weight models and alternative embedding backbones.

We added a 72B open-weight direct large language model (LLM) and two additional frozen embedding backbones under the same transfer protocol.

Larger open-weight direct LLM comparison. Revised Appendix D.1 and Table 12 include Qwen2.5-72B-Instruct alongside the two 8B-scale direct LLMs. All direct LLMs use the same test examples, prompting protocol, and averaging procedure as the leave-one-domain-out (LODO) comparison.

LODO results on representative semantic, ordering, and regret metrics. Table 12 reports all six metrics.

MethodSet-L1 ↓Pair Acc. ↑Lang. regret ↓
Qwen3-8B.4328.8521.0434
Ministral 3 8B.3651.8633.0402
Qwen2.5-72B-Instruct.3358.8943.0371
Language2Utility.2996.9173.0343

Qwen2.5-72B-Instruct improves over both 8B models on all six metrics, confirming the benefit of greater open-weight model scale. Language2Utility nevertheless obtains the better result on every metric in Table 12, including lower Set-L1, .2996 versus .3358, and lower language-conditioned regret, .0343 versus .0371. The transfer advantage therefore persists at the 72B scale.

Consistent gains across embedding backbones. Newly added Appendix F.3 and Table 20 repeat both in-domain and four-fold LODO evaluations using text-embedding-3-large and gemini-embedding-2 in addition to Qwen3-Embedding-8B. For each frozen encoder, we compare Language2Utility with its parameter-matched DeepSets w/ Joint counterpart over five seeds.

Set-L1 comparison across frozen encoders. Table 20 reports standard deviations and all six metrics.

EncoderEvaluationDeepSets w/ JointOursReduction
Qwen3-Embedding-8BIn-domain.1487.113623.6%
Qwen3-Embedding-8BLODO.3279.29968.6%
text-embedding-3-largeIn-domain.2540.228110.2%
text-embedding-3-largeLODO.5803.53088.5%
gemini-embedding-2In-domain.1298.106518.0%
gemini-embedding-2LODO.3171.29048.4%

Language2Utility achieves the better mean on all six metrics for every encoder and evaluation setting. Its Set-L1 reduction ranges from 10.2% to 23.6% in-domain and from 8.4% to 8.6% under LODO. Although absolute performance varies across encoders, the consistent within-backbone gains show that the architectural advantage extends beyond the default language encoder.

Overall, the expanded comparisons show that the transfer advantage over open-weight direct LLMs persists from 8B to 72B and that the architectural advantage persists across three frozen embedding backbones. Language2Utility also produces simplex-valued outputs without autoregressive decoding and achieves approximately 20× lower median latency than the locally evaluated 8B direct LLMs.

Revision. We updated Sections 4.1, 4.2, and 4.3.1 and Appendix D.1, and added Appendix F.3 with Table 20.

AuthorsOpenReview ↗

Response to W2: Semantic Validity and the Uniform Utility Reference

W2. Training accepts any weight vector in the admissible set, while Centroid-L1 measures distance to its centroid and the utility metrics average uniformly over the set. A valid prediction can therefore still receive a worse score. The paper should justify why this uniform weighting represents the uncertainty in users’ preferences.

Response to W2. We appreciate the reviewer’s careful observation about set-valued validity and centroid-based evaluation.

Uniform weighting is not intended to model user-specific uncertainty. Semantic validity is defined by compatibility with the admissible set, while uniform averaging provides a symmetric aggregate reference without introducing additional user-specific probabilities over admissible weights.

Semantic validity and complementary metrics. The admissible set W(c) represents numerical preferences compatible with the language, rather than a probability distribution over users’ latent preferences. This accommodates expressions that specify qualitative priorities without uniquely determining numerical trade-offs. For example, prioritizing passenger comfort over travel efficiency in autonomous driving establishes an ordering but leaves their numerical gap unspecified. Training therefore uses the nearest-set loss

min over w ∈ W(c) of ‖ŵ(x) − w‖₁,

which treats every member of W(c) as a valid target. Any admissible prediction receives zero Set-L1.

The metrics evaluate complementary aspects of a semantic specification. Set-L1 directly measures compatibility with the full admissible set, while the other metrics capture ordinal consistency, central tendency, and behavioral relevance.

Evaluation metricsReferenceInterpretation
Set-L1 ↓Full admissible set W(c)Semantic compatibility. Every admissible weight receives zero Set-L1
Ordinal Accuracy ↑Stated ordinal constraintsOrdinal consistency. Agreement with the qualitative priority relations
Pair Accuracy ↑Return pairs unanimously ordered by W(c)Preference-invariant behavior. Ordering shared by all admissible weights, independent of within-set averaging
Centroid-L1 ↓Admissible-set centroid w̄(c)Central tendency. Not a semantic-validity criterion
Language utility ↑Uniform average over W(c), equivalently w̄(c)Realized behavioral utility. Utility of the controller return under a symmetric aggregate reference, not an estimated user-specific distribution
Language regret ↓Best empirical utility under the same aggregate referenceBehavioral shortfall. Gap to the best return available in the fixed controller library

An admissible prediction away from the centroid can therefore obtain zero Set-L1 but nonzero Centroid-L1. This is not a contradiction: the former evaluates semantic validity with respect to the full set, whereas the latter summarizes distance from its center. Pair Accuracy further evaluates only preference-invariant behavioral comparisons.

Uniform utility reference and its scope. Uniform averaging provides a symmetric card-level reference across compatible realizations without introducing additional user-specific probabilities over admissible weights. It serves as an aggregate evaluation reference rather than an estimate of a user-specific uncertainty distribution. Under linear utility,

mean over w ∈ W(c) of (w · R) = w̄(c) · R,

so the uniformly averaged utility is exactly the utility evaluated at the admissible-set centroid. We now state this equivalence in Appendices A.3 and C.3 and clarify that it defines the aggregate utility reference, not semantic validity.

Controlled comparison with centroid supervision. Table 2 and Appendix D.3 compare the full model with Ordinal + Centroid, which retains the same architecture and ordinal loss but replaces admissible-set supervision with regression to w̄(c). In-domain, set-valued supervision achieves lower Set-L1, .1136 versus .1272, with similar Ordinal Accuracy and Pair Accuracy. Centroid regression obtains lower Centroid-L1 and utility regret because its target directly matches the centroid-based evaluation reference. Under leave-one-domain-out (LODO) transfer, the two variants have similar Set-L1, while set-valued supervision yields slightly better average Ordinal Accuracy, Pair Accuracy, realized utility, and utility regret. These results make the expected metric alignment explicit rather than treating the centroid as the unique semantically correct target.

Overall, uniform weighting provides a symmetric aggregate utility reference, while Set-L1, Ordinal Accuracy, and Pair Accuracy evaluate the full admissible set and its ordinal implications.

Revision. The revised Sections 3.1 and 4.1 and Appendices A.3, C.3, and D.3 make these distinctions explicit. Table 2 reports the controlled comparison.

AuthorsOpenReview ↗

Response to W3: Expanded Semantic-Card Coverage and Statistical Uncertainty

W3. Table 11 contains only 38 test specifications, including one for DST and four for Minecart. The 174 test sentences are different expressions of these specifications. Results across five training seeds do not address the uncertainty caused by such a small set of test preferences.

Response to W3. We thank the reviewer for highlighting the importance of distinguishing linguistic variation, semantic-card coverage, and training-seed variability.

We expanded the leave-one-domain-out (LODO) evaluation from 38 held-out semantic cards (174 language realizations) to all 386 benchmark cards (1,759 language realizations), and added stratified paired card-level bootstrap intervals to quantify variation across semantic specifications.

Aggregation unit and sources of variation. In both the original and expanded evaluations, the semantic card is the aggregation unit. Metric values are first averaged over the language realizations of each card, after which all cards receive equal weight. The original 174 language realizations therefore evaluate robustness to different phrasings of 38 distinct semantic preferences. The five training seeds quantify training variability, while the expanded card-level bootstrap analysis additionally quantifies variation across semantic specifications.

Expanded coverage and paired analysis. Under LODO evaluation, the entire target domain is excluded from training, so every evaluated target-domain card is unseen by the grounding model. The expanded evaluation covers all 386 semantic cards retained in the benchmark. This includes all 7 Deep Sea Treasure (DST) cards and all 43 Minecart cards induced by the importance-role rules, together with 144 Lunar Lander and 192 Fruit Tree cards. For each card, we average over its language realizations and compute the paired difference between Language2Utility and DeepSets w/ Joint, the closest set-based alternative, across five training seeds. We estimate pointwise 95% percentile intervals using 100,000 paired card-level bootstrap replicates stratified by domain.

Expanded LODO results from Table 18 over all 386 semantic cards (1,759 language realizations). Paired gain is defined in the favorable direction, so positive values favor Language2Utility. Intervals are pointwise 95% percentile intervals from the stratified card-level bootstrap. Bold indicates the better mean.

Semantic metrics

MethodSet-L1 ↓Centroid-L1 ↓Ordinal Accuracy ↑
Language2Utility.3237.3458.7763
DeepSets w/ Joint.3418.3672.7576
Paired gain Δ.0181.0215.0187
95% card-level interval[.0071, .0289][.0102, .0325][.0088, .0290]

Behavioral metrics

MethodPair Accuracy ↑J(lang) ↑Reg(lang) ↓
Language2Utility.9191.6439.0328
DeepSets w/ Joint.8942.6350.0417
Paired gain Δ.0249.0089.0089
95% card-level interval[.0187, .0314][.0058, .0121][.0058, .0121]

Language2Utility obtains favorable paired gains on all six metrics, and every card-level 95% interval lies entirely above zero. It reduces Set-L1 and Centroid-L1 by .0181 and .0215 and improves Ordinal Accuracy and Pair Accuracy by 1.87 and 2.49 percentage points. This more than tenfold expansion in semantic-card coverage supports the aggregate transfer advantage under broader preferences and directly quantifies specification-level uncertainty beyond training-seed variation.

Revision. We added this analysis in Appendix E.2 with Table 18 and clarified the card-level aggregation procedure in Appendix C.1.

Reviewer_KVgUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of transferable grounding of natural-language preferences across domains in heterogeneous objective spaces, and proposes the Language2Utility method. The method encodes preference–objective semantic relationships and employs a permutation-equivariant set attention mechanism to model a variable number of objectives in context, thereby mapping natural-language preferences to numerical utility weights. In addition, it combines ordinal constraints with semantically compatible numerical preference sets for supervised learning to handle the semantic underdetermination of natural-language preferences. Furthermore, the paper decouples the cross-domain shared semantic mapping from domain-specific behavioral realization, and derives an upper bound on the utility regret from semantics to behavior, linking the language-conditioned utility regret to the semantic mismatch error and the behavioral realization error. Overall, the problem studied in this paper has certain theoretical significance and practical application value.

Critical strengths
  1. The problem of how reinforcement learning can understand human preferences expressed in natural language is valuable.

  2. The proposed method's integration of cross-task preference transfer, relational semantic representations, and the decoupling of semantic grounding from behavioral realization is clear and well-articulated.

Critical weaknesses

1.The paper provides a relatively clear mathematical definition of the natural-language preference grounding task, but some key concepts still require clarification. The authors are suggested to further explain the semantic soundness of this set construction, and to discuss whether the discretization rules and threshold choices affect the problem definition and the cross-domain generalization evaluation of the model.

2.This paper primarily relies on existing techniques, including a frozen language model, joint preference–objective representations, shared neural networks, and self-attention. Although there are many ablation studies, the novelty of the proposed transfer mechanism remains unclear. The authors are suggested to clarify the respective contributions of the pretrained language representations and the trainable grounding network to cross-domain generalization.

3.When reporting the experimental results, the authors only describe the data; it is suggested that they add analytical discussion.

4.The paper allows the same natural-language preference to correspond to multiple admissible weights, yet produces only a single weight vector at inference time. The authors are suggested to explain the rationale for this design and to clarify whether it loses the uncertainty information inherent in the preference itself (i.e., the underdetermination reflected by the admissible weight set).

5.The paper involves multiple hyperparameters (e.g., the weight discretization rule and thresholds). Although the appendix provides the specific settings, there is a lack of analysis of the basis for parameter selection and their sensitivity; further supplementation is suggested.

6.The paper proposes a theoretical upper bound linking the semantic mapping error and the downstream utility regret, but lacks empirical validation of its tightness. The authors are suggested to report the empirical values of each error term and compare them with the empirical utility regret, so as to verify whether the bound is tight and whether it has practical guiding significance.

Questions for authors
  1. How do the discretization rules and threshold choices affect the semantic soundness of the admissible weight set and cross-domain generalization?

  2. What is the key novelty of the proposed transfer mechanism, and how do the pretrained language model and grounding network contribute to generalization?

  3. Could the authors provide deeper analysis and explanations of the experimental results rather than merely reporting performance differences?

  4. Why does the model predict a single weight vector when multiple weights may be admissible? Does this lead to a loss of preference uncertainty?

  5. What motivates the hyperparameter choices, and how sensitive is the model's performance to these settings?

  6. How tight is the proposed theoretical regret bound in practice, and can the authors validate it by comparing the empirical error terms with the observed utility regret?

12 comments
AuthorsOpenReview ↗

Response to W1 and Q1: Semantic Soundness and Construction Sensitivity

W1. The paper provides a relatively clear mathematical definition of the natural-language preference grounding task, but some key concepts still require clarification. The authors are suggested to further explain the semantic soundness of this set construction, and to discuss whether the discretization rules and threshold choices affect the problem definition and the cross-domain generalization evaluation of the model.

Q1. How do the discretization rules and threshold choices affect the semantic soundness of the admissible weight set and cross-domain generalization?

Response to W1/Q1. We thank the reviewer for examining the semantic basis and sensitivity of the admissible-set construction.

The admissible set groups numerical weights that express the same qualitative objective roles. The cutoff and discretization choices determine the finite benchmark representation of those semantics, while the main ranking and cross-domain behavioral conclusions remain stable across the tested settings.

Shared qualitative semantics. We discretize the preference simplex and characterize each objective as negligible, minor, secondary, major, or dominant relative to the other objectives. We group weights with identical role signatures into one admissible set. Every member therefore satisfies the same dominance, ordering, and sparsity relations while allowing numerical gaps not uniquely specified by language. Ordinal relations are recorded separately.

For example, prioritizing passenger comfort over travel efficiency specifies an ordering without fixing numerical weights. The admissible set contains the grid points sharing that qualitative meaning, thereby representing semantic underdetermination through multiple valid numerical preferences. These choices instantiate the finite benchmark rather than alter the formal grounding task. The same normalized role rules apply across domains and objective cardinalities.

Sensitivity to cutoff choices. We independently increased or decreased the dominant, major, and secondary cutoffs by one grid step, producing D⁻, D⁺, M⁻, M⁺, S⁻, and S⁺. Language inputs, model predictions, and ordinal labels remained fixed. Figure 17 reports the complete rankings under in-domain and leave-one-domain-out (LODO) evaluation, while the table below summarizes Language2Utility.

Ranks across the baseline and six cutoff perturbations. Lower is better.

MetricIn-domain baselineIn-domain rangeLODO baselineLODO range
Set-L11122
Language-conditioned utility21–322–3

Set-L1 rank is unchanged in both evaluations. Across all cutoff settings, Language2Utility remains above Ordinal + Centroid, DeepSets w/ Joint, and w/o Objective Interaction on both LODO metrics. Figure 18 further shows absolute changes below approximately .04 for Set-L1, .025 for Centroid-L1, and .011 for language-conditioned utility, with Pair Accuracy (Pair Acc.) changing by at most 1.2 percentage points.

Sensitivity to simplex resolution. We use U = 12 because it is the smallest common resolution that exactly represents the uniform preference for objective cardinalities 2, 3, 4, and 6. We also evaluate the same fixed predictions at U = 24 and 36.

Values are means over five training seeds. Table 19 reports standard deviations.

In-domain

USet-L1 ↓Centroid-L1 ↓Ord. Acc. ↑Pair Acc. ↑Lang. utility ↑Lang. regret ↓
12.1136.1418.9211.9796.6602.0122
24.0748.1407.9211.9956.6595.0119
36.0627.1413.9211.9969.6597.0118

LODO

USet-L1 ↓Centroid-L1 ↓Ord. Acc. ↑Pair Acc. ↑Lang. utility ↑Lang. regret ↓
12.2996.3284.8007.9173.6381.0343
24.2274.3278.8007.9500.6373.0341
36.2114.3269.8007.9544.6377.0338

Set-L1 decreases because denser grids approximate each admissible region more closely, not because predictions change. Pair Acc. changes because refinement alters the grid-specific comparison set, while accuracy on pairs shared across resolutions remains unchanged. Centroid-L1 varies by at most .0015, Ord. Acc. remains unchanged, and both language-conditioned utility and regret vary by less than .001. These results show that the qualitative semantic and cross-domain behavioral conclusions do not depend materially on the tested resolutions.

Overall, the construction groups numerical preferences by shared qualitative meaning, and both sensitivity studies preserve the paper’s main conclusions.

Revision. We clarified the set-valued semantics in Section 3.1 and the construction rules in Appendices B.2 and B.3. Newly added Appendix F.1 presents the cutoff analysis with Figures 17 and 18, and newly added Appendix F.2 presents the discretization analysis with Table 19 and Figure 19.

AuthorsOpenReview ↗

Response to W2 and Q2: Novelty and Sources of Cross-Domain Transfer

W2. This paper primarily relies on existing techniques, including a frozen language model, joint preference–objective representations, shared neural networks, and self-attention. Although there are many ablation studies, the novelty of the proposed transfer mechanism remains unclear. The authors are suggested to clarify the respective contributions of the pretrained language representations and the trainable grounding network to cross-domain generalization.

Q2. What is the key novelty of the proposed transfer mechanism, and how do the pretrained language model and grounding network contribute to generalization?

Response to W2/Q2. We thank the reviewer for the opportunity to clarify the novelty of the transfer mechanism and the distinct roles of its frozen and trainable components.

The novelty lies in transferring preference-relative objective roles across changing objective sets. Frozen Joint representations expose this relational semantic structure, and the permutation-equivariant grounding network learns its set-dependent numerical interpretation.

Relational transfer across changing objective sets. Language2Utility represents each supplied objective through how the language preference prioritizes it, rather than through a domain-specific identity. The grounding network contextualizes these objective-wise relations within the current objective set. This relational, permutation-equivariant formulation allows one shared grounding model to operate across unseen objective compositions and cardinalities without domain-specific parameters. The contribution therefore lies in the transferable relational formulation and its learning principle, not in any individual architectural primitive.

Contribution of frozen representations. We test whether transferable structure exists before grounding-network adaptation using linear probes trained on frozen source-domain representations and evaluated directly on each held-out domain. Figure 4 reports the complete analysis, summarized below.

Spearman correlation of source-trained linear probes evaluated on each held-out domain. Higher is better. Joint gain reports the improvement over Separate.

Held-out domainObjectivesSeparate preference/objective ρJoint preference-objective ρJoint gain
DST2−1.0001.000+2.000
Minecart3−.337.946+1.283
Lunar Lander4−.039.554+.593
Fruit Tree6.035.503+.468
Overall2–6−.335.751+1.086

Joint representations improve recoverability in every held-out domain and raise the overall correlation from −.335 to .751. This provides representation-level evidence that preference-relative objective importance remains accessible across domain and cardinality shifts before grounding-network adaptation. Appendix E.4 provides the probe protocol and additional representation-space analyses.

Contribution of trainable grounding. The grounding network learns across source domains how these relations translate into numerical trade-offs within the supplied objective set. Permutation-equivariant objective interaction provides the set-dependent contextualization required for this interpretation. The controlled leave-one-domain-out (LODO) ablations are summarized below.

Means over five seeds for representative semantic, ordering, and regret metrics. Table 3 reports all six LODO metrics.

VariantSet-L1 ↓Pair Acc. ↑Lang. regret ↓
Full model.2996.9173.0343
No objective interaction.3800.8835.0450
No Joint representation.6555.6808.1162

Removing Joint representations produces the largest degradation, supporting relational semantic structure as the primary cross-domain signal. Retaining Joint representations while removing objective interaction also degrades all three metrics, showing the additional contribution of learned set-level contextualization.

Together, the probe and controlled ablations identify complementary sources of transfer. Frozen Joint representations make preference-relative importance recoverable in unseen domains, while the trainable grounding network converts those relations into set-dependent numerical preferences. Their combination enables a single model to generalize across new objective compositions and cardinalities.

Revision. We clarified the relational transfer mechanism in Section 3.2.2 and expanded the representation-level and end-to-end attribution analysis in Section 4.3.2. Figure 4 and Table 3 provide the corresponding evidence.

AuthorsOpenReview ↗

Response to W3 and Q3: Analytical Interpretation of Experimental Results

W3. When reporting the experimental results, the authors only describe the data; it is suggested that they add analytical discussion.

Q3. Could the authors provide deeper analysis and explanations of the experimental results rather than merely reporting performance differences?

Response to W3/Q3. We thank the reviewer for this constructive suggestion to deepen the interpretation of the experimental results.

We expanded the analysis to explain the sources of grounding and transfer gains, their behavioral relevance, and generalization to human-authored language.

Grounding and transfer mechanisms. Section 4.2 and Table 2 now interpret the controlled ablations rather than only reporting their values. Removing Joint preference-objective representations produces the largest architectural degradation, increasing Set-L1 from .1136 to .3037 and reducing Pair Accuracy (Pair Acc.) from .9796 to .9000. Removing objective interaction yields a smaller but consistent degradation. These results indicate that preference-relative semantic representations provide the primary grounding signal, while set-level contextualization contributes by interpreting each relation within the supplied objective set. We also explain why centroid regression improves centroid-aligned metrics through direct target alignment, whereas set-valued supervision achieves lower Set-L1 by retaining multiple semantically valid targets.

Section 4.3.2 and Figure 4 now analyze the distinct roles of pretrained representations and the grounding network under leave-one-domain-out (LODO) evaluation. Linear probes trained on source domains and evaluated on the held-out domain increase the overall Spearman correlation from −.335 with separate representations to .751 with Joint representations, with positive gains in every held-out domain. Combined with the LODO ablations in Table 3, this shows that Joint representations expose transferable preference-objective structure, while the trainable set model learns how that structure translates into numerical trade-offs within a new objective set. Appendix E.1 further analyzes the domain-level pattern, and Appendix E.2 confirms the aggregate advantage on an expanded set of 386 cards using paired confidence intervals.

Behavioral relevance of semantic accuracy. Section 3.4 and Proposition 1 identify semantic grounding error as a model-dependent term in the downstream utility-regret bound. Appendix D.4 and Figure 6 complement this theoretical connection empirically: within the learned model families, lower Set-L1 generally coincides with higher preference-consistent Pair Acc. and lower language-conditioned utility regret. This explains why improvements in preference geometry are behaviorally meaningful rather than merely numerical differences in predicted weights.

Human-authored language and unseen cardinality. Section 4.4 and Table 4 now discuss the structural pattern in the external evaluation. Language2Utility reduces Set-L1 by 10.3% relative to DeepSets w/ Joint and attains the lowest Set-L1 in all three application scenarios, including the unseen five-objective setting. The degradation caused by removing either Joint representations or set contextualization also persists, indicating that the proposed relational semantics remain informative beyond the controlled benchmark language.

Overall, the revised analysis shows that Language2Utility's gains arise from transferable preference-objective representations, their contextual interpretation within heterogeneous objective sets, and set-valued semantic supervision.

Revision. We incorporated these explanations throughout revised Sections 4.2 to 4.4 and the corresponding analyses in Appendices D and E.

Reviewer_HBq9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to turn natural-language preferences into weights for multi-objective reinforcement learning (MORL). A shared model handles different objective sets and learns from multiple valid weight vectors to account for ambiguous preferences. The experiments examine how the predicted weights affect controller behavior and include tests on unseen environments and human-written preferences.

Critical strengths
  1. Flexibility in the choice and number of objectives

    It extends previous methods that map natural language to weights. It uses the same model to interpret preferences even when the set of objectives changes. The results in held-out environments and with unseen numbers of objectives suggest that the approach can generalize to out-of-domain settings.

  2. A sensible treatment of ambiguous preferences

    The method allows multiple valid weight vectors for a given preference statement, rather than assuming that there is a single correct answer. It also accounts for the relative ordering of objectives.

  3. Evaluation beyond weight prediction

    The paper compares the utility achieved by different weight prediction methods using the same controller and also includes ablation studies. The shorter inference time is also a practical advantage.

Critical weaknesses
  1. Figure 1 is a bit too dense for an overview.

    I found Fig. 1 difficult to take in before reading the details of the method. Some of the content may also be hard to read without zooming in. This is a minor issue, but I would prefer a simpler overview of the main idea, with the details moved to a separate figure in the Method section.

  2. The transfer comparison with stronger LLMs needs clarification.

    I’m not sure whether Tables 3 and 12 use the same test examples and averaging. If they do, GPT-5.6 Luna has the lower Set-L1 score. I’d like to see this result discussed separately from the inference-time results

  3. A direct comparison with single-target regression is missing.

    Table 2 suggests that the ranking and set losses work well together. However, it does not compare the set loss with regression to a single target weight vector using the same model. I’m not sure how much of the gain comes from using multiple targets.

Questions for authors
  1. Following up on W2, do Tables 3 and 12 use the same test examples and averaging procedure?

    The 8B LLM results are identical in both tables. If the evaluation setup is the same, could the stronger LLM results in Table 12 also be used as baselines for LODO?

  2. Would the methods rank differently if the cutoffs for the importance ranges changed?

    Or is there evidence that methods with more accurate weight predictions also better predict which outcomes users choose?

  3. How does performance change if the set loss is replaced with regression to the centroid of the valid weight set?

    I’d like to see this comparison with the same architecture, ranking loss, and other settings. That would help show how much of the gain comes from using multiple valid targets.

7 comments
AuthorsOpenReview ↗

Response to W1: Simplifying Figure 1

W1. Figure 1 is a bit too dense for an overview.

I found Fig. 1 difficult to take in before reading the details of the method. Some of the content may also be hard to read without zooming in. This is a minor issue, but I would prefer a simpler overview of the main idea, with the details moved to a separate figure in the Method section.

Response to W1. We thank the reviewer for this constructive suggestion regarding the overview figure.

We simplified Figure 1 and separated the architectural details into Figure 2 in Section 3.2.

Simplified overview. Figure 1 now focuses on the high-level distinction among direct large language model (LLM) generation, task-specific adapters, and Language2Utility when the available objective set changes across tasks. It also retains only the common downstream separation between semantic grounding and domain-specific behavioral realization. The two application examples serve to illustrate grounding over different four- and six-objective spaces.

Separated method details. Figure 2 in Section 3.2 presents the preference-objective relational representations, permutation-equivariant set contextualization, simplex-valued prediction, and ordinal and set-valued supervision.

Revision. We also revised the captions and surrounding text so that Figure 1 introduces the research problem and central idea, while Figure 2 provides the grounding architecture and training objectives.

AuthorsOpenReview ↗

Response to W2 and Q1: Direct LLM Comparison

W2. The transfer comparison with stronger LLMs needs clarification.

I’m not sure whether Tables 3 and 12 use the same test examples and averaging. If they do, GPT-5.6 Luna has the lower Set-L1 score. I’d like to see this result discussed separately from the inference-time results.

Q1. Following up on W2, do Tables 3 and 12 use the same test examples and averaging procedure?

The 8B LLM results are identical in both tables. If the evaluation setup is the same, could the stronger LLM results in Table 12 also be used as baselines for LODO?

Response to W2/Q1. We appreciate the reviewer’s careful cross-check of Tables 3 and 12.

Yes, Tables 3 and 12 use the same test examples and averaging procedure, and the stronger direct large language model (LLM) results can serve as leave-one-domain-out (LODO) baselines.

Shared evaluation protocol. Direct LLMs perform no task-specific grounding training, so their predictions do not depend on whether the learned grounding model is trained in-domain or under LODO. All direct LLMs and Language2Utility are evaluated on the same test examples using the same card-level and language-level averaging procedure. Direct LLM results report mean and standard deviation over five independent inference runs, whereas learned grounding methods report five training seeds.

Accordingly, revised Table 12 reports each direct LLM once alongside both Ours (LODO) and Ours (in-domain), avoiding duplicate direct LLM rows across the main tables. The remaining learned in-domain methods are compared in Table 1, while the remaining learned LODO methods are compared in Table 3. We have made this organization and the shared evaluation protocol explicit in Appendix D.1 and the caption of Table 12.

Stronger direct LLM comparison. At the comparable 8B scale, Ours (LODO) outperforms Qwen3-8B and Ministral 3 8B on all six metrics. We further added Qwen2.5-72B-Instruct as a larger open-weight baseline in the revision. Although it consistently improves over both 8B models, Language2Utility remains better on every metric, including lower Set-L1 (.2996 vs. .3358) and language-conditioned utility regret (.0343 vs. .0371). Language2Utility also outperforms Gemini 2.5 Flash on all six metrics, with Set-L1 of .2996 versus .3279 and regret of .0343 versus .0367.

The more capable frontier models, including Gemini 3.5 Flash, GPT-5.4 mini, and GPT-5.6 Luna, achieve stronger zero-shot LODO results. In particular, GPT-5.6 Luna attains Set-L1 of .2044, compared with .2996 for Ours (LODO). We now discuss this result explicitly in Section 4.3.1 and Appendix D.1. With target-domain supervision, Ours (in-domain) surpasses all compared direct LLMs on five of the six metrics. Relative to GPT-5.6 Luna, Ours (in-domain) reduces Set-L1 by 44.4% (.1136 vs. .2044), Centroid-L1 by 26.6% (.1418 vs. .1931), and regret by 15.3% (.0122 vs. .0144), while improving Pair Accuracy (Pair Acc.) (.9796 vs. .9683) and realized utility (.6602 vs. .6579).

Separate efficiency evidence. We revised the manuscript to separate transfer accuracy from inference characteristics. Table 12 reports prediction quality. Table 13 provides a controlled local efficiency comparison with the 8B direct LLMs on the same RTX A6000, while Table 14 separately reports application programming interface (API) latency, cost, and first-attempt output validity for the frontier LLMs. In the controlled local comparison, Language2Utility reduces median latency from 2073.90 ms for Qwen3-8B and 2575.68 ms for Ministral 3 8B to 116.84 ms by avoiding autoregressive decoding. Its softmax output also guarantees a valid simplex-valued preference vector without retries.

Overall, Language2Utility first improves over direct LLMs at the comparable 8B scale, and this advantage persists against the newly added 72B open-weight model and Gemini 2.5 Flash. It additionally provides structured simplex outputs through non-autoregressive inference, with approximately 18 to 22 times lower latency than direct LLMs under the controlled local setting. The stronger frontier models establish a higher zero-shot accuracy level, while the in-domain results show that dedicated target-domain grounding can surpass them on five of the six metrics.

Revision. We clarified transfer accuracy in Section 4.3.1 and Appendix D.1, organized the shared baselines in Table 12, and reported local and API efficiency separately in Tables 13 and 14.

AuthorsOpenReview ↗

Response to W3 and Q3: Centroid Regression

W3. A direct comparison with single-target regression is missing.

Table 2 suggests that the ranking and set losses work well together. However, it does not compare the set loss with regression to a single target weight vector using the same model. I’m not sure how much of the gain comes from using multiple targets.

Q3. How does performance change if the set loss is replaced with regression to the centroid of the valid weight set?

I’d like to see this comparison with the same architecture, ranking loss, and other settings. That would help show how much of the gain comes from using multiple valid targets.

Response to W3/Q3. We thank the reviewer for proposing this precise controlled comparison.

We added a controlled Ordinal + Centroid variant that retains the architecture and ordinal loss while changing only the numerical supervision target. Set-valued supervision achieves better mean admissible-set fidelity and ordering accuracy, while centroid regression primarily favors metrics directly aligned with its single target.

This pattern is consistent with the two supervision objectives: the full model preserves multiple semantically valid weights, whereas the centroid variant directly optimizes a single reference point.

Controlled supervision comparison. Ordinal + Centroid changes only the supervision target: it replaces the nearest-admissible-weight set loss with L1 regression to the admissible-set centroid, defined as the arithmetic mean of the admissible weights, while retaining the ordinal loss and all other settings.

The full model instead uses set-valued supervision since practical preference expressions may admit multiple semantically compatible numerical trade-offs, treating each admissible weight as a valid target rather than privileging the centroid. The table below reports means over five training seeds:

Values are means ± standard deviations over five training seeds. Bold indicates the better mean within each evaluation setting.

In-domain

MethodSet-L1 ↓Centroid-L1 ↓Ord. Acc. ↑Pair Acc. ↑J-lang ↑Reg-lang ↓
Ordinal + Centroid.1272 ± .0037.1183 ± .0081.9206 ± .0046.9786 ± .0013.6617 ± .0012.0106 ± .0012
Ours (full).1136 ± .0050.1418 ± .0049.9211 ± .0072.9796 ± .0008.6602 ± .0008.0122 ± .0008

Leave-one-domain-out (LODO)

MethodSet-L1 ↓Centroid-L1 ↓Ord. Acc. ↑Pair Acc. ↑J-lang ↑Reg-lang ↓
Ordinal + Centroid.3004 ± .0071.3244 ± .0075.7910 ± .0101.9142 ± .0059.6361 ± .0029.0362 ± .0029
Ours (full).2996 ± .0125.3284 ± .0092.8007 ± .0149.9173 ± .0079.6381 ± .0042.0343 ± .0042

The full model attains the better mean on Set-L1, Ord. Acc., and Pair Acc. in both settings, and on five of the six metrics under LODO. The centroid variant’s remaining gains are concentrated in centroid-aligned metrics, consistent with direct optimization of its single numerical target.

In-domain semantic and reference alignment. Admissible-set supervision reduces Set-L1 by 10.7%, from .1272 to .1136, while maintaining similar Ord. Acc. and Pair Acc. This result shows that retaining multiple valid targets improves recovery of the full admissible preference region. Centroid regression achieves lower Centroid-L1 and better in-domain utility and regret. This expected advantage follows from direct target-metric alignment: Centroid-L1 measures distance to the centroid, and under linear utility, uniformly averaged utility over the admissible set is equivalent to utility evaluated at its centroid.

Ordering and behavior under transfer. Under cross-domain zero-shot transfer, the two variants achieve similar Set-L1, .2996 versus .3004. The full set-supervised model nevertheless obtains higher Ord. Acc. (.8007 vs. .7910), Pair Acc. (.9173 vs. .9142), and realized utility (.6381 vs. .6361), together with lower language-conditioned regret (.0343 vs. .0362). Centroid regression retains a small advantage only on Centroid-L1.

Overall, the comparison distinguishes the two supervision objectives. Centroid regression is directly aligned with metrics defined by the central reference, whereas admissible-set supervision preserves the semantic multiplicity of the language preference. The latter yields more faithful recovery of the admissible region in-domain and stronger ordering and behavioral relevance under LODO transfer.

Revision. We added the new ablation to Table 2, the complete in-domain and LODO comparison to Table 16 in Appendix D.3, and the corresponding analysis to Section 4.2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Core Contributions, New Evidence, and Revisions

I. Acknowledgments

We thank the three reviewers and Area Chair for their careful evaluation. We appreciate their recognition of the problem's value, flexibility across changing objective spaces, treatment of ambiguity, and behavioral evaluation, as well as their constructive suggestions.

II. Core Contributions and Reviewer-Recognized Strengths

  1. Problem formulation and community value. We formulate transferable natural-language preference grounding across heterogeneous objective identities, compositions, and cardinalities as a simplex-valued prediction problem and benchmark for language-driven multi-objective reinforcement learning (MORL).
  2. Relational transfer mechanism and scientific insight. Preference-relative objective roles supply transferable structure. Frozen Joint representations encode semantic relations, and a permutation-equivariant grounding network learns their set-dependent numerical interpretation. Representation probes and leave-one-domain-out (LODO) ablations test these roles separately.
  3. Set-valued semantics and semantic-to-behavior connection. Admissible-set and ordinal supervision preserve multiple numerical trade-offs compatible with language. Set-L1, Pair Accuracy, utility regret, and Proposition 1 connect semantic grounding to behavioral consequences.
  4. Benchmark, evaluation, and reproducibility. Four MORL domains with 2, 3, 4, and 6 objectives, semantic-specification-level splits, human-authored evaluation, complementary metrics, and anonymized code make the claims reproducible and testable. The model guarantees simplex-valued outputs with efficient non-autoregressive inference.

III. Key Concerns and New Evidence

Key concernReviewer(s)Response and supporting evidence
Transfer novelty and roles of pretrained representations and grounding networkR2W2/Q2, R3W5Section 3.2.2, Figure 4, and Table 3 distinguish transferable semantic structure from learned numerical grounding. The full model leads all six aggregate LODO metrics among learned methods.
Stronger direct large language models (LLMs) and embedding-backbone dependenceR1W2/Q1, R3W1Added Qwen2.5-72B-Instruct and three-backbone comparisons in Tables 12 and 20. Language2Utility leads all six LODO metrics against the evaluated 8B and 72B open-weight direct LLMs. Frontier LLMs set a higher zero-shot accuracy level, while target-domain grounding surpasses all direct LLMs on five of six metrics.
Set-valued supervision versus centroid regressionR1W3/Q3, R3W2With matched architecture and ordinal loss, set-valued supervision improves admissible-set fidelity and ordering accuracy, while centroid regression favors metrics aligned with its single target (Table 2 and Appendix D.3).
Set-valued semantics versus point-valued inferenceR2W4/Q4Nearest-set supervision preserves multiple valid interpretations during learning, while simplex-valued inference produces a controller-compatible realization for immediate execution. Sections 3.1 and 3.3 clarify this deliberate separation between semantic multiplicity and downstream realization.
Benchmark construction, threshold sensitivity, and preference coverageR1Q2, R2W1/Q1, R2W5/Q5, R3W3Figures 17–19 and Table 19 test construction sensitivity. Expanded LODO evaluation over 386 distinct semantic preference specifications represented by 1,759 language realizations yields favorable gains on all six metrics, with every stratified 95% specification-level bootstrap interval above zero (Table 18).
Domain-level variation versus aggregate transferR3W4Revised Table 17 reports the Minecart exception and higher-cardinality gains. Tables 3 and 18 support the leading aggregate transfer result.
Scope and empirical tightness of Proposition 1R2W6/Q6, R3W5Sections 3.4 and 4.3 separate the architecture-independent consequence bound from transfer-design evidence. Empirical components in Table R1 show a conservative bound with diagnostic value.

IV. Revision Summary

We simplified Figure 1 and separated method details into Figure 2. We added a 72B direct LLM, two embedding backbones, and a centroid-regression control. We expanded LODO evaluation to 386 semantic preference specifications represented by 1,759 language realizations with stratified specification-level bootstrap intervals. We also added construction-sensitivity studies, deeper experimental analysis, and empirical regret-bound components. The expanded analysis yields favorable paired gains on all six metrics, with every 95% interval above zero.

Together, these revisions reinforce the central contribution: a principled and empirically supported interface for transferring natural-language preference semantics across changing objective spaces while retaining a direct connection to downstream behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.