acceptodds
Desk rejected at ICLR 2027

AutoValueWorld: Learning Value Dynamics for Consequence-Aware Decision Making

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

In multi-turn interactions, language model assistants can shape users' deliberation and the conditions for subsequent decisions. Responses with similar immediate ratings may support or violate human values differently, changing the relative utility of later actions. Response-level judgments alone don't specify how these value-relevant states evolve under successive responses. We introduce AutoValueWorld, an action-conditioned value world model that separates stable value importance from a dynamic, ten-dimensional value-support state. Training combines procedural supervision of state transitions and simulated reactions with human-rating supervision from PRISM. Candidate-difference supervision learns relative value consequences and recursive state prediction supports a frozen gain-based selector between one-step and three-step policies. Experiments show candidate-difference supervision reduces held-out candidate-contrast RMSE by 13.1% and 13.8% at 3B and 7B. In a study of 5,000 procedural episodes, selective lookahead improves mean realized return over one-step selection for value alignment at 3B and weighted reaction at 7B. These results support explicit value-dynamics modeling and selective lookahead for consequence-aware decision making in procedural environments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Models, Human Value Modeling, Model Grounding, Human Response Prediction, Transfer Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_z5sPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AutoValueWorld proposes a framework for choosing assistant responses based on their consequences for later decisions. It separates a user’s fixed priorities over ten human values from a changing state describing how the interaction supports or violates those values. The authors jointly train Qwen2.5-Instruct 3B and 7B models using a hand-designed simulator to supervise value-state transitions and simulated reactions, and PRISM human ratings to supervise immediate-rating prediction. Their main training contribution is a candidate-difference loss that explicitly teaches the model how alternative responses differ in their value consequences, reducing relative-state prediction error by 13.1–13.8%. They separately train a small tree-based selector to choose between immediate response selection and three-step planning using predicted consequences; across 5,000 synthetic episodes, selective planning produces small return improvements for two model–objective combinations. They also evaluate rating prediction on held-out PRISM participants and test transfer to HelpSteer2 human helpfulness labels.

Critical strengths
  1. The paper provides a new angle to candidate response selection in multi-turn conversations. It separates stable user value priorities from the interaction’s changing support or violation of those values. Their trained predictor could predict value-support state in longer horizon and provides the planner a signal to choose potentially based on long term value rather than immediate.
  2. The paper introduces a new candidate-difference loss and the control run comparing candidate-difference weights demonstrates that it is useful for consequence prediction.
Critical weaknesses
  1. The method's empirical performance improvement seems small and inconsistent. Based on the results, selective planning improves return for only two of four model–objective combinations and the gain is small. Therefore, this experiment results show that the method helps with consequence prediction (improve quite a bit) but only translate little to selection regret. This reads as an evidence that this selective planning is not necessary.
  2. The advantage over simpler baselines is not established. Reading from Table 8 , 9 and 10, structured ridge models seem to perform strongly on some dynamics and planning measures even outperforming AutoValueWorld. The paper also does not compare against preference or reward training that already uses multi-turn outcomes, leaving unclear when explicit value-state prediction is necessary.
Questions for authors
  1. See weakness 1. How do authors interpret that the method improves on consequence prediction but not selection itself.
  2. See weakness 2. If the simpler method already outperforms language-based predictor on several simulator metrics, what is the benefit of using this more complex method?
Reviewer_DsVeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors use a structural causal model (SCM) that generates synthetic data. The authors propose AutoValueWorld where it uses the output of the SCM and uses it to infer next value state (ten dimension vector), human rating and simulated reaction. Two models: Qwen 2.5 3B and 7B were used as the backbone of AutoValueWorld model. Model Performance was measured in various tasks such as candidate-difference prediction, action-order prediction,... etc. PRISM dataset was used as the primary dataset in this paper.

Critical strengths

The hypothesis to think about human values to improve decision making is exciting and it is also a promising direction where more work is needed.

Critical weaknesses

Validity of the data:

Synthetic data is used in this paper but data validation is not done. PRISM does not have Schwartz value annotations so there needs validation about data synthesis. There are other datasets such as ValueNet and ValueKaliedoscope that have value annotations. Also there are other datasets related to human values which could be used for decision making such as daily dilemmas. Without validation, relying heavily on PRISM especially with synthetically generated data questions the readers whether the findings are really generalizable.

Performance improvement:

The results show AutoValueWorld shows better scores on several metrics but the improvement seems very small. For example, the selective planning gain on 3B and 7B is +0.000617 and +0.002298. Similar for the value-ranking reversal accuracy. Whether this result is coincidence or actually meaningful improvement should be further discussed in the paper. Since improving decision making is the key claim of the paper this is a critical problem.

Questions for authors

How is the SCM configured such as the architecture, transition process and the parameters? Is it done through validation set sweep or is there intuition behind the architecture?

Does SCM and AutoValueWorld really improve decision making in realistic scenarios?

Why are recent text-based world models not heavily discussed in the related work section?

Reviewer_QA4yOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a value-recognized world model called AutoValueWorld, designed for agents in an environment of multi-round conversations. In each turn, the environment provides several (three in experiment) text options for response and the agent choose one to respond, the state change of the environment is controlled by a structural causal model.

Following the Schwartz value system, the paper first model the value status of each human user as a stable value importance and a context-dependent value current state, for each Schwartz value dimension. This paper trained a discriminative model from Qwen2.5-4B/7B by LoRA, estimating the next value status (based on the given text response), three-dimensional human reaction (acceptability, trust, and continuation) and human rating, from the inputs: context, action, prior, state, masks. The training ground truth is generated by the structural causal model of the environment on the first two output types and the PRISM dataset on human rating.

In decision making, two objectives are considered. One is the overall weighted human response (acceptability, trust and continuation) measures how positive the human user is on the response. The other is value alignment, measuring the importance-weighted positivity of the 10 value dimensions.

Two experiments are conducted to verify the model, a decision-making test in the environment simulator, and an estimation of the human ratings in PRISM and HelpSteer2.

Critical strengths
  1. The paper separates stable value importance from the current value support state. Previous work usually does not distinguish the two. It is useful for value-aware decision making, because what a user cares about and whether the current reply supports it could be different things.

  2. The paper does not focus on minimizing the average state error alone or optimizing the long-term effect only. The candidate difference loss focuses on the gap between replies, and the three-step plan is used only when the predicted gain is positive. It is a reasonable compositional strategy.

Critical weaknesses
  1. The environment, including the interaction pattern, the inner dynamics, are too simple to support "a world model". In my knowledge, the environment looks like a "converesation tree" in some games, which moves according to some particular rules (SCM) may have too few options in choosing actions.

  2. Since each action is a multiple-choice in given texts, the choices on value-state change effect is then quite limited. I guess changing the generator of response into an LLM and do the rejection-sampling is not an efficient solution since the sampling procedure is processed in a 10-dimensional space of values.

  3. Structure of value attributes might be more complicated in reality, not just an importance and a state.

  4. It is not very clear why the utilities are the appropriate targets to optimize in decision-making. It is known that following human value preference may lead to real world problems, and optimizing user's trust / continuation may lead to user's addictions to the AI tools. Moreover, in real applications, if decision is made to solve problems, simply aligning with human preference may not work.

  5. The model uses both the envrionment and the human rating as supervision signals, and in experiment verified both. But it is not clear for me how they are in fact connected to each other, not just as signals in the fine-tuned Qwen discriminators.

Questions for authors

As shown in the Weaknesses part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.