acceptodds
Withdrawn from ICLR 2027

EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Making

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Large language models (LLMs) are increasingly deployed as agents for multi-step decision-making, yet transfer poorly to unseen environments. World-model methods address this by training agents to predict future observations, at the cost of additional training and errors that compound when predictions are used for planning. However, for LLM agents operating in digital environments, much of this world knowledge is already internalized during pretraining, which shifts the problem from acquiring it to eliciting it. We argue that typical post-training provides little pressure for such elicitation, since supervision under a single goal at each visited state inadvertently drives policies to rely on superficial contextual habits. We introduce EVOKE, a post-training method that supplies this pressure through goal diversity at fixed states. Motivated by theory shows that an agent competent across diverse goals must encode a world model recoverable from its action preferences, EVOKE holds the environment state and interaction history fixed and evaluate the same candidate actions under alternative goals, forcing action preferences to change, so that a policy relying on contextual habits or single-goal correlations cannot rank actions correctly. This implicitly elicits the policy's pretrained world knowledge to inform decisions. We evaluate EVOKE across diverse tasks in three backbones, demonstrating improved task performance, unseen environment generalization, and sample efficiency. We further conduct controlled analyses to better understand what drives these gains. These findings offer a new perspective on eliciting internalized world knowledge for transferable action through direct decision supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent, LLM Agent, Decision-making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ppgvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EVOKE, a post-training method that that inject goal-diversity in policy optimization, and using contrastive ranking to train the model. The paper shows not the strongest but promising results under three small open source models and three types of benchmarks. The author also provides ablation to study how goal intervention and diversity affects the model's performance.

Critical strengths
  1. The entire methodology is simple yet effective. Between all four stages, they are fairly easy to implement and do not depends on specific hardcoded conditions too much. E.g. Multi-positive ranking within each goal is a simple and intuitive design but it aligns with the entire storyline, and the ablation shows that we need multi-positive ranking.

  2. The authors put significant efforts on benchmarking the idea. We can see that this training method does not perform the best for all tasks, but there are a large amount of settings where EVOKE performs the best or seconds best, comparing to different post training variants and world-model methods.

  3. Good ablation studies to present the components of EVOKE.

Critical weaknesses
  1. It looks like the algorithm is RL related, and it requires goal intervention to update the model, so that it might end up more cost. I think the paper lacks discussion on the cost estimation (dollars & FLOPs), comparing to other baselines.

  2. The entire section 3 is about EVOKE, where I did not find out where the exact algorithm of EVOKE is. I think the authors would provide a clearer version of algorithm description. Differences between EVOKE and other RL algorithms can be potentially clearer, in section 2 or section 3.

Questions for authors
  1. Any comparison to iterative DPO or something similar? If there are baselines that is an variant of iterative DPO, it's great to point it out. The reason is that the entire looks to me like a variant of iterative DPO, where DPO can use Plackett-Luce model ( a more general version of Bradley Terry model) to deal with preference under multiple choices. Other than that, step2 and step3 serves a ranking of the choices.

  2. Follow question 1, it would be great if the author clarifies more on the algorithm part. Give the figure, I saw policy rollout. Are we using any policy gradient algorithm over here (e.g. PPO, GRPO, etc)? Or it's the same pipeline of SFT, but with ranking and a customized objective? If it's latter, than I personally would categorize it as a variant of DPO, where the authors should explain more clearly, either mention the connection between EVOKE and DPO, or correct me over here for clearer context.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ppgv W2 and Q2: Training Algorithm (Part 1/3)

Thank you for recognizing the simplicity of the framework and the breadth of the experiments. Your questions help us explain the optimization procedure and its relationship to iterative preference learning more directly.

W2 / Q2) What is the exact algorithm, and are rollouts used for PPO or GRPO updates?

EVOKE uses supervised preference updates, not a policy-gradient update. Rollouts collect states visited by the current policy; candidate executions provide grounded outcomes for preference labels. Training then optimizes action rankings conditioned on each goal.

The complete loop is:

Initialize the policy with demonstration SFT.
For each round:
  1. Roll out the current policy in training environments.
  2. Collect decision states and their interaction histories.
  3. At each state, retain the original goal and propose achievable alternatives.
  4. Execute candidates from the shared state and history.
  5. Label goal-advancing actions and select policy-favored competitors.
  6. Aggregate preference groups from all rounds.
  7. Update the policy with the set-plus-pair ranking loss.
At deployment, execute the trained policy directly.

For each goal, action scores are mean token log-probabilities. The set term increases the total probability mass of positive actions under the candidate softmax; the pairwise logistic term ranks every positive above every competitor. There is no PPO/GRPO return or advantage estimator, and no frozen reference-policy log-ratio in EVOKE's objective.

Updated policies visit new states and expose new high-probability errors, which supply the next round's preference groups. This iterative data collection is the role of policy rollouts in our framework.

AuthorsOpenReview ↗

Response to Reviewer ppgv Q1: Relationship to DPO and Additional Comparison (Part 2/3)

Q1) How does EVOKE relate to iterative DPO and preference models over multiple choices?

We agree that EVOKE is closely related to iterative preference learning, and that Plackett–Luce preference models can handle multiple candidates. EVOKE's contribution centers on fixed-state goal interventions that organize action preferences across goals, together with iterative collection of the current policy's hard negatives.

With standard single-goal supervision, a familiar context can be associated with a routine action. EVOKE instead holds the state, history, and candidate actions fixed and changes the goal. When preferred actions differ, a goal-blind ranking cannot satisfy both contexts. This supervision structure can also be used with standard DPO.

We provide an additional comparison using Qwen2.5-3B on ALFWorld. Both DPO variants use β = 0.1, the same initialization and matched update counts. Evaluation counts successful games directly: 140 Seen and 134 Unseen games.

Table R2. Controlled comparison of goal interventions and preference objectives.

Training settingSeenUnseen
Original goals + standard DPO82.14% (115/140)78.36% (105/134)
Goal interventions + standard DPO90.00% (126/140)85.82% (115/134)
Goal interventions + EVOKE ranking90.00% (126/140)91.79% (123/134)

Adding goal interventions improves standard DPO by 7.46 percentage points on unseen games. With goal-intervention supervision, EVOKE's ranking objective gives a further 5.97-point improvement.

The two DPO arms use the collected two-round data, testing the intervention and objective within our data pipeline. At the objective level, standard DPO uses a frozen-reference log-ratio, whereas EVOKE uses reference-free, length-normalized scores with a positive-set term and a pairwise term. These results clarify both the connection to DPO and the contribution of the fixed-state supervision design.

AuthorsOpenReview ↗

Response to Reviewer ppgv W1: Compute and Interaction Costs (Part 3/3)

W1) Could goal interventions make the method more expensive than RL or world-model pipelines?

We provide an estimated resource comparison at a common 7B policy scale. EVOKE's data-construction budget includes goal and candidate generation, 72B outcome verification, environment restoration or replay, and short continuations.

Table R1. Estimated resource budgets. Setup includes acquisition and policy training, including initialization; deployment compute is per 1,000 decisions.

ResourceEVOKEGRPOITP-R
Data acquisition (EFLOPs)2.116.893.74
Policy training (EFLOPs)1.1635.149.65
Total setup (EFLOPs)3.2742.0313.39
Environment interactions164.4k775.7k73.4k
Deployment (PFLOPs / 1,000 decisions)14.5614.56130.8

All methods use the same compute approximation: approximately 2NT FLOPs for forward computation and 6NT for training, with N parameters and T processed tokens.

EVOKE uses approximately 12.9× less setup compute than GRPO and 4.1× less than ITP-R, 4.7× fewer interactions than GRPO, and 9.0× less deployment compute than ITP-R. Its branching expense is incurred while constructing preferences. Training operates on those preference groups, and deployment runs the policy without an annotator or lookahead world model.

This separates the computational effect of goal interventions from the much larger cost of repeated online RL optimization or lookahead inference, and makes the efficiency comparison explicit beyond the number of labeled states.

Reviewer_irGtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EVOKE, a post-training framework that aims to elicit the world knowledge already encoded in pretrained LLM agents rather than learning an explicit world model. EVOKE keeps the environment state, interaction history, and candidate actions fixed while introducing alternative goals, and trains the policy to rank the same actions differently using grounded preference labels derived from real environment executions. Experiments on ALFWorld, WebShop, and seven search-based QA benchmarks across three Qwen backbones show improved task performance, generalization to unseen environments, and data efficiency.

Critical strengths
  1. By introducing alternative goals while keeping the environment state, interaction history, and candidate action set fixed, EVOKE can induce preference reversals over the same actions. This design effectively isolates the influence of the goal and provides a more direct way to examine whether the policy is genuinely goal-conditioned or merely following context-dependent habitual actions.

  2. The empirical evaluation is broad and covers multiple agent settings and model scales. The paper evaluates EVOKE on ALFWorld, WebShop, and seven search-based QA benchmarks, using Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B as backbones. This coverage across household interaction, web navigation, and information-seeking tasks provides stronger evidence than an evaluation restricted to a single embodied benchmark or a single model size.

Critical weaknesses
  1. The method relies heavily on a strong Qwen2.5-72B-Instruct annotator. The annotator is used not only to generate alternative goals, but also to suggest candidate actions and classify executed actions as advancing, worse, or uncertain. This introduces a potential teacher-knowledge confound: part of the improvement may come from transferring task knowledge from the 72B model to the smaller policy, rather than purely from eliciting knowledge already encoded in the policy itself.

  2. The training-time data construction pipeline may be expensive, but its cost is not sufficiently reported. For each state–goal context, multiple candidate actions must be executed from the same state, sometimes followed by a short continuation to reveal their consequences. Figure 2 further indicates that the environment is reset for each candidate action. Although EVOKE does not require an additional module at deployment, the paper does not report the number of environment transitions, state resets, continuation steps, annotator calls or tokens, GPU hours, wall-clock time, or total financial cost. Without such information, it is difficult to assess the scalability and practical efficiency of the method.

  3. The ablation of the proposed ranking objective is incomplete. The final objective combines a multi-positive listwise loss �0��0� and a pairwise ranking loss �1��1�, with fixed values of �2��2� and �3��3�. However, the paper does not separately evaluate the two loss components or report sensitivity to �4�,�5��4�, �5�, the number of positive and negative actions, or the preference-group size. Consequently, it remains unclear which component of the objective is primarily responsible for the observed improvements and how robust the method is to these design choices.

Questions for authors
  1. Could the authors provide a complete accounting of the training and data-construction costs of EVOKE? In particular, please report the total number of environment transitions, state resets, short-continuation steps, annotator calls and tokens, GPU hours, wall-clock time, storage requirements, and, where applicable, API or inference costs. It would also be helpful to compare these costs with those of the RL, distillation, and world-model baselines under a comparable performance or interaction budget.

  2. How frequently does genuine preference reversal occur in the training data? Since the central mechanism of EVOKE is that changing the goal at a fixed state and history changes the preferred action, please report statistics such as the average number of proposed and retained alternative goals per state, the fraction of state–goal pairs for which the preferred action actually changes, the fraction with disjoint positive-action sets, and the distribution of preference reversals across task categories.

  3. The current pipeline requires executing several candidate actions from exactly the same state and resetting the environment after each branch. This assumption is convenient in simulators such as ALFWorld, but may not hold for dynamic websites, real external APIs, irreversible operations, stochastic environments, or physical robots. Could the authors clarify how state restoration and branching are implemented in WebShop and search-based QA, and discuss what modifications would be required in non-resettable or stochastic environments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer irGt W1: Annotator Role and Controlled Comparisons (Part 1/3)

Thanks for your detailed feedback and for highlighting the value of isolating goal changes at a fixed state and history. We address the annotator, data construction, preference statistics, and optimization questions below.

W1) Could the gains come from knowledge transferred by the 72B annotator?

The 72B model proposes alternative goals and candidate actions, and assesses goal progress from observed outcomes after executing the candidate actions. Its assessments construct preference labels; the policy receives the goal, history, and available actions as input. The annotator is not used at deployment.

Our contribution concerns how this supervision is organized. EVOKE holds the state and history fixed while varying the goal, then trains the policy to prefer actions whose executed consequences advance that goal. Two comparisons help establish the contribution of this design under shared annotator supervision:

ComparisonShared componentsUnseen success
Source replay72B outcome verifier, step-level ranking, matched updates; original goals85.9%
Positive SFTSame goal contexts and positive actions as EVOKE84.3%
EVOKEGoal interventions and multi-positive ranking91.8%

EVOKE improves over Source replay by 5.9 percentage points, and over Positive SFT by 7.5 points. These results show the value of goal interventions and preference ranking beyond original-goal supervision or imitation of the same positives.

Our broader motivation is supported by Richens et al., General agents contain world models [1]: sufficiently general goal-conditioned policies contain extractable environmental predictive information under the paper's assumptions. In our setting, consequence probes already achieve 97.7–99.6% balanced accuracy before ALFWorld post-training. EVOKE studies how training can connect such existing consequence representations to goal-conditioned action choices, and our controlled comparisons test the effect of the supervision design.

References

[1] Richens, Jonathan, et al. "General agents contain world models." ICML, 2025.

AuthorsOpenReview ↗

Response to Reviewer irGt W2 and Q1: Data-Construction and Deployment Costs (Part 2/3)

W2 / Q1) What is the cost after accounting for annotation, branching, short continuations, and policy training?

We provide a four-part resource accounting at a common 7B policy scale. The acquisition budget includes alternative-goal and candidate generation, 72B outcome verification, state restoration or action-prefix replay, and short continuations.

Table R1. Estimated resource budgets. Setup combines acquisition and policy training, including initialization; deployment compute is per 1,000 decisions.

ResourceEVOKEGRPOITP-R
Data acquisition (EFLOPs)2.116.893.74
Policy training (EFLOPs)1.1635.149.65
Total setup (EFLOPs)3.2742.0313.39
Environment interactions164.4k775.7k73.4k
Deployment (PFLOPs / 1,000 decisions)14.5614.56130.8

Compute is estimated using the same dense-model convention across methods: approximately 2NT FLOPs for forward computation and 6NT for training, where N denotes parameters and T processed tokens.

The resulting comparison is 12.9× / 4.1× lower setup compute than GRPO / ITP-R, 4.7× fewer interactions than GRPO, and 9.0× lower deployment compute than ITP-R. This accounting separates labeled-data efficiency from the computational cost of constructing and learning from those labels.

The deployment procedure is particularly simple: one policy selects the next action from the current goal and history, without invoking the annotator, predicting future states with an auxiliary model, or conducting lookahead search.

AuthorsOpenReview ↗

Response to Reviewer irGt W3 and Q2–Q3: Preference Structure, Branching, and Ranking (Part 3/3)

Q2) How often do goal changes produce disjoint positive-action sets?

The two-round ALFWorld training pool contains 767 states and 1,482 goal contexts. Among the 686 states with one original-goal context and one alternative-goal context, each represented by a complete preference group, 419 have disjoint positive-action sets: 61.1%. The other 267 pairs have overlapping positive sets. This reports the structure of the training labels at fixed states, directly complementing the held-out counterfactual decision test.

Q3) How are fixed-state branches implemented?

  • ALFWorld: restore the recorded environment state before executing each candidate.
  • WebShop: replay the recorded action prefix to reconstruct the branch point.
  • Search-based QA: reuse the same retrieval-history prefix and evaluate candidate continuations from that shared context.

For non-resettable external systems, training-data construction would require a replayable sandbox or independently reproducible contexts; stochastic branching would require repeated samples to assess outcomes. These requirements concern preference collection. Deployment uses the policy directly and requires no branching or restoration.

W3) What are the roles of the two ranking terms?

The set term increases the aggregate probability mass of goal-advancing actions, allowing several valid positives. The pair term increases every positive action's score relative to each competitor. Action scores are length-normalized policy log-probabilities.

In the additional fixed-β DPO comparison, standard DPO with goal-intervention data achieves 85.82% unseen success, while EVOKE's set-plus-pair ranking achieves 91.79%. The existing negative-selection comparison also holds goals, positives, and the ranking objective fixed:

Competitor selectionUnseen success
Policy-favored actions91.8%
Other goals' positive actions79.9%
Random valid actions74.6%
Least-likely actions67.2%

These comparisons establish the practical value of the ranking design and of targeting mistakes the current policy is likely to make.

Reviewer_kBCDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EVOKE trains agents to rank candidate actions under different goals while holding the state and history fixed. It combines preferences grounded in executed outcomes with iterative training, aiming to elicit pretrained world knowledge for transferable decisions.

Critical strengths
  1. Changing goals at the same state directly discourages goal-insensitive action habits, while deployment requires only the trained policy.
  2. On Qwen2.5-3B, ALFWorld unseen micro success reaches 91.8%, versus 85.9% for Source replay and 84.3% for Positive SFT. Ablations address training updates, supervision volume, and negative selection.
Critical weaknesses
  1. The central claim is hard to be verified directly by the experimental results. Solving ALFWorld tasks and observing better goal-conditioned decisions do not establish that improvements causally arise from using pretrained consequence knowledge. I think that learning better decision rules mainly from additional preference supervision remains a plausible explanation.
  2. Results on WebShop and QA do not establish transfer across environment families or unfamiliar action semantics, limiting the broader contribution to transferable world knowledge.
  3. The practical advantage over world-model training remains unclear. EVOKE requires a 72B annotator and repeated execution of candidate actions from the same state. Efficiency measured by labeled-state counts does not establish lower total cost, and comparisons do not establish matched interaction, annotation, and training budgets.
Questions for authors

See above

3 comments
AuthorsOpenReview ↗

Response to Reviewer kBCD W1: Motivation and Evidence (Part 1/3)

Thanks for your careful review and for recognizing the value of fixed-state goal interventions and the controlled ablations. We address your concerns about the interpretation of our results, generalization, and practical cost in three parts.

W1) Do the improvements reflect the use of pretrained consequence knowledge, or simply additional preference supervision?

Our motivation builds on Richens et al., General agents contain world models [1], which establishes, under its formal conditions, that sufficiently general goal-conditioned policies contain extractable predictive information about their environments. This motivates studying how information represented within a policy can support flexible decisions across goals.

EVOKE addresses the gap between representing action consequences and choosing actions according to the active goal. At a fixed state and history, an action has the same environmental consequence across goals, but that consequence can be useful for one goal and counterproductive for another. Supervising these differing preferences discourages context-only action habits. Our evidence examines this from three complementary perspectives:

  1. Consequence representations before post-training. On same-action transitions with different outcomes, probes on the pretrained Qwen2.5-3B backbone achieve 97.7–99.6% balanced accuracy across five consequence attributes. For Held and Contained, the pretrained model achieves 97.7% and 98.4%, compared with 65.0% and 68.0% for the same architecture with random weights.
  2. Controlled supervision. Source replay matches training updates while retaining the same 72B outcome verifier and step-level ranking on original goals. EVOKE achieves 91.8% unseen success versus 85.9% for Source replay. Positive SFT uses the same goal contexts and positive actions but achieves 84.3%. In a further fixed-budget comparison using 600 states and matched updates, replacing half of the original goals with alternative goals increases unseen success from 78.8% to 84.5%.
  3. Goal-conditioned behavior. On 812 counterfactual goal pairs in unseen environments, where the state, history, and candidate actions are shared across goals, EVOKE reduces habitual errors from 23.3% for the initialization policy to 6.4%, a 72% relative reduction. The corresponding error rates are 13.3% for Source replay and 10.7% for Positive SFT.

Together, these results support our practical contribution: fixed-state goal interventions improve goal-conditioned decisions in models that already encode action-consequence information. The probes characterize the available representations; the controlled comparisons and counterfactual tests establish the behavioral effect of our training design.

References

[1] Richens, Jonathan, et al. "General agents contain world models." ICML, 2025.

AuthorsOpenReview ↗

Response to Reviewer kBCD W2: Generalization Settings (Part 2/3)

W2) What kind of transfer is supported by the ALFWorld, WebShop, and QA experiments?

The experiments evaluate generalization within each domain and the applicability of the same training framework across domains. Each domain has its own trained policy. ALFWorld evaluates the resulting policy on unseen environments with parameters frozen, while WebShop and search-based QA use their respective held-out evaluation settings.

The method is evaluated across household interaction, web navigation, and seven search-based QA benchmarks using three backbones. For Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B, respectively, WebShop success rates are 82.8%, 85.9%, and 80.5%; average QA scores across the seven benchmarks are 45.3%, 49.2%, and 43.9%.

The counterfactual analysis provides a more targeted test of the intended transfer behavior: at the same unseen state and history, the policy must respond differently when the active goal changes. This directly tests whether it follows the goal rather than reusing a familiar action. Joint correctness across both goals increases from 24.4% for Source replay and 43.6% for Positive SFT to 46.8% for EVOKE.

Thus, our empirical contribution concerns improved decisions on held-out tasks and environments, together with a training procedure that works across three distinct agent domains. These are separate domain-specific training and evaluation experiments, rather than deployment of an ALFWorld-trained policy directly in WebShop.

AuthorsOpenReview ↗

Response to Reviewer kBCD W3: End-to-End Cost Accounting (Part 3/3)

W3) Does the practical advantage remain after including annotation, branching, and training costs?

Thank you for requesting a fuller accounting. We provide estimates at a common 7B policy scale, separating data acquisition, policy training, environment interaction, and deployment. The data-construction accounting includes goal and candidate generation, 72B outcome verification, state restoration or replay, and short continuations.

Table R1. Estimated compute and interaction budgets. Setup compute is acquisition plus policy-training compute, including initialization. Deployment compute is per 1,000 decisions.

ResourceEVOKEGRPOITP-R
Data acquisition (EFLOPs)2.116.893.74
Policy training (EFLOPs)1.1635.149.65
Total setup (EFLOPs)3.2742.0313.39
Environment interactions164.4k775.7k73.4k
Deployment (PFLOPs / 1,000 decisions)14.5614.56130.8

We use a common dense-model FLOP approximation, with forward computation proportional to 2NT and training computation to 6NT, where N is parameter count and T is processed tokens. This compares the resource requirements of the respective pipelines under a shared accounting convention.

EVOKE requires approximately 12.9× less setup compute than GRPO and 4.1× less than ITP-R, 4.7× fewer environment interactions than GRPO, and 9.0× less deployment compute than ITP-R. At deployment, EVOKE uses only the trained policy, with no annotator or lookahead world model.

These quantities distinguish the sources of efficiency: compact preference construction and training reduce setup compute; avoiding online RL rollouts reduces interactions relative to GRPO; policy-only execution reduces deployment compute relative to lookahead planning. We appreciate your suggestion to make these resource dimensions explicit.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response and Summary of Key Clarifications and Results

Dear Area Chair and Reviewers,

Thank you for your careful reviews and constructive feedback. We appreciate your recognition of fixed-state goal interventions, the breadth of the evaluation, and the controlled ablations. Our responses clarify the research motivation and training algorithm, provide additional DPO and resource comparisons, and report training-data preference statistics. We highlight the key clarifications and results below. The companion post, Summary of Consolidated Reviewer Feedback, organizes the reviewers' strengths and concerns and points to the corresponding detailed replies.

  1. From consequence representations to goal-conditioned decisions (kBCD, Part 1/3):

    Our motivation builds on Richens et al., General agents contain world models [1], which links sufficiently general goal-conditioned behavior to extractable environmental predictive information under its formal conditions. EVOKE studies how fixed-state goal interventions can connect action-consequence information to the active goal during decision-making. Our evidence combines pretrained consequence probes (97.7–99.6% balanced accuracy), controlled supervision comparisons, and counterfactual behavior tests. On 812 unseen-environment goal pairs, habitual errors decrease from 23.3% for the initialization policy to 6.4%, a 72% relative reduction. See kBCD, Part 1/3.

  2. Supervision design under shared annotator supervision (irGt, Part 1/3):

    The 72B annotator proposes goals and candidates and labels observed post-execution outcomes. Source replay shares the outcome verifier, step-level ranking, and training-update budget, achieving 85.9% unseen success versus 91.8% for EVOKE. Positive SFT receives the same goal contexts and positives and achieves 84.3%. These comparisons show the value of goal intervention and ranking beyond original-goal supervision or imitation of the same positives. See irGt, Part 1/3.

  3. Algorithm clarification and an additional standard-DPO comparison (Table R2 in ppgv, Part 2/3):

    EVOKE uses rollout for data collection and supervised preference ranking for policy updates. At fixed DPO β = 0.1, unseen success increases from 78.36% with original goals to 85.82% with goal interventions; EVOKE's set-plus-pair ranking reaches 91.79%. Thus, goal interventions improve standard DPO, and our ranking objective provides a further gain. See ppgv, Parts 1/3 and 2/3, including Table R2.

  4. Resource accounting across acquisition, training, interaction, and deployment (Table R1 in kBCD, Part 3/3):

    At a common 7B policy scale, estimated setup compute is 3.27 / 42.03 / 13.39 EFLOPs for EVOKE / GRPO / ITP-R. EVOKE's acquisition budget includes goal and candidate generation, 72B verification, state restoration or replay, and short continuations. EVOKE uses 4.7× fewer interactions than GRPO and 9.0× less deployment compute than ITP-R. The full resource breakdown appears in Table R1 in the cost response under each review.

  5. Preference statistics and environment implementation (irGt, Part 3/3):

    Among 686 training states with complete original/alternative-goal preference groups, 419 (61.1%) have disjoint positive-action sets. Fixed-state branches use state restoration in ALFWorld, action-prefix replay in WebShop, and shared retrieval-history prefixes in search-based QA. The three-domain experiments evaluate domain-specific policies on their respective held-out settings. See irGt, Part 3/3, and kBCD, Part 2/3.

We thank the reviewers for helping us present more clearly how EVOKE's fixed-state goal interventions, iterative hard negatives, and grounded action preferences improve goal-conditioned decisions across agent tasks.

References

[1] Richens, Jonathan, et al. "General agents contain world models." ICML, 2025.

AuthorsOpenReview ↗

Summary of Consolidated Reviewer Feedback

Below, we summarize the strengths and key concerns raised across the reviews, together with our corresponding responses. This post consolidates overlapping points; detailed evidence appears in the individual reviewer replies.

Strengths

  1. Fixed-state goal interventions (kBCD, irGt). Both reviewers recognize that changing goals while keeping the state and history fixed discourages goal-insensitive action habits and offers a direct test of goal-conditioned behavior.

  2. Broad empirical evaluation (irGt, ppgv). The reviews highlight the evaluation across household interaction, web navigation, and seven search-based QA benchmarks using three model backbones.

  3. Simple framework and informative controls (kBCD, ppgv). Reviewers appreciate the simple training pipeline and ablations addressing supervision, update budgets, and negative selection. kBCD also highlights policy-only deployment.

Key Concerns and Responses

  1. Interpretation of pretrained knowledge and the 72B annotator (kBCD, irGt).

    Concern: Whether gains reflect better use of pretrained consequence information, additional preference supervision, or knowledge supplied by the annotator.

    Response: We connect the theoretical motivation to three complementary sources of evidence: consequence probes before post-training, controlled supervision comparisons, and counterfactual goal-conditioned behavior. Source replay shares the outcome verifier, ranking objective, and update budget; Positive SFT shares the goal contexts and positives. These comparisons identify the contribution of the supervision design under shared annotation.

    Detailed responses: kBCD, Part 1/3; irGt, Part 1/3.

  2. Full resource accounting (kBCD, irGt, ppgv).

    Concern: Retained-state counts alone do not capture annotation, environment branching, training, and deployment costs.

    Response: We provide a common-scale accounting of acquisition and training compute, environment interactions, and deployment compute. Acquisition includes goal and candidate generation, 72B verification, restoration or replay, and short continuations. Each efficiency comparison specifies the resource and baseline concerned.

    Detailed responses: Table R1 in kBCD, Part 3/3; irGt, Part 2/3; ppgv, Part 3/3.

  3. Algorithm, DPO relationship, and ranking design (ppgv, irGt).

    Concern: The exact role of rollout, the relationship to iterative DPO and multi-candidate preferences, and the role of the ranking objective.

    Response: We give the complete training loop and distinguish rollout-based collection from supervised preference updates. The additional fixed-β DPO comparison separates the gain from goal interventions from the further gain obtained with EVOKE ranking. We explain the set and pair terms and report the existing controlled comparison of competitor sources.

    Detailed responses: ppgv, Parts 1/3 and 2/3; Table R2 in ppgv, Part 2/3; irGt, Part 3/3.

  4. Preference reversals and environment branching (irGt).

    Concern: How often training goals induce different positive sets, and how branches are reconstructed across environments.

    Response: We report disjoint-positive statistics on complete paired training contexts and describe ALFWorld state restoration, WebShop action-prefix replay, and shared QA retrieval histories. We also distinguish the requirements of preference collection from policy-only deployment.

    Detailed response: irGt, Part 3/3.

  5. Scope of generalization (kBCD).

    Concern: What the three-domain evaluation establishes about transfer.

    Response: We specify the domain-specific training and held-out evaluation settings, and distinguish the framework's applicability across domains from each policy's generalization within its evaluation domain. The unseen-state counterfactual test directly examines adaptation of action choice to a changed goal.

    Detailed response: kBCD, Part 2/3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.