acceptodds
Desk rejected at ICLR 2027

See Better, Foresee Better, Act Wiser: Physically Grounded Proactive Modeling and Decision Making

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable human-centered decision making in physical environments requires proactive agents not only to choose what action to take, but also to judge whether the current evidence is sufficient to act. This judgment is not an isolated property of the decision module; it directly reflects how deeply the agent understands the task context of the physical world. We study proactive retail service from sparse third-person video. Before a customer makes an explicit request, the agent must use limited human–object interaction evidence to choose between intervening to guide the interaction and remaining silent to continue observing. We use physical grounding narrowly: converting real human–object observations into a task-relevant retail state, not modeling low-level physical dynamics. We introduce the Proactive Intent World Model (PIWM), organized around three functional capabilities: See constructs the perceptual basis, Foresee models counterfactual consequences, and Act selects the subsequent action. Our experiments reveal that performance is poor when the agent must extract information from raw video and decide directly, but improves substantially when it receives structured inputs extracted and annotated from a professional retail perspective. Reliable intervention therefore requires role- and goal-directed attention that selects and organizes the cues that matter for service decisions. Ablations involving AIDA-stage constraints and BDI state structure further support this finding. PIWM's counterfactual prediction remains strong in standalone evaluation, yet planning methods that explicitly query these forecasts at inference time degrade sharply. Locally useful consequence prediction therefore does not yet translate reliably into better action selection. This gap may reflect incomplete process understanding, uncertainty in fine-grained single-step outcomes, and insufficient joint modeling of the physical scene and its temporal evolution. remains the hardest action in structured-state evaluation, exposing the same challenge of temporal awareness in proactive decisions. PIWM thus advances static intent recognition toward intent world modeling: an agent must organize physical observations under professional task knowledge, anticipate how candidate interventions affect interaction state, and treat intervention and non-intervention as joint decisions. Future work will introduce long-horizon interaction trajectories and temporal consequence supervision to improve sustained reasoning about evolving situations and intervention timing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
grounded vision-language models, proactive intervention, world models, counterfactual reasoning, multimodal decision making, human-centered AI, retail video, risk-aware non-intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jHKLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Targeting proactive retail assistance, this paper proposes a three-stage See–Foresee–Act framework instantiated as the PIWM model. The See module extracts customer behavioral evidence from sparse videos, and outputs a structured customer representation with both explicit observation fields and inferred states, integrating the AIDA purchasing stage model and the BDI cognitive framework. Conditioned on the current state, the Foresee module predicts one-step normative consequences (including next-stage state, expected benefits and risks) for five candidate actions: Greet, Elicit, Inform, Recommend, and Hold. The Act module outputs the final intervention decision by explicitly comparing predicted consequences across all candidates. In addition, the authors construct GuidanceSalesBench, a benchmark consisting of rule-guided hierarchical synthetic annotations and a small set of real-store samples, supporting both component-level evaluation and end-to-end full-pipeline assessment. Experimental results show that the consequence-based decision paradigm achieves an 18.27% accuracy improvement over direct action output from states, and verify the positive effects of explicit evidence retention and non-intervention consequence modeling on decision performance.

Overall, this work delivers a systematic pipeline design and benchmark construction for proactive retail assistance with sound engineering completeness. However, debatable issues remain regarding the rigor of core concepts, the increment of academic novelty, and the rigor of experimental design.

Critical strengths

1.Systematic solution design for retail scenarios This work clarifies the task definition and decision boundary of proactive retail assistance, and provides a complete structured pipeline from visual observation to intent modeling and intervention decision-making. It also includes non-intervention as an equivalent decision option in the framework. With self-consistent design logic, it has clear reference value for the development of retail intelligent customer service and in-store shopping guide systems.

2.Instrumental contribution of the hierarchically annotated benchmark GuidanceSalesBench adopts a four-layer annotation structure (observation-state-consequence-action), supporting independent evaluation and optimization of perception, prediction and decision modules. Different from traditional end-to-end datasets, it helps follow-up research accurately locate module bottlenecks and has good resource value for the field.

Critical weaknesses

1.Imprecise core concept definition with limited academic novelty and technical depth Terminology usage deviates from field consensus: the paper frames the single-step state classification Foresee module as a world model (Section 3.3) and labels candidate outcome comparison as counterfactual reasoning (Table 2), neither aligning with mainstream field definitions and indicating insufficient terminological rigor. Meanwhile, the overall framework is essentially a scenario transfer of the classic perception-prediction-decision pipeline. Both AIDA and BDI are well-established theories; the model only applies LoRA fine-tuning on a general large model (Section 3.5) and restricts the task to 5-way classification at a single decision point (Section 1), with no new theory or structural innovation proposed and limited academic depth.

2.Insufficient experimental rigor weakening credibility and generalizability of core conclusions First, benchmark annotations are rule-generated (Section 4.1), so model performance largely reflects fitting to annotation rules, with an unclear generalization boundary to real scenarios. Second, the evidence ablation modifies inputs to both Foresee and Act simultaneously (Section 5.3), and variable confounding leads to ambiguous attribution of performance changes. Third, test sample sizes differ between baselines and the proposed model in real-scenario evaluation (Appendix Table 10), with inconsistent comparison calibers, which undermines the empirical persuasiveness of the conclusions.

Questions for authors
  1. Clarification on concept definition The paper refers to the Foresee module as a world model, but it only performs single-step structured state classification, which differs greatly from the mainstream world model paradigm of latent-space dynamics modeling and multi-step trajectory generation. How do the authors define the connotation of world model in this work? Have you considered adjusting the wording to better align with field consensus and avoid ambiguity? Please provide the necessary explanations or make the necessary revisions.

  2. Generalization boundary clarification Section 4.1 states that core annotations of GuidanceSalesBench are generated from “27 compact principles and 78 source-linked conditional rules”, constituting rule-guided synthetic data, so model performance essentially reflects fitting to annotation specifications. To what extent do the authors believe conclusions from this dataset can be transferred to real retail scenarios? Are there further plans for real-scenario validation? Please provide the corresponding explanations.

  3. Ablation experimental design As described in Section 5.3, the evidence ablation removes explicit evidence fields from inputs to both Foresee and Act and regenerates all candidate consequences, making it impossible to precisely locate the source of performance degradation. Have the authors supplemented a control condition where evidence fields are removed only from Act’s input while Foresee inputs remain unchanged? If not, do you acknowledge that the current attribution conclusion has ambiguity? Please provide the corresponding experiments and explanations.

  4. Fairness of real-scenario comparison According to Table 10 in the appendix, baseline large models are evaluated on only 237 samples in the real-store experiment, while PIWM covers all 418. Test sample sizes differ between baseline models and PIWM. What are the filtering criteria for the 237 matched samples? Could you supplement PIWM’s accuracy on the same 237-sample subset to ensure fair horizontal comparison? Please provide the corresponding experiments and analyses.

  5. Reproducibility details Section 3.5 only states that the model is implemented with a “Qwen2.5-VL-7B-Instruct backbone and LoRA adaptation” without specific configurations. Please supplement the complete training configurations and inference prompt templates for all three modules, including key hyperparameters such as LoRA target layers, rank, learning rate schedule, and training steps, to improve the reproducibility of the work.

2 comments
AuthorsOpenReview ↗

# Response to Reviewer jHKL

We thank the reviewer for the careful reading.

Q1. Terminology. We use "world model" in its functional sense: a model that predicts the next state from the current state and an action. PIWM is an intent world model. Its state is the customer's intent state (AIDA stage and BDI) rather than pixels or a learned latent, and Foresee predicts how this state changes under each candidate action. It therefore differs from latent-dynamics world models in its state space and horizon, but in essence it performs action-conditioned next-state prediction. We will state this definition and its scope (structured intent state, one-step horizon) explicitly in Section 3.3. Likewise, "counterfactual" refers to comparing the predicted consequences of alternative candidate actions from the same current state; we will define it at first use.

Q2. Generalization boundary. The conclusions are drawn in the rule-guided synthetic setting. Results on the 418 Real records for all models are given below.

Q3. Evidence ablation. We ran the requested control: the four evidence fields are removed from one module at a time.

SettingAccuracy (509)Change
Full state to both modules63.65%—
Evidence removed from Foresee only59.95%−3.7
Evidence removed from Act only (Foresee unchanged)58.55%−5.1
Evidence removed from both57.15%−6.5

Evidence contributes through both modules, with the larger share through Act.

Q4. Fairness of the real-scene comparison. All models are now evaluated on the same 418 Real records: GPT-4o 39.71%, Gemini 2.5 Flash 40.91%, PIWM 63.16%.

Q5. Reproducibility.

ItemSetting
BackboneQwen2.5-VL-7B-Instruct
AdaptationLoRA, rank 16, alpha 32, dropout 0.05, all linear layers of the language model; vision encoder and aligner frozen
Trainable parameters40.37M (0.48%)
OptimizerAdamW (β1 0.9, β2 0.95), weight decay 0.1, gradient clipping 1.0
Learning rate2e-5, cosine decay, no warm-up
Batch1 × 8 gradient accumulation
Training5,305 examples × 2 epochs = 1,326 steps; one adapter for See, Foresee and Act
Input3–5 frames per example, at most 401,408 pixels per frame; max length 8,192 tokens
DecodingGreedy; at most 640 / 384 / 320 new tokens for See / Foresee / Act
Hardware1x A100

The prompt templates of the three modules are identical in training and inference and will be listed verbatim in the appendix together with the released code.

Thank you for the detailed response. The separate Foresee and Act ablations address my concern about attribution, and evaluating all three models on the same 418 records resolves my concern about the real-store comparison. The clarification of what Foresee predicts is also helpful. I see the paper’s main contribution in the retail decision task and its benchmark. Since the predicted consequences reflect rule-guided expectations, the contribution to modeling how customer intent changes after an intervention remains limited. I therefore maintain my original rating.

Reviewer_xeT6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes See–Foresee–Act and implements it as PIWM for proactive retail assistance. See extracts observable evidence and AIDA–BDI customer states from sparse video; Foresee predicts one-step consequences for candidate actions; Act compares these consequences, including Hold, to decide whether and how to intervene. The authors introduce GuidanceSalesBench, built mainly from synthetic, rule-guided supervision. On 509 Full test examples, using candidate consequences improves decision accuracy from 45.38% to 63.65%. The task is relevant, and the pipeline is easy to follow. However, the consequence targets are normative annotations, not observed outcomes of real interventions, which limits the evidence for world modeling. The paper also leaves questions about training inputs, data splits, and overlap across task-specific resources. The comparisons do not control for additional inference cost, and the ablations change the input conditions. Results on the Real set are missing. These issues need to be addressed before drawing stronger conclusions from the results.

Critical strengths
  1. The paper considers both whether and how to intervene before a customer asks for help. Including Hold alongside Greet, Elicit, Inform, and Recommend makes sense, since assistance may interrupt independent exploration.
  2. See keeps explicit evidence about gaze, engagement, posture, and object interaction alongside AIDA–BDI states. This makes it easier to see what supports the inferred customer state. The Full results also show that candidate-consequence comparison improves decision accuracy from 45.38% to 63.65% in this setting.
Critical weaknesses
  1. Foresee learns normative consequence targets rather than outcomes observed after real interventions. Its evaluation only measures next-AIDA-stage accuracy, leaving the predicted benefits, risks, and relative reward untested. The paper also introduces a Real set of 418 real-store records but reports no results on it.
  2. Compared with direct decision output, the full pipeline uses additional inference to generate five candidate forecasts. Without a comparable inference budget, it is unclear how much of the gain comes from consequence prediction itself. Likewise, withholding only the Hold forecast creates an uneven comparison; removing each active-action forecast would help establish whether the effect is specific to non-intervention.
Questions for authors
  1. How do you assess the predicted benefits, risks, and relative reward beyond next-AIDA-stage accuracy? Can you report results on the 418-example Real set to show whether the findings extend to real-store observations?
  2. Does candidate-consequence comparison still outperform direct decision output under a comparable inference budget? Also, what happens when you withhold the forecast for each active action instead of Hold? This would help clarify whether the reported drop is specific to non-intervention or reflects missing candidate information more generally.
  3. Were the figures generated using AI? The resolution seems low, and the background colours around the borders look uneven. Some manual adjustments would help improve their clarity and consistency.
1 comment
AuthorsOpenReview ↗

# Response to Reviewer xeT6

We thank the reviewer for the detailed questions.

W1 / Q1. Predicted benefit, risk and reward.

We've evaluated them separately.

MetricAnnotated state (132 Foresee test rows)State predicted by See (1,592 candidates)
Next-stage accuracy84.9%59.4%
Risk accuracy84.9%81.2%
Benefit accuracy78.0%73.9%
Reward mean absolute error0.0750.139
Reward Spearman correlation0.8020.605
Pairwise ordering of candidates within a scene75.3%73.8%

W1 / Q1. Results on the Real set.

All models are now evaluated on the same 418 Real records: GPT-4o 39.71%, Gemini 2.5 Flash 40.91%, and PIWM 63.16%.

W2 / Q2. Comparable inference budget.

MethodCalls per exampleSeconds per exampleAccuracy (509)
Fine-tuned frames → action, greedy17.532.42%
Fine-tuned frames → action, 5-sample vote5≈ 5×33.01%
See → Act (no Foresee)220.561.69%
See → Foresee → Act5.3535.863.65%

With approximately five calls, direct decision output reaches 33.01%, 30.6 points below the full pipeline. The full pipeline is also 2.0 points above See → Act.

W2 / Q2. Withholding each action's forecast.

Forecast withheldAccuracyChange
None63.65%—
Hold59.92%−3.7
Inform61.49%−2.2
Recommend62.48%−1.2
Elicit62.67%−1.0
Greet63.65%0

Withholding the Hold forecast causes the largest drop; withholding an active-action forecast also lowers accuracy.

Q3. Figures.

We will redraw the figures at higher resolution with consistent backgrounds in the revised version.

Reviewer_kgdnOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the See–Foresee–Act framework for determining whether and how to proactively assist customers in retail environments. It begins by extracting behavioral evidence and customer states from video, proceeds to predict the outcomes of five candidate actions, and concludes by comparing these outcomes to select the optimal course of action. The authors developed GuidanceSalesBench, a benchmark primarily composed of synthetic data, and fine-tuned the Qwen2.5-VL-7B model using it. On a full test set of 509 samples, the inclusion of outcome predictions for candidate actions improved accuracy from 45.38% to 63.65%.

Critical strengths
  1. The problem formulation is sound; since proactive service carries the risk of disturbing customers, incorporating a non-intervention option is highly practical.
  2. The ablation study directly addresses the core design; the results confirm that both explicit behavioral evidence and descriptions of potential consequences are beneficial for the task at hand.
  3. Results are reported separately for state recognition, consequence prediction, action selection, and the end-to-end process, facilitating an assessment of performance gaps between the individual modules.
Critical weaknesses
  • Comparisons based on real-world scenarios are insufficient. In the appendix, authors' proposed method is evaluated on 418 real-world samples, whereas other models cover only 237 and their accuracy scores cannot be directly compared.
  • Foresee already predicts a reward for each action, but the paper does not compare against simply choosing the highest-reward action, leaving Act’s added value unclear. It is also unclear whether all records from the same parent scene stay in one split across tasks. Related records could otherwise appear in both training and test sets, creating a risk of data leakage.
Questions for authors

please see the weakness

1 comment
AuthorsOpenReview ↗

# Response to Reviewer kgdn

We thank the reviewer for the constructive comments.

W1. Real-scene comparison uses different sample counts (418 vs. 237).

All models are now evaluated on the same 418 Real records.

ModelReal (418)
GPT-4o39.71%
Gemini 2.5 Flash40.91%
PIWM63.16%

W2a. Baseline: choose the action with the highest Foresee reward.

We added this baseline. On the 509 Full test examples:

SettingHighest predicted rewardPIWM (Act compares consequences)
State predicted by See38.11%63.65%
Annotated state46.37%62.87%

PIWM is 25.5 points above the reward-argmax rule under predicted states. Act does not receive the numeric reward; its input contains the predicted next stage, BDI, risk and benefit of each candidate.

W2b. Do records from the same parent scene stay in one split?

Yes. The dev and test splits of See, Foresee, Act and Full share no parent scene with the training split of any task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.