acceptodds
Under review as a conference paper at ICLR 2027

Staying Consistent, Staying on Target: Revisable References for Visual Reasoning

Abstract

Visual references enable vision-language models to reuse object-level information across successive questions. However, when an early localization is incorrect, reusing it can propagate the mistake through later reasoning, even when the resulting answers remain mutually consistent. Recovering from such errors requires revising the localization while preserving the object intended by the task. We address this requirement through revisable references, which separate task-defining information from the model's localization estimates. In a binary repair model, we characterize when rewards for historical agreement oppose task-beneficial corrections. This motivates Referent-Preserving Policy Optimization (), a reinforcement learning framework that learns reference revisions from their consequences for subsequent reasoning. jointly optimizes revision and execution: edits are assessed against retaining the original reference under the same executor, while continuations are compared within the same edited state. These comparisons connect both decisions to downstream task utility. Answer accuracy alone, however, does not establish whether the maintained reference has been repaired. We therefore design a controlled evaluation protocol that separately assesses reference recovery, preservation of the intended object, and answer correctness, using paired reference semantics and shared starting states under matched information, supervision, and computational budgets.

Primary area:
reinforcement learning
Keywords:
vision-language models, visual reasoning, reinforcement learning, revisable references

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nS4COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how vision-language models can correct cached visual references while preserving the intended referent. It proposes Referent-Preserving Policy Optimization (RP²O), which jointly trains reference revision and downstream reasoning using separate credit signals derived from answer rewards. Controlled experiments on Qwen2.5-VL-7B show improved reference recovery and joint reference–answer correctness, while maintaining answer accuracy comparable to answer-reward GRPO.

Critical strengths

The controlled evaluation clearly distinguishes reference recovery from answer correctness, using shared starting states and frozen-executor comparisons to support the reported gains.

Critical weaknesses
  1. The methodological novelty is still unclear to me. The design builds on ideas from ReMax (ICML 2024), RLOO (ACL 2024), and local credit assignment methods such as GiGPO (NeurIPS 2025). However, the main RL comparison is against standard GRPO. I would like to see these closer alternatives adapted to the same interface and compared under matched budgets. Without that, it is hard to tell how much the proposed estimator adds.

  2. There is a gap between rewarding correct answers and recovering correct references. The executor can still access the original image and description, so it may answer correctly while leaving the cached reference wrong. If keeping and repairing the reference yield the same answer reward, the objective cannot distinguish them. The unchanged-state baseline does not change the expected policy gradient, so it does not by itself resolve this issue. The paper needs a clearer explanation of why this training setup reliably encourages reference recovery.

  3. The ablations do not fully explain where the gains come from. In Table 2, removing the revision control still gives 71.40% JR, compared with 72.58% for the full method. This suggests that much of the improvement remains without this component. I would like to see separate tests of the within-edit execution credit, sampling structure, and advantage normalization, along with comparisons against simpler baselines. These would help establish which parts of the method are actually needed.

  4. Joint training seems to involve a robustness tradeoff that needs more discussion. In Table 9, with corrupted caches, the RP²O executor reaches only 17.34% average answer accuracy, versus 42.96% for the GRPO executor. Given the same RP²O-edited references, the GRPO executor also achieves higher answer accuracy with nearly identical JR. This raises the concern that joint training may make the executor more dependent on the cache. I would like to understand what happens when editing fails and whether this tradeoff is necessary for the recovery gains.

  5. Releasing the implementation would improve reproducibility and make the method easier to evaluate and adopt. While not essential for acceptance, it would strengthen the overall impression of the work.

Questions for authors

See the above weakness.

Reviewer_PFKCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper improve the localization problem in multi-turn grounded visual dialogue, where an initial bounding box (stored as a reference) could be erroneous, which can potentially harm the downstream reasoning chains. The authors tackle this problem by splitting into fixed task-defining information and a revisable cached box. They also introduced a role tag that says whether the box is the model's guess or the user's own choice. They proposed an RL-based method, RP2O, that adds one KEEP/REBIND call before the follow-up questions and trains revision and answering jointly in Qwen2.5-VL-7B with RL on answer correctness.

Critical strengths
  1. The metrics that authors report are grounded by three seeds with standard deviations, and image-cluster bootstrap, which improve the transparency along with the training budgets.
  2. The paper's idea of separating what the user meant from the model's current estimate is an intuitive and easily understandable design for potential revision later.
Critical weaknesses
  1. My biggest concern about this work is its significance and if it fits the general direction of how models and agentic systems are evolving. The general trend of the model evolution to solve this issue would be training for having models that are better at grounding the user requests in the correct visual segments on the first try, or relying on tools or subagents that can provide more trustworthy intermediate visual evidence. The models getting stronger and stronger with better first pass grounding would shrink the value of the proposed method. The authors could argue that their method facilitates a better localization mechanism than directly optimizing on the localization target itself, in which case they would need to demonstrate superiority against a baseline that directly performs RL on better localization in the beginning. Also the evaluation has been focusing on the conditions where the localization is already incorrect, which limits the scope as well.
  2. The authors need to show benchmarking against a much more comprehensive set of baselines to claim robust performance and efficacy aside from the one mentioned above. For example, the proposed RP2O is more inference expensive with one extra call, and it should be compared with different methods that use the same inference cost budget like a self-verification prompt or best of 2 selection. Figure 5’s budget matching covers training tokens, not inference. Furthermore, there needs to be a vanilla method that uses base Qwen2.5-VL with the Prompt F.1 KEEP/REBIND instruction and no training, which would show how much the gain comes from the RL. Also the two-call design can be justified by having a model that emits revision and the first answer in one generation, which will showcase how much the credit assignment actually requires this extra step.
  3. Some metric gains that were shown in the paper are small or negative. Specifically, endpoint answer in table 1, average answer in table 2, and natural trajectories in table 7C all show that the proposed method is inferior. At the matched 60.4M-token budget, GRPO (Answer) is higher.Furthermore, since JR requires the stored box to be correct, most of the JR gain comes from fewer right answer & wrong box cases, which is a headline metric that favors the design choice. Even in cases where the method are superior on a larger scale like trajectories in table 7c, the performance is conditional on an error already existing, not the end-to-end.
  4. Some adapted baseline methods that are being used for comparison is way behind the frontier method results, namely RegionReasoner, which scores far behind the GRPO (Answer). Can you try comparing the native RegionReasoner-7B and VisionReasoner-7B on the shared-start recovery protocol? They are currently only appearing in table 3 and on native benchmarks. Also can you try remove the cached box from the prompt and have the model resolve “the boy referenced by R1” from the original description on every turn? This test the fact if maintaining a reference is necessary at all.
Questions for authors
  1. Can you show comparison against a model trained only to localize correctly on the first turn (e.g., VisionReasoner or RegionReasoner), used without any revision step and evaluated on full dialogues?
  2. Can you run other inference-cost-matched baseline methods as comparison on the natural-trajectory evaluation?
  3. Can you try comparing the native RegionReasoner-7B and VisionReasoner-7B on the shared-start recovery protocol?
Reviewer_hLdoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies weather a model can remain consistent with an earlier bounding box while continuing to refer to the wrong object in the multi-turn conversation setup. It proposes separating the information that establishes the intended referent from the cached localization that the model may revise. The proposed Referent-Preserving Policy Optimization (RP2O) jointly trains an editor and an executor using a shared vision-language model. Revision credit compares downstream answer utility after an edit against utility from retaining the original cache. Execution credit uses a leave-one-out comparison between continuations from the same edited state. The central empirical result is an improvement in the probability that both the maintained reference and the subsequent answer are correct, rather than an improvement in answer accuracy alone.

Critical strengths
  1. The problem formulation identifies a meaningful failure mode and the distinction between preserving an intended object and preserving a cached localization is clear and practically relevant.
  2. Several design choices improve interpretability: common starting states prevent differences in initial-error distributions from looking like recovery gains, correct-cache starts measure harmful revisions, semantic pairs test different task definitions, and reference correctness is evaluated separately from answer correctness.
  3. The authors distinguish an unbiased score estimator from the actual clipped, KL-regularized optimizer update. They also explicitly acknowledge that estimating the control baseline can increase or decrease total gradient variance. These qualifications are appropriate and prevent the analysis from promising more than it establishes.
Critical weaknesses
  1. The paper changes several aspects of training relative to ordinary answer-reward GRPO: the sampling structure, averaging over continuations, revision baseline, execution baseline, and advantage transformation. The current ablation removes the revision control but retains the within-edit execution credit. It therefore does not isolate which of the remaining changes accounts for most of the improvement.
  2. Reproducibility details remain incomplete despite the extensive appendix. I could not identify an explicit training-start count, training-image count, or exact mixture proportions for clean, corrupted, and coherent-error starts. The manuscript specifies the types of starts, but not their numerical composition.
  3. The method is tested only on a single backbone i.e Qwen-VL-2.5-7B. This is not enough to show that the method generalizes to other backbones as well.
Questions for authors
  1. Which estimator component drives the gain? Under identical nested rollouts and update weighting, what happens when the revision baseline, execution baseline, or both are removed?
  2. Table 9 shows higher average answer accuracy for RP2O edits combined with the GRPO executor, at almost the same mean JR. Can reference-correctness-stratified results explain this difference?
  3. Does preservation generalize beyond keeping correct defining caches? What happens when a definition-role cache is corrupted while its immutable selection remains fixed, and how does RP2O compare with a deterministic keep-on-definition baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.