acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Temporal Instance-State Binding and Its Stage-Dependent Repairability in Vision-Language Models: A Controlled Study

Abstract

Temporal visual understanding requires more than detecting that a state change occurred. A model must also identify which persistent instance underwent that change. We introduce TIB-Bench, a controlled synthetic benchmark designed to isolate event recognition from instance-state binding. Its counterfactual pairs end in pixel-identical final frames, but differ in history and in the correct event owner. On Qwen3-VL-2B, perception and event occurrence are both correct on 210 of 480 episodes. Yet the owner is wrong in 109 of those cases (51.9%), revealing a failure that event-level accuracy does not capture. Layerwise probing shows that owner-related information remains accessible in downstream representations, while late-layer activation interventions perturb binding decisions substantially more than event judgments. We then examine how this failure can be repaired. Replacing matched ordinary examples with counterfactual-generated examples raises counterfactual-history consistency from 0.006 to 0.850 without an explicit pairing objective. Under matched eight-layer LoRA budgets, early and middle adaptation also substantially outperform late adaptation. Temporal-order perturbations further show that repaired models rely more strongly on the canonical history. Together, these results separate event recognition, owner-related representation, and history-sensitive owner attribution within a controlled temporal visual setting.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, temporal reasoning, temporal instance-state binding, video understanding, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ybKqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates temporal instance-state binding failures in vision-language models, where models correctly recognize an event but attribute it to the wrong object. The authors introduce TIB-Bench, a controlled synthetic benchmark using counterfactual histories to distinguish event recognition from owner attribution. Through behavioral evaluations, layer-wise probing, and activation patching, the paper examines these failures and the accessibility of owner-related information in internal representations. The experiments further demonstrate that counterfactual-generated supervision substantially improves binding performance, while early and middle-layer LoRA adaptation is more effective than late-layer adaptation.

Critical strengths
  1. The paper investigates an important limitation of VLMs by distinguishing temporal event recognition from instance-state binding. The proposed TIB-Bench uses counterfactual histories to systematically evaluate history-dependent owner attribution.

  2. The paper combines behavioral evaluations, layer-wise linear probing, and activation patching to investigate binding failures, providing interesting insights into the relationship between internal representations and model predictions.

  3. The experiments demonstrate substantial improvements through counterfactual-generated supervision and reveal an interesting discrepancy between representation decodability and layer-dependent LoRA repairability.

Critical weaknesses
  1. Limited validity of the benchmark for evaluating temporal instance-state binding. TIB-Bench primarily consists of three-frame synthetic storyboards with a small number of objects and predefined event types. For certain events, such as attribute changes and state toggles, the correct owner may be identified by directly comparing the initial and final frames without requiring persistent object tracking across the full temporal sequence. Therefore, it remains unclear to what extent the benchmark evaluates temporal instance-state binding rather than simpler visual state differencing. Additional controls using only the first and last frames would help establish the necessity of temporal reasoning.

  2. Insufficient isolation of instance-state binding failures from other perception and tracking errors. The proposed conditional binding metric evaluates owner attribution only when perception and event-recognition questions are answered correctly. However, these preliminary questions do not necessarily verify whether the model correctly identifies object states at every timestep or maintains accurate cross-frame identity correspondences. Consequently, the reported binding errors may still arise from intermediate-state perception failures or object-tracking errors rather than a distinct instance-state binding failure. More fine-grained diagnostic questions would help disentangle these potential sources of error.

  3. Limited causal evidence from the representation analysis. Although the linear probing results demonstrate that owner-related information is decodable from intermediate representations, this does not establish that the model functionally uses such information during inference. Furthermore, the activation-patching experiments show substantial answer changes even with same-answer or random donors, suggesting that the observed effects may reflect general sensitivity to hidden-state perturbations rather than owner-specific causal mechanisms. While the authors acknowledge these limitations, the current evidence provides limited mechanistic insight into the underlying causes of binding failures.

  4. Potential confounding factors in counterfactual training. The improvement achieved through counterfactual-generated supervision is substantial, and the authors provide size-matched comparisons with ordinary training data. However, the two data-generation procedures may still differ in higher-order statistical properties, such as state-transition diversity, history-owner correlations, and sample difficulty. Therefore, it remains unclear whether the improvements specifically result from counterfactual structure or more general changes in training-data coverage and diversity. Additional distribution-matched controls would strengthen this conclusion.

Questions for authors

See my weaknesses.

Reviewer_aYPwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate temporal instance-state binding in vision-language models, focusing on whether a model can correctly identify which object underwent a particular state change. They introduce TIB-Bench, a synthetic benchmark containing 480 episodes, including counterfactual pairs that share identical final frames but have different temporal histories and event owners. The authors show that VLMs frequently recognize events correctly while assigning them to the wrong objects. They further analyze internal representations through linear probing and activation patching, finding that owner-related information remains accessible even when the model makes incorrect binding predictions. Finally, they investigate whether counterfactual training data and different LoRA placement strategies can repair these failures. Their experiments show substantial improvements from counterfactual supervision, with early and middle layer adaptation outperforming late layer adaptation.

Critical strengths

The paper studies an interesting and relatively specific failure mode in temporal visual understanding. I appreciate that the authors distinguish event recognition from instance-state binding rather than treating them as the same capability. The counterfactual benchmark design is useful because it controls the final visual observation while changing the relevant temporal history. The experimental finding that Qwen3-VL-2B makes incorrect owner predictions in 109 out of 210 cases despite correctly recognizing the event is interesting. Another strength is the analysis of different supervision and adaptation strategies. The size-matched comparison between ordinary and counterfactual-generated training data provides useful evidence that data composition matters for learning temporal associations. The large performance difference between early and late LoRA placement is also an interesting observation that deserves further investigation.

Critical weaknesses

My main concern is the novelty and generalizability of the proposed benchmark. Several recent works have already investigated object identity, temporal consistency, and state tracking in vision-language models. In particular, TOC-Bench and STEMO-Bench study closely related problems involving persistent object identities and state changes across video frames. VLM2-Bench also evaluates the ability of multimodal models to associate corresponding visual information across images and videos. The authors propose a controlled counterfactual benchmark, but I think the paper should more clearly explain what new capability it measures beyond these existing benchmarks. The evaluation itself is also quite limited, with only three objects, three timesteps, and four predefined event families in relatively simple synthetic scenes. Although Appendix A.7 includes an additional real-video diagnostic, it does not evaluate whether the proposed repair methods transfer to realistic visual environments. My second concern is the interpretation and generality of the mechanistic findings. The authors report that owner information remains linearly decodable in later layers, but this does not necessarily mean that the model has learned a correct temporal binding representation. Similarly, the activation patching results are difficult to interpret causally because same-answer and random donor interventions also cause substantial changes in predictions, as the authors acknowledge. The early versus late LoRA results are interesting, but the repair experiments are conducted primarily on Qwen3-VL-2B. It remains unclear whether this stage-dependent behavior generalizes to other VLM architectures. I am also concerned about the relatively large variance in the counterfactual repair experiments, where CFC varies from 0.650 to 0.967 across three seeds. Overall, the paper contains interesting observations, but I am not fully convinced that the results support broad conclusions about temporal binding mechanisms or their repairability.

Questions for authors

Could the authors clarify the main differences between TIB-Bench and existing temporal object consistency benchmarks, particularly TOC-Bench, STEMO-Bench, and VLM2-Bench? I would also like to understand whether the proposed benchmark can be extended to more realistic sequences involving longer temporal histories, more objects, and complex occlusions. Second, can the authors evaluate whether counterfactual supervision improves binding performance on real videos rather than only on synthetic examples from the same generation framework? This would substantially strengthen the practical significance of the work. Third, regarding the mechanistic analysis, how can the authors distinguish an owner-specific causal effect from general perturbations of the model's internal representations, given that random and same-answer donor interventions also affect the predictions? Finally, do the early versus late LoRA findings hold for other models such as Qwen3-VL-4B or InternVL3.5? It would also be helpful to discuss the observed variation across training seeds and whether the improvement is stable under different training data distributions.

Reviewer_YeDsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a vision-language model assigns a recognized state change to the persistent object that underwent it. TIB-Bench contains 480 synthetic three-frame episodes spanning movement, attribute changes, state toggles, and occlusion followed by reappearance. Of these, 240 form 120 counterfactual pairs with pixel-identical final frames but different histories and correct owners. Separate questions check visible objects, event occurrence, and ownership. Evaluation reports owner accuracy conditional on the first two checks and counterfactual-history consistency, CFC, the fraction of pairs with both owners correct. The primary four-option logit interface avoids free-generation parsing failures. Qwen3-VL-2B passes the checks on 210 episodes but gets the owner wrong on 109; other Qwen sizes, InternVL, and an auxiliary real-video diagnostic provide additional tests.

Linear probes and activation patching examine owner-label decodability and answer sensitivity to state replacement. Repair experiments then vary training data and LoRA placement. Arm A contains 14,000 ordinary episodes; size-matched A* and B contain 20,000, adding either 6,000 ordinary or counterfactual-generated episodes. B uses independent QA supervision without a paired loss. Each run samples 1,500 episodes, and all three fixed training seeds are reported. CFC rises from 0.006 for A* to 0.850 for B; partner cross-entropy reaches 0.958, while an additional hinge loss gives no demonstrated consistent benefit. With matched eight-layer rank-16 LoRA budgets, B achieves early, middle, and late CFC of 0.850, 0.736, and 0.069. Separate validation-based tuning preserves the early versus late gap. Order perturbations and single-seed structural shifts assess history sensitivity and transfer within the synthetic generation setting.

Critical strengths
  1. The 120 counterfactual pairs hold the final frame fixed while changing the correct owner, making the final frame alone insufficient to determine the owner across both histories. The primary model makes 109 owner errors on 210 episodes passing the perception and event checks, identifying a concrete omission in event-level evaluation. Cross-model and real-video diagnostics further motivate the problem. CFC is explicitly treated as a stricter paired success criterion rather than a capability independent of pointwise accuracy.

  2. A* and B match pool size, objective, training steps, and observed marginals for events, object count, similarity, and answer position. The CFC increase from 0.006 to 0.850 supports counterfactual-generated coverage beyond simply adding ordinary examples. Only 13 to 22 complete pairs are encountered in a run, supporting the distinction from explicit paired training. Placement experiments also report individual seeds, independent validation-based tuning, and budget controls. The early versus late gap remains substantial after these checks.

Critical weaknesses
  1. The perception gate checks visible objects and event marginals, without requiring complete cross-frame identity correspondence or trajectory recovery. Owner errors after the gate may therefore include tracking difficulties and do not isolate a distinct binding process. Owner labels can be decoded by a probe, but same-answer and random patch donors also substantially alter predictions. The tasks have different output cardinalities, so the 5.3-fold graphic in Figure 3 is not a calibrated causal-effect ratio. The authors explicitly acknowledge these limits. I accept the restricted behavioral diagnosis, while finding the additional explanation of the processing failure limited.

  2. Table 2 establishes the placement profile only for Qwen3-VL-2B and the tested LoRA interface. Cross-model and real-video experiments replicate the diagnosis rather than the repair profile; structural shifts use only seed 111. Main-run variation is substantial: B early CFC is 0.650, 0.933, and 0.967. The existing placement-specific validation tuning rules out one important alternative explanation, but does not establish a repair principle across architectures. The contribution is best assessed as a convincing controlled case.

Questions for authors
  1. How many episodes pass an additional requirement for correct frame-by-frame identity correspondence or full trajectories, and what is their owner error rate? This concerns the model’s tracking, rather than the correctness of generated trajectory annotations.

  2. Which joint-distribution or difficulty differences remain between A* and B beyond the matched marginals? Could event- and similarity-stratified repair gains show whether the effect concentrates in particular generation patterns?

  3. Are early versus late comparisons available for another architecture or adaptation interface? If not, please consistently retain the primary-model and LoRA scope when summarizing the placement result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.