acceptodds
Under review as a conference paper at ICLR 2027

Beyond Average Success: Outcome Reversals and Delayed Divergence in Quantized Vision-and-Language Navigation

Abstract

Quantization is widely used to compress models, reduce memory requirements, and potentially accelerate inference. When average performance is preserved, a natural expectation is that quantization introduces only local numerical perturbations while leaving model outputs and most individual outcomes largely unchanged. However, our experiments in streaming vision-and-language navigation (VLN) reveal substantial behavioral differences behind similar average scores. Changed trajectories can also alter subsequent observations and histories. Using instruction-paired R2R-CE episodes, we compare four-bit weight quantization of StreamVLN against its unquantized bfloat16 (BF16) model as the full-precision baseline. Success rate decreases by only 0.11 percentage points, yet 18.5% of outcomes reverse and 87.0% of action sequences change, as gains nearly offset losses. To examine this discrepancy, fixed-state probes separate numerical perturbations from immediate decisions and distinguish the effects of current computation and historical content. Further, controlled continuations show that history perturbations can induce later divergence even when the initial action queue is held fixed and subsequent computation returns to BF16. Together, these findings show that similar average performance does not imply behavioral fidelity, and immediate action agreement does not guarantee subsequent agreement in the tested states. Behavioral differences need not reduce task success. Quantized VLN deployment therefore calls for evaluating retained task outcomes and subsequent behavior alongside average quality, realized memory savings, and end-to-end latency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model quantization, vision-and-language navigation, decision-level analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YNZSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether quantized vision-and-language navigation (VLN) models preserve behavioral consistency beyond conventional average performance metrics. The authors reveal that similar success rates after quantization can hide substantial behavioral changes, including outcome reversals and delayed trajectory divergence. Through paired evaluations, fixed-state probing, and controlled continuation experiments, the paper provides a more comprehensive analysis framework for understanding behavioral changes introduced by model compression.

Critical strengths
  1. The paper addresses an important but underexplored issue: performance metrics alone may fail to capture behavioral instability after model compression.
  2. The proposed analysis reveals interesting phenomena such as outcome reversals and delayed divergence.
  3. The experiments provide convincing evidence that quantization can alter agent behaviors even when average success remains similar.
Critical weaknesses
  1. The work mainly focuses on evaluation and analysis rather than proposing new quantization algorithms, training strategies, or mitigation approaches. The practical contribution could be strengthened by exploring how to preserve behavioral consistency during compression.
  2. Although the paper demonstrates behavioral changes after quantization, the connection between these changes and real-world deployment risks remains insufficiently investigated. More discussion or experiments on safety-critical embodied scenarios would strengthen the impact.
  3. The analysis is mainly conducted on specific VLN models and benchmarks. It remains unclear whether similar behavioral inconsistencies widely exist in other embodied AI tasks, such as vision-language-action models or robotic control systems.
  4. Some observed behavioral differences do not necessarily indicate performance degradation. Further analysis is needed to distinguish harmless trajectory variations from meaningful failures or undesirable behaviors.
Questions for authors
  1. Are the observed behavioral inconsistencies specific to the evaluated VLN models, or are they a general property of quantized vision-language agents?
  2. How should practitioners balance compression efficiency and behavioral fidelity when deploying embodied AI systems?
  3. Can the proposed evaluation framework be extended to other embodied tasks beyond VLN, such as manipulation or autonomous interaction?
Reviewer_L5mTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a critical but often overlooked phenomenon in the deployment of Vision-and-Language Navigation (VLN) models: the discrepancy between aggregated average performance and individual behavioral fidelity after weight quantization. The authors evaluate the StreamVLN model under 4-bit weight quantization, revealing that while the overall success rate drops only marginally, the underlying trajectory outcomes and action sequences exhibit severe volatility, with successes and failures canceling each other out. Through meticulously designed fixed-state probes and queue-controlled continuation experiments, the authors validate that numerical perturbations not only alter immediate action decisions at narrow margins but also induce delayed divergence via contaminated historical context (KV cache), even when initial physical execution is forcefully matched.

Critical strengths

1.The paper makes a convincing case that current compressed-model evaluations share a systemic blind spot. Through a careful paired-outcome analysis, the authors show that similar aggregated success rates can hide severe instability on specific trajectories. This matters for reliable embodied agent deployment, where real-world applications require predictability and fidelity, not merely statistical equivalence. 2.The queue-controlled continuation experiment is well designed. By enforcing identical initial action executions while preserving the quantized branch’s history, the authors isolate the causal effect of altered historical context on future divergence. This provides clear evidence that immediate behavioral agreement is not sufficient to guarantee long-term trajectory fidelity.

Critical weaknesses

1.The paper does not provide a mathematical account of how quantization error propagates through the history buffer over time. A quantitative decision model, for example formulating the expected error or bounding the divergence probability given a specific initial conditional probability margin, would make the argument more rigorous. Without such a formulation, the current analysis remains largely empirical. 2.The experimental presentation makes the evaluation difficult to follow. The paper introduces several evidence families, including current/history substitutions, single-call pulses, and queue-controlled continuations, but there is no overview diagram explaining how they relate or what state transfers are involved. The three main figures mainly summarize outcome counts and disagreement rates, while much of the supporting evidence is left in dense tables and appendices. Representative paired trajectories and aligned timelines of history intervention, queue execution, and first divergence would help connect the aggregate findings to concrete navigation behavior. In addition, all navigation experiments are conducted in simulation, so it remains unclear whether the observed effects persist in real-world navigation, where sensor noise, actuation uncertainty, and execution timing may affect subsequent decisions. Clearer diagnostic visualizations and a small controlled real-world experiment would improve both the interpretability of the findings and the support for their deployment implications.

Questions for authors

1.In the fixed-state probes, you show flips concentrate at small reference margins. Can we mathematically bound the expected probability of an outcome reversal or delayed divergence based on the initial action's conditional probability margin?

2.Using the existing paired results, can you characterize whether lost successes concentrate in particular instruction types, navigation situations, or stopping decisions? Such an analysis would help determine whether turnover reveals a structured deployment concern rather than approximately offsetting variation.

3.Could the authors add real-world experiment to assess whether outcome reversals and delayed behavioral divergence also occur in real-world navigation? Such validation would strengthen the practical relevance of the findings and support the paper’s deployment implications.

Reviewer_H9TvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether quantization preserves the behavior of vision-and-language navigation agents while maintaining average task success. In the StreamVLN experiments, BF16 and W4 achieve nearly identical success rates, yet 18.5% of task outcomes reverse and 87.0% of action sequences change. Fixed-state experiments distinguish the effects of perturbing current computation from those of perturbing historical inputs. After a history intervention, the reference model’s initial action queue is enforced and subsequent computation returns to BF16. Nevertheless, 16 of 128 continuations diverge, while the unperturbed queue control exactly reproduces the reference results.

Critical strengths

The continuation experiment with a controlled initial execution queue is the most valuable part of the paper. By enforcing the reference model’s initial actions while retaining each branch’s historical state, the authors rule out differences in initial execution as the direct cause of subsequent trajectory divergence.

Critical weaknesses
  1. The main concern is the substantial overlap with prior work. The distinction from directly relevant research needs to be clarified. Dutta et al., Accuracy is Not All You Need (NeurIPS 2024), already show that aggregate accuracy can remain similar after quantization because correct-to-incorrect and incorrect-to-correct transitions offset each other. Section 5 of that paper also examines the probability gap between the top two candidate answers and finds that predictions with smaller gaps are more likely to flip. This work is not cited. The navigation experiments extend these observations to an interactive setting, but this extension is not clearly presented as the central contribution and, in my view, is insufficient on its own to constitute one.

  2. The history intervention supports the claim that historical differences can cause delayed divergence, but the explanation remains broad. The HQ branch replaces only the execution queue while retaining its own post-generation state. Subsequent differences may therefore involve the incoming KV perturbation, tokens generated during the intervention call, and the historical state resulting from that generation.

  3. The experiments support the proposed history-related pathway, but do not identify which part of the state carries this effect. Since this is the clearest contribution beyond existing observations, I would like to see a more concrete analysis of the divergent cases.

  4. The key result is based on a subset of 128 routes from an already inspected experimental population. How well this finding generalizes to other states remains unclear.

Questions for authors
  1. Relative to Dutta et al. (NeurIPS 2024), what is new in Findings 1 and 2? Which conclusions confirm existing observations in navigation, and which establish previously unreported findings?

  2. How were the original 256 intervention states selected within their respective routes? The paper describes the sampling procedure for the 128-route subset, but the selection of the original intervention events and their representativeness of ordinary navigation calls need further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.