- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Credit-Guided Policy Improvement (CGPI), a feedback-free test-time adaptation framework for Vision-Language Navigation. It derives reference-relative decision credits from action-induced observation transitions to guide lightweight policy updates, which are selectively retained or rolled back through historical credit-based verification.
Critical strengths
- The identify–propose–verify–retain framework provides a clear and intuitive approach to test-time policy adaptation.
- The experiments cover multiple VLN benchmarks and navigation backbones.
Critical weaknesses
-
The decision credit relies on fixed semantic-similarity measures and handcrafted evidence heuristics, which appear somewhat tailored to the evaluated benchmarks (e.g., goal-oriented REVERIE versus fine-grained instruction-following R2R/R2R-CE). Moreover, the paper lacks individual ablations of the four evidence components and quantitative real-world evaluations (e.g., SR) under more diverse scenes and instructions. These limitations raise concerns about the reliability of the proposed credit assignment and weaken the evidence for its generalizability beyond the evaluated benchmarks.
-
The proposed Verification Mechanism relies on historical decision credits derived from the same heuristic evidence functions used for policy updates. Although the authors introduce a warm-up stage to alleviate memory sparsity, this does not ensure the reliability of the accumulated credits. Incorrect credits from early interactions may contaminate the Verification Memory, potentially reinforcing harmful updates or rejecting beneficial ones. Despite the robustness analysis under random credit noise, the paper does not investigate this potential self-reinforcing failure mode, raising concerns about the reliability of the Verify-and-Retain mechanism.
Questions for authors
-
Could the authors provide individual ablations of the four evidence components (Goal, Progress, Grounding, and Risk), particularly removing Goal/Grounding on REVERIE and Progress on R2R/R2R-CE, to clarify their respective contributions?
-
Could the authors provide quantitative real-world evaluations (e.g., SR) against the frozen backbone, particularly in scenarios involving more abstract or complex instructions and challenging instruction–observation semantic alignment, to better demonstrate generalizability?
-
How robust is the Verification Mechanism when the initial memory contains systematically incorrect credits? Could the authors evaluate this scenario, analyze how frequently such errors occur on real benchmark trajectories, and investigate whether the resulting impact depends on the pretrained backbone's capability?
3 comments
I thank the authors for the detailed rebuttal and revision. The new component ablations (Table 4), shared-context controls (Table 19), real-world evaluations, and memory-corruption analyses (Apps. G.7 and H.3) have adequately addressed my main concerns. I also appreciate the transparent reporting of negative results.
Some limitations remain regarding verification reliability under strong systematic credit errors and the limited scale of real-world evaluations. I encourage the authors to discuss these limitations more explicitly in the main text and Appendix K, and to report confidence intervals for the real-world results.
Overall, the original submission did not provide sufficient experimental evidence to fully support its claims. However, the substantial additional experiments have addressed my main concerns and significantly strengthened the paper. I therefore raise my rating accordingly.
Thank You for Your Constructive Feedback and Reassessment
Dear Reviewer XZUM,
We sincerely appreciate your careful reassessment of our revised manuscript and the time you have devoted to reviewing our additional experiments.
We are particularly grateful that you acknowledge the new evidence addressing your main concerns. Your detailed feedback has been invaluable in helping us strengthen the experimental validation and clarify the scope of our contributions.
We also appreciate your remaining suggestions regarding verification reliability under systematic credit errors and the statistical uncertainty of real-world results. We agree that these limitations deserve clearer discussion and will explicitly address them in the main text and Appendix K, together with confidence intervals for the real-world evaluation.
Thank you again for your constructive comments, thoughtful engagement, and willingness to reconsider your assessment in light of the new evidence.
Best regards,
The Authors
Thank you for recognizing the clarity of the identify–propose–verify–retain framework. We added evidence-design controls, component ablations, paired robot trials and historical-credit audits. References below use the revised PDF.
W1. Evidence design and benchmark dependence
CGPI uses frozen learned representations with shared evidence, reference-relative credit and update-verification rules. The task-dependent Goal context is explicit: REVERIE uses the full goal description, while R2R/R2R-CE use the final instruction segment (App. D).
We directly test dependence on this choice by using the full instruction as Goal context across all benchmarks, including its STOP computations. REVERIE already uses this context. On R2R/DUET and R2R-CE/ETPNav, the shared-context variant reaches 77.6/66.5 and 59.8/50.9 SR/SPL, compared with frozen 72.0/60.0 and 57.0/49.0 (B.9, Table 19). Deduplicating the resulting Goal/Grounding signal while keeping denominator four gives 77.8/66.7 and 60.0/51.1. The table also includes denominator-three and fixed-STOP controls; normalization uses source data and other hyperparameters remain fixed. These controls retain adaptation gains without benchmark-dependent Goal context.
W2. Historical-credit contamination and self-reinforcement
We added natural-error audits, controlled initial-memory corruption and tracking after FIFO eviction. These measure both harmful-update acceptance and beneficial-update rejection, together with later memory errors and navigation outcomes (G.7, Table 32; Q3 below).
A complementary test applies persistent semantic distractor bias to both update evidence and verification-memory writes. At bias strength ρ=0.25, CGPI achieves 56.8 SR versus 55.4 for Direct Commit. At ρ=0.50, the results are 53.4 versus 53.8 (H.3, Table 38). The retention benefit persists at the moderate setting and is absent at the stronger setting.
Q1. Contributions of Goal, Progress, Grounding and Risk
Table 4 now contains individual removals across all three benchmarks. Component removal retains denominator four and fixes the complete STOP branch. On REVERIE/DUET, Goal and Grounding coincide by construction: removing either gives 58.1 SR, removing both 56.9, versus 58.76 for Full CGPI. The three-term deduplicated mean gives 58.8, distinguishing semantic removal from aggregation changes.
Removing Progress reduces R2R/DUET SR 78.9→76.8 and R2R-CE/ETPNav 61.2→59.0. Progress measures local semantic change over fixed instruction segments, without completion tracking. We also added reference-relative counterfactual diagnostics on these instruction-following benchmarks: Spearman 0.589/0.551, sign errors 20.4%/22.7% (B.13, Table 27).
Q2. Quantitative real-world evaluation
We compare the same R2R-CE-pretrained BEVBert with and without CGPI on Unitree Go2, sharing starts, candidates, controller and execution budgets. Four environments and 30 instructions × four repetitions give 120 attempts per method, in ten independently initialized paired blocks. All failures and interventions count.
CGPI improves SR 21.7%→30.8% (26/120→37/120, +9.2 points). Abstract/ambiguous tasks improve 8/40→12/40; ordered multi-stage tasks improve 4/40→8/40, with success requiring the specified stage order. H.1 reports all 94/83 failures by mutually exclusive cause (Tables 35–36). H.2 stratifies the same paired trials by visual conditions: reduced illumination 5/24→8/24, occlusion 4/24→6/24, and both 3/24→5/24 (Table 37).
Q3. Error frequency, initial corruption and backbone dependence
Across 20 REVERIE task orders, initial-memory movement sign errors against the reference-relative oracle, excluding neutral records, are 24.8% for HAMT and 15.2% for DUET. Subsequent insertions have error rates of 20.1%/12.1%. We then reverse 0/2/4 initially oracle-consistent credits, preserving magnitudes and other states. This paired experiment uses the 16 orders eligible for both backbones, with the original warm-up of eight and FIFO capacity of 32.
SR for 0/2/4 reversals is 36.9/35.9/34.7 for HAMT and 58.6/58.0/56.9 for DUET. Four reversals therefore cost 2.2 versus 1.7 SR points. For DUET, harmful-update acceptance rises 11.8%→31.2% and beneficial-update rejection 18.3%→35.7% (G.7, Table 32).
After the injected records are evicted, new-credit errors remain 25.7%/15.9%, versus unmodified runs' 20.1%/12.1%, with SR deficits of 1.6/1.0 points. An oracle-assisted diagnostic corrects subsequent memory-write signs in both unmodified and corrupted runs, leaving proposal credits unchanged; the SR gaps shrink to 0.5/0.3 points. This supports error propagation through later memory writes after the original contaminated records are evicted (G.7).