- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how post-quantization adaptation changes the structure of a VLA policy's error. Using a task-adapted OpenVLA-OFT teacher on a simulated cube-stacking task, the authors build 8×6 teacher-relative residual maps for 13 ternary and three 3-bit layer-selection configurations, and compare rank-standardized maps across configurations before and after GT+KD adaptation. They report that adaptation substantially increases shared residual structure.
Critical strengths
Overall, the idea of studying how post-quantization adaptation changes the structure of a VLA policy’s error, rather than only its magnitude, is interesting.
Critical weaknesses
- All results come from a single task-adapted OpenVLA-OFT policy, a single simulated cube-stacking task, a selected set of layer configurations, and open-loop outputs on fixed inputs. Since the paper contributes an empirical observation rather than a method, the empirical scope is too narrow for an analysis paper.
- The paper shows that the remaining error becomes organized in a shared way, but does not show why this matters. There is no link between residual geometry and closed-loop task success, and no demonstration that the observation can inform quantization or adaptation choices.
- The paper is hard to follow because the main text is dense with numbers and switches between three residual representations.
Questions for authors
- Does the shared residual structure appear for another task or another VLA policy?
- How do the authors envision residual geometry being used in practice, and is it related to closed-loop task success?
2 comments
Thanks for the responses. I feel like the rebuttals does not fully address my concerns. Besides, although the authors updated the manuscript, the paper is still not easy to follow as most of them are wordy discussions with statistical analysis. Therefore I maintain my original score.
Response to Reviewer fdPC
Thank you for recognizing the motivation to study error organization beyond magnitude. Our finding is adaptation-associated reorganization of teacher-relative action residuals, not a quantization method or success predictor.
[W1, Q1] What is established, and how far does it generalize?
We study one adapted OpenVLA-OFT policy, one simulated cube-stacking task, and fixed-input predictions. No second policy, task, simulator, or closed-loop evaluation is reported; we do not claim transfer.
Nevertheless, across 13 ternary configurations (depth-based, random, and projection-based layer selections), mean pairwise Full-map Spearman alignment rises from 0.037 to 0.557 after adaptation (demonstration-bootstrap median gain +0.520, 95% CI [0.472, 0.568]). A new analysis of the submitted held-out predictions finds a positive alignment increase in all 19 held-out demonstrations (mean per-demonstration gain +0.437, 95% CI [0.387, 0.495]). This provides additional evidence of consistency within the task, not of cross-task generalization.
Controls refine the interpretation. Across six matched configurations, GT-only adaptation also raises Full-map similarity (0.186 to 0.445, versus 0.497 under GT+KD), as reported in the original submission. A distinct 3-bit quantizer and additional optimization seeds also support the within-setting trend. Crucially, matched full-precision controls achieve 0.421–0.843 alignment against counterpart-excluded adapted ternary references. Thus, quantization supplies heterogeneous initial perturbations but is not necessary for the shared adapted structure. We will make adaptation-associated residual geometry—not quantization specificity—the paper's headline.
[W2, Q2] What does this reveal that scalar recovery misses?
Uniform error shrinkage cannot explain this change. For B72, scalar recovery is 60.7%, yet the raw-to-adapted 48-cell rank correlation is −0.043 and a multiplicative scaling fit gives R² = −0.509. The relative ranking of residual cells changes substantially despite scalar recovery. Moreover, after removing additive chunk-position and action-dimension effects, Interaction alignment increases from 0.053 to 0.321 (bootstrap gain CI [0.191, 0.374]). Thus, the result is neither uniform shrinkage nor solely a common additive profile.
The practical output is an offline error-localization diagnostic: inspect the teacher-relative 8×6 residual map alongside T6. A post-hoc comparison of submitted held-out predictions shows why:
Despite nearly identical overall T6, the two configurations have different coordinate-level residual profiles. This identifies where their remaining discrepancies differ; the scalar average cannot. Alignment quantifies shared structure; residual maps locate the discrepancies. Neither is a policy-quality score.
We have not linked 48-cell alignment to closed-loop success or demonstrated better quantization selection, adaptation, or deployment. The demonstrated diagnostic value is coordinate-level error localization, complementary to—not a substitute for—task-level evaluation.
[W3] Making the contribution easy to follow
We will reduce the density of numbers and competing representations. We will reorganize the main story around three steps: (1) Full, the rank-comparable teacher-relative 8×6 residual map, as the primary result; (2) Interaction, used specifically to test whether alignment survives removal of additive chunk/dimension effects; and (3) Nuisance, moved to supporting sensitivity analysis. We will foreground full-precision controls and the B72/S2 comparison, moving secondary numbers out of the main narrative.
We will also demote template prediction from a headline contribution: its mean-final Spearman is 0.678 versus 0.658 for the additive baseline (difference +0.019, 95% CI [−0.001, 0.036]). This focuses the contribution on observed reorganization, without claiming an established rank-correlation advantage over the additive baseline.
We will state this within-task adaptation-associated structural finding, its diagnostic use, and its limits clearly in the revision.