acceptodds
Under review as a conference paper at ICLR 2027

Controlling Action-Head Co-Adaptation to VLM Representation Anisotropy

Abstract

Modern learning systems increasingly combine pretrained models with downstream heads. Despite the growing importance of such modular designs, how the downstream model co-adapts to the geometry of pretrained representations remains relatively underexplored. We study this problem through vision-language-action models (VLAs), where a pretrained vision-language model (VLM) conditions an action model for control. We find that the action model progressively concentrates its sensitivity on high-variance dimensions of the VLM outputs. Through controlled experiments across different backbone-head models, we further show that counteracting this sensitivity concentration can be beneficial under representation–task mismatch settings. Motivated by these observations, we introduce Adaptive Interface Equalizer (AdaIE), a parameter-free mechanism that reweights pretrained representations according to the evolving downstream sensitivity during training. AdaIE improves VLA success rates on simulation benchmarks, with gains of up to on RoboCasa-GR1 and on RoboTwin 2.0, and achieves an average gain of across five real-world manipulation tasks. It further generalizes to diffusion policies and world models, improving downstream performance in both settings. Our results highlight interface co-adaptation as an important optimization problem when coupling pretrained representations with downstream models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Multimodal Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_Z1VDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how action heads adapt to pretrained vision-language representations. The authors observe that action heads become increasingly sensitive to high-variance feature channels during training. They propose Adaptive Interface Equalizer (AdaIE), which rescales feature channels according to the evolving sensitivity of the downstream model without introducing additional trainable parameters. Experiments on simulated and real-world robotic manipulation, diffusion policies, and world models show overall performance improvements.

Critical strengths
  1. The paper addresses a practically relevant question about the interface between pretrained representations and downstream action models. AdaIE is simple to integrate into existing architectures and introduces modest training overhead.

  2. Evaluation spans multiple simulation benchmarks, five real-world tasks, and world models. The comparison between equalizing and reinforcing sensitivity provides useful initial evidence for the proposed approach.

Critical weaknesses
  1. The proposed mechanism lacks theoretical grounding and sufficient causal evidence. The paper links the benefits of sensitivity equalization to a mismatch between pretrained representations and downstream tasks. However, it does not establish when sensitivity concentration harms learning or why inverse-sensitivity reweighting should improve optimization or control. The aligned and mismatched settings in Section 2.2 differ in models, datasets, and tasks, making it difficult to attribute the observed differences specifically to alignment. The authors acknowledge this limitation. A theoretical guarantee is not necessary to establish empirical usefulness, but the current evidence supports the effectiveness of reweighting more strongly than the proposed explanation for that effectiveness.

  2. Comparisons and ablations do not fully establish the distinctive benefit of AdaIE. The main VLA experiments lack direct comparisons with variance-based reweighting and simple normalization, and the contributions of EMA and norm matching are not isolated. Limited reporting across independent training seeds and only 20 real-world trials per task per method also leave uncertainty about the robustness of the gains.

Questions for authors
  1. Is there evidence that representation–task mismatch determines the benefit of equalization when the model architecture and downstream task are held fixed? Can theoretical analysis or additional diagnostics substantiate this explanation?

  2. How does AdaIE compare with variance-based reweighting and simple normalization in the main VLA experiments? Does norm matching alone explain part of the improvement?

  3. How many independent training seeds were used for Tables 2 and 3? Can the authors report variability across training runs or confidence intervals to clarify the reliability of the gains?

6 comments

Thank you for your thoughtful review! We address the concerns below:

1. Mismatch metrics

Thanks for raising this point. We looked through the literature for ways to measure this gap. We found two measures in the transfer learning literature that might be helpful: LogME [1] and linear probing [2]. Specifically, LogME estimates how suitable frozen backbone features are for a downstream task, and linear probing measures how much target variance a simple linear readout can explain. They are used to estimate transferability from a backbone to a downstream task. Higher LogME and linear probing mean that downstream targets are more easily predictable from the frozen features. We report these two metrics below:

InterfaceLogME ↑Linear probing↑
CIFAR: DeiT-S → labelAligned−0.8250.717
NYUv2: DINOv2-S → patch depthAligned−0.6740.808
Push-T: ResNet-18 → DPMismatched−1.2700.289
RoboCasa: Qwen3-VL-2B → GR00TMismatched−1.2890.158

The two aligned interfaces have higher scores than the two mismatched interfaces, reflecting that they are easier to adapt to the downstream tasks, empirically. However, a more comprehensive understanding of representation–task alignment, particularly its theoretical foundations, requires further investigation, as acknowledged in our Limitations.

2. Var-based experiments for VLAs and simple normalization.

We conduct additional experiments on a subset of RoboCasa-GR1 (12 random subset tasks), training for 30K steps and evaluating with 60 episodes per task. Due to computational resource constraints, we use this subset for the additional ablations. For the variance-based baseline, we follow the same approach used in our LeWM experiments.

MethodBaselineα=0.1α=0.25α=0.5
AdaIE24.226.4 (+2.2)27.8 (+3.6)24.6 (+0.4)
Variance-based24.218.1 (-6.1)22.9 (-1.3)24.0 (-0.2)

As shown above, the variance-based variant does not improve over the baseline (but instead degrades performance) at either tested α\alpha, while AdaIE consistently outperforms the baseline.

Regarding simple normalization, our baseline already includes LayerNorm and a learnable linear projection before the VLM features enter the DiT. AdaIE is applied after these operations, which are shared by both variants. Furthermore, NormMatch only preserves the overall RMS magnitude after channel-wise reweighting. Without reweighting (α=0), NormMatch reduces to the identity transformation. Thus, the improvement cannot be explained by the addition of normalization or norm matching alone.

3. Training seeds were used for Tables 2 and 3

As in our Limitations, due to computational resource constraints, we were only able to conduct multi-seed training for a subset of our experiments. Tables 2 and 3 report results from a single training run, while Tables 1 (which also evaluates RoboCasa-GR1, but on a subset of the tasks in Table 2) and 5 report mean and standard deviation across multiple training seeds. Additionally, our new ablation results above show consistent improvements across different α\alpha values. This provides further evidence of AdaIE's reliable gains.

Thank you again for your time and thoughtful review. We look forward to your feedback.

[1] LogME: Practical Assessment of Pre-trained Models for Transfer Learning, ICML 2021.

[2] Scalable Diverse Model Selection for Accessible Transfer Learning, NeurIPS 2021.

Thank you again for your thoughtful review.

With the discussion period approaching its conclusion, we wish to kindly check whether our response has addressed your questions.

If you find our responses helpful, we would greatly appreciate your acknowledgment. If you have any further questions or would like additional clarification, please do not hesitate to reach out. We are committed to providing any further responses during this discussion phase.

We sincerely appreciate your time and thoughtful consideration, and we look forward to your feedback.

Dear Reviewer Z1VD,

Thank you again for your time and consideration.

As the discussion period ends tomorrow, we would greatly appreciate it if you could take a moment to review our rebuttal and additional experiments when you have a chance. We look forward to hearing your feedback.

Best,

The Authors

Reviewer_tG2MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the co-adaptation between pretrained VLM representations and downstream action heads during joint training. The authors observe that action-head sensitivity progressively concentrates on high-variance channels of VLM representations, potentially introducing an undesirable bias when pretrained representations are not well aligned with downstream control objectives. To address this, they propose Adaptive Interface Equalizer (AdaIE), a parameter-free mechanism that periodically estimates downstream sensitivity using Jacobian probing and adaptively reweights conditioning features during training. Experiments across VLA models, diffusion policies, and world models demonstrate performance improvements, including gains on simulation benchmarks and real-world manipulation tasks.

Critical strengths
  1. The paper investigates how action-head sensitivity evolves in relation to the anisotropic representations of a pretrained VLM during joint fine-tuning. While co-adaptation and feature reliance have been studied in prior work, examining their dynamics specifically at the VLM–action-head interface provides an interesting perspective on optimizing modular VLA systems.

  2. AdaIE provides a lightweight intervention without introducing additional learnable parameters or modifying the underlying architectures. Its sensitivity-guided reweighting mechanism can be integrated into existing training pipelines with relatively modest computational overhead.

  3. The paper evaluates the method across multiple architectures and tasks, including two VLA simulation benchmarks, real-world humanoid manipulation, diffusion policies, and world models. The real-world experiments are particularly valuable, and the improvements suggest practical potential beyond a single benchmark.

Critical weaknesses
  1. I think the proposed perspective on co-adaptation is interesting, and the empirical improvements are encouraging. Figure 1 provides an interesting observation of how action-head sensitivity evolves during training. However, what happens to the representations and sensitivity distributions after introducing AdaIE? The paper lacks a corresponding analysis showing how AdaIE changes the feature distributions and the relationship between representation variance and downstream sensitivity. Such an analysis would help establish whether AdaIE actually mitigates the phenomenon identified in Figure 1, rather than simply improving performance through feature rescaling. Additionally, given that co-adaptation is central to the paper, its definition and relationship to existing literature deserve a more detailed discussion, particularly in the Related Work section.

  2. While the proposed Jacobian-based reweighting is interesting, the improvement on RoboTwin 2.0 is relatively modest (+1.68 percentage points). This raises a question: would Jacobian-based reweighting still demonstrate an advantage when compared with simpler feature reweighting baselines? The authors provide a variance-based comparison on LeWM, but not on the main VLA benchmarks. Such comparisons would help clarify whether the improvements specifically come from Jacobian sensitivity or from feature rescaling in general.

Questions for authors
  1. Section 2.2 categorizes depth estimation and classification as aligned settings, while diffusion policies and VLAs are considered mismatched. Could the authors clarify the criteria underlying this distinction? Is there a way to quantify representation–task alignment beyond the differences between pretraining and downstream objectives?

  2. The overall results are encouraging, but the improvement on RoboTwin 2.0 is relatively modest, particularly under randomized conditions. Could the authors comment on the sensitivity of these results to training randomness or checkpoint selection?

3 comments

Thanks for the comprehensive review! It helps us to improve the paper. We answer the questions below:

1. What does AdaIE change?

We measure the correlation between channel variance and action-head sensitivity across the 2,048 conditioning channels, defined as ρz=corrj(log⁡Var[zj],log⁡gj).\rho_z = \mathrm{corr}_j(\log \mathrm{Var}[z_j], \log g_j).

α\alphaStep 01K2K5K30K
0 (baseline)+0.02+0.43+0.49+0.43+0.48
0.5 (equalize)+0.02+0.24+0.27+0.19+0.07
-0.5 (exploit)+0.02+0.54+0.56+0.49+0.71

Without AdaIE the head's sensitivity becomes aligned with the high-variance channels within the first 1K steps and stays aligned, whereas with AdaIE this alignment is removed from the input the head receives, and the alpha=-0.5 strengthens it.

2. Var-based experiments for VLAs

We conduct additional experiments on a subset of RoboCasa-GR1 (12 random subset tasks), training for 30K steps and evaluating with 60 episodes per task. Due to computational resource constraints, we use this subset for the additional ablations. For the variance-based baseline, we follow the same approach used in our LeWM experiments.

MethodBaselineα=0.1α=0.25α=0.5
AdaIE24.226.4 (+2.2)27.8 (+3.6)24.6 (+0.4)
Variance-based24.218.1 (-6.1)22.9 (-1.3)24.0 (-0.2)

As shown above, variance-based reweighting does not improve over the baseline, while AdaIE consistently outperforms the baseline.

3. Clarification for mismatched/align assumption.

As discussed in the paper’s limitations, our aligned/mismatched grouping is based on an a priori assumption. During this rebuttal, we also looked at the transfer learning literature and found two measures that can be helpful, LogME[1] and linear probing[2]. LogME estimates how suitable frozen backbone features are for a downstream task, and linear probing measures how much target variance a simple linear readout can explain. They are used to estimate transferability from a backbone to a downstream task (higher is easier to transfer). We use them as quantitative proxies for representation–task alignment.

InterfaceLogME ↑Linear probing↑
CIFAR: DeiT-S → labelAligned−0.8250.717
NYUv2: DINOv2-S → patch depthAligned−0.6740.808
Push-T: ResNet-18 → DPMismatched−1.2700.289
RoboCasa: Qwen3-VL-2B → GR00TMismatched−1.2890.158

The two aligned interfaces have higher scores than the two mismatched interfaces, reflecting that they are easier to adapt to the downstream tasks. However, we believe a comprehensive analysis requires further research, which is why we discuss this point in our limitations.

4. Random Robotwin2.0 results

Thank you for the question. As discussed in the caption of Table 14, the smaller gain under Randomized may be due to the additional perceptual and generalization challenges, which are not directly addressed by AdaIE. Regarding training randomness, we acknowledge in the Limitations that computational constraints have allowed us to conduct multi-seed training for only a subset of experiments. We agree that further evaluation across training seeds and checkpoints would help better understand the robustness of the results.

Thank you again for your time and thoughtful review. We look forward to your feedback.

[1] LogME: Practical Assessment of Pre-trained Models for Transfer Learning, ICML 2021.

[2] Scalable Diverse Model Selection for Accessible Transfer Learning, NeurIPS 2021.

Thank you again for your thoughtful review.

With the discussion period approaching its conclusion, we wish to kindly check whether our response has addressed your questions.

If you find our responses helpful, we would greatly appreciate your acknowledgment. If you have any further questions or would like additional clarification, please do not hesitate to reach out. We are committed to providing any further responses during this discussion phase.

We sincerely appreciate your time and thoughtful consideration, and we look forward to your feedback.

Dear Reviewer tG2M,

Thank you again for your time and consideration.

As the discussion period ends tomorrow, we would greatly appreciate it if you could take a moment to review our rebuttal and additional experiments when you have a chance. We look forward to hearing your feedback.

Best,

The Authors

Reviewer_XE1iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how an action head adapts to the geometry of pretrained vision-language representations. It observes that, during training, the action head becomes increasingly sensitive to high-variance feature channels. The authors propose Adaptive Interface Equalizer (AdaIE), which periodically estimates this sensitivity using stochastic Jacobian probes and inversely rescales the conditioning channels while preserving their overall magnitude. AdaIE introduces no learnable parameters and improves performance on two simulated manipulation benchmarks and five real-robot tasks, with additional experiments on diffusion policies and world models.

Critical strengths
  • Simple intervention targeting an important interface: The method addresses how pretrained representations are consumed by an action head, offering an improvement without redesigning the backbone or policy architecture.
  • Broad empirical evaluation: Experiments cover two VLA benchmarks, multiple backbone sizes and families, real hardware, and extensions to diffusion policies and world models.
  • Meaningful hardware results: Overall task success increases from 30% to 49% across five tasks, with twenty trials per task and method. Separately reporting interaction success helps distinguish reaching or grasping from completing the task.
  • Practical computational cost: The paper reports a 3.7% training-time overhead in one large-scale setting, and sensitivity estimates are frozen during inference. The equalize-versus-exploit comparison also provides a useful initial test of the proposed motivation.
Critical weaknesses
  • The benefit of sensitivity-based weighting over simpler rescaling remains unclear in the main VLA setting. The paper includes a variance-based baseline for world models, but not for VLAs. Please extend this comparison to one representative VLA benchmark under comparable tuning budgets. This would help establish whether downstream sensitivity estimation is necessary for the reported gains.
  • The experiments do not isolate representation–task mismatch as the reason equalization helps. The “aligned” and “mismatched” groups differ in their tasks, backbones, heads, and training objectives. Their contrasting results therefore cannot attribute the benefit specifically to mismatch. The authors acknowledge this limitation, but the explanation remains central to the paper. Please vary backbone pretraining or task alignment within an otherwise fixed setup, or present mismatch as a hypothesis rather than an established explanation.
  • The importance of adaptation and hyperparameter robustness is unclear for VLAs. The α\alpha sweep is presented for LeWM, and the authors note that the best value varies across tasks. However, the main contribution concerns VLA training. Please show a compact VLA ablation of equalization strength and compare continuously updated weights with weights frozen after an initial calibration period. This would clarify whether AdaIE is robust enough to use with a common setting and whether ongoing updates are necessary.
  • The real-world generalization claim needs a clearer definition of OOD. Table 4 separates in-distribution and out-of-distribution tasks, but the main text does not explain precisely what is unseen: object identity, appearance, instructions, manipulation behavior, or some combination. Please specify the train–test distinction and describe the corresponding training coverage. The large OOD improvement is promising, but its significance depends on the difficulty and nature of that shift.
Questions for authors
  • Please clarify what makes the Table 4 conditions out of distribution: for example, whether they involve unseen object instances or categories while retaining familiar manipulation tasks.
5 comments

Thank you for your comprehensive review. We address the concerns below:

1. Alpha sweep and Var-based experiments for VLAs

We conducted the requested alpha sweep and variance-based comparison on VLA. Due to computational resource constraints, we can only run additional experiments on a subset of RoboCasa-GR1 (12 randomly selected tasks), training for 30K steps and evaluating with 60 episodes per task. For the variance-based baseline, we follow the same reweighting approach used in our world models experiments.

MethodBaselineα=0.1α=0.25α=0.5
AdaIE24.226.4 (+2.2)27.8 (+3.6)24.6 (+0.4)
Variance-based24.218.1 (-6.1)22.9 (-1.3)24.0 (-0.2)

As shown above, the variance-based variant does not improve over the baseline (but instead degrades performance) at either tested α\alpha, while AdaIE consistently outperforms the baseline. These results suggest that downstream sensitivity provides a more effective reweighting signal than feature variance alone. However, AdaIE's improvement becomes smaller at α=0.5\alpha =0.5. As acknowledged in our Limitations, the optimal α may vary across tasks and architectures, requiring task-specific tuning.

2. Clarify aligned/mismatched as a hypothesis

Thank you for pointing this out. As discussed in the paper’s limitations, our aligned/mismatched grouping is based on an a priori assumption. We will make this clearer in the revision.

We also looked at the transfer learning literature and found two measures that can be helpful, LogME[1] and linear probing[2]. LogME estimates how suitable frozen backbone features are for a downstream task, and linear probing measures how much target variance a simple linear readout can explain. They are used to estimate transferability from a backbone to a downstream task (higher is easier to transfer). We use them as quantitative proxies for representation–task alignment:

InterfaceLogME ↑Linear probing↑
CIFAR: DeiT-S → labelAligned−0.8250.717
NYUv2: DINOv2-S → patch depthAligned−0.6740.808
Push-T: ResNet-18 → DPMismatched−1.2700.289
RoboCasa: Qwen3-VL-2B → GR00TMismatched−1.2890.158

The two aligned interfaces have higher scores than the two mismatched interfaces.

3. Comparison between continuously updated and frozen sensitivity weights

Thank you for raising this question. We conduct this ablation using the same setting as in our response to the first question, with α=0.25\alpha=0.25 for both variants. Specifically, we use the first 10% of training (3K steps) as the calibration phase, after which we freeze the statistics for the remaining 27K steps.

MethodSuccess Rate (%)
Baseline24.2
Frozen@3K24.2
Continuos27.8

As shown above, freezing the statistics after 3K steps provides no improvement over the baseline. This suggests that ongoing sensitivity adaptation is important.

4. Clarification for OOD

Thank you for the question. In Table 4, OOD refers to unseen object categories. Specifically, Toy Car, Cheese Ball, and Milk Bottle were entirely absent from the training data, while the underlying manipulation behaviors (e.g., pick-and-place) were covered by the training tasks. We will clarify this distinction in the revised paper.

Thank you again for your time and thoughtful review. We look forward to your feedback.

[1] LogME: Practical Assessment of Pre-trained Models for Transfer Learning, ICML 2021.

[2] Scalable Diverse Model Selection for Accessible Transfer Learning, NeurIPS 2021.

Thank you again for your thoughtful review.

With the discussion period approaching its conclusion, we wish to kindly check whether our response has addressed your questions.

If you find our responses helpful, we would greatly appreciate your acknowledgment. If you have any further questions or would like additional clarification, please do not hesitate to reach out. We are committed to providing any further responses during this discussion phase.

We sincerely appreciate your time and thoughtful consideration, and we look forward to your feedback.

Dear Reviewer XE1i,

Thank you again for your time and consideration.

As the discussion period ends tomorrow, we would greatly appreciate it if you could take a moment to review our rebuttal and additional experiments when you have a chance. We look forward to hearing your feedback.

Best,

The Authors

Reviewer_M99WOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the representation interface between pretrained vision-language models (VLMs) and downstream action heads in vision-language-action (VLA) models. The authors identify "action-head co-adaptation," a phenomenon where downstream action models progressively concentrate their sensitivity on high-variance, anisotropic feature dimensions of pretrained VLM representations during training. Through controlled experiments on aligned versus mismatched tasks, the paper demonstrates that counteracting this sensitivity concentration improves performance when the pretrained representation objective differs from downstream control. To regulate this interface, the paper proposes the Adaptive Interface Equalizer (AdaIE), a parameter-free mechanism that estimates action-head sensitivity online using Hutchinson's random probe and inversely rescales VLM conditioning channels while preserving global RMS feature magnitude. AdaIE is evaluated across simulation benchmarks (RoboCasa-GR1, RoboTwin 2.0), dual-arm real-world humanoid manipulation tasks (IRON-R01-1.11), and non-VLA predictive frameworks (Diffusion Policy, LeWM, DINO-WM).

Critical strengths

The reviewer appreciates the following paper's strengths:

A. Method Novelties

  • Identifies a previously uncharacterized optimization phenomenon in modular robot learning: downstream action heads co-adapting to the anisotropic feature geometry of a pretrained VLM backbone. I appreciate this finding a lot.

  • The author also proposes a parameter-free mechanism (AdaIE) that dynamically equalizes conditioning interfaces online without modifying backbone weights or introducing learnable adapter layers.

  • The method establishes a useful diagnostic framework by separating "aligned" (e.g., depth estimation, classification) and "mismatched" (e.g., continuous control) interfaces to isolate when representation anisotropy hurts learning.

B. Strengths of Experiments:

  • Demonstrates broad architectural and task applicability across simulation benchmarks (RoboCasa-GR1, RoboTwin 2.0), real-world humanoid manipulation (IRON-R01-1.11), and world models (LeWM, DINO-WM).

  • Real-world evaluation across five physical tasks yields a +19.0% average task success rate gain (49.0% vs. 30.0%) over the baseline. Additionaly, the additional overhead is minimal.

C. Presentation

  • The paper provides clear motivational visualizations (Figure 1) tracking per-channel variance against action-head sensitivity across training iterations. System architecture diagrams (in Figure 2) map directly to the mathematical formulations of sensitivity probing and norm matching.
Critical weaknesses

The paper has the following points need to be addressed:

  • First, the AdaIE's effectiveness relies on the equalization strength α\alpha. The ablation studies show that the optimal value for α\alpha varies across tasks and architectures, requiring manual search. Please discuss a general strategy to tune these values.

  • Architectural Scope Constraints: The current formulation operates exclusively on single-layer conditioning interfaces. Modern multimodal architectures frequently employ multi-layer conditioning (e.g., cross-attention across multiple Transformer layers or Mixture-of-Transformers), where single-layer equalization may be insufficient.

  • The paper relies on an a priori qualitative assumption to categorize tasks as "aligned" or "mismatched". It might be better to have a quantitative metric to measure representation-task mismatch prior to training.

Questions for authors

I have the following questions:

a. Given that the optimal equalization strength α\alpha varies across tasks, is it possible to formulate an adaptive schedule for α\alpha based on online feature statistics to eliminate manual tuning?

b. How does AdaIE perform when applied to multi-layer conditioning interfaces (e.g., injecting VLM representations across multiple layers of a DiT action head) rather than a single input layer?

c. Can you propose a quantitative metric (e.g., based on Jacobian-feature alignment or mutual information) to measure "representation-task mismatch" before training to predict whether AdaIE will benefit a specific backbone-head pair?

3 comments

We thank the reviewer for the positive and encouraging feedback! We appreciate the recognition of our contributions and the constructive suggestions. We address each point below.

1. α\alpha without manual tuning:

Since α\alpha is differentiable, we also tried to make it learnable and jointly optimize it with the action head, rather than manually tuning its value. We evaluate this approach on a 12 random tasks subset of RoboCasa-GR1, with 30K training steps and 60 evaluation episodes per task.

DatasetBaselineLearnable α\alphaα\alpha=0.25α\alpha=0.1α\alpha=0.5
RoboCasa-GR124.222.427.826.424.6

Interestingly, directly learning α\alpha performs even worse than the baseline. We hypothesize that this may be related to the co-adaptation phenomenon studied in our paper. These results suggest that simply making α learnable is not sufficient to eliminate manual tuning. Developing an adaptive schedule based on online sensitivity statistics remains an interesting direction for future work.

2. Metrics for alignment:

Thanks for raising this point. We looked through the literature for ways to measure this gap. We found two measures in the transfer learning literature that might be helpful: LogME [1] and linear probing [2]. Specifically, LogME estimates how suitable frozen backbone features are for a downstream task, and linear probing measures how much target variance a simple linear readout can explain. They are used to estimate transferability from a backbone to a downstream task. Higher LogME and linear probing mean that downstream targets are more easily predictable from the frozen features. We report these two metrics below:

InterfaceLogME ↑Linear probing↑
CIFAR: DeiT-S → labelAligned−0.8250.717
NYUv2: DINOv2-S → patch depthAligned−0.6740.808
Push-T: ResNet-18 → DPMismatched−1.2700.289
RoboCasa: Qwen3-VL-2B → GR00TMismatched−1.2890.158

The two aligned interfaces have higher scores than the two mismatched interfaces. These results provide evidence of differences in representation–task alignment. However, we believe a comprehensive analysis requires further research, which is why we discuss this point in our limitations.

3. Multi-layer interfaces

Thank you for the question. Our current experiments focus on single-layer conditioning interfaces. In principle, AdaIE could be extended to multi-layer conditioning by estimating sensitivity and applying channel-wise reweighting separately at each conditioning interface, allowing it to account for layer-specific sensitivity patterns. However, architectures such as MoT differ from conventional VLAs, as VLM and action tokens interact through joint self-attention rather than a separate conditioning interface. Extending AdaIE to such architectures would therefore require a more substantial redesign.

Thanks again for reviewing our paper.

[1] LogME: Practical Assessment of Pre-trained Models for Transfer Learning, ICML 2021.

[2] Scalable Diverse Model Selection for Accessible Transfer Learning, NeurIPS 2021.

Thanks, author, for the rebuttal. It would be good if you could integrate these clarifications into the revised PDF. Overall, I keep the current positive score.

We sincerely thank you again for your thoughtful and positive review. We will revise our paper accordingly.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for the review. The authors have responded to the points you raised, and the response period closes on 18 November (AoE). I would be grateful if you could post a brief reply on whether your concerns are resolved, partly resolved, or still open, so that the authors have a fair chance to address them. If anything still needs clarifying, please ask now while they can still respond.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.