acceptodds
Under review as a conference paper at ICLR 2027

When Policy Representations Fail: Diagnosis and Repair through Behavioral Evidence

10 views

Abstract

Compact policy representations adapt to downstream tasks by optimizing low-dimensional coordinates rather than the full policy parameter vector, reducing adaptation overhead. However, these representations are typically kept fixed after training. When adaptation to a new task fails, expanding the representation is a natural response. We show that such failure is only a symptom, not evidence of insufficient capacity: similar performance degradation can result from limited target-relevant information, inadequate coordinate search, poor organization of behavior within the existing representation, or genuinely missing adaptation directions. We prove that final performance alone is insufficient to uniquely identify the source of failure or determine how the representation should be modified. Because these latent factors are not directly observable, we introduce a behavioral witness: an independently verified successful target adaptation that provides both behavioral and directional evidence. Building on this reference, we develop a witness-guided hierarchical diagnosis and repair framework. Under fixed budgets, the framework tests pre-specified interventions from weaker to stronger and selects the weakest verified repair that satisfies both target-performance and behavior-preservation requirements, expanding the representation only when fixed-capacity alternatives remain insufficient. We evaluate the framework in continuous control and vision-language-action policies. The results show that behavioral evidence supports different levels of sufficient repair, turning default expansion into an evidence-guided, on-demand process that avoids unnecessary representation growth. At the repair stage, the resulting approach also substantially reduces trainable-parameter and peak-memory overhead relative to full fine-tuning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Compact Policy Representations, Policy Adaptation, Continual Learning, Representation Expansion, Behavioral Witness, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_W7nMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies failed adaptation in compact policy representations and proposes using a successful adaptation as a behavioral witness for repair. The witness supplies action targets for fixed-capacity fitting and a parameter direction for expansion. A hierarchy tests stronger coordinate search, witness-informed fitting, decoder updates and joint paths, then aligned expansion, selecting the weakest tested intervention that meets target-performance and archive-preservation criteria. Experiments include controlled failure constructions, natural HalfCheetah reward variants, and SmolVLA/LIBERO adaptation. The MuJoCo experiments demonstrate different sufficient repairs, while the VLA experiments evaluate witness-aligned directions rather than the full hierarchy.

Critical strengths

1. Useful problem formulation: low return does not distinguish unsuccessful coordinate search from an inadequate policy representation, so expansion should not be the default response. The paper appropriately limits its conclusions to sufficient repairs within a declared intervention family and budget. The joint-only case in Table 7 also shows why testing individual interventions alone can miss a viable fixed-capacity repair.

2. Meaningful controls for witness information: archive-only refitting matches the structure and training budget of witness-conditioned fitting, while random expansion controls match direction norm and, in VLA, layer-wise rank and spectrum. On Random-20, aligned residuals achieve 69/200 successes versus 0/200 for each of three random controls (Table 23). This supports the utility of task-aligned evidence beyond additional capacity alone.

Critical weaknesses

1. The main practical claim is not convincingly established. In the original prospective panel, hierarchical selection and always-expand both pass 12/12 cases, but the hierarchy avoids only one expansion while requiring approximately 4.9 times as many interactions (Table 11). The later targeted panel resolves 10/12 cases without expansion, but consists of two rewards selected after development inspection. VLA does not evaluate the full hierarchy. Its efficiency comparison covers only 50 continuation steps after a shared 5,000-step candidate, and Aligned is slightly more memory-intensive and slower than Direct LoRA, with five additional successes out of 120 episodes (Tables 31–32). These results establish examples of successful repair, but not a compelling performance–cost–storage advantage for the proposed procedure.

Key controls and consistency checks remain incomplete. MuJoCo expansion uses the candidate-minus-base direction, so Gnew(zb,1)=θwG_{\mathrm{new}}(z_b,1)=\theta_w by construction; final confirmation coefficients remain close to one. Without fixed-α=1\alpha=1 witness replay or scalar-search-only controls, the contribution of the subsequent fitting/PPO procedure is unclear. In the untruncated VLA experiment, Aligned should equal Direct algebraically, yet seven paired rollout outcomes differ (Table 24). Loaded-weight and same-input action checks remain outstanding. The manuscript acknowledges both issues, but disclosure does not resolve them: the former limits the interpretation of expansion results, and the latter leaves part of the VLA validation unreliable.

Questions for authors
  1. What advantage does the hierarchy offer over always-expand and direct witness reuse when candidate acquisition, diagnosis, evaluation, repair, and persistent storage are all included? Can this be demonstrated on a prospectively fixed task panel?

  2. How do fixed-α=1\alpha=1 witness replay and scalar-search-only expansion compare with the full repair procedure under the same target and archive criteria?

  3. What explains the Direct–Aligned discrepancies in Table 24? Please verify loaded weights, adapter scaling, same-input actions, and rollout agreement under matched inference randomness.

1 comment

This paper decomposes adaptation failures in compact policy representations into representation loss and exploration error, and proposes using a verified successful adaptation (here, behavioral witness) to identify the minimum sufficient repair by testing weaker interventions first. The problem formulation is useful: low return alone cannot distinguish the causes of failure. However, as currently written, the paper makes it difficult to understand which hypothesis each experiment is intended to support. Although the questions about the discrepancies in Table 24 and the replay controls have been resolved, the ablations are insufficient to establish whether the higher-level stage that uses witnesses is necessary. The presentation is also still not intuitive, making it difficult to justify an Accept recommendation.

Reviewer_zAfzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission proposes a behavioral framework for diagnosing and repairing failures when adapting compact robot policy representations to new settings. It distinguishes between different causes of failure (e.g. inadequate search, lack of relevant information in training data, poor policy mapping organisation, and representation capacity limitations) and applies targeted interventions for each case, instead of blindly expanding the model capacity. This is achieved through a witness-guided hierarchical diagnosis and repair framework, that independently evaluates successful adaption through two forms of evidence, namely behavioral (actions) and directional (param displacement) reference. Using this, the framework tests a predefined family of increasingly expensive interventions (i.e. from stronger coordinate search to representation expansion) and selects the weakest representation that satisfied both a target-performance requirement and a behavior-preservation constraint.

Critical strengths
  • The manuscript tackles a timely and interesting problem, from a novel perspective. The key idea of framing representation expansion as a conditional intervention (following a principled diagnosis), rather than a default response across all adaptation failures is insightful.
  • The formal separation of adaptation failures into representation error and finite-budget search is also an interesting idea that motivates additional interventions, combined with the proposed novel behavioral-witness concept and hierarchical repair mechanism. This formulation is highly interpretable and can guide future work to build more robust robot policies.
  • The paper overall considers computational costs and parameter efficiency aspects of the deployed policies, enhancing its practical applicability in real-world deployment;
Critical weaknesses
  • The process of generating and verifying a generalisable behavioural witness seems to be a significant research challenge itself, also incurring significant computational cost. A deeper analysis on the underlying trade-offs is required to indicate the scalability and practical applicability of this strategy (e.g. if more radical intervention approaches need to be added etc.). As such, the "difficulty of adaptation" (the key motivation behind the design of the proposed mechanism) is not adequately quantified; and may be shifted to the task of finding a witness.
  • The evaluation of the proposed approach is solely conducted in simulation. Real-world validation of the findings would make the contribution of the proposed approach more convincing, while exposing potential limitations and trade-offs in real-world applicability.
  • The conducted evaluation does not seem to establish whether the adopted repairs preserve the performance on the previously learned tasks (an important property for reliable deployment in continual learning scenarios).
Questions for authors

Consider replying to the concerns raised in the weaknesses section above.

Reviewer_HvQnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to repair policy representation after unsuccessful target-task adaptation. It argues that low return is not an evidence of missing capacity, and proposes a framework for deciding how to repair a compact policy representation when adaptation to a new task fails, rather than immediately adding capacity.

Critical strengths

None

Critical weaknesses

The writing needs substantial improvement. Key concepts are used before they are adequately defined or explained. For example, the meanings of “compact policy representations” and “adaptation” are not sufficiently clear.

The problem formulation also needs to be made explicit, including the setting, assumptions, and objectives.

Overall, the manuscript reads like an incomplete draft. I encourage the authors to undertake a substantial revision before resubmission, focusing particularly on clarifying the foundational concepts and formalizing the problem.

Questions for authors

Please refer to the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.