acceptodds
Under review as a conference paper at ICLR 2027

Acting Is Easier Than Adapting: The Geometry and Design of Adaptation-Sufficient Representations

Abstract

A representation can be sufficient for acting in the training environment yet insufficient for adapting after the environment changes. The missing object is not the nominal action but its prospective response: how the optimal action moves with the environment. For smooth continuous actions, the curvature-weighted projection of this response onto a frozen code yields a local conservation law, , \(\Omega=\Gamma_\phi+\Lambda_\phi\), decomposing prospective sensitivity into irrecoverable representation debt and executable motion. That motion is useful only to the extent that deployment evidence can identify which response to execute. An exact three-policy Gaussian frontier prices this evidence and shows that representation refinement is discounted by the resolvable fraction of newly executable motion. Representation rankings are therefore evidence-dependent and can reverse even between equal-capacity codes with identical nominal control. AP-GFAR operationalizes this principle by learning executable prospective responses and selecting representation–evidence programs from calibrated branch values, with freezing as an explicit option. Controlled reward systems verify the refinement law and ranking reversal, and equal-capacity learned encoders reproduce it. In native HalfCheetah, pre-evaluation complete-program pair-value estimates predict heterogeneous rankings under fixed interfaces; a prospectively locked, cross-fitted interface design then yields the predicted reversal across all ten declared systems. Fresh learned-POMDP and fixed-checkpoint VariBAD studies extend the evidence beyond the analytic model, showing gains from complete-program valuation over frozen/no-probe and native online deployment, respectively. Representation value is therefore a property of the representation–evidence pair, not of the representation alone.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, representation learning, deployment-time adaptation, adaptive control, control-sufficient representations, partial observability, active information acquisition, learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_61KCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when a representation sufficient for control in a nominal environment becomes insufficient for adaptation after an environment change. It separates local changes in the optimal action into a component recoverable from the representation and a residual, and quantifies the resulting value loss (p.4, Theorem 2). In a quadratic setting, it then relates the value of the recoverable response to the available task-only evidence (p.5, Proposition 4). Controlled experiments and learned systems examine evidence-dependent representation rankings and the selection of information-acquisition and adaptation programs.

Critical strengths
  1. The paper connects information lost through compression with uncertainty about the environment. It defines the loss that remains even when the environment parameter is revealed (p.3, lines 110–128), and connects it to a curvature-weighted decomposition of local action responses (p.4, lines 162–184). The conditions for evidence-dependent ranking reversals make this connection precise by relating the responses a representation retains to the directions the evidence can identify (p.5, lines 240–248).

  2. The controlled experiments test the proposed mechanism, and the paper states their limitations. The synthetic experiments directly compare the rewards of optimal actions chosen with full information and with compressed information (p.22, Appendix C.5, lines 1150–1167). The paper also explicitly limits the freezing result to its three-policy library (p.5, lines 216–222) and reports that a matched generic acquisition head selects the same acquisition counts on the interpolation evaluation (p.34, lines 1815–1822).

Critical weaknesses
  • W1. I am not yet convinced of the significance of the theoretical advance over prior work. The related-work discussion identifies representation-level sensitivity and its coupling to evidence as the contribution (p.9, lines 466–481), but does not explain sufficiently what this characterization lets us understand or predict beyond existing theory. The Introduction's early formal counterexample (p.1, lines 46–53) also leaves the motivation for this coupling underdeveloped (p.2, lines 72–93). The quantitative characterization could itself be a substantial contribution; my concern is that its significance is not yet well established. (See Q1 and Q3.)

  • W2. What the geometric theory explains in the learned-system experiments could be clearer. In HalfCheetah, the authors predict representation rankings before evaluation and select measurements that reverse the rankings on held-out data (p.8, lines 420–431). The predictions use estimates of the returns from complete adaptation procedures; they do not use the quantities in the local analysis (p.27, lines 1431–1447). I see the value of these results, but I am less sure what they establish about the geometric theory. (See Q1, Q2, and Q3.)

Questions for authors
  • Q1. What is the main conclusion or prediction enabled by the proposed characterization that prior theory does not provide (p.9, lines 466–481)? For me, a clearer account of the theoretical insight would help assess the contribution, even without practical performance gains. Which existing results most directly support that insight? (See W1 and W2.)

  • Q2. How should I interpret the HalfCheetah results in relation to the geometric theory? In particular, how does the theory inform the predictions and the choice of measurements when the predictions are computed from estimated returns for complete adaptation procedures (p.27, lines 1431–1447)? (See W2.)

  • Q3. Which adaptation settings does the theory aim to explain, and why are PEARL and VariBAD suitable representatives of that scope (p.2, lines 101–106)? For methods such as DPT, AD, RL², and SSM-based meta-RL, how would the representation, evidence, and response in the theory correspond to their components—or which assumptions put them outside scope? This asks for scope clarification, not additional experiments or comparisons. (See W1 and W2.)

    The methods referred to here are:

    • DPT: Lee et al. (2023). Supervised Pretraining Can Learn In-Context Reinforcement Learning. NeurIPS.
    • AD: Laskin et al. (2023). In-Context Reinforcement Learning with Algorithm Distillation. ICLR.
    • RL²: Duan et al. (2016). RL²: Fast Reinforcement Learning via Slow Reinforcement Learning.
    • SSM-based meta-RL: Lu et al. (2023). Structured State Space Models for In-Context Reinforcement Learning. NeurIPS.
Reviewer_RDcLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper looks into the question of how the optimal action moves with the environment. It For smooth continuous actions, the curvature-weighted projection of this response onto a frozen code yields a local conservation law, Ω = Γϕ + Λϕ.

From my own reading, it is not clear what the exact contribution of this paper has. Honestly, the paper is hard to read.

The paper's abstract is just composed of a collection of plain, isolated statements, and it is not easy to make out what their logic connections are .

In the abstract, there are no definitions of the notations, and there are no definitions of the AP-GFAR.

Critical strengths

None.

Critical weaknesses
  1. This paper is extremely hard to follow. Many things are not defined. Even the abstract is poorly written, making me unable to follow its logic.

For example, notations are not well defined, and the abbreviations are not defined.

  1. The paper's writing needs to be significantly improved. The introduction, the algorithm description need to be better written and be more friendly to readers.

It is not even clear what the problem definition is.

Questions for authors
  1. Can you explain the problem formulation in languages which are more accessible to general readers, like introducing the problem formulations and definitions more explicitly.

That is the main difficulty of reading this paper.

Since it is really hard to follow the abstract and introduction, it is hard to evaluate the technical contents of the paper.

Reviewer_VakBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors addresses the problem of the fact that a frozen representation can be enough to act in the training environment and fail after the env distribution shifts as there is no enough signal for the sensitive algorithm to act optimally. The authors propose to split that sensitivity into a part the representation cannot recover and a part it can still execute, and a selector that either buys a little evidence or freezes. In the simple quadratic cases this split shows up, and so does a reversal of which code is better once the evidence changes.

Critical strengths
  • the theory looks truthful in the exact class the authors described and applied in the work. The risk sits on the frontier that is being derived by the conservation identity with negligible numerical error. In our current state of the research community with repositories of AI maths proofs, that makes the geometry feel like a real object of the method, not only a story in the introduction
Critical weaknesses
  • The difference between the two interfaces is consistent with zero, and only a couple of systems actually cross. The crossing discussed later comes from picking the most extreme interfaces in a family whose size is never stated. Is HalfCheetah really a test of the geometry or a transfer of interfaces?
Questions for authors
  • before you pick the extreme interfaces, how big is that family, and how many of each predicted sign does each run actually see?
  • on the later systems, what in the method changes the selected program, if the head that sees the response features and the generic head pick the same count every time?
  • is the HalfCheetah protocol meant to test the conservation law, or only whether a value computed before evaluation predicts a ranking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.