acceptodds
Under review as a conference paper at ICLR 2027

What Must Be Learned to Adapt? Decision-Directed Identification Under Environment Shift

Abstract

Pretraining is useful only when the knowledge it supplies rules out alternatives that would change the downstream decision. Yet adaptation is commonly described by overall source–target distance or by how much uncertainty pretraining removes, quantities that can be dominated by variation irrelevant to acting well. We formulate post-shift adaptation as : after observing a source artifact, an agent should distinguish only target hypotheses that require incompatible decisions. This formulation yields a residual information radius and an instance-wise lower bound on target interaction. We operationalize the principle with Decision-Directed Adaptation (DDA), which probes decision-conflicting alternatives and stops once all plausible targets share an -good decision. Controlled experiments show that DDA can require substantially fewer probes than random exploration, full system identification, and decision-agnostic information gain. They also show that environmental distance can reverse the true ordering of adaptation difficulty and that retaining abundant nuisance information may provide little decision value. Robot experiments across seven Panda-to-XArm PickCube shifts and more than 200,000 held-out episodes expose the same practical structure: zero-shot transfer ranges from sufficient to ineffective, scratch learning and fine-tuning exchange advantage across shifts, and action history resolves an important latency-induced information deficiency. The resulting perspective turns transfer from a vague similarity judgment into a concrete question: which decision conflicts remain, and what evidence is needed to eliminate them?

Primary area:
reinforcement learning
Keywords:
transfer learning, learning theory, sample complexity, decision theory, active learning, reinforcement learning, system identification, robot learning, pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FrncOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Decision-Directed Identification framework, define the residual information radius, and derive a theoretical lower bound on the number of interactions required with the target environment. Building on this, the authors introduce the Decision-Directed Adaptation (DDA) algorithm, which reduces unnecessary environment identification by actively exploring environment hypotheses that present decision conflicts.

Critical strengths

This paper emphasizes that environmental differences do not necessarily imply differences in decision-making, and it characterizes the target hypotheses that truly require identification through the radius of residual information. This perspective offers a clear theoretical framework for understanding the transfer value of pre-trained knowledge.

Critical weaknesses

The interaction efficiency of DDA was validated only within the controlled model. The robotics component primarily compared zero-shot transfer, training from scratch, fine-tuning, and action history; it did not actually execute DDA, nor did it estimate the residual information radius or compare the method against other active exploration strategies.

The robotics experiments were limited to the PickCube task and involved a small number of independently trained policies, which constrains the generalizability of the conclusions.

Questions for authors

How does DDA select exploration actions when the sets of ϵ-good decisions across multiple environments intersect pairwise but share no globally common decision?

Is it possible to run DDA directly in a robotic environment and compare the number of interactions required to reach the goal against random exploration, information gain, and full system identification methods, in order to validate the practical advantages of the core approach?

Reviewer_HJ65OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Ιn this work, the authors argue that transfer difficulty should be measured by decision-relevant uncertainty and not by source-target distance or the uncertainty that pretraining removes. The authors model this adaptation as identification over a set of candidate settings (environments), where the artifact zz (e.g., pretrained weights, data, etc.) is used to restrict the set to feasible adaptations. Two candidates need to be distinguished only if there is no ϵ\epsilon-good decision that works for both; otherwise, they do not, since one decision is acceptable for both. The authors also define a residual information radius that measures how quickly conflicting targets can be separated. Hence, if all remaining candidates in the set share a good decision, then no target data is needed. In the case of conflicting candidates, the paper introduces Decision-Directed Adaptation (DDA), a likelihood-based procedure that probes only these candidates and stops once a common good decision exists. The authors evaluate the DDA in a controlled setup that consists of finite and LQR problems, where DDA needs fewer probes than random, round-robin, full system identification, and total-information baselines. They also show that in constructed counter examples that parameter distance can misorder adaptation difficulty. Lastly, a study on robotic transfer from Panda to XArm6-Robotiq robot in ManiSkill is conducted, which shows that some shifts are solved zero-shot, scratch training and fine-tuning trade places, and action history substantially helps under latency.

Critical strengths

Neat reframing of transfer difficulty. The central framing of the paper is that only uncertainty that affects the acceptable decision needs to be resolved. This perspective is simple and well-motivated. Under this perspective, no target data are needed when every target still consistent with the source shares an ϵ-good decision, while many interactions are needed when the remaining targets call for different decisions but are hard to discriminate, even if they are close to the source.

The controlled experiments isolate the mechanism properly. The authors provide evidence for the central focus of the paper that full identification wastes samples on nuisance variables, which is correctly evaluated in methods that share the same likelihood update, confidence set, and stopping rules. So the evaluation is restricted only to probe selection.

Critical weaknesses

Inadequate overall evaluation. While the controlled experiments serve as a good toy example for the proof of concept for the approach, DDA is never run in ManiSkill, the residual radius is never estimated there, and no probing baselines are compared. So, the robot section is a standard comparison between zero-shot, scratch, and fine-tuning, and the findings are not specific to the framework and are expected. The authors acknowledge most of this in the limitations. Additionally, the statistical evidence is weak with only three scratch seeds, intervals such as [0, 1] for latency-4 History-4 scratch and for combined-shift scratch mean the comparisons cannot support conclusions. Nonetheless, the abstract builds a narrative on them, while the main text abstains from this. The abstract states ‘more than 200,000 held-out episodes,’ but the actual evidence comes from only 3–6 trained policies per condition. Hence, the method’s actual evidence comes only from small synthetic problems.

Limited theoretical novelty and assumptions that conflict with the motivation. The lower bound is the standard fixed-confidence change-of-measure argument, applied to a smaller set of alternatives, and the authors themselves call the formula “classical in spirit.” Moreover, DDA is a greedy surrogate with no guarantee of getting close to this bound. Also, the bound itself can be loose, since the set of alternatives only contains targets that conflict with θ itself. So, the residual complexity can be zero even when two other plausible targets conflict with each other and still need to be distinguished. In my opinion, a bigger concern is that the framework relies on a predefined and well-specified set of candidate targets as well as the ϵ\epsilon-good decision for each one of them. Additionally, in continuous spaces like robotic settings, which motivate the approach, the modeling is probably misspecified and thus, finding good policies for each candidate target will need full RL training. There is a chance that this might be more costly than fine-tuning directly on the target. Regarding the update-0 action-scale result, it also requires some target interaction in order to verify that the policy is already sufficient, which the framework does not account for. Lastly, while the limitations section notes the finite, well-specified assumption, the authors do not present ways to deal with the approximation, misspecification, or computational cost.

Questions for authors
  1. Could the authors show what is the computational cost of finding the ε-good decision set for each candidate in a robotics setting, and provide a comparison with simple fine-tuning in the target?
  2. Could DDA be run on at least one ManiSkill shift with a small, discrete candidate set? This experiment will help show one instance where the approach works beyond the synthetic problems, and whether DDA uses fewer target interactions than fine-tuning or random probing.
Reviewer_tvmoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues for adaptation to critically factor how behavior/policy changes rather than modeling all aspects that differ between a source and target environment. Given a known decision loss function (task objective) for hypothesized domain parameters, the paper describes a greedy probing procedure based on pair-wise comparison to arrive at acceptable policy/behavior for the target domain. The paper presents results on stylized identification tasks.

Critical strengths

The problem posed is interesting. While the insight of task-aware adaptation broadly exists, the paper takes a step towards formalizing this (however see weaknesses concerning presentation and clarity).

Critical weaknesses
  • The paper suffers from substantial exposition and clarity issues. I found several sections unnecessarily abstract, with many terms introduced before their use/interpretation is established.
    • I found the phrasings of examples (Intro L043-049) very convoluted and the casual use of terms like $\epsilon$-good decision and information radius without grounding what that means. Additionally, the introduction or abstract does not provide any intuition for what the experiments do or what the results mean, making this paper unappealing to read any further.
    • The method setup does not carefully motivate or concretely instantiate the decision set $\mathcal{D}$ and the corresponding loss function $L_\theta$ – it would make sense to give a concrete example from the get go.
      • It is unclear how easily a family of loss functions for a target domain $\theta$ can be obtained and this feels like a very strong assumption – particularly for settings like robotics.
      • The setup doesn’t do a good job of carefully outlining the assumptions here. The method section reads like a dump of mathematical notation that makes sense locally but does not provide any intuition to a reader.
  • While the paper presents an added perspective that favors task-aware probing for adaptation – the proposed abstraction and algorithm DDA do not demonstrate scalability beyond the stylized identification experiments. The robot experiments in the paper admittedly don't attempt to test the proposed formulation.
    • Additionally, the paper doesn’t do a good job about carefully outlining the decision sets, the task objective etc., for these experiments.
    • The paper also makes some explicit assumptions by considering only pair-wise decision conflicts to construct the basis of the proposed DDA probing algorithm. While they make no claims of global optimality, this construction can fail in scenarios where there are no globally compatible decisions (e.g. pairwise intersections non-empty, but intersection of all domains being empty). This appears to be a limitation in the formulation and it would be good to highlight this explicitly even if they are pathological.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.