acceptodds
Under review as a conference paper at ICLR 2027

Tasks Do Not Tensorize: Learning When to Reset or Continue

Abstract

Meta-reinforcement learning usually treats the grouping of source trajectories into tasks as fixed. We study an acquisition model in which the learner decides whether the next controlled trajectory should retain the current hidden task or begin an independent one. This changes the statistical experiment. For every , we construct finite meta-MDPs in which every episodewise experiment has zero information about two target-incompatible models while one bounded persistent experiment is informative; under perturbation the persistent-to-episodewise rate ratio is arbitrarily large. For every fixed , the zero-terminal-value -step receding-horizon information optimizer can likewise be arbitrarily inefficient. For finite persistent meta-MDPs with bounded task depth, a finite observational quotient, and disjoint single-answer cells, we characterize the exact fixed-confidence source cost over all randomized history-dependent experiment trees. The characteristic information is a Bellman–Chernoff game: a dynamic program chooses the next episodic policy and whether to stop, while a least-favorable target alternative prices information. Certified column generation computes a sparse design, and vanishing certified optimization error preserves the characteristic constant. A cross-modal calibration model illustrates how continuation reveals correspondence while resetting supplies independent breadth. Thus task grouping is not bookkeeping: it is an information-control action.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, meta-reinforcement learning, generalization in reinforcement learning, task persistence, reset-or-continue decisions, sample complexity, fixed-confidence identification, sequential experimental design, pure exploration, optimal stopping, Bellman–Chernoff theory, active data acquisition, fresh-target transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Z8WHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats the grouping of source trajectories into latent tasks as a decision: before each episode the learner either keeps the current hidden task or draws a new one. It argues that this choice changes the experiment. A parity construction seems to show that short same-task views can be uninformative while a long enough continuation is informative, and that a short-horizon information planner can miss that. For finite models with a depth limit and disjoint answers, the fixed-confidence cost looks like a Bellman–Chernoff game, computed by a Track-and-Stop rule with column generation. A small cross-modal calibration model is solved in closed form.

Critical strengths

The parity task seems to separate another fresh episode from one more view of the same latent, which ordinary one-episode access cannot see. Giving that choice an exact fixed-confidence price, rather than leaving it as part of the access model, looks like the part that would justify acceptance.

Critical weaknesses

The exact result seems to assume a known finite tree class and known likelihoods, so it does not yet say what a meta-learner without that model should do. The calibration section looks like a closed-form special case of the same game, rather than evidence about the meta-RL settings named in the introduction.

Questions for authors
  1. Why characterize a learner that already knows the quotient and the tree likelihoods, instead of the usual meta-RL interface, where the trial boundary is given and the agent does not get that model?
  2. Why leave task-boundary detection and latent-context change-point methods out of the comparison, since those already decide whether the current hidden task is still the one they are in?
Reviewer_B84JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a learner should continue collecting episodes from the same hidden task or reset to a newly sampled task. It evaluates these choices through the information they provide relative to their costs, with the objective of learning a strategy that performs well on a fresh target task. The paper constructs examples in which collecting multiple observations from the same task is necessary for identifiability. Under finite-model and bounded-depth assumptions, it also establishes information-theoretic cost bounds and develops an asymptotically optimal experimental design and stopping procedure.

Critical strengths

The paper has an interesting motivation and develops a coherent information-theoretic approach to deciding when to reset or continue. The examples showing that premature resetting can prevent identification are particularly useful. The main exact-design bounds appear sound based on the AI-assisted review, although this should not be interpreted as a complete verification of every proof in the 132-page manuscript.

Critical weaknesses

1.The paper should clarify its contribution relative to treating reset as an additional action in a suitably augmented POMDP. This appears to be a natural formulation of the decision problem, but the manuscript does not sufficiently explain which technical difficulties remain under that formulation and how the proposed framework resolves them.

2.The approximate-design guarantee in Theorem 4.2 should explicitly require the optimization errors to be bounded by a deterministic, confidence-independent envelope whose Cesàro mean vanishes, as in Theorem E.17, or provide an alternative quantitative tail condition and proof. If the errors are random, almost-sure convergence of their average to zero alone need not imply the claimed expected-cost asymptotics. This concern, identified by GPT Pro, appears fixable by clarifying the assumptions and aligning the main statement with the appendix.

Questions for authors

as above

Reviewer_CN11OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies meta-RL data acquisition when the learner controls whether successive source episodes share a hidden task or begin independent tasks. It constructs target-relevant persistence hierarchies in which fewer than rr same-task observations provide no information, while an rr-episode experiment is informative. For finite models with bounded task depth and disjoint answer cells, it characterizes the asymptotically optimal fixed-confidence source cost through a Bellman–Chernoff game over stopped experiment trees. Group-action examples and a calibration environment illustrate the complementary roles of continuation and resetting.

Critical strengths
  1. The paper identifies a meaningful acquisition decision that changes the statistical experiment itself. The arbitrary-order persistence hierarchy establishes that controlling which observations share a latent task can determine target-relevant identifiability, even under adaptive acquisition. This gives a precise and consequential explanation of why episodewise acquisition can fail.

  2. The bounded-model characterization is comprehensive within its stated domain. The reduction of randomized history-dependent experiments to mixtures over deterministic stopped trees supports matching lower and upper bounds over a clearly specified learner class. The Bellman–Chernoff representation also yields an interpretable continuation rule that accounts for future information rather than only immediate information gain.

Critical weaknesses
  1. The approximate-design theorem requires clarification. Main-text Theorem 4.2 states a Cesàro-vanishing optimization-error condition, while Appendix Theorem E.17 requires a deterministic Cesàro-vanishing envelope, 0≤ζk≤ζˉk0\le\zeta_k\le\bar\zeta_k with n−1∑k=1nζˉk→0n^{-1}\sum_{k=1}^n\bar\zeta_k\to0. This distinction matters because pathwise convergence alone does not exclude rare, long periods of poor experimental design that can inflate expected stopping cost. The main statement should explicitly include the stronger condition or provide an alternative argument controlling these tails. This concern affects the approximate-design extension; it does not invalidate the exact-design characterization.

  2. The statistical and computational scope is restrictive relative to the broader meta-RL motivation. The flagship theorem requires finite alphabets, bounded same-task depth, a finite observational quotient, disjoint answer cells, and exact likelihood and optimization access. The experiment-tree space can be enormous, and sparse optimal support does not guarantee efficient discovery of that support. Column generation provides certification and finite termination, but the manuscript does not establish scalable computation in general. The calibration examples illustrate the acquisition principle without demonstrating that these requirements can be handled in more realistic meta-RL problems.

Questions for authors
  1. In Theorem 4.2, are the errors ζk\zeta_k intended to be deterministic or bounded by a deterministic sequence? Please reconcile its statement with Theorem E.17 and clarify the condition needed for asymptotic optimality in expected source cost.

  2. Which parts of the observational quotient, answer partition, and canonical target strategies are assumed available to the learner? Does the computational characterization include constructing these objects, or does it assume they have already been supplied?

  3. What computational guarantee does the existing column-generation analysis provide beyond finite termination and a certified optimization gap? Please clarify the worst-case cost of Bellman pricing and whether any analyzed subclass admits polynomial computation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.