acceptodds
Under review as a conference paper at ICLR 2027

DETECTOR-COUPLED SURROGATE REGRET FOR TEST-TIME ADAPTATION IN PIECEWISE-STATIONARY CMDPS

Abstract

Constrained reinforcement-learning policies deployed in provisioning and finance must determine not only how to adapt under environmental shifts, but also when adaptation is warranted. Keeping a policy frozen can reduce reward and move its cost behavior away from the levels it was trained to satisfy, whereas indiscriminate test-time adaptation can overfit transient noise. We study this timing problem in piecewise-stationary constrained Markov decision processes (CMDPs) with imperfect change-point scores. Existing detection-and-restart analyses price detection delay and false alarms but omit constraints, while online-CMDP analyses price constraint costs but assume no external detector. Neither targets the timing decision. We propose AdaptiveCMDP, which closes both gaps: constraint exposure is fixed before deployment, and detector error is priced inside the regret bound. Specifically, it leaves the trained policy frozen and adapts only a bounded adapter whose radius, set from the Slater margin, fixes that exposure. Updates are reward-free, so adaptation can start before returns are informative; each detector firing restarts the step-size clock. The surrogate dynamic-regret bound then prices the detector directly—a detector tax that is square-root in horizon, regime count and expected false alarms, plus linear total detection delay. Under explicit value sufficiency and realizability, this transfers to policy value on an adapter-independent target stream. A frozen exploitability–delay rule scores every registered window: it correctly keeps the policy frozen in 15 of 16 financial panels where adaptation does not pay, and fires on all three provisioning domains—public traffic, bike-sharing and electricity-load data. There a capacity budget replaces the financial drawdown limit; strict regret wins exclude zero, at the cost of more stale-capacity breaches in two. A matching lower bound shows these terms are unavoidable: detector quality, not adapter capacity alone, decides when adaptation pays.

Primary area:
reinforcement learning
Keywords:
reinforcement learning theory, test-time adaptation, constrained MDPs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HpSyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when test-time adaptation is beneficial in piecewise-stationary constrained Markov decision processes. It analyzes how detection delay and false alarms affect surrogate dynamic regret, and evaluates detector-coupled adaptation on synthetic, financial, and provisioning tasks.

Critical strengths

The paper separates adaptation timing from update frequency through matched-budget controls. It also reports negative financial results, constraint-violation trade-offs, and the limits of its theoretical guarantees.

Critical weaknesses
  1. The main theorem applies to a convex last-layer predictor updated by projected mirror descent, whereas the implemented method uses LoRA and AdamW. The policy-value result requires additional sufficiency assumptions, and the cost bound concerns expected per-step cost rather than realized drawdown or cumulative violations.
  2. The principal positive real-data result in Table 3 compares an exogenous calendar schedule with CUSUM; it does not establish broad superiority of a learned detector. Violations also increase on the traffic and electricity tasks. Although the appendix reports an approximately 1.1% relative regret improvement for the trained policy on concentrated traffic, the practical evidence remains limited.
  3. The abstract describes the lower bound as “matching” and suggests that the detector-related terms are unavoidable. Theorem 5 establishes a delay lower bound and a generic surrogate-regret lower bound, but not a lower bound matching the false-alarm term. This claim should be narrowed.
Questions for authors
  1. Can the authors validate the theoretical predictions on a practical task satisfying the theorem’s assumptions, or further characterize the gap between the theorem and the LoRA/AdamW implementation?
  2. Can an online learned detector be compared with the calendar schedule and CUSUM under matched update budgets, reporting both regret and constraint violations?
Reviewer_UggVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a mechanism to adapt constrained RL policies to new environments. The main contribution is theoretical and involves a regret bound on the proposed policy. I cannot exactly point out the full contributions, and they are not even mentioned explicitly in the introduction, where one would expect them to be mentioned. The paper's main results are theoretical, although some experiments have been done.

Critical strengths
  • The paper is driven by theoretical results on the regret. The paper provides lower and upper bounds on the regret for the method they discuss. Assumptions are also outlined.
Critical weaknesses
  • The paper is very difficult to read. The terms are not clear, and I don't think it is because the paper is directed at a certain domain. I believe the terms used are not very common.
  • It is not clear what the main contributions of the paper are. Usually, papers have a demonstrative figure for the method. This is also absent. There is not even a place that states that this is the main method that is being developed and analyzed.
  • Despite having a lot of theoretical results, it is not clear what insights can be drawn from them.
  • I cannot understand the motivation of the paper well. I do understand the issue of adapting the RL policy, but beyond that, the flow becomes very difficult to follow.
  • The experiments are not clear. I do understand that theory took a large part of the paper, but for example, the proof sketches could have been substituted with more experiments.
  • I am not sure why there is a specific link to LoRA adapters, and whether this means that the paper does not address any constrained MDP.
Questions for authors
  • How does this paper relate to [1]? [1] is a paper tackling a similar test-time problem where a policy must adapt to risks it has not seen beforehand?

  • Does the use of LoRA in the paper imply that we are dealing with a foundation model framework? That is, not a general RL policy, which might be DQN or a simple feedforward policy network trained with PPO?

  • I do believe key insights from the theoretical results must be added, and that better motivations, figures and common terms must be added/substituted into the paper.

[1] https://arxiv.org/abs/2408.08812

Reviewer_g4i8OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper studies detector-triggered adaptation in piecewise-stationary CMDPs, deriving a surrogate-regret bound with detection-delay and false-alarm terms and evaluating an empirical timing rule on financial and provisioning tasks.
Critical strengths
  • Deciding when adaptation is worthwhile is an important deployment problem. Explicitly accounting for detector quality provides a useful perspective beyond adapter design alone.
  • Equal-update-budget controls help isolate timing effects. The paper also transparently reports negative results, retrospective analyses, and the limits of its theoretical guarantees.
Critical weaknesses
  • Practical applicability remains unclear. The framework assumes adapter-independent targets and a base policy that remains feasible across all regimes, limiting its relevance to closed-loop systems where actions affect future observations and shifts can invalidate safety. A prospective evaluation with causally estimated exploitability and matched constraint levels is needed to establish deployment value.
Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.