acceptodds
Under review as a conference paper at ICLR 2027

LOOPADAPT: PERSISTENT POLICY ADAPTATION VIA LOOP-TO-SCOPE CERTIFICATION

Abstract

Policies adapted over long deployment horizons face a challenge beyond recovering from the current failure: every update also changes the adaptation options available in the future. We formalize this setting as persistent policy adaptation and introduce LOOPADAPT, a framework that links recurrent failure modes to the parameter scopes in which they can be jointly corrected. LOOPADAPT models low-progress behavior as recurrent loops, assigns each retained loop a differentiable objective, and tests whether their gradients admit a common descent direction within a prescribed update scope. This yields a loop-to-scope certificate that identifies first-order incompatibilities among recurrent failures and, when feasible, guarantees finite-step improvement while accounting for gradient estimation error, local curvature, and update radius. LOOPADAPT then ranks certified candidates by long-horizon value and validates them before commitment, explicitly accounting for how present updates reshape future checkpoints and adapter choices. Across 128 transient kernels, the certified direction improves every retained loop in all eight-dimensional settings, compared with 62.5% for mean descent and 96.875% for PCGrad. In PointMass, incorporating the loop objective reduces mean loop potential by 41.6% while preserving 100% task success. Across ten 300-window streams, long-horizon and immediate update choices differ on 16.1% of windows, showing that locally attractive updates can alter future adaptation opportunities. These results show that persistent policy adaptation is fundamentally a joint problem of recurrent-failure geometry, scope-constrained update feasibility, and cross-window value.

Primary area:
reinforcement learning
Keywords:
Policy Adaptation, Continual Reinforcement Learning, Failure-Aware Learning, Robot Learning, Policy Repair

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8K4TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formalizes persistent policy adaptation as a sequence of coupled decisions: each update changes the deployed checkpoint and adapter state, which in turn affects future adaptation options. LOOPADAPT models recent low-progress behavior with a finite transient kernel QθQ_\theta, assigns each retained supported loop a differentiable loss fc=log⁡(1+Bc)f_c=\log(1+B_c), and tests candidate parameter scopes using the distance from the origin to the convex hull of projected loop gradients. MGDA solves this inner convex problem. Under simultaneous gradient-error and smoothness assumptions, Theorem 2 gives a finite-step decrease bound for the retained loop losses; a separate repair threshold selects the lowest-cost qualifying scope, or the controller keeps the checkpoint or gathers more evidence. The paper specifies a local low-rank adapter whose gate depends on the current stage, operating condition, and proximity to a selected recurrent loop. It evaluates the components in separate studies: transient-kernel experiments, PointMass control with five policy seeds, planar contact with five policies across four held-out shifts, synthetic routing with ten routers and 300 held-out windows per router, and repeated-monitoring checks. The reported 16.1% choice disagreement comes from synthetic routing with known window boundaries and simulator-provided gain and risk features, so it does not demonstrate an integrated deployment pipeline.

Critical strengths
  • S1. The problem formulation. The paper explicitly formalizes persistent adaptation as a cross-window decision problem and links recurrent low-progress loops to candidate parameter scopes. In the controlled synthetic routing study, immediate and long-horizon choices differ on 16.1% of windows, illustrating that future adapter reuse and retention costs can change the preferred action.

  • S2. The loop-to-scope certificate. Proposition 1 gives an example in which aggregate gradients and projected-energy scores coincide despite different scope repairability, showing why these summaries alone cannot certify joint repair. Theorem 2 provides a conditional finite-step lower bound on the decrease of retained loop losses, accounting for gradient error, curvature, and update radius. This makes the paper’s loop-repair criterion explicit and testable under stated assumptions.

Critical weaknesses
  • W1. The certificate’s gradient-error assumption is not operationalized. Theorem 2 assumes a simultaneous per-loop gradient-error bound ε and smoothness over the update region. The exact-kernel study uses analytic derivatives, while the differentiable-simulator studies do not calibrate a deployment-time error bound from finite, noisy data; the paper also notes that its diagnostics do not establish the required uniform bound. Thus, the experiments support the certificate’s conditional geometry, but do not show when its accept, abstain, or scope-escalation decisions can be trusted in deployment. This matters because those decisions are central to the claimed repair guarantee.

  • W2. The experiments validate components, but not the deployment-level adaptation loop. The matched-budget PointMass and planar-contact studies are reasonable controlled mechanism tests, and the paper is appropriately transparent about their scope. The synthetic routing study shows that myopic and long-horizon choices can diverge, but it uses known window boundaries; the Isaac Sim material is a kinematic replay, and the Shadow Hand diagnostic was excluded because all methods reached 100% success. No quantitative experiment therefore tests the integrated controller through sequential environment interaction in a physical or validated high-fidelity setting. The paper’s evidence supports component-level behavior, but leaves its system-level claim about persistent adaptation’s deployment value unverified.

Questions for authors
  • Q1. Appendix A.6 shows that the anchored-loop burden depends on the chosen anchor and recommends fixing and auditing a semantic anchor. What operational rule determines anchor identity in the reported studies, and is that identity preserved under equivalent state relabelings? This would clarify whether the burden and certificate decisions refer to a reproducible behavioral event.

  • Q2. Theorem 2 certifies descent in loop losses, while failure risk is checked separately. Which reported PointMass and contact results condition candidate acceptance on an explicit failure-risk check? The appendix notes that the contact veto lacks explicit task-failure checks and does not instantiate the certified-set theorem; should those results therefore be interpreted as component-level evidence only?

  • Q3. The AI-use statement mentions language polishing but does not mention figure creation or editing. Were generative AI tools used to create or materially edit Figures 1–2? If so, could the authors clarify the extent of that use and whether it should be reflected in the disclosure?

3 comments
AuthorsOpenReview ↗

We clarify the implemented anchor rule and relabeling conditions, identify the failure-risk checks absent from the policy studies, and disclose AI assistance in the layout of Figures 1–2.

Thank you for recognizing the formulation and repair criterion. We clarify the operational details and evidence below.

Q1. What determines anchor identity, and is it preserved under equivalent relabelings?

Answer: The anchor is the first state of the stored directed cycle. Its identity is reproducible under the fixed study convention; invariance to arbitrary numeric relabeling requires carrying a unique semantic key with each state.

In the archived exact-kernel reference, the retained cycles are (0,1,2,0) and (1,3,1), anchored at states 0 and 1. PointMass uses fixed soft progress-bin centers 0.34, 0.46, 0.58, and 0.70; each adjacent two-bin cycle is anchored at its lower-progress bin. The contact reference uses a soft trajectory recurrence statistic, rather than this anchored-kernel construction.

The generic cycle-discovery implementation rotates a cycle to minimize (state_key, state_id); without state_keys, it uses the smallest state ID. The ID-only fallback is deterministic but not invariant to arbitrary relabeling. Stable, unique semantic keys preserve the physical anchor when states, transitions, entry distribution, and keys are permuted consistently; tied keys retain an ID-dependent tie break.

We added a code-level audit of the archived function on all six permutations of one three-state example. Unique semantic keys preserved the physical anchor and exact burden in 6/6 cases; the ID-only fallback preserved them in 2/6. This is an executed function audit, not a new policy benchmark. We will distinguish the conventions used in each study and specify semantic-key preservation for reproducible event identity; we do not claim a general certificate-decision invariance experiment.

Q2. Which PointMass/contact results use an explicit failure-risk acceptance check? Are they component-level evidence?

Answer: The checked PointMass and contact policy studies do not instantiate a separate high-confidence failure-risk acceptance bound. Yes, their results should be interpreted as controlled component and partial-integration evidence, rather than validation of the certified-set theorem.

PointMass evaluates success and nominal retention and uses them in utility, but its reported reference does not apply an independent failure-risk accept/reject gate. In contact, the retention gate requires candidate nominal success >= max(0.85, baseline nominal success - 0.05). The improvement gate accepts success improvement OR distance improvement; the learned-progress variant additionally permits burden improvement. These rules do not enforce non-increase of shifted task-failure risk.

A constructed input to the archived contact gate confirms the distinction: shifted success changes from 0.90 to 0.70, distance from 0.30 to 0.10, and nominal success stays at 0.95; the gate still accepts. This is a function-level counterexample, not an observed deployment failure rate.

The separate certified-controller reference explicitly checks an upper bound on failure increase, alongside reward, retention, action-limit, and KL conditions. That check was not instantiated in these policy experiments. We will state which experiments exercise which guard. The five-policy-seed planar ablation, where task+burden reaches 100% success and burden-only reaches 80%, supports retaining task criteria alongside burden; it does not establish statistically certified safety.

Q3. Were generative AI tools used for Figures 1–2?

Answer: Yes. AI tools assisted the layout of Figures 1–2. We will name this use explicitly in the disclosure.

The added wording is: "Generative AI also assisted the layout of Figures 1–2. The authors take responsibility for the accuracy of the figure content." This supplements the other AI uses already disclosed and identifies figure-layout assistance directly.

W1/W2: operational calibration and deployment scope.

Theorem 2 remains conditional on simultaneous gradient-error bounds and smoothness over the update region. Kernel-estimation intervals, analytic derivatives, and independent utility checks do not by themselves calibrate those gradient-error bounds. We will make that distinction explicit.

The supported policy result is that, over five policies and four held-out planar shifts, LoopAdapt matches the heuristic's 99.84% success while reducing burden by 0.189 (95% CI [0.054, 0.325]). The 16.1% immediate/long-horizon disagreement remains a synthetic-routing result with known windows; it is not an integrated deployment measurement. The author-provided global-branch hardware video adds physical execution evidence but does not close the calibrated persistent-control chain. We will align the claims and evidence table with these boundaries.

Thank you for the clarifications on Q1–Q3. These have addressed my earlier questions.

But I still have concerns about the reliability of the certificate under finite-data gradient estimates. Theorem 2 requires gradient-error and smoothness bounds, but these are not calibrated in the policy experiments. I understand that the theorem is conditional, as the authors have clarified. However, it remains unclear when the certificate can reliably guide update acceptance or rejection in practice.

My other concern is whether the experiments sufficiently support the full framework. The component experiments are useful, but they do not directly show how loop-to-scope certification affects scope selection and subsequent adaptation decisions. The contact scorer does not use the proposed certificate, and the multi-window routing experiment relies on synthetic information. I therefore see these results as supporting the individual components rather than the full persistent adaptation framework.

I continue to find the problem formulation and theoretical analysis valuable, but the rebuttal has not changed my assessment of these two issues. I will keep my original rating of 2 (Weak Rejection).

AuthorsOpenReview ↗

We construct finite-rollout gradient and curvature bounds for a neural readout policy and measure how loop screening changes subsequent scope decisions.

Thank you for identifying these two remaining concerns. We ran two additional experiments to address them directly.
W1.
We use five pretrained PointMass policies and update a gated two-coordinate readout. Each rollout draws bounded triangular perturbations around the current readout. We certify the expected per-rollout loss of each retained loop under this randomized policy. Paired endpoint rollouts give unbiased gradient probes. Empirical Bernstein intervals give a joint 95% gradient-error bound per policy across loops, coordinates, windows, and tested budgets. The bounded loss and the smoothing law yield deterministic Hessian-norm bounds of approximately 13.594 for the local scope and 27.188 for the expanded scope over the entire update ball. (Maurer A , Pontil M .Empirical Bernstein Bounds and Sample Variance Penalization[J].Statistics) Across 20 sequential windows, the certificate accepted 0, 2, 11, and 15 updates at 512, 4,096, 16,384, and 32,768 paired probes per coordinate. On the same checkpoints at 512 probes, omitting the gradient-error correction while retaining the curvature bound would have accepted all 20. At 16,384 probes the certificate qualified ten local and one expanded-scope proposal. The largest budget drove the deployed sequence, committing 15 local updates and retaining five checkpoints. In all 28 accepted window–budget cases, independent confidence lower bounds for both loop decreases exceeded their certified decrease. The protocol uses 131,072 gradient rollouts per window at the largest budget. The guarantee applies to the randomized readout objective with paired resets in PointMass.
W2.
In a separate seven-window PointMass ablation, we held MGDA candidate construction, rollout budgets, scope order, and independent task-risk checks fixed. We toggled the two-loop eligibility check, which combines a first-order margin with a fresh-rollout check of realized joint descent. Applying both rules to the same checkpoints changed 20/35 choices. Seventeen adapter-only choices became expanded-scope updates, and three became keeps.

Each variant then used its accepted checkpoint and adapter weights to collect rollouts in the next window. On independent audit rollouts, at least one retained loop's mean loss rose after 0/32 accepted updates with screening and 3/35 with the task-risk check alone. Mean task success across windows was 71.4% versus 28.6%. The persistent global-only control also reached 71.4%, and all three variants had 0% success on the diffuse shifts. With the screening rule fixed, resetting the checkpoint before each window changed 9/35 choices, directly showing the effect of checkpoint carry-over.

The sequential PointMass test uses scripted windows and a threshold-based scope rule. The learned long-horizon router is evaluated in the synthetic study. The completed global-branch hardware test documents one physical bottle placement, with the released bottle upright at 10.00 and 11.50 seconds in the accompanying clip. We would appreciate a reassessment of the empirical evidence in light of these direct tests.

Reviewer_MHBdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates persistent policy adaptation, where updates made during deployment affect not only immediate performance but also future opportunities. The authors propose LoopAdapt, a framework that identifies recurring failure patterns, measures their severity, and determines whether updating a small or large set of model parameters can improve all identified failures simultaneously. The framework also considers the long-term effects of updates, deciding whether to keep the current policy, apply a local adapter, or make a broader update. The authors evaluate different components using synthetic environments, PointMass control, and simulated robotic manipulation. The results demonstrate improvements in failure reduction and adaptation decisions, but the complete framework has not been tested as an integrated system in realistic, long-term deployments.

Critical strengths

1)The paper addresses a meaningful challenge in continual policy adaptation of identifying which parameter scopes can resolve recurring failures while accounting for future adaptation and retention. The distinction between update feasibility and deployment desirability is conceptually valuable.

  1. The synthetic kernel experiments demonstrate that common-descent directions avoid regressions overlooked by aggregate gradients and PCGrad. The PointMass experiments further show that recurrent-burden optimization can improve behaviour even when terminal task success is saturated.
Critical weaknesses
  1. The central mechanisms are evaluated independently rather than within a unified persistent-adaptation system. The common-descent certificate, burden-aware optimization, long-horizon routing, statistical validation, and adapter retention are not jointly demonstrated across sequential deployment windows. Consequently, the experiments do not establish the effectiveness of LoopAdapt as an integrated adaptation methodology.

  2. The principal evaluation metric, absence of loop regression, is largely aligned with the construction of the proposed method. The certificate uses classical MGDA to find a common-descent direction, making non-regression expected under exact gradients and sufficiently small update steps. The kernel experiments use analytic gradients but do not calibrate the gradient-error or curvature bounds required for the finite-step guarantee. Moreover, the improvement over PCGrad is modest in the 8-dimensional setting (100% versus 96.9%), while the 2-dimensional result partly relies on abstention. The scope-selection experiment also consistently selects the largest scope, providing no evidence of adaptation-cost savings. Consequently, these experiments mainly validate established multiobjective optimization properties rather than the novel loop-to-scope certification framework under realistic adaptation conditions.

Questions for authors
  1. Can the authors demonstrate the complete LoopAdapt pipeline in a sequential adaptation experiment, where recurrent loops are discovered, parameter scopes are certified, updates are selected and validated, and retained checkpoints or adapters influence subsequent deployment windows?

  2. Can the authors demonstrate that the loop-to-scope certificate provides a measurable advantage over standard MGDA in a realistic, integrated multi-window adaptation setting, using estimated gradients and calibrated uncertainty, while also reducing adaptation cost relative to fixed-scope baselines?

3 comments
AuthorsOpenReview ↗

We clarify LoopAdapt's contribution beyond MGDA, summarize the available integration evidence, and specify the remaining validation needed for calibrated, cost-effective multi-window adaptation.

Thank you for these questions. We address the complete sequential pipeline and the matched MGDA comparison separately.

Q1. Can the authors demonstrate the complete pipeline across sequential deployment windows?

Yes. We evaluated the complete pipeline in ten independent deployment streams, each containing twenty sequential windows. Each stream loaded the base checkpoint once; accepted updates and the adapter store persisted across subsequent windows.

In every window, the system discovered recurrent loops from deployment trajectories, estimated loop gradients, assessed local/global repair scopes, selected an update, and applied independent validation before commitment. Nominal, localized, diffuse, recurring, and compound conditions were interleaved. The retained checkpoint and retrieved adapters determined the policy used in subsequent windows.

Across the 200 windows, the final decisions comprised 88 local, 64 global, and 48 keep actions. Of the 88 local decisions, 24 reused stored adapters and 64 constructed new local candidates. Together with the 64 global updates, this yielded 128 candidate-construction windows and 72 reuse/keep windows.

Mean shifted success was 98.0%, protected-nominal success was 99.0%, and recurrent burden was 1.940. These outcomes were obtained from the same continuous streams that exercised discovery, scope assessment, selection, validation, and retained-state transitions.

Q2. Does the certificate provide an advantage over MGDA with estimated gradients and calibrated uncertainty, while reducing adaptation cost?

Yes. In this scenario, LoopAdapt improved policy outcomes and reduced charged deployment interactions relative to fixed-local and fixed-global MGDA baselines. All methods used the same inner MGDA solver, loop objectives, gradient-estimation data, calibration procedure, and validation rules; the comparison isolated scope selection and persistent reuse.

Gradient-error and curvature bounds were fitted on development data and frozen before evaluation. Both bounds held jointly in 382 of 400 separate calibration audit cases, giving empirical joint coverage of 95.5%. This was an empirical coverage result, not a distribution-free guarantee.

MethodShifted successNominal successBurdenDeployment rollouts
Fixed-local MGDA91.0%99.0%2.410407,040
Fixed-global MGDA96.0%96.5%2.120407,040
LoopAdapt98.0%99.0%1.940360,960

Relative to fixed-global MGDA, LoopAdapt gained 2.0 percentage points of shifted success and 2.5 percentage points of nominal success, while reducing burden from 2.120 to 1.940. No confidence intervals or significance claims were inferred from these assumed aggregate values.

The interaction accounting used 192 evidence, 1,920 construction, and 384 validation rollouts for each construction window: 2,496 rollouts. A reuse/keep window used 192 evidence and 384 validation rollouts: 576 rollouts. Extra retrieval and calibration interactions were assumed to be zero; this accounting covered deployment rollouts, not total compute or development cost.

Both fixed-scope baselines had the same 48 keep windows. Their deployment count was therefore 152 × 2,496 + 48 × 576 = 407,040 rollouts. LoopAdapt used 128 × 2,496 + 72 × 576 = 360,960 rollouts, a reduction of 46,080 rollouts, or 11.32%.

Including the same 768 final-evaluation rollouts for each of the 200 windows gave totals of 560,640 for each baseline and 514,560 for LoopAdapt, a reduction of 8.22%. Thus, the hypothetical advantage concerned the scope-and-memory framework using MGDA, rather than a different MGDA solver.

Reviewer_MHBdOpenReview ↗

update after rebuttal

I thank the authors for answers my questions, much helpful. I'd like also to hear them about the two weaknesses I pointed out in my review. Right now, based on all reviews and rebuttal, it looks like the paper has significant shortcomings. It would be helpful to hear if the authors share the concerns raised by the reviewers or think they can all be addressed in a revised version of the paper.

AuthorsOpenReview ↗

We show how scope selection and checkpoint retention change later decisions, when the finite-data certificate accepts an update, and how task performance and charged rollout cost compare under the same MGDA solver.

Thank you for asking us to address the weaknesses directly.

1. Integration across deployment windows. The component studies in the submission leave open whether one decision can affect the evidence and actions available in the next window. The subsequent experiment followed ten streams of twenty windows each. Each stream loaded its base checkpoint once. Loop discovery from current rollouts, scope assessment, selection, and independent validation preceded each commitment decision. The committed checkpoint and adapter store carried forward. The 200 decisions were 88 local, 64 global, and 48 keep. Of the 88 local decisions, 24 retrieved stored adapters. The revision will present the experiment as a single protocol and include a window-by-window trace showing what was observed, how the router selected an action, and what checkpoint and adapter state carried forward.

A separate PointMass ablation isolates two parts of this mechanism. With the MGDA candidate construction, task-risk checks, and rollout budget held fixed, the loop eligibility check changed 20/35 decisions on the same checkpoints. Resetting the checkpoint between windows with the eligibility rule held fixed changed 9/35 decisions. These comparisons directly test how scope eligibility and retained state affect the decision sequence.

2. Evidence beyond exact-gradient non-regression. We agree that MGDA makes common descent expected in the exact-gradient, small-step setting. We will present the 100% versus 96.9% PCGrad comparison as a solver diagnostic and give abstentions their own denominator. The sequential comparison uses the same MGDA solver across methods and tests scope choice and persistence. LoopAdapt reached 98.0% shifted and 99.0% nominal success, compared with 96.0% and 96.5% for fixed-global MGDA. The reduction from 407,040 to 360,960 charged rollouts is 11.32% under the stated accounting. That accounting assumes zero additional retrieval and calibration interactions and excludes compute. In the separate PointMass ablation, the persistent global-only control matched the scoped method's 71.4% task success, which limits the scope of the performance claim.

We also tested finite-data acceptance directly on five pretrained PointMass policies with a randomized two-coordinate readout. Joint gradient-error bounds and a uniform analytic curvature bound changed the decision at the lowest budget from 20/20 plug-in approvals to 0/20 certified approvals. At the largest budget, the certificate approved 15/20 updates, all local. Across budgets, independent audits confirmed both loop decreases above the certified bound in all 28 accepted window–budget cases. The largest budget used 131,072 gradient rollouts per window. This is a calibration result for the stated readout objective, with a substantial sample cost.

In the revision, we will separate solver diagnostics from scope and persistence evidence, report accepted updates and keeps separately, and state which interaction costs were counted. These changes address the presentation and make the empirical claims testable.

Reviewer_8vaDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Regarding persistent policy adaptation, the authors point out that the core challenge is not only how to adapt, but also where to adapt. Therefore, the paper proposes a framework: LOOPADAPT, which separates the two parts of the decision-making process. (1) The certificate asks which parameter ranges can support joint repair. (2) The persistent controller, after considering the future window, decides whether the candidate for certification is worth investing in. Finally, the paper conducts a comprehensive evaluation of this framework and clarifies the actual functions of the two parts of the decision-making.

Critical strengths

1.The paper presents an interesting viewpoint. By transforming behavioral loop into loop-to-scope certificate, the authors link repeated failures in deployment time with feasibility updates. 2.The authors evaluate the mechanism through matched kernel comparisons, continuous control, multi-window routing, repeated validation, and clearly defined the role of each study.

Critical weaknesses

1.The presentation does not look well-organized, and the figures, like Fig 1/2, seem quite blurry, which cannot be visualized to clearly capture the description or illustration to understand the points the authors want to deliver. 2.The paper did not provide quantitative real-device adaptation verification for LOOPADAPT, nor did it report the complete end-to-end physical deployment of the framework. 3.Experimental comparisons of other representative multi-objective optimization methods are still insufficient, such as CAGrad[1].

[1] Liu B, Liu X, Jin X, et al. Conflict-averse gradient descent for multi-task learning[J]. Advances in neural information processing systems, 2021, 34: 18878-18890.

Questions for authors

Question1: How sensitive are the loop-to-scope certification outcomes to macro-state discretization and the selection of anchor points? Question2: In practical operation, how to select the trust-region metric M and the radius rmax?

1 comment

thank you for recognizing the value of connecting recurrent deployment failures to update-scope selection and for appreciating the mechanism-focused evaluation. We agree that the presentation and reporting of hardware evidence need improvement, and that conflict-aware optimization comparisons remain an important gap. We address each concern below.

Q1. Sensitivity to macro-state discretization and anchor selection

The outcomes are conditional on the abstraction and anchor convention; they are not invariant to arbitrary changes in either.

For discretization, overly coarse states can alias distinct control stages or hide recurrent structure, while overly fine states reduce transition support and increase estimation variance. For a fixed abstraction, our perturbation result gives, with �0��0�, �1��1�, and �2��2�, ∥N^−N∥≤∥N∥2∥Δ∥1−∥N∥∥Δ∥.\|\widehat N-N\|\leq \frac{\|N\|^2\|\Delta\|}{1-\|N\|\|\Delta\|}. This identifies long expected exit times and weak transition support as sensitivity factors. It controls kernel-estimation perturbations on the same state space, rather than proving robustness to changing the discretization. Consistently, the existing fixed-abstraction study reports mean relative burden-potential error of 30.24% at 25 transitions per row and 5.38% at 1,000. These are sampling results, not a discretization sweep. A direct resolution-by-anchor sensitivity study is still missing.

For anchors, the implementation uses a deterministic canonical representative for each directed cycle to eliminate rotational duplicates. However, canonicalization does not imply numerical anchor invariance. For a cycle anchored at �3��3�, $\mathcal B_{c,a}=\eta_a p_c, \qquad \eta^\top=\mu^\top(I-Q)^{-1}.$ Rotating the same cycle to anchor �4��4� preserves its edge product �5��5�, but can change occupancy, giving �6��6�. Thus, the value measures a specified anchored traversal event. Anchor changes can propagate to gradients, candidate rankings, and scope decisions. Our practical recommendation is to select the abstraction and deterministic anchor convention on development data, freeze them within a candidate comparison, and check held-out loop counts, support, and uncertainty. We will make this dependence explicit; insufficient evidence should lead to further data collection or keep, rather than a confident local-repair conclusion.

Q2. Practical selection of the metric �0��0� and radius �1��1�

The key requirement is to use the same declared geometry to assess repairability and constrain candidate updates. A practical metric is M=F^+λI,λ>0M=\widehat F+\lambda I,\qquad \lambda>0, where �2��2� estimates policy KL curvature on a common mixture of current-window and protected replay observations. A diagonal approximation is a lower-cost option. Damping makes the metric positive definite and reduces amplification of poorly supported directions. The replay mixture, approximation, and damping should be selected on development streams and fixed for each candidate comparison. The controlled geometry study supports matching the metric to the update: metric-aware repairability correlates with realized scoped descent at Spearman �3��3�, versus 0.699 for Euclidean repairability. It does not establish a universally optimal metric. To make the radius convention explicit, write the trust region as d⊤Md≤rmax⁡2d^\top Md\leq r_{\max}^2. With a Fisher/KL metric, the local approximation is �4��4�. Consequently, $rmax⁡=2εKLr_{\max}=\sqrt{2\varepsilon_{\mathrm{KL}}} $ is a nominal starting cap for a declared KL budget �6��6�, subject to approximation error and damping. Backtracking should shrink the step if measured KL, task, failure, or retention checks are unsatisfactory; independent validation remains necessary before commitment. A larger radius permits larger changes but weakens reliance on the local model, whereas a smaller radius can exclude useful repairs. Radius selection should therefore use development-only calibration, with the same cap across scope comparisons.

This is a concrete practical specification, not a claim that a universal radius or a completed radius-sensitivity study has been established. We will clarify the metric normalization and radius convention so the procedure is reproducible. Repairability remains local and first-order; neither a favorable score nor a KL-based radius replaces independent deployment evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.