acceptodds
Under review as a conference paper at ICLR 2027

When Edits Accumulate: Constraint-Aware Policy Learning for Lifelong Model Editing

Abstract

Lifelong model editing aims to continuously update knowledge in large language models without full retraining. However, sequential parameter updates can accumulate interference and progressively damage previously edited or unrelated knowledge. To address this challenge, we propose SeqConEdit, a constraint-aware policy learning framework that formulates lifelong editing as a constrained sequential decision problem. Conditioned on the current editing request and accumulated editing history, SeqConEdit adaptively coordinates knowledge incorporation and preservation through a state-conditioned policy. Specifically, we introduce a Preservation-Key Constraint (PKC) to suppress interference along protected knowledge directions and a Preservation Trace Buffer (PTB) to retain historical preservation information across editing steps. Building on these mechanisms, constraint-aware policy optimization dynamically balances editing and preservation according to feedback from the sequential editing process. Extensive experiments show that SeqConEdit consistently improves lifelong editing performance, outperforming HiEdit by up to 11.94% in average performance while substantially enhancing long-term retention of edited knowledge. Further analyses demonstrate improved knowledge preservation, cross-dataset generalization, and policy optimization stability under long edit streams. The source code and implementation details are publicly available at https://anonymous.4open.science/r/SeqConEdit.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
knowledge edit, large language model, lifelong edit

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_SKDVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents SeqConEdit, a framework for lifelong model editing that addresses cumulative interference from sequential updates. It formulates editing as a constrained sequential decision problem and uses a state-conditioned policy to adapt editing decisions based on the current request and editing history. The Preservation-Key Constraint (PKC) limits interference along protected knowledge directions, while the Preservation Trace Buffer (PTB) retains historical preservation keys. Constraint-Aware Policy Optimization (CAPO) adjusts editing and preservation based on feedback from the editing process.

Experiments on ZsRE and CounterFact with two LLM backbones show improvements over existing methods, including HiEdit, with gains of up to 11.94 percentage points in average performance. Additional analyses examine knowledge retention, cross-dataset generalization, and policy optimization stability.

Critical strengths
  • The paper addresses cumulative interference in lifelong model editing, where sequential updates can damage previously edited and unrelated knowledge. Balancing knowledge incorporation and preservation over long edit sequences is an important challenge.

  • SeqConEdit provides a coherent framework that combines state-conditioned editing with persistent preservation constraints. PKC and PTB protect existing knowledge, while CAPO adapts editing decisions and preservation strength as edits accumulate.

  • The experiments show consistent improvements over strong baselines, particularly in edited-knowledge retention. The evaluation also covers knowledge preservation and cross-dataset generalization. Component and policy ablations support the effectiveness of the proposed mechanisms.

Critical weaknesses
  • The methodological novelty is limited. PKC builds on existing preservation methods, and CAPO combines established optimization techniques. The main contribution is their integration, with limited technical advances over AlphaEdit, RLEdit, HiEdit, and LyapLock.

  • The evaluation provides limited evidence for constrained optimization. CAPO adjusts preservation pressure according to a target Specificity level, but the paper lacks an analysis of constraint satisfaction over long edit sequences and an ablation of adaptive Lagrangian weighting.

  • The FIFO strategy in PTB lacks sufficient justification. The capacity analysis shows the benefit of retaining historical preservation keys, but the experiments omit the effect of evicting older keys and comparisons with alternative retention strategies.

  • The experiments lack efficiency and statistical analyses. The computational and memory costs of CAPO and PTB are not reported, and the main results provide averages over three runs without standard deviations.

Questions for authors
  • What are the main technical advances over AlphaEdit, RLEdit, HiEdit, and LyapLock, particularly in history-aware editing and adaptive preservation?

  • How well does CAPO maintain the target preservation level over long edit sequences, and how does adaptive Lagrangian weighting compare with a fixed Specificity weight?

  • Why does PTB use FIFO, and how does evicting older preservation keys affect long-term retention compared with alternative strategies?

5 comments
AuthorsOpenReview ↗

Response to Reviewer SKDV (Part I)

We thank Reviewer SKDV for the careful review and for recognizing the importance of accumulated interference, the coherent integration of state-conditioned editing with persistent preservation, and the retention improvements reported in our evaluation. We address the novelty, constraint behavior, buffer replacement policy, computational cost, and statistical variability below.

RW1 & RQ1: Technical advances in history-aware editing and adaptive preservation

SeqConEdit's technical contribution is a preservation-aware control architecture in which editing history affects both the action selected by the controller and the protected directions entering the update. We distinguish this contribution from the established capabilities of the related methods.

Table R1. Methodological comparison with the four approaches raised by the reviewer.

Related methodEstablished mechanismSpecific advance pursued by SeqConEdit
AlphaEdit [1]Null-space projection for reference knowledge, together with protection of previously edited keys.Soft protection over current and buffered unrelated keys, with preservation strength selected jointly with update strength, regularization, and module routing.
RLEdit [2]Trajectory-level hypernetwork training with memory backtracking, locality KL, and update regularization.A separate controller learns explicit update controls from the current request and historical feedback while the trained writer is frozen; preservation keys constrain the resulting update directly.
HiEdit [3]Hierarchical policies for layer selection and parameter updating, with historical loss feedback and a sparsity-oriented intrinsic reward.Adaptive routing is coupled to explicit control of the preservation-aware update operator and persistent historical protection directions.
LyapLock [4]Virtual queues and Lyapunov optimization adapt preservation pressure under a long-term preservation-loss constraint.A learned policy selects routing and continuous controls using factual editing utility, a Specificity target, and distributional feedback, with long-stream preservation evaluated empirically.

The coupling can be expressed directly through the update. Suppressing the module and time indices on edit keys KK and residuals RR, the update applied to module ℓ\ell is

ΔWℓ,t=mt,ℓαtRK⊤(KK⊤+ρtPℓ,tPℓ,t⊤+βtλℓI)−1,Pℓ,t=[Kℓ,tunr,Bℓ,t].\Delta W_{\ell,t}=m_{t,\ell}\alpha_t RK^\top\left(KK^\top+\rho_tP_{\ell,t}P_{\ell,t}^{\top}+\beta_t\lambda_\ell I\right)^{-1},\qquad P_{\ell,t}=\left[K_{\ell,t}^{\mathrm{unr}},B_{\ell,t}\right].

The history summary hth_t, including recent editing outcomes and accumulated update magnitude, conditions the action (αt,βt,ρt,mt)(\alpha_t,\beta_t,\rho_t,m_t). PTB separately determines which historical unrelated-key directions enter Pℓ,tP_{\ell,t}. Consequently, history influences both the intervention decision and the directions that resist the resulting update. The learned ρt\rho_t adjusts the local preservation penalty jointly with update strength, regularization, and intervention location. The training multiplier μ\mu has a different role: it adjusts the additional Specificity weight in the trajectory reward.

This architecture makes the role of history explicit in the update operator. The Specificity feedback measures observable locality, while DLR supplies information about predictive-distribution drift. Standard Lagrangian weighting and group-relative normalization are optimization ingredients within this design. Our contribution concerns their coordination with the history-conditioned action space and persistent protected directions; it does not require claiming a new general optimization principle or a stronger long-term guarantee than LyapLock.

We evaluated these additions through component and controller ablations.

Table R2. Component and controller comparisons after 8,000 edits. Results are means over three runs.

ConfigurationRet. ↑Avg. ↑
Full SeqConEdit55.1855.14
Without PKC51.8251.24
Without PTB; controller retrained53.9053.48
Adaptive routing only; controller retrained53.8054.40
Adaptive continuous controls only; controller retrained54.1054.60
Explicit history summary removed; controller retrained54.5354.51

Removing PKC reduces mean Ret. by 3.36 points, and removing the historical buffer reduces it by 1.28 points. Restricting adaptation to routing or continuous controls yields lower Ret. than jointly adapting both. The explicit history summary also contributes to the observed mean performance. These comparisons support the roles of protected update directions, their persistence, and history-conditioned control. They quantify the utility of the specific design without assuming that component gains are additive or that every component improves every metric.

AuthorsOpenReview ↗

Response to Reviewer SKDV (Part II)

RW2 & RQ2: Long-sequence constraint behavior and fixed-weight comparison

Equation (2) specifies an expected trajectory-average preservation constraint,

\left[ \frac{1}{N}\sum_{t=1}^{N}\mathrm{Spe}_t \right] \geq \tau, \qquad \tau = 0.5.$$ The implementation updates the multiplier using feedback from groups of four 200-step training rollouts: $$\widehat{P}_{H,u} = \frac{1}{GH}\sum_{g=1}^{G}\sum_{t=1}^{H}\mathrm{Spe}_{t,u}^{(g)}, \qquad \mu_{u+1} = \Pi_{[0,10]}\left[ \mu_u + 0.5\left(0.5 - \widehat{P}_{H,u}\right) \right].$$ Specificity uses the 0–1 scale in these equations. This is a training-batch update; it does not impose a lower bound on every deployment step or on the final checkpoint. We therefore measure preservation over the 8,000-edit trajectory separately from training feedback and terminal performance. Table R3. Preservation measured at different stages and over different time ranges. | Preservation measurement | Adaptive weighting | Validation-selected fixed weighting | |---|:---:|:---:| | Training-support mean Specificity | 78.42 | 78.03 | | Independent validation mean Specificity | 52.16 | 51.29 | | Online-feedback trajectory mean Specificity | 51.81 | 50.55 | | Independent-probe trajectory mean Specificity | 51.06 | 49.77 | | Final Specificity after 8,000 edits | 46.57 | 45.93 | The adaptive configuration's independent-probe trajectory mean is 51.06%, exceeding the target by 1.06 percentage points; the fixed configuration obtains 49.77%, falling below it by 0.23 points. Both feedback averages exceed 50%, which also shows why measuring independent probes adds information beyond the feedback signal. The empirical independent-probe average supports preservation over the evaluated sequence, while its margin above the target should not be read as a statistical feasibility guarantee. Late checkpoints remain challenging. The adaptive configuration's checkpoint Specificity is 53.50%, 50.50%, 48.30%, and 46.57% at 2,000, 4,000, 6,000, and 8,000 edits, respectively. Its terminal values for seeds 0, 123, and 456 are 46.18%, 51.77%, and 41.75%. Thus, one endpoint exceeds the 50% reference threshold, while two fall below it by 3.82 and 8.25 percentage points. A trajectory average above 50% can coexist with these lower endpoints. The results distinguish average preservation from per-step or terminal guarantees. In the training diagnostics, support-set Specificity remained above the target and the final additional multiplier was zero in all three runs. Under the projected update, above-target feedback decreases the additional penalty toward its lower boundary. At zero, the base Specificity coefficient, DLR, PKC/PTB, and the policy's per-edit $\rho_t$ remain active. The observation identifies the gap between satisfying the monitored short-rollout criterion and maintaining the same level at late deployment checkpoints. We also compared adaptive weighting with a fixed additional Specificity weight selected on validation data from $c\in\{0,1,3\}$. All three seeds selected $c^\star=0$. The fixed reward is $$r_t^{\mathrm{fixed}} = \mathrm{Eff}_t + \mathrm{Gen}_t + (\eta_{\mathrm{spe}} + c^{\star})\mathrm{Spe}_t - \kappa\\,\mathrm{DLR}_t.$$ Selecting zero therefore retains the base Specificity reward and the other preservation mechanisms. The comparison matches the writer, feedback, request streams, and per-controller training budget. Fixed-weight selection evaluates three candidate controllers per seed, so its total search cost differs from training one adaptive controller. Table R4. Adaptive versus validation-selected fixed Specificity weighting. Results are means over three runs after 8,000 edits. | Training penalty | Selected additional fixed weight | Ret. ↑ | Avg. ↑ | |---|---|:---:|:---:| | Adaptive multiplier | Updated during training | 55.18 | 55.14 | | Validation-selected fixed weight | 0 for all three seeds | 53.40 | 54.95 | | Adaptive − fixed | — | +1.78 | +0.19 | The adaptive configuration improves mean early-edit Ret. by 1.78 points and Avg. by 0.19 points, while the fixed configuration has higher Eff. and Gen. We believe this pattern reflects the trade-off between retaining early edits through many subsequent updates and preserving flexibility for other edits. The independent-probe difference of 1.29 points in Table R3 provides additional evidence of the observed preservation benefit. The small Avg. difference and the multiplier's boundary behavior support a measured claim about these training configurations, without implying a uniform advantage on every stream or attributing the entire benefit to a persistently positive multiplier.
AuthorsOpenReview ↗

Response to Reviewer SKDV (Part III)

RW3 & RQ3: FIFO, historical-key eviction, and alternative retention rules

We adopted FIFO in the original implementation to maintain a bounded, deterministically refreshed set of historical preservation keys. Newly collected unrelated-knowledge keys replace the oldest stored rows, allowing protection directions from recent edits to remain active in subsequent updates. This rule provides a simple way to maintain protection across consecutive edits without an additional importance estimator. The reviewer's question motivated us to examine how this preference for recent history affects long-term retention.

PTB stores directions used to constrain subsequent updates, while edited knowledge resides in the model parameters. In the preservation term ρt∥ΔWP∥F2\rho_t\|\Delta WP\|_F^2, each key column pjp_j contributes ρt∥ΔWpj∥22\rho_t\|\Delta Wp_j\|_2^2. Evicting an old key removes its contribution from the protection objective. Other current or buffered keys can continue to constrain the same direction if they cover it; otherwise, that direction becomes less protected against later updates. Thus, the replacement rule determines which historical directions continue to resist subsequent interference.

We compared FIFO with uniform reservoir sampling over all historical token-key rows and a hybrid rule retaining 250 recent rows plus 250 rows sampled from older history. Within each paired run, all three strategies use the same frozen controller and a capacity of 500 token-key rows per editable module. Only the deployment-time retention rule changes, allowing us to examine the effect of allocating the same buffer capacity over different time ranges.

Table R5. Buffer replacement policies after 8,000 edits.

Retention ruleMean key ageRet. ↑Spe. ↑Avg. ↑
FIFO226.1155.1846.5755.14
Uniform reservoir1294.5257.3345.8955.55
250 recent + 250 historical rows1006.6355.9046.1655.26

Each buffered token-key row corresponds to one key column in PP, and an edit can insert multiple rows. With FIFO, the mean key age is 226.11 editing steps, and the oldest keys at the end of the three runs are approximately 552–586 steps old. Reservoir and hybrid retention increase the mean key age to 1294.52 and 1006.63 steps, respectively. These measurements show that FIFO concentrates protection on recent history, whereas historical sampling maintains directions collected much earlier in the editing sequence.

Reservoir improves mean Ret. from 55.18 to 57.33 and Avg. from 55.14 to 55.55. Ret. improves in all three paired runs, by 0.40, 4.55, and 1.50 percentage points. The hybrid rule also improves mean Ret. and Avg. over FIFO, reaching 55.90 and 55.26. We believe the broader historical coverage helps explain these improvements: directions that have left the FIFO buffer can remain represented under historical sampling and continue to constrain later updates, helping reduce accumulated interference with earlier knowledge.

The fixed capacity also determines how much coverage can be devoted to recent directions. FIFO concentrates its 500 rows on the most recently collected keys and obtains a Spe. of 46.57, compared with 45.89 for reservoir. We believe this emphasis on recent protection directions contributes to its slightly higher Specificity. The hybrid rule divides the capacity between recent and older history; its Ret. of 55.90 and Spe. of 46.16 lie between the two alternatives, consistent with this allocation of protection across time.

These findings complement the PTB ablation in Table R2, where removing the historical buffer reduces Ret. and Avg. Together, the experiments support carrying historical protection directions into subsequent updates and identify their temporal coverage as a useful way to strengthen the mechanism. FIFO provides the rolling, recent-history protection used in the original experiments. Reservoir extends that coverage further into the editing history and improves early-edit retention at the same buffer capacity in this comparison. We will clarify this distinction in the method description and include the replacement-policy comparison in the revised paper.

Reviewer_zsMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SeqConEdit for lifelong model editing. A controller uses the current editing request and a summary of previous edits to choose update strength, regularization, preservation strength, and editable modules. Each selected module is updated through a regularized key–value objective that penalizes changes along preservation keys. A FIFO buffer carries unrelated preservation keys across editing steps. The controller is trained using an adaptive specificity reward, distributional locality regularization, and group-relative trajectory advantages.

Critical strengths
  1. The reported improvements in early-edit retention are substantial. Table 1 shows the highest average score in all four settings. On Gemma-2-9B, retention increases from 68.73 to 83.24 on ZsRE and from 54.00 to 78.00 on CounterFact relative to HiEdit. Appendix A.2.3 defines this metric using the first 500 edits after the complete 8,000-edit stream. This directly addresses forgetting of edits exposed to many subsequent updates, rather than only the success of recent edits.

  2. The paper includes useful controls for several parts of the method. Tables 2–3 examine the preservation components, Table 4 tests the policy’s state information and action branches, and Table 5 distinguishes group-relative advantages from simpler return normalization. In Table 5, group-relative estimation achieves an average of 60.90, compared with 58.38 for global normalization and 59.94 for sequence centering. These experiments help identify which design choices contribute to the reported performance.

Critical weaknesses
  1. The routing surrogate is not sufficiently justified as part of the policy-optimization objective. Appendix A.5 samples exactly six modules using Gumbel-Top-K, but Equation 25 scores the selected modules by summing their log-softmax values. This is not the log-probability of that without-replacement selection: its probability depends on the successive remaining choices, or on a sum over orders if the action is treated as an unordered set. Appendix B.5 acknowledges that this term is a surrogate, so I am not treating it as an undisclosed approximation. Nevertheless, Equations 69–71 insert it into the trajectory score without explaining its relationship to the expected-return objective or testing the effect of this approximation. The reported gains may still be real, but the central policy-learning argument needs either a justification of this surrogate or a comparison with a sampling-consistent likelihood.

  2. The provenance of the baseline results is unclear, making the main comparison difficult to assess. Section 5.1 states that all methods are evaluated on the same shuffled sequence in each of three runs. Appendix A.4.2 instead states that published results are used when available, with other methods reproduced from official implementations. Table 1 does not identify which entries fall into each category or explain how imported results share the stated edit sequences. Edit order matters in lifelong editing, so matching the backbone and nominal 8,000-edit protocol does not by itself establish a matched comparison. This is particularly consequential for the comparisons with RLEdit and HiEdit, which support the main performance claims. I am not concluding that those comparisons are invalid; the current reporting does not allow me to verify them.

Questions for authors
  1. Routing optimization: What is the justification for the selected-log-softmax surrogate in Equations 25 and 68? How does performance change when using the likelihood corresponding to the sampled without-replacement route?

  2. Baseline provenance: Which Table 1 entries are reproduced and which are imported from prior publications? Were RLEdit and HiEdit evaluated on the same three edit streams as SeqConEdit? Please provide the corresponding per-run results or variability estimates.

  3. Constraint satisfaction: What preservation level is actually achieved under Equation 2, and how does the multiplier evolve during training? Since training uses 200-step rollouts but evaluation uses 8,000 edits, please distinguish rollout-average preservation from final-model Specificity. A control with a tuned fixed specificity weight would also help establish the contribution of the adaptive multiplier.

  4. Evaluation feedback: Appendix A.5 constructs the history state using efficacy, generalization, and locality feedback from completed edits. Are these feedback prompts also used in the final evaluation, or are there separate held-out probes? What feedback is supplied to the closest baselines? This matters for interpreting both the comparison and the intended deployment setting.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zsMd (Part III)

RQ4: Feedback access and independent evaluation

We examined preservation measured on the online feedback queries and on the independent diagnostic probes. The adaptive configuration achieves an independent-probe trajectory average of 51.06%, compared with 49.77% for the validation-selected fixed-weight control. The corresponding feedback averages are 51.81% and 50.55%. The improvement therefore also appears in the independent measurement, with a gain of 1.29 percentage points. These are trajectory averages; final Specificity is 46.57% and 45.93%, respectively.

The full configuration assumes a monitored editing environment. At step tt, the history summary contains efficacy, generalization, and locality feedback from completed edits, together with progress and accumulated-update statistics. It is constructed before choosing the next action. This provides causal feedback from previous edits while avoiding access to the outcome of the current action in advance. The controller parameters remain frozen during deployment, while the history inputs and selected actions evolve.

RLEdit and HiEdit already incorporate preservation-related supervision in their published training procedures. RLEdit combines an editing objective with locality KL regularization and memory backtracking [3]. HiEdit combines editing and locality losses with its hierarchical reward and layer-selection mechanism [2]. These mechanisms differ from SeqConEdit's explicit summary of previous ES/GS/LS outcomes. Consequently, shared task supervision and method-specific use of feedback should be distinguished; using a common editing benchmark does not require inserting SeqConEdit's history representation into the baseline algorithms.

An edit request and its desired answer are legitimate inputs to an editor, and using that request for efficacy evaluation is part of the task. Generalization concerns alternative formulations of the same edited fact, whereas locality evaluates other knowledge. The additional independent-probe measurement evaluates preservation beyond the online feedback score.

The deployment assumption is therefore explicit: the full method requires feedback queries and their evaluation references for completed edits. A frozen controller avoids online policy retraining, but feedback acquisition and evaluation still have a cost. This is a monitored editing setting; performance in a setting without such feedback is a separate question.

References

[1] Ancestral Gumbel-Top-k Sampling for Sampling Without Replacement. Journal of Machine Learning Research, 21, 2020.

[2] HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning. ACL, 2026.

[3] Reinforced Lifelong Editing for Language Models. 2025.

Dear Reviewer zsMd,

Thank you once again for your valuable feedback on our work. We hope our rebuttal has provided useful clarification on the points you raised.

Before the rebuttal period concludes, we would greatly appreciate any further thoughts or questions you may have. Please feel free to let us know if any concerns remain; we would be glad to provide additional explanations.

We sincerely appreciate your time and thoughtful consideration.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer uKmM (Part II)

RW2 & RQ2: Training resources, online execution, and policy reuse

We measured both training stages on a single NVIDIA H100-80GB GPU. Stage A trains the writer; Stage B freezes the writer and trains the controller, while the target LLM receives knowledge updates.

Training stageGPU time, H100-hPeak allocated memory, GiBTraining exposure
Stage A: writer6.2021 ± 0.603920.6434 ± 0.0185One epoch over 10,499 distinct facts
Stage B: controller1.7769 ± 0.019621.0104 ± 0.000512,000 editing transitions
Total7.9790 ± 0.6056Reported by stageStage A followed by Stage B

Stage B uses 15 policy updates with G=4 rollouts of H=200 steps: 15 × 4 × 200 = 12,000 transitions. Each group shares one request sequence, giving 3,000 request positions before replication; facts may repeat across updates. Thus, transitions are not distinct facts. Controller training accounts for 22.27% of total training time.

MethodTotal training time, H100-hAvg. ↑Ret. ↑
HiEdit10.10 ± 0.1552.0853.47
RLEdit6.27 ± 0.1952.2352.13
SeqConEdit7.98 ± 0.6155.1455.18

Under these training procedures, SeqConEdit uses 2.12 fewer H100-h than HiEdit and 1.71 more than RLEdit, improving Avg./Ret. over RLEdit by 2.91/3.05 points. These measurements quantify the quality–cost trade-off under each method's training procedure.

At deployment, continuous controls use their predicted means and routing uses deterministic Top-K (Appendices A.5–A.6). Grouped rollouts and policy-gradient updates occur only during training, so G=4 does not multiply online execution by four. The frozen controller adapts its actions to the evolving request and history.

MethodLatency, ms/editMeasurement scope
HiEdit124.83 ± 17.60Native validation/editing loop; excludes subsequent audit evaluation
RLEdit152.00 ± 25.11Native validation/editing loop; excludes subsequent audit evaluation
SeqConEdit138.62 ± 1.442Online editing loop, including support feedback and PTB maintenance

SeqConEdit's measured latency is 13.79 ms higher than HiEdit and 13.38 ms lower than RLEdit, including support-feedback computation and PTB maintenance. These absolute timings place its loop cost between the baselines under the listed scopes; matched instrumentation is needed to isolate component overheads.

The controller has 1,186,322 parameters (approximately 4.53 MiB). PTB logical tensors occupy 93.75 MiB, with 93.94–94.88 MiB of unique backing storage. These storage figures quantify the additional data structures; the stage-level GPU memory peaks cover the complete training process.

We also examined controller-training budgets:

Group size GHorizon HPolicy updatesEditing transitionsRet. ↑
42001512,00055.18
82001524,00055.56
45001530,00055.73
4500612,00055.34

Doubling transitions through G=8 improves Ret. by 0.38 points. Extending H to 500 with 15 updates improves it by 0.55 points at 2.5 times the interactions. At 12,000 transitions, H=500 with six updates improves Ret. by 0.16 points. The limited gains support G=4, H=200 as a practical default.

For policy reuse, we froze the same trained writer and controller and applied them to two CounterFact streams excluded from controller training and selection. Each stream-specific comparator uses the same writer and a controller trained for that stream.

Target streamReused Avg.Stream-specific Avg.Reused Ret.Stream-specific Ret.Controller training avoided
12356.6153.9155.4051.701.7784 H100-h
45658.6559.0661.2062.401.8224 H100-h

Reuse changes Avg./Ret. by +2.70/+3.70 points on stream 123 and −0.41/−1.20 on stream 456, avoiding 3.6008 H100-h of controller training in total. Stream-specific training can help, but retraining is not required for deployment on a new stream. Serving S streams with one writer–controller pair amortizes its initial training cost to (TA+TB)/S(T_A+T_B)/S per stream, plus online editing costs. This demonstrates reuse across streams within the tested backbone and domain.

Dear Reviewer uKmM,

We appreciate your careful review and constructive comments.

With the rebuttal period nearing its end, we wanted to check whether any points in our response would benefit from further explanation. We hope our clarifications have been helpful in addressing your questions, and we would welcome the opportunity to discuss any remaining concerns.

Thank you again for the time and effort you have devoted to reviewing our work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.