acceptodds
Under review as a conference paper at ICLR 2027

Exact Undo: Inverse-Bound Writes Make Recurrent States Provably Revisable

Abstract

Sequence models can write, but they cannot unwrite. Attention appends keys and values that are never removed; delta-rule networks overwrite by decay. Revision–undo a write exactly, in constant-size state, at unbounded nesting depth–has no architectural primitive. We supply one: inverse-bound writes, where a token and its inverse share materialized floating-point parameters at the bit level, so executing the inverse composes the exact inverse of the forward write into the state. Exactness is constructive and parameter-free: one round returns the state to O(u) for every parameter value (Theorem 1); K rounds accumulate at most K*O(u), while a repeated learned mismatch accumulates at first order (Theorem 3); the construction ports into DeltaNet at a proved price (Theorem 6), and a corollary proves exact cancellation unreachable in that family's open parameter domain; measured, its depth-128 undo stacks collapse (0.207) where the orthogonal carrier stays exact. Closure under inverses further makes the state an exact O(1)-state solver for group word problems, with zero-error 8x length extrapolation. Measured on a group-program benchmark, the taxonomy orders itself as the theory demands: append-only transformers plateau at 0.320; approximate-inverse DeltaProduct shares capability but not cancellation; inverse-closed-but-unbound OS reaches exactness with 5-10x more samples; inverse-bound OS is exact at the matched schedule, its state exact from initialization, through depth-128 undo stacks. What compute buys is the readout channel–a linear probe decodes depth-128 answers from the unbound model's contracted signal at 1.000 while its first-layer operator is off by 0.8, an error the trained readout's margin tolerates; what the binding guarantees, in the vocabulary-bound layer, is the whole operator, every direction, any probe. A dose-response boundary chart closes the paper: each inverse-bound layer monotonically degrades byte-level language modeling–composition, not compression. The scope is the strictly nested (LIFO) core of revision.

Primary area:
learning on time series and dynamical systems
Keywords:
recurrent neural networks, linear attention, reversibility, floating-point error analysis, delta rule, vector symbolic architectures, state tracking, fast weights, group word problems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dNfXOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces inverse-bound recurrent updates to support reversible state changes. It combines algebraic constructions and floating-point error analysis with synthetic undo tasks, depth generalization experiments, and a small language-modeling evaluation.

Critical strengths
  1. The distinction between correct task outputs and accurate operator restoration is useful. Matched bound/unbound comparisons reveal differences that task accuracy alone would miss.
Critical weaknesses
  1. Limited novelty. Explicitly tying operations to their inverses is a direct structural constraint; the paper does not clearly establish the advance beyond existing structured recurrent models.

  2. The experimental evidence is not sufficiently solid. Gains are concentrated on synthetic tasks tailored to the construction. The unbound baseline matches task accuracy with more training, while language modeling degrades, leaving the practical benefit of precise state restoration unproven.

Questions for authors
  1. Does accurate operator restoration improve performance on unseen readouts or subsequent operations when the unbound baseline already achieves correct task outputs?
  2. Can the authors reconcile the numerical-saturation explanation with the difference between a rounded squared sum and the exact squared sum of stored floating-point values?
  3. Is the estimated training requirement a fitted extrapolation rather than a lower bound, and how sensitive is it to the fitting interval?
  4. Do the guarantees extend beyond the vocabulary-bound layer to contextualized updates in a multilayer model?
1 comment

Thank you for the clarifications. They address some of my concerns, although the presentation still has substantial room for improvement. I have raised my rating to 2.

Reviewer_gWoeOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces inverse-bound writes: designated inverse tokens reuse the parameters of forward writes in an affine operator-state recurrent model. It derives floating-point rollback bounds, analyzes accumulated mismatch, and constructs a DeltaNet inverse with a conditioning cost. Synthetic group-program experiments demonstrate improved sample efficiency, near-roundoff restoration of the first-layer state through nesting depth 128, and 8× length extrapolation. Additional diagnostics distinguish state restoration from prediction accuracy. Language-modeling experiments show weaker performance, although inverse binding is inactive there.

Critical strengths
  • A simple construction with a meaningful guarantee. Parameter sharing removes the need to learn inverse consistency and provides a state-restoration guarantee independent of task training, within the stated assumptions. Preserving all operator directions is stronger than achieving correct predictions through one readout.

  • Useful controlled diagnostics. The equal-parameter ablation, three-seed comparisons, longer training schedules, and layer-specific probes distinguish sample efficiency from state fidelity. Reporting that the unbound model eventually matches task accuracy makes this distinction particularly informative.

Critical weaknesses
  • The closest inverse-capable baseline is missing. Corollary 7 correctly excludes �0��0�, but this endpoint gives an orthogonal Householder reflection, and two reflections can implement a plane rotation, as discussed in DeltaProduct, §4. This suggests a direct control: fix �1��1�, share the forward factors, and reverse their order for inverse tokens. Algebraically, this avoids counter-decay. Without this comparison, the experiments do not establish the advantage of the proposed carrier over an existing orthogonal parameterization equipped with the same inverse information.

  • The consequential benefit of full-operator precision remains insufficiently demonstrated. The experiments establish sample-efficiency gains on symbolic tasks with predefined inverse pairs. After longer training, however, the unbound OS also achieves 1.000 accuracy through depth 128 despite substantial state error. The guarantee covers only the vocabulary-bound layer, and the language-modeling experiment does not exercise binding. Thus, the paper demonstrates a strong local numerical property but provides limited evidence that its additional precision improves downstream functionality once task learning converges.

Questions for authors
  • How does an inverse-bound, two-Householder model with �0��0� compare under the same training schedule, inverse metadata, and nesting evaluations? This would materially clarify the contribution relative to DeltaProduct.

  • What evidence shows a downstream query or continuation benefiting from the tighter operator restoration after bound and unbound models both reach perfect benchmark accuracy?

  • Can the numerical claims distinguish exact arithmetic from floating-point execution consistently? In particular, please reconcile Appendix C’s explanation with the exact stored-matrix defect, and qualify “unbounded-depth exactness” using the horizon-dependent bounds.

1 comment

Thank you for the additional experiments and clarifications. The β=2\beta=2 control substantially addresses my baseline concern, and the numerical corrections resolve my concern about floating-point exactness. These additions strengthen the paper, but my concern about the significance of full-operator preservation beyond the specialized symbolic setting remains. The reported downstream evaluations are relevant, yet do not sufficiently change my overall assessment. I therefore maintain my recommendation of 2: Weak rejection.

Reviewer_PjMjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces inverse-bound writes, a mechanism that allows recurrent neural networks to undo previous memory updates without having to learn how to reverse them. Each token is paired with an inverse token that restores the previous memory state, up to small floating-point errors. With theoretical guarantees, the authors show that their method outperforms transformer and learned-inverse baselines on synthetic tasks involving sequences of operations and nested undo actions. They also extend the idea to DeltaNet, but with numerical stability limitations. The proposed mechanism only supports undoing operations in reverse order (LIFO), and incorporating it into language models worsens their performance. So, the method is suited to tasks requiring reversible operations rather than general language modeling.

The authors compare their method against Transformer, DeltaProduct, and recurrent models without explicit inverse binding on synthetic and nested-undo tasks. Their method achieves 100% accuracy on undo sequences of nesting depth 128 and generalizes to sequences eight times longer than those used during training. However, models without explicit inverse binding require much more training to achieve comparable accuracy. The authors also show that accurate predictions do not necessarily imply accurate internal memory states.

Critical strengths
  1. The paper introduces inverse-bound writes, which allow recurrent networks to undo previous memory updates by construction, a parameter-efficient alternative to learning inverse operations. The authors also provide theoretical guarantees on the numerical error of these updates and its accumulation over repeated undo operations. They also extend the construction to the DeltaNet model and analyze the associated numerical stability limitations. This is a meaningful contribution because it implies that the conventional range for β\beta in DeltaNet does not permit exact inverse writes within the same parameter domain.
Critical weaknesses
  1. The theoretical guarantee applies only to explicitly paired inverse tokens processed in reverse order (LIFO). The method does not guarantee exact revision of the entire network or arbitrary earlier memory updates, limiting its applicability to more general revision tasks.

  2. The experiments primarily focus on synthetic group-composition and nested-undo tasks. The method performs well on these tasks, however, its practical benefits remain unclear. The unbound model eventually achieves the same task accuracy despite its internal state errors, which raises the question of whether exact state restoration is necessary for downstream performance. The language-modeling experiments show worse performance than the Transformer baseline. These do not provide enough evidence for the method's broader usefulness.

Questions for authors
  1. (Clarification Request) The inverse-binding guarantee applies only to the first vocabulary-bound layer, while deeper contextualized layers are not covered and exhibit substantial state errors. Could the authors clarify whether the proposed mechanism can guarantee exact restoration of the entire recurrent network's state?

  2. (Clarification Request) The method assumes that the appropriate inverse token is supplied to undo the previous memory update. However, in practical applications, it may not be known in advance when a memory update should be reversed. The authors acknowledge that addressing this is outside the scope of this work, however, could they clarify how they envision the proposed mechanism being used in practical applications where information about when to apply the inverse is not explicitly available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.