acceptodds
Under review as a conference paper at ICLR 2027

How Long Is a Recurrent Circuit Faithful? Retained Memory and Fading Error

Abstract

A recurrent circuit must carry its own state, not receive a corrected state before each short test. Certifying persistent execution is difficult when retained units hold information indefinitely: the usual contraction test on the model–circuit comparison can fail even when their output discrepancy is small. We separate retained circuit memory from fading error. An exact finite memory transducer, combined with protocol-constrained error contraction, gives convergent all-time risk bounds and failure histories that can be replayed without requiring the retained memory to contract. We quantify the error that remains when memory summaries are coarse and give a complementary reset–stutter certificate for token-conditioned affine recurrences. The method builds on invariant-set and differential-verification principles rather than replacing them. A coupled write/hold example exercises the new premise. Six fully trained parity recurrences produce smaller circuits certified at tolerance under a reset-bounded token grammar; five of six refreshed selections at that tolerance fail exact persistent evaluation. Separate lower-calibrated scaling tests compare adaptive suffix refinement against block-aware invariant bounds, including difficult unresolved cases. Finally, tolerance reanalysis of fixed dense nonlinear networks show that local success followed by later failure depends strongly on the chosen metric and operating point.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, recurrent circuits, persistent execution, formal verification, invariant sets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FUnXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper tackles circuit extraction of recurrent systems. To obtain certificates on the discrepancy between the model and the extracted circuit, an error term is explicitly tracked. As the recurrency could result in a blow-up of the error term, the paper ensures that this stays small by forgetting no-longer relevant error terms.

Critical strengths
  • Bounding the error term, especially over infinite horizons, is very useful for safety critical applications.
  • The mathematics are rigor; although a Lean formalization would be nice.
Critical weaknesses
  • It is not clear to me what kind of input tokens exist and what they do exactly. There are mentions of some such as HOLD and RELAX, but they are never introduced.
  • Variables and their meaning could be better introduced, e.g., all variables in (1): A∈Rn×nA \in \mathbb{R}^{n\times n} etc.
  • The paper could benefit from some illustrations and/or a running example
  • Some sentences read very AI-generated (overly exhaustive, typical formulations); these could be refined for precision. For example, I find "A recurrent circuit must carry its own state, not receive a corrected state before each short test." an odd sentence to introduce readers into the topic in the abstract.
  • Shouldn't Table 2 be in an algorithm environment? If so, use variables as introduces in the paper and pseudo code instead of text.
  • Related work in appendix, no comparison against it; also odd to have a proposition in this section
Questions for authors
  • Do you have a concrete application in mind where you want to use this method?
3 comments
AuthorsOpenReview ↗

Response to Reviewer FUnX

Thank you for the concrete suggestions. We agree with all six points.

W1: tokens are not introduced. The new Table 2 in §2 lists each token of the latch, the parity models and the §5.2 systems, with its effect and its protocol restriction. Appendix D.4 defines the tokens of the new generated systems of §5.3. HOLD and RELAX keep the latch's stored value and multiply the omitted error by 9/8 and 1/4, respectively. The caption of Figure 1, where they first appear, now points to §2 for this. §2 also defines an input protocol (a graph whose paths are the allowed token sequences) and a legal word (one such sequence).

W2: variables are not introduced. The sentences around Eq. (1) now define its symbols, and Table 1 lists the main symbols with their dimensions, such as Aa∈Rn×nA_a\in\mathbb{R}^{n\times n}.

W3: running example. The coupled latch is introduced in §2 and reused in §3 and §5.3. For coupling strength g=1/32g=1/32, the new Figure 2 shows (left) a legal sequence whose discrepancy approaches the exact all-time risk 68g/23≈0.09268g/23\approx0.092 and (right) the bounds of our procedure (Algorithm 1) converging to it.

W4: wording. We removed the sentence you quoted, and the abstract now opens by defining a circuit and stating our question. We also rewrote the abstract, introduction and section openings in plain words, defined each term where it first appears, and removed repeated disclaimers.

W5: Table 2 as an algorithm. The submission's Table 2 procedure is now Algorithm 1 in §4, written as pseudocode with the paper's variables.

W6: related work and comparison. Related work is now §6, with an extended version in Appendix B. The submission's Table 1, which compared our assumptions with control-theory methods, is now Table 6 in §6, and the proposition you noted in that section (now Proposition 7) has moved to Appendix C. The recurrent-network verifiers (POPQORN, Prover, Cert-RNN, DiffRNN, RnnVerify) bound only a fixed number of steps, so they cannot rule out later failures, and a head-to-head run would not test the same claim (§6). Circuit-discovery methods complement ours, because they supply the circuits that we verify. When integrated-gradient gating and attribution patching (Syed et al., 2023) keep as many units as our persistent selector (which tests circuits on their own state), they pick exactly its units in all six parity models, so their circuits certify too (§5.1, Appendix D.3). On the §5.2 systems, we also added a classical control-theory bound, which certifies different cases from ours (Appendix D.6).

Formalization. The proofs in Appendices A and C are not machine-checked, and a Lean formalization does not fit in the rebuttal period. All certificates and counterexamples use exact rational or rigorous ball arithmetic, and every counterexample is replayed independently. Floating point only proposes candidates for exact checking, except in the uncertified diagnostics listed at the start of Appendix D and the empirical study of §5.4.

Q1: application. We do not target a specific deployed system. The immediate use is in interpretability research, and §5.1 shows why it matters. In five of the six affine models, a circuit that passes a validation with periodic copying of the model's state still fails when run on its own state. Proposition 4, the certificate we use for the trained models, suits episodic use: the system is reset exactly between episodes, and each episode contains at most a fixed number of informative tokens (tokens other than resets and state-preserving tokens such as QUERY). A certificate supports a safety argument only if the original model is trusted, and only for the declared inputs (new "Intended use" paragraph in §1).

Reviewer FUnX, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

AuthorsOpenReview ↗

Response to Area Chair 1Z2k (1/2)

Thank you for following up with Reviewer FUnX.

The revised PDF addresses each of FUnX's points. The new Table 2 in Section 2 lists the tokens with their effects and protocol restrictions, and Table 1 and the text define the symbols of Equation (1). The coupled latch example now runs through Sections 2, 3 and 5.3, with the new Figure 2. We rewrote the abstract and introduction in plain words, and the old Table 2 procedure is now Algorithm 1 in Section 4.

Related work is now Section 6 of the main text. It explains why verifiers that check recurrent networks for a fixed number of steps cannot certify a claim for all time, and its Table 6 compares assumptions with the closest control-theory methods. The new "Intended use" paragraph in Section 1 answers the application question.

Reviewer_Z2JAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how to formally certify the long-term faithfulness of reduced recurrent neural network (RNN) circuits. It shows that short-term evaluations with periodically refreshed hidden states can be misleading, as errors may accumulate during persistent execution. The authors propose a verification framework that separates persistent memory from fading error, enabling all-time guarantees under specified input protocols without requiring the entire system to contract. Experiments on trained recurrent models demonstrate that locally faithful circuits can fail during longer executions, while the proposed approach can certify persistent faithfulness under its stated assumptions.

Critical strengths
  1. The paper introduces a memory-conditioned verification framework that separates persistent memory from fading error, enabling all-time faithfulness guarantees without requiring whole-system contraction. This addresses an important limitation of conventional stability-based verification approaches.

  2. the paper combines formal guarantees with experiments on trained recurrent models, demonstrating that short-term circuit faithfulness can be misleading. The inclusion of feasible counterexamples, baseline comparisons, and unresolved cases strengthens the credibility of the evaluation.

Critical weaknesses
  1. The main certification framework requires exact finite-state retained memory and contractive error dynamics under constrained input protocols. The trained-model experiments rely on resets after at most 12 informative tokens, substantially limiting the generality of the claimed all-time guarantees, particularly for realistic nonlinear RNNs.

  2. The proposed approach leaves all 96 comparable-slow scaling requests unresolved, highlighting significant computational limitations. Moreover, the nonlinear RNN experiments provide only empirical evidence rather than formal certification, leaving the practical effectiveness of the framework on realistic architectures insufficiently demonstrated.

Questions for authors

To what extent can the proposed certification approach be extended to nonlinear RNNs or recurrent architectures with continuous hidden states? Is the exact finite-memory assumption essential, or could sound approximations provide similarly meaningful guarantees?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Z2JA

Thank you for the clear questions.

W1: finite memory, contraction and the 12-token cap. We agree that this is the main limitation, and §7 now says so. The new Table 3 lists what each all-time result requires and gives. Only Proposition 4, which needs exact resets and at most LL informative tokens between resets, applies to our trained models, and its cost grows exponentially with LL.

The method accepts any finite protocol graph, even one that allows every sequence. A narrower protocol can make a claim certifiable, but the certificate then covers only the sequences it allows. For example, the write/hold latch passes the error-contraction test only if two HOLDs in a row are forbidden.

For the affine models the cap is essential (§5.1, Appendices D.1 and D.2). All six circuits selected on their own state (persistent circuits) certify at 0.1 for every cap up to 13 tokens. Two fail, at 14 and 19 tokens, and the other four certify at the largest caps we enumerated, 18 or 20 (Fig. 3). Without resets, every circuit exceeds 0.1 on an exactly replayed periodic input after 22 to 239 tokens, and the premises of Theorem 3 provably fail (Appendix D.1, Table 7).

W2: the 96 unresolved requests. We agree, and we report it as a negative result for our adaptive refinement (Algorithm 1) alone. The 96 requests (certification decisions) come from 24 comparable-slow systems, in which all modes decay slowly, each run with 2 tolerance margins and 2 boundary sets (a ball and a box). At the 6000-node cap, the median ratio of upper to lower bound is 7.00 with the ball and 3.71 with the box, at both margins. In the slow–fast regime, where one mode is slow and the others are fast, refinement certifies 52 of 96 requests, while none of the three non-adaptive comparators certifies any (Table 5).

For the 12 comparable-slow systems with n≤32n\le32 (48 of the 96 requests), we added a classical ellipsoid bound from a linear matrix inequality (LMI) and verified it exactly (Appendix D.6, Table 10). With the ellipsoid used alongside the ball or the box inside refinement, the combined method certifies, in both cases, all 12 ten-percent requests and 2 of the 12 one-percent ones. It also falsifies 2 more one-percent requests, whose preset tolerances were below the true risk. The other 8 in each case (16 in all) stay unresolved, with upper bounds 1.017 to 1.052 times the lower bounds, against 2.26 to 11.9 for refinement alone. For these systems, the gap therefore lay mostly in the upper bound. The 48 requests with n=64n=64 and 128128 remain unresolved, because our LMI implementations did not complete at n=64n=64.

W2: only empirical evidence for nonlinear RNNs. Because Proposition 4 uses no linearity, the revision now checks, in rigorous ball arithmetic, the circuits of six trained tanh token recurrences (§5.1, Appendix D.5, Table 9). At tolerance 0.1, five of six persistent circuits certify, as do four of six refreshed ones, whose validation periodically copies the model's state into the circuit. These small models have a fixed, unlearned RESET. The dense tanh networks of §5.4 have no such reset, so Proposition 4 does not apply, and certifying them remains open.

Q: is exact finite memory essential, and does the approach extend? Exact finite memory is neither essential nor sufficient on its own. A general all-time check needs a restriction such as contraction of the error along allowed sequences or a cap on informative tokens per episode.

  • Continuous hidden states are allowed, because finiteness concerns only the values of the retained memory, and continuous inputs may still drive the error. The memory is finite only if it starts from finitely many values and receives no continuous input (Appendix A.4). It is finite, for example, when each token sets it to one of finitely many values or keeps it.
  • With contraction of the error along allowed sequences, finite memory makes Theorem 3's bounds converge. A certificate does not need finite memory, however, because Proposition 4 needs only exact resets and the cap.
  • The new Proposition 8 (Appendix C) shows that without such contraction or a cap, the question whether the all-time risk is at most ε\varepsilon is undecidable. This holds even for linear recurrences whose memory takes a single value, and even when the tolerance is far from the risk.

Sound approximations give weaker guarantees. For a continuous memory, Proposition 1 gives only an upper bound from a verified invariant set (Appendix A.3), and by Proposition 5 a coarse memory can leave an error floor that does not vanish. For a masked tanh RNN, such a bound would need every allowed product of error matrices to contract, for every choice of tanh slopes (§4). We have not implemented this, and we make no claim for selective recurrences such as Mamba (§7).

Reviewer_Z2JAOpenReview ↗

thanks

I thank the authors for replying my questions. It partially solves my concerns, I will keep my positive score.

AuthorsOpenReview ↗

Response to Reviewer Z2JA (follow-up)

Thank you for reading our response and for keeping your positive score. We agree that two limitations remain, and Section 7 of the revision states both. First, Theorem 3 requires a memory with finitely many values and an error that shrinks along the allowed token sequences, so we certify our trained models with Proposition 4 under a cap of 12 informative tokens between resets (Section 5.1, Appendices D.1 and D.2). Second, certifying dense tanh networks without an exact reset remains open (Sections 5.4 and 7). If a specific concern remains, we would be glad to answer it before the discussion closes.

Reviewer_fA5iOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper asks whether a smaller network, made by removing some neurons from the original, still gives nearly the same outputs as the original when it runs on its own for a long time. The authors argue that local tests, which keep resetting the small network's state back to the original's, can make it look better than it really is. They propose a way to prove a long-term guarantee. Their method keeps exact track of the information the network has to remember for a long time, and only requires the remaining errors to fade over time. Under certain conditions, this gives an upper bound on how far the two outputs can drift apart. They also run a series of experiments to show the method works.

Critical strengths

I think the question itself is worth discussing, and I agree the view of author that existing local evaluation methods may indeed overestimate how well a pruned model performs.

Critical weaknesses

I recommend rejecting this paper for the following reasons.

  1. I think the paper's main point is its proof method for "long-term memory" (theorem 3), but this method is mostly demonstrated on systems the authors constructed themselves. I did not see it applied to any of the six trained models. I would like to see how the core method performs on more cases where it applies.

  2. For the 48 randomly generated test systems, the baselines are simple methods that the authors implemented themselves within the same pipeline. I think the paper should include some other thorough comparisons with existing methods from the community. The authors acknowledge this gap but did not address it.

  3. The paper is hard to follow. Many terms are used without being defined or introduced first, as if every reader already knows the terminology, which is not friendly for reader. The writing is also overly defensive, frequent disclaimers break up the main line of the argument and super seriously hurt the reading flow.

  4. The figures and tables did not help me understand what the authors are trying to show, and many table columns are not explained. The main text is currently only 7 pages, so the authors should use more space to explain what the tables mean and what claims they support.

In summary, the paper addresses a clearly defined problem. However, the experiments lack a thorough comparison with other methods and on their settings, and the core claim is demonstrated only on a single small constructed example. I do not think this is enough to support the effectiveness of the method. The writing and figures also make the paper hard to follow

Questions for authors
  1. The six trained models use the reset and stutter certificate rather than theorem 3. Can theorem 3 be applied to these models? If not, which assumptions are not satisfied or have not been verified?

  2. Could you clarify how the baselines in section 5.2 represent the closest existing verification methods? What do the current comparisons establish about the advantages and limitations of your approach?

  3. I think the question you observed and proposed is worth to discuss, but your writing made the paper very difficult for me to read. Would you be willing to rewrite it to make it easier to follow and provide additional experiments as I mentioned above? Also, please provide more understandable figure and table. I would be happy to reconsider my score if these concerns are addressed.

6 comments
AuthorsOpenReview ↗

Response to Reviewer fA5i (1/2)

Thank you for the detailed review and for saying that the question is worth discussing.

W1 and Q1: can Theorem 3 be applied to the six trained models? It cannot be applied usefully, and Appendix D.1 now shows which premises fail. Theorem 3 needs (a) a retained memory that takes finitely many values and (b) an error that contracts along the token sequences the protocol allows, called legal (Lemma 2).

  • In the trained models' reset protocol, the QUERY token leaves the state unchanged, so repeating it never shrinks the error, and (b) fails.
  • If QUERY is deleted and resets are kept, both premises hold, but only trivially. With at most 12 informative tokens between resets, every legal 13-token word contains a RESET, which restores the initial error and memory. Hence the error matrices along any such word multiply to zero, and the memory takes at most 8191 values. Theorem 3 then adds nothing to Proposition 4, our reset-based certificate.
  • Without resets, both premises provably fail for all six circuits. Repeating one token moves the memory toward a fixed point without repeating a value, so the memory takes infinitely many values. For each circuit, the error matrix of some legal word of at most five tokens has a real eigenvalue below −1-1, so the error does not contract, and no exact reduction removes this expanding direction. The all-time claim at tolerance 0.1 is then false, because each circuit exceeds 0.1 on an exactly replayed periodic input after 22 to 239 tokens.

For these models, the cap on state-changing tokens between resets is therefore essential, and Proposition 4 is the right certificate. All six circuits selected on their own state certify at 0.1 for every cap up to 13 (Fig. 3).

W1: more cases where Theorem 3 applies. We added 48 new generated systems with finite retained memory (§5.3, Appendix D.4), separate from the 48 memory-free systems of §5.2. Their memories take 3, 5 or 9 values, and their omitted dynamics are dense, with dimension 2 to 16. On all 48, the error contracts along legal sequences but not along arbitrary ones, and the whole model–circuit difference system does not contract. These outcomes hold by construction, and Theorem 3 still has no application to a trained model (§7).

  • With a search budget of 60,000 nodes per request, our search procedure (Algorithm 1) finds a certified upper bound and a replayed lower bound within 1% of each other for 30 of the 48 systems, against 4 with 6000 nodes. Systems that allow two error-expanding HOLD tokens in a row contract slowly and keep wider gaps (median 35%).
  • The certified upper bound is a median 11 times tighter than the bound of Lemma 2, which ignores the memory.
  • We also fixed each tolerance in advance at 1% above a precomputed lower bound. With the same budget, 13 of the 48 one-percent requests certify, 9 remain unresolved, and 26 are falsified, meaning that their preset tolerance was below the true risk (Appendix D.4 also gives the 10% counts). The bound of Lemma 2 certifies none of the 96 requests at the two margins.

W2 and Q2: how do the §5.2 baselines relate to existing methods? The three baselines are our re-implementations of the principles behind the closest methods, not the published algorithms. They are (i) the norm bound of Lemma 2, (ii) an invariant box from a linear program, verified exactly, and (iii) an exact sum over a fixed number of recent steps plus a bound on the older remainder. The last is the principle behind minimal robust positively invariant (mRPI) set approximation (Raković et al., 2005) and invariant multisets (Athanasopoulos et al., 2017). Below, "refinement" means our method, the adaptive suffix refinement of Algorithm 1. All three baselines use the same matrix products along legal token sequences, the same stability test (Lemma 2) and the same 6000-node budget as refinement. On the 24 slow–fast systems (one slow mode, the others fast), each run at two tolerance margins and with two boundary sets for the error (a ball and a box), refinement certifies 52 of the 96 resulting requests and no baseline certifies any (Table 5). We did not run the published algorithms, because the mRPI algorithm assumes a single time-invariant system (Table 6) and we found no public implementation of invariant multisets.

AuthorsOpenReview ↗

Response to Reviewer fA5i (2/2)

W2 and Q2, continued: a classical control-theory method. We also added the invariant ellipsoid for switched systems computed from a linear matrix inequality (LMI; Boyd et al., 1994; Abedor, Nagpal & Poolla, 1996). We verified it exactly on the 24 §5.2 systems with n≤32n\le32 (Appendix D.6, Table 10). The LMI is stronger on comparable-slow systems, where all modes are slow: it certifies 10 of 12 ten-percent requests, while refinement certifies none. Refinement is stronger on slow–fast systems at 1%, where it certifies 8 of 12 requests with the box as boundary set (4 with the ball) and the LMI certifies none. Used inside refinement as an extra boundary set, the ellipsoid lets the combination certify every ten-percent and slow–fast request. Of the 12 comparable-slow 1% requests per boundary set, the combination certifies 2 and falsifies 2 whose preset tolerances were below the true risk. The other 8 per boundary set (16 in all) stay unresolved, with upper bounds 1.017 to 1.052 times the lower bounds.

The comparisons also show the limits of our approach. Refinement alone resolves none of the 96 comparable-slow requests (Table 5), and the 48 with n=64n=64 and 128128 remain unresolved, because our LMI implementations did not complete at n=64n=64. We claim no advantage over other certificates for switched systems, such as path-complete Lyapunov functions, multinorms, or exact support and ℓ∞-gain linear programs. Any verified invariant set can be used inside refinement in this way (Appendix A.4), so the paper presents the two methods as complementary. We did not compare with recurrent-network verifiers such as POPQORN, Cert-RNN and DiffRNN, because they bound a fixed number of steps and so cannot certify an all-time claim (§6).

W3: readability and defensive tone. We agree, and we rewrote the abstract, introduction and section openings in plain words, defining each term where it is introduced. For example, Theorem 3 now says what a suffix and a seed are, §5.4 defines its selectors and "cells", and the new Appendix D.7 gives the training and selection protocols. Tables 1 and 2 give the notation and the tokens, and the coupled latch is a running example (§2, Fig. 2). On tone, we removed repeated inline disclaimers, such as "Success is not built in", and moved most caveats to §7.

W4: figures and tables. Every table and figure caption in the main text now says what the item shows and defines its columns. For example, Table 4 (formerly Table 3) defines kk, Local, R∞\mathcal{R}_\infty and "Fails at", and Table 5 (formerly Table 4) defines Switching, Margin, Boundary and the ratio column. Fig. 3 (formerly Fig. 2) plots each model separately, extends past the cap, marks where two circuits first exceed 0.1, and uses larger fonts. The main text uses all 10 pages.

Q3. Yes. The rewrite and the clearer figures and tables are described above, and the three analyses you asked for are in Appendices D.1, D.4 and D.6.

Reviewer fA5i, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: revised PDF and new results

We thank all three reviewers and have uploaded a revised PDF.

As FUnX and fA5i asked, we rewrote the paper for readability. We also added five requested analyses: our main certificate (Theorem 3) on 48 new finite-memory systems (Appendix D.4), why its premises fail for the trained parity models (Appendix D.1), certificates for trained nonlinear circuits (Appendix D.5), a comparison with a classical control-theory method (Appendix D.6), and a proof that finite memory alone does not make all-time faithfulness decidable (Proposition 8, Appendix C). Each reply gives details and locations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.