- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper tackles circuit extraction of recurrent systems. To obtain certificates on the discrepancy between the model and the extracted circuit, an error term is explicitly tracked. As the recurrency could result in a blow-up of the error term, the paper ensures that this stays small by forgetting no-longer relevant error terms.
Critical strengths
- Bounding the error term, especially over infinite horizons, is very useful for safety critical applications.
- The mathematics are rigor; although a Lean formalization would be nice.
Critical weaknesses
- It is not clear to me what kind of input tokens exist and what they do exactly. There are mentions of some such as HOLD and RELAX, but they are never introduced.
- Variables and their meaning could be better introduced, e.g., all variables in (1): etc.
- The paper could benefit from some illustrations and/or a running example
- Some sentences read very AI-generated (overly exhaustive, typical formulations); these could be refined for precision. For example, I find "A recurrent circuit must carry its own state, not receive a corrected state before each short test." an odd sentence to introduce readers into the topic in the abstract.
- Shouldn't Table 2 be in an algorithm environment? If so, use variables as introduces in the paper and pseudo code instead of text.
- Related work in appendix, no comparison against it; also odd to have a proposition in this section
Questions for authors
- Do you have a concrete application in mind where you want to use this method?
3 comments
Reviewer FUnX, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?
Response to Area Chair 1Z2k (1/2)
Thank you for following up with Reviewer FUnX.
The revised PDF addresses each of FUnX's points. The new Table 2 in Section 2 lists the tokens with their effects and protocol restrictions, and Table 1 and the text define the symbols of Equation (1). The coupled latch example now runs through Sections 2, 3 and 5.3, with the new Figure 2. We rewrote the abstract and introduction in plain words, and the old Table 2 procedure is now Algorithm 1 in Section 4.
Related work is now Section 6 of the main text. It explains why verifiers that check recurrent networks for a fixed number of steps cannot certify a claim for all time, and its Table 6 compares assumptions with the closest control-theory methods. The new "Intended use" paragraph in Section 1 answers the application question.
Response to Reviewer FUnX
Thank you for the concrete suggestions. We agree with all six points.
W1: tokens are not introduced. The new Table 2 in §2 lists each token of the latch, the parity models and the §5.2 systems, with its effect and its protocol restriction. Appendix D.4 defines the tokens of the new generated systems of §5.3. HOLD and RELAX keep the latch's stored value and multiply the omitted error by 9/8 and 1/4, respectively. The caption of Figure 1, where they first appear, now points to §2 for this. §2 also defines an input protocol (a graph whose paths are the allowed token sequences) and a legal word (one such sequence).
W2: variables are not introduced. The sentences around Eq. (1) now define its symbols, and Table 1 lists the main symbols with their dimensions, such as Aa∈Rn×n.
W3: running example. The coupled latch is introduced in §2 and reused in §3 and §5.3. For coupling strength g=1/32, the new Figure 2 shows (left) a legal sequence whose discrepancy approaches the exact all-time risk 68g/23≈0.092 and (right) the bounds of our procedure (Algorithm 1) converging to it.
W4: wording. We removed the sentence you quoted, and the abstract now opens by defining a circuit and stating our question. We also rewrote the abstract, introduction and section openings in plain words, defined each term where it first appears, and removed repeated disclaimers.
W5: Table 2 as an algorithm. The submission's Table 2 procedure is now Algorithm 1 in §4, written as pseudocode with the paper's variables.
W6: related work and comparison. Related work is now §6, with an extended version in Appendix B. The submission's Table 1, which compared our assumptions with control-theory methods, is now Table 6 in §6, and the proposition you noted in that section (now Proposition 7) has moved to Appendix C. The recurrent-network verifiers (POPQORN, Prover, Cert-RNN, DiffRNN, RnnVerify) bound only a fixed number of steps, so they cannot rule out later failures, and a head-to-head run would not test the same claim (§6). Circuit-discovery methods complement ours, because they supply the circuits that we verify. When integrated-gradient gating and attribution patching (Syed et al., 2023) keep as many units as our persistent selector (which tests circuits on their own state), they pick exactly its units in all six parity models, so their circuits certify too (§5.1, Appendix D.3). On the §5.2 systems, we also added a classical control-theory bound, which certifies different cases from ours (Appendix D.6).
Formalization. The proofs in Appendices A and C are not machine-checked, and a Lean formalization does not fit in the rebuttal period. All certificates and counterexamples use exact rational or rigorous ball arithmetic, and every counterexample is replayed independently. Floating point only proposes candidates for exact checking, except in the uncertified diagnostics listed at the start of Appendix D and the empirical study of §5.4.
Q1: application. We do not target a specific deployed system. The immediate use is in interpretability research, and §5.1 shows why it matters. In five of the six affine models, a circuit that passes a validation with periodic copying of the model's state still fails when run on its own state. Proposition 4, the certificate we use for the trained models, suits episodic use: the system is reset exactly between episodes, and each episode contains at most a fixed number of informative tokens (tokens other than resets and state-preserving tokens such as QUERY). A certificate supports a safety argument only if the original model is trusted, and only for the declared inputs (new "Intended use" paragraph in §1).