- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies Markov chains whose transition kernels share one stationary distribution and switch hidden among a finite set of such kernels. It splits the problem into estimating the constrained kernel and identifying the active one. It shows that Laplacian smoothing is strictly suboptimal under the constraint, gives a closed form of the constrained Bayes predictor for two states, and shows that the difference lies in the readout rather than the count state. It reports that trained Mamba models close most of the estimation gap, and that the optimal filter keeps a strictly positive residual loss that trained models approach but do not beat. It concludes that provable unpredictability against trained sequence models must rest on hidden switching rather than on estimation difficulty.
Critical strengths
- Clean, correct theory. The two-state closed form and the strict suboptimality for any number of states are correct, and the constructive Gauss–Jacobi readout and its width lower bound are consistent with each other.
- Careful probe design. The three-state probe fixes sequence length and current-row counts, and the state-patching test has untrained-model and orthogonal-direction controls whose weaker results are reported.
- Candid limitations. Conjectures are labeled, the exact-representation result is proved only for two states, and the appendix points out that the two-state experiments cannot separate a cross-row readout from a within-row readout with position.
Critical weaknesses
W1. The design implication ignores an i.i.d. baseline. The kernel whose rows all equal belongs to , and every predictor has expected log loss at least on it. Under any stationary process, predicting while ignoring history already achieves , so the optimal filter under hidden switching has loss . With the E3 parameters () I get nats against – (my own simulation). Within the paper's model, an i.i.d. kernel is therefore at least as hard to predict as any hidden-switching scheme, and the claim that provable unpredictability "must rest on" hidden switching does not follow. The graph-topology constraint in App. C.7 might change this, but it enters no theorem, and (A2) and Proposition 2 exclude sparse supports. The authors should report as a reference and either model the constraint that rules out i.i.d. kernels or present the defense angle only as motivation.
W2. The residual loss does not isolate identification. The oracle knows , so already includes whether a switch occurs at the next step. The proof of Theorem 7 only uses that the predicted belief has full support, which also holds for a predictor that knows exactly, so alone does not show a limit on identification. In my simulation of the E3 setting (two kernels, , ), the optimal filter has at , in line with Figs. 5–6, while a predictor that knows has . Roughly 30% to 70% of the residual loss thus comes from switch-time unpredictability rather than detection delay. The remark after Theorem 7 mentions both sources, but the abstract and Sec. 2.2 call the residual an identification limit. A known- benchmark would let the paper split the residual and place the trained models.
W3. The estimation and identification halves are studied in different settings. The switching analysis of the estimation gap (Theorem 3, Sec. 3.3) assumes the counts restart at each switch, which App. C.6 describes as "known switching times", whereas hidden switching is the premise of the paper. The identification half (Sec. 5, E3) assumes and are known, so the trained models need the strategy kernels in their weights if the strategy set is fixed across sequences, and E3 then contains no estimation problem. No theorem or experiment covers unknown constrained kernels with hidden switches, yet the abstract and conclusion merge the two halves. The paper should state what is resampled per sequence in E3, that Theorem 3 applies to visible switches, and either add a joint experiment or split the conclusion.
W4. The evidence for "closing the estimation gap" and a cross-row readout is mostly at two states. At the gap is 0.0125 nats (0.5273 for add-β vs. about 0.5148 for , Fig. 3(a)), 2.4% of the loss. The upper bound for readouts that use the current row's counts plus position is about 98% (Fig. 4(a)), above every trained model, and App. C.3 itself says the closed fraction cannot tell the two readouts apart. For , App. C.5 reports that trained models at are worse than add-β and that is only slightly above zero at . The cross-row probes use one model family (full Mamba, , one length-12 family, five seeds; patching slope 0.86 with 95% interval 0.64–1.08, and orthogonal perturbations have effects of similar size). The paper should report, for , the trained models' loss relative to the best within-row-plus-position predictor and to with seed intervals, include Gated M-2 in the probes, and scope the abstract to the settings tested.
Questions for authors
- Were the four architectures given comparable tuning? They use different optimizers, learning rates and widths under one 8000-step budget, and the Transformer is still improving at the end of training (Fig. 3(b)), so it is unclear whether the differences between them (e.g., 95% vs. 40% closed) reflect architecture or training.
- The appendix numbering is inconsistent (Theorem B.1 appears as Theorem B.2 in later references, and Corollary B.2 is used twice). Which statements are intended by each label?
- Should the Limitations paragraph state that the switching analysis assumes known and and, for the estimation gap, visible switches?
11 comments
Response to Reviewer ocx3 (2/4)
A1.2 (response to Q1, part 2)
Comparison between architectures. At 128000 steps, we computed paired bootstrap intervals for the differences in closed fraction between the six pairs of architectures in each group. This gives 12 intervals over the two groups, each taken at the 99.58% level by the Bonferroni correction. The difference between Gated M-2 and full Mamba is not significant in either group. Gated M-2 is about 1.2 percentage points higher than the Transformer, and the intervals in both groups exclude zero. The difference between full Mamba and the Transformer is not significant in the equal-width group, and in the equal-parameter group the lower end of its interval is slightly above zero. The MambaZero variant is 44 to 47 percentage points below the other three architectures. Therefore, with sufficient training, Gated M-2, full Mamba and the Transformer differ by only about 1 percentage point, whereas the shortfall of the MambaZero variant does not disappear with training.
Cross-row readout. At 128000 steps, the 95% intervals of the closed fractions of Gated M-2, full Mamba and the Transformer all lie above the upper bound for within-row readouts with position, which indicates that the readouts of these three architectures use cross-row information. Appendix C.3 of the initial submission notes that, with two states, the closed fraction cannot distinguish a cross-row readout from a within-row readout with position. The evidence that ruled out the positional explanation therefore came only from Probes 4 and 5 with three states (Section 4.1). The results under the unified training protocol add two-state evidence for this conclusion. The MambaZero variant stays between the two bounds, consistent with the capability of its normalized readout.
Effect on the conclusions. Theorems 1 to 7 and Propositions 1 and 2 concern the Bayes-optimal predictor, the representational capacity of readouts and the statistical limits. They do not involve training and are not affected by the training settings. The training settings affect only the interpretation of the differences between architectures in E2. Under the unified training protocol the Transformer also closes most of the estimation gap, which strengthens our conclusion that sufficiently trained models can close the estimation gap.
Revision. Section 3.4 has been revised to state that this difference partly comes from training and to give the results under the unified training protocol. Section 3.4 and Figure 3 still report the E2 results under the original settings. The new Appendix C.7 and Figure C.10 report all results under the unified training protocol and the comparison between architectures, and the settings of this protocol have been added to Appendix D.2. The statement on the two-state closed fraction in Appendix C.3 has been revised accordingly. These changes are shown in red in the revised version.
Response to Reviewer ocx3 (3/4)
A2 (response to Q2)
Thank you for your careful reading. The numbering in Appendices B.2 and A.12 of the initial submission is indeed wrong. We have thoroughly checked the whole paper and standardized the format of the appendices. In the revised version, the statements in the appendices have been renumbered by subsection, each proof is preceded by the corresponding statement, and labels, statements and proofs are in one-to-one correspondence.
Statements intended by each label in the initial submission. In the initial submission, Appendix B.2 contains one theorem and two corollaries, which are, in order, Theorem B.1, Corollary B.1 and Corollary B.2. “Theorem B.2” occurs in five places, namely the statements of Corollaries B.1 and B.2, the heading of Appendix A.12, the label of the proof of the theorem and the proof of Corollary B.2. All five refer to Theorem B.1. Appendix A.12 gives the proofs of the three statements in order. The first “Corollary B.2” in its heading refers to Corollary B.1, and the second refers to Corollary B.2. Likewise, the first proof labeled “Corollary B.2” is the proof of Corollary B.1, and the second is the proof of Corollary B.2.
Other numbering errors. While checking the labels, we also found problems in the initial submission such as Proposition B.1 being called Proposition B.6 elsewhere.
Revision. In the revised version, the statements in the appendices have been renumbered by subsection, consecutively within each subsection. For example, the three statements of Appendix B.2 have been renumbered as Theorem B.2.1, Corollary B.2.2 and Corollary B.2.3. The proofs of the results in Appendix B have been moved from Appendix A to the corresponding subsections of Appendix B. Each label in the revised appendices refers to exactly one statement, and every reference to these labels has been checked. The table below lists, for each label in the appendices of the initial submission, the corresponding label in the revised version.
| Initial submission | Title | Revised version |
|---|---|---|
| Theorem B.1 | Suffix-class decomposition | Theorem B.2.1 |
| Corollary B.1 | Class-wise closed form | Corollary B.2.2 |
| Corollary B.2 | Strict suboptimality of order- add- for | Corollary B.2.3 |
| Proposition B.1 | Sufficiency of cross-row readouts | Proposition B.5.1 |
| Lemma A.0.1 | Loss decomposition | Lemma A.0.1 |
| Lemma A.0.2 | One-dimensional likelihood-ratio order, strict version | Lemma A.0.2 |
| Lemma A.2.1 | Maximum and properties of the maximizer | Lemma A.1.1 |
| Corollary A.2.2 | Quantitative inclusion | Corollary A.1.2 |
| Lemma A.2.3 | Posterior concentration | Lemma A.1.3 |
Response to Reviewer ocx3 (1/4)
A1.1 (response to Q1, part 1)
Thank you for raising this question. The four architectures were not given comparable tuning in the initial submission, and a considerable part of the difference between 95% and 40% came from training. In the revised version, we have retrained the four architectures under a unified training protocol (new Appendix C.7 and Figure C.10). The results show that the lower closed fractions of full Mamba and the Transformer in the initial submission came mainly from the training settings, whereas the shortfall of the MambaZero variant comes from its architecture.
Original training settings of E2 in the initial submission. In the initial submission, E2 used a separate setting for each architecture (Appendix D.2). The MambaZero variant and Gated M-2 used d=8 and Adam with a constant learning rate. Full Mamba used d=8 and the Transformer used d=16, and both used AdamW with learning rate 10−3 and cosine decay. All four architectures were trained for only 8000 steps, and their learning rates were not selected on a common grid.
Unified training protocol added in the revised version. The four architectures use the same optimizer and learning-rate schedule, namely AdamW with a cosine schedule decaying to 0, and batch size 64. The learning rate is selected from a common grid by the mean validation loss over 10 seeds on a separate validation set, and further values outside the grid are tried when the selected value lies at an endpoint. The training budgets are 8000, 16000, 32000, 64000 and 128000 steps, and each budget is trained from scratch with 10 seeds. The widths are matched in two ways. In the equal-width group all four architectures use d=16, and in the equal-parameter group the parameter counts of the architectures lie between 8266 and 8608. The evaluation set is the same as in the original E2.
Results under the unified training protocol. At the same 8000 steps, the closed fraction of the Transformer rises from 41.1% under the original setting to 75.2%, and that of full Mamba rises from 86.7% to 98.9% (equal-width group). The learning rate selected for both is then 3×10−3, higher than the 10−3 of the original setting. The closed fractions of Gated M-2 and full Mamba exceed 97% at every training budget. The closed fraction of the Transformer keeps rising with the number of training steps and reaches 98.6% at 128000 steps, within about 1 percentage point of Gated M-2 and full Mamba. In both groups and at every training budget, the MambaZero variant stays at 47.1% to 54.3%, between the two upper bounds for within-row readouts in Theorem 4, fF1 (42.2%) and fF2 (97.2%).