acceptodds
Under review as a conference paper at ICLR 2027

What Algorithms Do Pre-trained LLMs Use for HMM Prediction?

Abstract

Large language models (LLMs) exhibit a striking ability to predict observations generated by Hidden Markov Models (HMMs) through in-context learning (ICL), yet the algorithms underlying this capability remain unclear. We investigate this question through a three-stage pipeline combining behavioral comparisons, constructive Transformer theory, and causal analysis of internal representations. First, we compare LLM behavior with a suite of candidate algorithms, including classical inference methods and algorithms motivated by empirical observations. Second, we establish theoretical connections among these algorithmic classes, construct in-context Transformer implementations, and validate a finite-window mechanism in a small trained Transformer. Third, we probe pre-trained LLMs for internal representations corresponding to the candidate algorithms. We identify low-dimensional linear representations that causally influence model predictions and whose strength tracks empirical ICL performance. These representations vary systematically across HMM regimes, with finite-window, gradient-descent-like computations predominating in most regimes. Together, our results connect the in-context behavior of LLMs to their internal algorithmic representations, providing a foundation for understanding how LLMs predict sequences governed by latent dynamics.

Primary area:
interpretability and explainable AI
Keywords:
Hidden Markov models, Pre-trained LLMs, In-context learning, Interpretability, Transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AoQDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the author investigated the underlying mechanism that enables the large language models to make prediction for hidden Markov models. First, the authors conducted empirical studies to identify three classical models, namely linear n-gram, Non-linear n-gram, and Spectral methods to be those that can approximate LLM better than others. Then, the authors show that indeed the LLM can implement them, finally, they conducted detailed layer-wise probing-and-intervention analysis aimed at identify causal relationship between low -dimensional algorithmic models and LLM's prediction. The overall conclusion suggest that LLM utilizes different models dynamically, although gradient-descent-like ones seem to play more important roles.

Critical strengths

The authors conducted extensive empirical analysis on different aspects of the problem. Especially the layer-wise probing-and-intervention analysis is quite insightful and of considerable impact.

The theoretical analysis, while not surprising, is also new and valuable to the community.

Critical weaknesses

My main concerns of the paper is that it is not conclusive. While many of the discoveries that have been discussed in the paper are interesting and beneficial to our understanding of LLM's predictive capability with respect to HHM, they do not provide any affirmative answer to the problem the authors raised at the beginning of the paper. It felt that while the paper has considerable potential, some work towards conclusions is still required.

Questions for authors

Q1: In the first stage of the studies, there seems to be no details on how exactly (π,A,B)(\pi, A, B) are generated, could you provide related information?

Q2: the argument for utilizing the coefficient of determination is rather weak, it would be great that references or some theoretical background can be provided.

Q3: The paragraph between L350-354, does not provide enough information on why it should be considered as a contribution. It now suggests that each of the individual mechanism is standard, and the contribution is on how they are applied.

2 comments

Thank you for your thoughtful review. We address your concerns and questions below.

Q1: In the first stage, how exactly are the HMMs generated?

Thank you for pointing this out. The generation code is included in the supplementary material (hmm.py). We fix M=4M=4 hidden states and vary E(A),E(B)∈{0,0.25,0.5,0.75,1}E(A),E(B) \in \{0,0.25,0.5,0.75,1\} (entropies normalized by MM and NN) and N∈2,4,8N \in {2,4,8}, giving 75 configurations.

  • Transition matrix. Aii=pA_{ii}=p and Aij=(1−p)/(M−1)A_{ij}=(1-p)/(M-1) otherwise, with pp set so the row entropy matches E(A)E(A). Its stationary distribution is uniform, and λ2=p−(1−p)/(M−1)\lambda_2=p-(1-p)/(M-1), so each E(A)E(A) induces a single mixing rate (λ2≈1,0.90,0.75,0.53,0)(\lambda_2\approx 1,0.90,0.75,0.53,0).
  • Emission matrix. Row ss of BB peaks at symbol s mod Ns \bmod N, with the remaining mass spread uniformly and the peak set by E(B)E(B).
  • Sampling and tokenization. We sample 4,096 sequences of 4,097 observations from a uniform initial distribution. Observations are encoded as single uppercase-letter tokens and fed without prompt or separators. At each position, we restrict the next-token logits to the NN observation tokens and renormalize.

We will add these details to Appendix A.

Q2: The argument for utilizing the coefficient of determination is rather weak; references or theoretical background would help.

Here is some background on the coefficient of determination. It is attributed to Wright [1], as the proportion of variance in one variable explained by others. In regression, it is defined as R2=1−SSres/SStotR^2 =1-SS_{res}/SS_{tot}, which in our notation is 1−MSE/Var(rt)1-MSE/Var(r_t). This is the form Kvalseth [2] identifies as the most common, and it can be negative on held-out data. It is also a standard metric for evaluating linear regression probes of neural representations [3].

We use it because normalizing by the target variance makes it comparable across candidate representations with different scales: 0 corresponds to predicting the mean and 1 to perfect recovery. Under label shuffling, held-out R2R^2 drops to ≈0\approx 0 (Appendix G.1), so high values reflect real alignment between activations and the target.

Q3: The paragraph at L350–354 does not explain why this is a contribution; it suggests each mechanism is standard and the contribution is only in how they are applied.

Our wording undersold the contribution. The individual tools are standard, but we combine them to address an open problem: discovering which algorithm a model implements when that algorithm is not known in advance.

Prior mechanistic work studies tasks where the optimal algorithm is known a priori, such as induction heads [5], modular addition [6], permutation tracking [7], and Markov-chain ICL [8]. There, the goal is to verify that the model implements a known algorithm. HMM prediction is different. The Bayes-optimal forward algorithm requires the unknown parameters, and multiple learning algorithms (spectral methods, finite-window regression, parameter estimation) all converge toward it with different inductive biases. The model is also pretrained rather than trained on the task, so there is no reason to expect it to implement any one of these exactly. Identifying which candidate the model implements therefore requires combining evidence: behavioral comparison to narrow the candidates, constructions to establish that each is implementable, and causal interventions to discriminate between them.


[1] Wright, Sewall. "Correlation and causation." Journal of Agricultural Research 20 (1921): 557–585.

[2] Kvalseth, Tarald O. "Cautionary note about R2R^2." The American Statistician 39.4 (1985): 279–285.

[3] Gurnee, Wes, and Max Tegmark. "Language models represent space and time." International Conference on Learning Representations (2024).

[4] Belinkov, Yonatan. "Probing classifiers: Promises, shortcomings, and advances." Computational Linguistics 48.1 (2022): 207–219.

[5] Elhage, Nelson, et al. "A mathematical framework for transformer circuits." Transformer Circuits Thread (2021).

[6] Nanda, Neel, et al. "Progress measures for grokking via mechanistic interpretability." International Conference on Learning Representations (2023).

[7] Li, Belinda Z., Zifan Carl Guo, and Jacob Andreas. "(How) do language models track state?" International Conference on Machine Learning (2025).

[8] Edelman, Ezra, et al. "The evolution of statistical induction heads: In-context learning Markov chains." Advances in Neural Information Processing Systems 37 (2024).

Thank you for your response, I will maintain my score.

Reviewer_rQzfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which algorithms pre-trained LLMs use when they predict sequences generated by Hidden Markov Models (HMMs) in context. It follows a three-stage pipeline. First, it compares several open-weight LLMs against classical predictors, like the forward-algorithm oracle, spectral methods, and n-gram regressors, across many synthetic HMM configurations. Second, it shows that the next-observation distribution can be written as a linear map of Kronecker features of recent observations. It then gives Transformer constructions for gradient descent on these features and for spectral learning. A small Transformer trained on a single HMM is also shown to learn a finite-window predictor. Third, it introduces the Principal Activations Probe (PAP), which combines PCA, ridge probing, and activation patching, and applies it to several LLMs. Interventions on the linear n-gram representation generally have the strongest causal effect. The paper concludes that this favors finite-window computation as an account of in-context prediction in LLMs.

Critical strengths
  • Combining behavioral, constructive, and causal evidence is a reasonable way to study which algorithm underlies in-context HMM prediction.

  • The PAP analysis gives a helpful illustration that decodability alone does not imply causal use.

  • The small-Transformer study is a useful controlled test that compares the learned readout to the optimal finite-window readout.

  • The paper repeats the intervention analysis across several model families and sizes, which helps support the generality of the finding.

Critical weaknesses
  • The paper is hard to follow because much of the evidence is placed in the appendix. The submission is 78 pages, and about 65 of them are appendix. Some of the main claims in the abstract are supported mainly by appendix results. This makes the paper hard to assess from the main body, and the paper relies on the appendix far more than a typical submission.

  • The theoretical contribution is limited. Several constructions combine existing results. For example, the spectral construction is assembled from prior Transformer constructions for trigram estimation, bigram-matrix inversion, and binary-tree composition. Some proof steps rely on informal assumptions, such as attention with an MLP forming a tensor product, or on idealized hard attention. Separately, the claimed exponential decay of the finite-window approximation error under mixing is stated but not proven.

  • The choice of candidate algorithms is qualitative. Stage 1 selects the candidates through visual similarity of context-length curves. The paper itself notes that the n-gram and spectral baselines each resemble LLMs in some regimes but diverge in others. Apart from the oracle reference, the intervention analysis only covers variants of these n-gram and spectral families. It is thus unclear whether candidates outside them would change the ranking that supports the finite-window conclusion (as the paper partly acknowledges). In addition, source-target pairs are selected using each candidate's own states, so the evaluated pairs can differ across candidates, which makes the ranking less direct.

Questions for authors
  • Is there a null control for the interventions, similar to the label-shuffling test used for probing?

  • Is the ranking among the linear n-gram, spectral, and bigram candidates stable across window sizes n?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rQzf (1/2)

Thank you for the careful review and constructive feedback. We appreciate your recognition of our approach. We address your questions and concerns below.

Q1: Is there a null control for the interventions, similar to the label-shuffling test used for probing?

Thank you for this suggestion. We did not originally include a separate null control because the intervention effects are already selective rather than uniform. IIA is high in late layers and at longer context lengths (Figs. 8, 58–60), Bigram is effective only in the observation-dominated regime (Fig. 11), and models that do not behave like the candidate algorithms show no effect despite high-R² probes (OLMo2-1B, Appendix G.3).

Following your suggestion, we have now added an explicit null control on Qwen3-1.7B (M=N=4M=N=4, E(A)=0.25\mathcal{E}(A)=0.25, E(B)=0.75\mathcal{E}(B)=0.75, Oracle belief, k=8k=8, 1,024 pairs per cell). We refit the ridge probe on row-permuted belief targets, apply the probe-inverse patch toward the same source beliefs, and rescale each shift to the norm of the corresponding real shift. Null values are averaged over two permutation seeds.

Window (pos ≥ 256)Probe R² (real / null)IIA (real / null)KL-Eff (real / null)
Layers 0–190.649 / −0.0080.296 / 0.0090.571 / 0.100
Layers 20–270.815 / −0.0080.824 / 0.0560.816 / −0.158
Layers 22–260.833 / −0.0091.000 / 0.0740.966 / −0.258

A shuffled probe with norm-matched shifts barely moves the prediction: null IIA stays below 0.08 in every window. In layers 20–27, the null's negative KL-Eff shows that a random push of the same size moves the output distribution away from the target. The causal effect therefore comes from the directions the probe identifies. We will add this control, with per-layer and per-position results, to Appendix G.

Q2: Is the ranking among the linear n-gram, spectral, and bigram candidates stable across window sizes n?

Among the three candidates, only Linear nn-gram has a window parameter. Bigram corresponds to a fixed window of one observation, and Spectral uses fixed uni-, bi-, and trigram moment statistics. The question is therefore whether Linear nn-gram remains the most causally effective representation as nn varies. Our main PAP results use n=4n=4, motivated by the behavioral window sweeps (Appendix F.4). We have now run PAP on Linear nn-gram for n∈{3,…,7}n \in \{3, \dots, 7\} (Qwen3-1.7B, M=N=4M=N=4, E(A)=0.25\mathcal{E}(A)=0.25, E(B)=0.75\mathcal{E}(B)=0.75, k=8k=8, 3×3 max-pooled as in Tables 5 and 7):

RepresentationProbe R²IIAKL-Eff
Bigram0.8900.5420.550
Spectral0.3850.6380.599
Linear nn-gram, n=3n=30.7940.9970.954
Linear nn-gram, n=4n=4 (main text)0.8040.9920.971
Linear nn-gram, n=5n=50.8100.9920.968
Linear nn-gram, n=6n=60.7980.9890.963
Linear nn-gram, n=7n=70.7820.9810.955

The ranking is stable: Linear nn-gram outperforms Bigram and Spectral in both IIA and KL-Eff at every window size. The effect is strongest for n∈{3,4,5}n \in \{3, 4, 5\} and declines only slightly for larger nn. We will add this sweep to Appendix G.

W1: The paper relies heavily on the appendix, and some main claims are supported mainly by appendix results.

Thank you for this feedback. Most of the appendix consists of full-grid figures, which we include for completeness rather than to support additional claims. We will make the main text more self-contained by adding compact summaries of the cross-model and cross-size PAP results (Tables 6–8) and the robustness checks from this rebuttal. We will also add a short guide at the start of the appendix that maps each claim to its supporting results.

AuthorsOpenReview ↗

Response to Reviewer rQzf (2/2)

W2: The theoretical contribution is limited; some proof steps rely on informal assumptions, and the exponential decay of the finite-window error is not proven.

Thank you for the careful reading. Our theory aims to show that each candidate is implementable in-context and to define the representations PAP probes. Prior work often treats a single construction as its main contribution, e.g., induction heads for Markov-chain ICL [1, 2]. We build on such results and add new elements, such as the unifying formulation in Lemma 3.1, which defines our PAP targets and is confirmed in the small trained Transformer (Appendix E.4).

  • Combining existing results. Please note that, we do not claim new algorithms for trigram estimation, matrix inversion, or binary-tree composition. Indeed, Remark 3.3 in Section 3 explicitly identifies the spectral realization as a combination of existing results, and Appendix D attributes its constituent constructions to prior work. The purpose of Remark 3.3 is to include spectral prediction among the computationally realizable hypotheses examined in our study.

  • Tensor product. We want to clarify that the “attention with an MLP forming a tensor product” is not an assumption. This is explicitly proved by Lemma C.3 for the one-hot inputs used in our construction: Attention copies two one-hot vectors into one token, and a single ReLU layer maps them exactly to their tensor product. Since tensor products of one-hot vectors are again one-hot, the same identity applies at every level of the binary tree. We will replace "assuming that" in the proof of Lemma C.6 with a direct invocation of Lemma C.3.

  • Hard attention. This is a standard idealization in Markov-chain ICL constructions, which scale softmax logits to infinity [1, 2]. In our construction (eq. 29), the matching score is exactly nn for matching histories and at most n−1n-1 otherwise, so the same limit recovers hard attention. The construction holds only approximately at finite scale, which we will discuss in Appendix C.

  • Exponential decay. This follows from the classical exponential forgetting of HMM filters [3, 4]. Lemma 3.1 gives qt+1=Gt,nϕt(n)q_{t+1} = \mathcal{G}_{t,n} \phi_t^{(n)}, where Gt,n\mathcal{G}_{t,n} depends on bt−nb_{t-n} (eq. 10). Replacing bt−nb_{t-n} with the stationary distribution μ\mu gives a fixed matrix Gn\mathcal{G}_n and qˉt+1:=Gnϕt(n)\bar{q}_{t+1} := \mathcal{G}_n \phi_t^{(n)}. If AA is strictly positive and BB has no zero columns, there exist C≥1C \geq 1, ρ∈(0,1)\rho \in (0,1) such that for all t≥n>1t \geq n > 1, ∥qˉt+1−qt+1∥1≤Cρn.\Vert \bar{q}_{t+1} - q_{t+1} \Vert_{1} \leq C \rho^n. We will state these assumptions and cite [3, 4] after Lemma 3.1.

W3: The choice of candidate algorithms is qualitative, limited to n-gram and spectral families, and the evaluated pairs differ across candidates.

Thank you for raising these points. To quantify behavioral similarity, we report the absolute top-1 accuracy gap between each algorithm and the LLM, integrated over log⁡2\log_2 context length (4 to 4096) and normalized by the log range (Qwen3-4B; 27 configurations with M=4M=4, N∈{2,4,8}N \in \{2,4,8\}, E(A),E(B)∈{0.25,0.5,0.75}\mathcal{E}(A), \mathcal{E}(B) \in \{0.25, 0.5, 0.75\}; mean ± s.e.m.; lower is more similar):

AlgorithmAUC |Δ accuracy|
Linear nn-gram0.028 ± 0.002
Non-linear nn-gram0.025 ± 0.002
Spectral0.029 ± 0.003
Bigram0.030 ± 0.002
Baum–Welch0.051 ± 0.002
Oracle0.060 ± 0.003

Baum–Welch and the Oracle are about twice as far from the LLM as every in-context candidate: both are accurate from the shortest contexts, whereas the LLM improves gradually. Baum–Welch thus behaves like an explicit parameter estimator rather than an in-context learner, so we do not treat it as a candidate mechanism.

Our candidates cover the established families for HMM prediction (Bayesian filtering, parameter estimation, method-of-moments, finite-window regression), and Appendix G.12 probes nine algorithms across them. We agree the analysis is limited to known algorithms and will state this in the limitations.

Finally, the evaluated pairs are identical across candidates. At each position, we draw 50,000 random source–target pairs and keep the 1,024 with the largest L2L_2 distance between their Oracle beliefs, a criterion independent of any candidate. All candidates use this same pair set and patching procedure; only the target prediction and IIA criterion differ. We will clarify this in Section 4.2.

AuthorsOpenReview ↗

Response to Reviewer rQzf: Reference


[1] Rajaraman, Nived, et al. "Transformers on Markov data: Constant depth suffices." Advances in Neural Information Processing Systems 37 (2024): 137521–137556.

[2] Edelman, Ezra, et al. "The evolution of statistical induction heads: In-context learning Markov chains." Advances in Neural Information Processing Systems 37 (2024).

[3] Cappé, Olivier, Eric Moulines, and Tobias Rydén. Inference in Hidden Markov Models. New York, NY: Springer, 2005.

[4] Le Gland, François, and Laurent Mevel. "Exponential forgetting and geometric ergodicity in hidden Markov models." Mathematics of Control, Signals and Systems 13.1 (2000): 63–93.

Thank you for the detailed response. The rebuttal has resolved my concerns. I also appreciate the planned changes to make the main text more self-contained. I will maintain my positive score.

Reviewer_2sYNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the in-context learning capabilities of LLMs through a three-stage research framework that integrates behavioral comparisons, constructible Transformer theory, and causal analysis of internal representations. In the first stage, the authors compare the behavior of large language models against a range of candidate algorithms, including classical reasoning methods and algorithms inspired by empirical observations. In the second stage, they establish theoretical connections between different classes of algorithms. In the third stage, they probe pretrained LLMs to identify internal representations corresponding to the candidate algorithms. Furthermore, the paper identifies a class of low-dimensional linear representations that causally influence model predictions, with the strength of these representations varying in tandem with the empirically observed performance of in-context learning.

Critical strengths
  1. The paper propose three dim: behavioral, constructive, and internal evidence, and is evaluated against the same set of candidate algorithms, which provides a strong body of empirical evidence.

  2. The authors employs interventions at three levels of granularity along with label-shuffling controls, strengthens the credibility of the experimental findings.

  3. The paper presents an extensive experimental evaluation, covering 12 models across 75 configurations and three dimensions.

Critical weaknesses
  1. The probed soft n-gram states are fitted to prefixes using the candidate algorithms, but the three candidate targets are not matched in terms of dimensionality or learnability. Therefore, it remains unclear how much of the observed causal advantage stems from the model genuinely relying on a finite context window or the target simply being easier to fit.

  2. The detailed results are largely limited to Qwen3-1.7B, while cross-model comparisons cover only a small number of configurations.

Questions for authors
  1. How sensitive are the intervention results to the choice of evaluation metrics and subspaces?

  2. Could the authors provide cross-model evaluation results across a broader range of configurations?

2 comments

Thank you for your insightful and helpful feedback. We respond to your questions and concerns below.

Q1: How sensitive are the intervention results to the choice of evaluation metrics and subspaces?

We checked the robustness of the intervention results, and the main conclusion holds across both metrics and subspaces.

For the choice of metrics, besides IIA, which checks the argmax prediction, we report KL-Eff, a distributional metric measuring how much of the KL gap between the unpatched model and the target distribution is closed by the patch (Appendix G.8). Both metrics identify soft n-gram as the most causally effective representation in all nine configurations (Table 5).

For the subspaces, we compare three patches of increasing specificity (full residual, PCA-subspace, probe-inverse) across all entropy configurations and N∈{2,4,8}N \in \{2, 4, 8\} (Figs. 8, 58–60), and the qualitative layer–position profile of IIA is consistent across them. For the PCA-subspace and probe-inverse patches, results are insensitive to ridge strength (mean IIA 0.94–0.95 for α≤107\alpha \leq 10^7; Table 4) and stable at small PCA ranks (0.91–0.97 for k∈{4,8,16}k \in \{4, 8, 16\}; Table 3).

We will add a compact summary of these robustness results to the main text to make them easier to find.

Q2/ W2: Could the authors provide cross-model evaluation results across a broader range of configurations?

Cross-model results are in Appendix G.9 and G.10. They cover all nine combinations of E(A),E(B)∈{0.25,0.5,0.75}E(A), E(B) \in \{0.25, 0.5, 0.75\} at M=N=4M = N = 4, which is the full non-degenerate entropy range. We report behavioral performance (Table 6) and PAP results (Table 7 and 8) for Llama3.2-3B, OLMo2-1B, and Gemma3-1B, and Qwen3 at 0.6B, 1.7B, and 4B.

The patterns are consistent across this sweep. Soft nn-gram is the most causally effective representation for all Qwen3 sizes in every configuration. Llama, which performs similarly to Qwen, shows the same pattern. OLMo and Gemma perform worse behaviorally and show much weaker intervention effects. The behavioral comparison itself spans all 75 configurations for 12 models (Appendix F).

We will add a pointer to these results in the main text.

W1: The candidate targets are not matched in dimensionality or learnability, so the causal advantage of soft n-gram may reflect a target that is easier to fit.

Thank you for raising this important point. We address dimensionality and learnability separately.

On dimensionality, each candidate’s probed belief is either its next-token distribution in RN\mathbb R^N or its hidden-state belief in RM\mathbb R^M. In all cases, it is decoded with the same probe rank (e.g., k=8k = 8) and patched through a subspace of the same dimension. In our main setting, M=N=4M = N = 4, so all candidate targets also share the same dimension.

On learnability, ease of fitting does not explain the causal advantage. As noted in the paper, a successful probe establishes decodability but not causal use (lines 418–420). In Table 7 (Qwen3-1.7B), Bigram has a higher probe R2R^2 than Soft nn-gram in all nine configurations, making it the easiest target to fit. Yet Soft nn-gram has a higher IIA in all nine.

The finite-window conclusion also rests on other evidence. The window-patching experiments (Appendix G.15) show that overwriting only the preceding 4–6 positions is often enough to transfer the source prediction. The small trained Transformer also attends to a window of about 6 recent tokens (Fig. 22).

Reviewer_2sYNOpenReview ↗

Response to the Authors' Rebuttal

Thank you for your rebuttal and for addressing my concerns. After considering your responses, I have decided to maintain my original score.

Comments on the paper

1 comment

We thank all reviewers and the AC for their thoughtful feedback and constructive suggestions, which have helped us strengthen both the empirical evidence and the presentation of our contributions.

Problem and contributions. Our work investigates a fundamental question: what algorithm do pre-trained LLMs implement when predicting Hidden Markov Model (HMM) sequences in context? More broadly, we address a central challenge in mechanistic interpretability: identifying an algorithm implemented by a model without assuming the correct algorithm is known in advance. Unlike prior studies of Markov chain prediction, where a well-defined optimal algorithm provides a natural mechanistic target, our setting admits multiple candidate algorithms that approach Bayes-optimal prediction through different inductive biases.

To distinguish among these candidates, we develop a three-stage framework that progressively combines (1) behavioral comparisons across 12 LLMs and 75 HMM configurations, (2) theoretical constructions showing that a Transformer can implement each candidate algorithm in context, and (3) causal interventions using our Principal Activations Probe (PAP). Together, these stages provide complementary evidence that pre-trained LLMs predominantly rely on finite-window computation, with the linear nn-gram representation exerting the strongest causal influence on predictions. Moreover, the causal strength of this representation tracks in-context prediction performance across model families and scales. Beyond HMMs, our framework offers a general methodology for algorithm discovery in deep models.

Clarifications in response to reviewer feedback. Most concerns were addressed by existing results in the paper or by clarifying our presentation:

  • Robustness and generality of the causal results (2sYN, rQzf): In response to questions about sensitivity to methodological choices and model selection, we highlight existing appendix analyses showing that candidate rankings remain stable across causal metrics, intervention subspaces, and hyperparameters, and that the main findings generalize across Qwen3, Llama, OLMo, and Gemma. Importantly, these analyses also demonstrate that causal effectiveness cannot be explained simply by ease of decoding.
  • Theoretical rigor (rQzf): We clarify the theoretical justification for the questioned proof steps, including a formal result already established in the appendix and the use of standard idealizations from prior Markov-chain ICL constructions. We also explain how the finite-window error decay follows from classical exponential forgetting results for HMM filtering.
  • Presentation and methodological details (AoQD): We direct the reviewer to the HMM generation code in the supplementary material, provide additional background on the R2R^2 probing metric, and clarify the intended contribution of L350–354.

New experiments motivated by reviewer suggestions. We conducted further analyses:

  • Null intervention control (rQzf): Shuffled patches have negligible effects on predictions (IIA < 0.08), compared with IIA values reaching 1.00 for the corresponding genuine interventions. This supports the conclusion that the observed causal effects arise from the identified representation directions rather than nonspecific perturbations.
  • Window-size robustness (rQzf): Linear nn-gram remains the most causally effective representation at every window size from 3 to 7.
  • Quantitative candidate selection (rQzf): We report an accuracy-gap AUC to quantify behavioral agreement. Baum–Welch and the Oracle exhibit approximately twice the discrepancy from LLM behavior as any in-context candidate algorithm, supporting the exclusion of explicit parameter estimators as candidate mechanisms.

We appreciate the reviewers' and AC's engagement with our work. We believe the additional analyses and clarifications reinforce our central findings and support the broader applicability of our framework to mechanistic algorithm discovery without a prespecified ground-truth algorithm.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.