acceptodds
Under review as a conference paper at ICLR 2027

Training Selects Among Behaviorally Equivalent Computations in In-Context Dictionary Learning

Abstract

Behavioral agreement with a known algorithm does not show which computation a network implements. We make this question exact in in-context complete dictionary learning, where a global inverse, a support-masked inverse, and a support-gated local decoder are provably identical on every training query and differ only beyond the sparse set. A sparse-trained Transformer does not learn the global inverse: fed queries denser than its training set it returns sparse-shaped outputs, and its derivative on held-out directions lies nearest the gated-local decoder, the minimum-norm endpoint of the equivalent class. Training design, not the task, sets where in that class the network lands. At matched sparse-task accuracy, removing the auxiliary support losses of the standard objective moves the learned map toward the support-masked inverse and widening the training query support moves it toward the global one. One support step or a 0.5% dense-query admixture removes most of the sparse-to-dense performance gap; the admixture acts within hundreds of steps on a trained network and reverts once withdrawn, while the derivative on sparse queries keeps its position in the class. As dictionaries grow, errors concentrate in selecting the support rather than in decoding on it. When supervised behavior leaves the computation unidentified, training design selects the extension.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, in-context learning, dictionary learning, sparse coding, identifiability, sparse autoencoders

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_33YdOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates which computational extension a Transformer learns when multiple candidate mappings agree on every sparse training query. It constructs three reference mappings:a global inverse, a support-masked inverse, and a gated-local decoder. and then distinguishes them using Jacobians, held-out directional responses, dense queries, and interventions on training.

Critical strengths
  • The experimental setting makes computational ambiguity directly testable. The authors first establish that the candidate mappings are equivalent on the supervised sparse query set, then construct probes on which they disagree. This gives a examining about matching an algorithm’s training-distribution behavior

  • The paper provides an informative evidence chain. Changes to auxiliary losses, query support size, and dense-query admixtures affect the learned extension in distinguishable ways. The finding that local derivatives and more distant behavior can change separately is informative

Critical weaknesses
  • Being closest to the gated-local reference does not show that the network implements that mapping. Table 8 reports substantial residuals outside the reference family, but they do not fully characterize the learned computation.

  • The central causal result is partly expected, and the theory does not explain it.

Questions for authors
  • When the residual outside the reference family is substantial, which identification claims remain justified?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 33Yd

Thank you for the positive review and for noting that the derivative at training queries and the behavior farther away can change separately. That separation is central to our claims. The task is also a harder form of in-context learning than in-context regression. Our context has no labels, and we prove that the single-layer linear-attention models studied by in-context regression theory stay wrong on some dictionaries, however long the context (App. O), so the result does not follow from that theory.

W1: closest is not implements. We agree, and our claim is about the learned map, not about how it is implemented. Within the three references, the learned map is closest to gated-local at k = 5, 8, 12, 16 and 20. No network of this architecture can equal either support-dependent reference exactly (App. P), and the learned map is far from the global inverse (Fig. 9). "Closest" has a plain meaning: the network reacts less than either inverse to directions that leave the training subspace, and gated-local does not react at all. The network's reaction is 34% of the global inverse's at k = 5 and 7% at k = 20 (general response). The ranking also predicts new behavior: when we move new queries slightly along a direction where the three references disagree, gated-local predicts the network's response best at k = 5, 8 and 12 (p. 5; Table 5). The Table 8 residual does not change which reference is nearest: it adds the same amount to all three squared distances on every query (Sec. 3.1, p. 5). At k = 5 the residual includes drift (App. J, p. 37): the network's derivative at checkpoints 500 steps apart differs by about two thirds of its distance to gated-local. The change lies mostly in directions that sparse data do not constrain and that the three references do not span, and with the training design fixed, the quantities that set the ranking stay put.

W2: partly expected; theory. Two parts of the result do not follow from the setup. First, removing the support-side losses does not change which maps solve the training task exactly, nor which of them has the smallest Jacobian, yet it moves the network: the nearest reference switches at k = 5, and at k = 8 alpha, the coordinate that runs from gated-local (0) to support-masked (1), rises by 0.08 in all five seeds (App. F.2). So training decides the selection, not the set of solutions. The standard model is much closer to the smallest-Jacobian solution than the global inverse is, but a preference for that solution does not predict when the nearest reference switches (App. I). Second, the far part reacts to very small changes in training, and separately from the near part. Making 0.5% of the training queries dense from the start removes about 82% of the sparse-to-dense gap at k = 5 (Fig. 3). On a trained network, dense queries take effect within hundreds of steps, the effect reverses once they are removed, and the near part keeps its place among the three references (Sec. 4.3). We agree that the direction is intuitive, and the paper's analysis explains it: denser training queries force more directions of the derivative to agree with the global inverse (App. I, remark 3).

Explaining the selection itself is the next question for theory (Outlook, p. 10). The result also does not follow from the closest prior work. There the competing algorithms fit the training data differently, so the training loss can favor one of them (Raventos et al., NeurIPS 2023; Park et al., ICLR 2025). Here all three fit every training query exactly, so the loss cannot favor any, yet the training design moves the network between them (general response, point 4). The setting also makes an account of the selection testable: it has to predict which design change moves the network (p. 10).

Question: which identification claims remain justified? These, now listed in App. L (p. 41): "Proved: the three extensions agree on every training query (Proposition 1), so training behavior cannot identify the extension. Measured at k = 3, 5, 8 and 12: sparse-trained networks extend differently from the global inverse. Their response to directions leaving the training subspace is smaller than both inverses' (Fig. 9), and at k = 5, 8 and 12 their outputs on dense queries are sparse-shaped (Fig. 7). Measured: training design moves the extension. At k = 5 the objective and the query support each switch the nearest reference, at k = 8 they move the family coordinates in the same directions, and at k = 5 and 8 the training-query distribution sets the response to dense queries (Sec. 4)." The discussion runs extend the response comparison and the dense-query result to k = 20 (App. Q). These claims rest on distances, Jacobian norms and outputs on dense queries; none of them assumes that the network's derivative is a combination of the three references.

thanks for your reply. i keep the positive score

AuthorsOpenReview ↗

Thank you to Reviewer 33Yd

Thank you for your reply and for keeping your positive score.

Reviewer_wX9JOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether matching a known algorithm on the training distribution is enough to conclude that a neural network implements that algorithm. Using in-context complete dictionary learning, the authors construct three computations that behave identically on all supervised sparse queries but differ outside that set. Their probes suggest that a sparsely trained Transformer is closer to a support-gated local decoder than to the global inverse. They further show that changing the objective or the query distribution can move the learned function toward different extensions, even when sparse-task performance remains similar.

Critical strengths

The paper studies an interesting and well-motivated question, which validates a long-held opinion of mine. Prior work argues that a Transformer has learned a specific algorithm, such as gradient descent, because their predictions match. This paper argues that multiple computations can agree on every observed example while differing elsewhere. This is intuitive because in-context learning is a data-driven property. I personally interpret this finding as demonstrating the “limited generalizability of the in-context learning algorithm“.

Critical weaknesses
  1. The presentation is difficult to follow in places. The authors should improve the clarity and accessibility of the writing, particularly when introducing key concepts and the central ideas.

  2. The term computation feels stronger than what is actually identified. The analysis mainly characterizes the model’s input-output function, local Jacobian, and behavior on dense queries. It does not show that the Transformer internally performs support selection followed by local inversion, or identify a corresponding circuit. I would therefore describe the result as identifying a gated-local-like function extension, rather than the algorithm implemented inside the network.

  3. The three references do not cover the full space of possible solutions. Many nonlinear functions could agree with all three references on sparse queries while behaving differently elsewhere. In several settings, the model is only closer to one reference than to the others, rather than being particularly close to that reference in absolute terms.

  4. The scope and explanatory depth remain limited. The study relies on small, noiseless, complete dictionaries with exact sparsity, so it is unclear how directly the conclusions transfer to overcomplete dictionaries, real sparse autoencoders, or more realistic ICL tasks. Moreover, the paper shows that training choices affect the learned extension, but does not explain why optimization produces these preferences. The work is therefore a strong characterization of the phenomenon, but not yet a general theory of the underlying implicit bias.

Questions for authors

Please refer to the weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer wX9J (1/2)

Thank you for the review. The reviewer insightful reading pointing out that "limited generalizability of the in-context learning algorithm", is what the results show in this setting. Our setting is also a harder form of in-context learning than in-context regression: our context carries no labels, and the single-layer linear-attention models that in-context regression theory analyzes provably cannot solve the task for every admissible dictionary, no matter how long the context (App. O; W4). The extension has a near part, the derivative at sparse queries, and a far part, the response to denser queries. A small dense-query admixture removes most of the sparse-to-dense gap (the drop in output cosine from sparse to dense queries) within hundreds of steps on a trained network, and the gap returns once the admixture is withdrawn, while the near part keeps its position (Sec. 4.3).

W1: presentation. We took your suggestion and rewrote where key concepts first appear: the definition of a computation where the question is posed (p. 1); the near part, the far part and the dense-query gap at first use (pp. 2-3); the objectives (pp. 3, 7); a worked example opening Sec. 3 (p. 4); alpha and beta by their vertices (p. 5); and plain terms in place of coined ones in the text, the captions and the titles inside Figs. 3 and 4. We would welcome pointers to any passage that remains hard to follow.

W2: "computation" versus function extension. We adopt the function-level description you propose. Thanks again, The abstract now says that the network learns "a different extension of the same training behavior", and Sec. 2.2 defines the term at this level: "a computation is the query map itself, including its extension beyond MsM_s; identifying one means pinning down this map to the resolution the data allow, not reconstructing the circuitry that implements it." The title and the opening question use the term in this sense, and p. 2 now says "the map the network computes".

More interesting details here: We came to the function level after testing the solver we first expected. At the start we hoped the network would run a classical solver, for example a primal-dual method that identifies the dictionary under mild conditions, such as the volume-minimization algorithm of Sec. 5. If so, that would have been the most benign outcome. Such a solver estimates one demixing matrix from the whole context and reads every token's code, the query's included, through it; that is the global inverse of Sec. 2, and the measured extension is far from it (Sec. 3). We make no circuit claim, but the measured extension is what any circuit-level account of these networks has to reproduce.

AuthorsOpenReview ↗

Response to Reviewer wX9J (2/2)

W3: other solutions and absolute closeness. The comparison also holds on an absolute scale. At almost every training query, every function exact on the training queries agrees with the references along the training subspace and has Jacobian norm at least the gated-local floor (Proposition 3); the network's squared Jacobian norm stays within 1.16-3.17 times that floor at k = 3-12, against up to 20.58 and 157 times for the support-masked and global inverses (Fig. 9), and at k = 20 the network's reaction to unseen directions is 7% of the global inverse's (general response; App. Q). We agree that the references do not cover all solutions. Each is the canonical member of its class (Sec. 2): the unique query-linear extension, the Jacobian of a TopK sparse-autoencoder encoder, and the minimum-norm exact interpolant. For a fixed context our networks are real-analytic in the query (App. I), so they cannot equal either support-dependent reference: matching its derivative on open parts of the training subspaces of two supports would force two different derivatives at the origin, which both subspaces contain. Among the three references only the global inverse is real-analytic, so we report how close the network comes, in units of the Proposition 3 floor, with the margins to the alternatives. App. P gives the remark with its proof.

W4: scope, and why training selects. The paper establishes where the choice is made: removing the support-side losses leaves the exact solutions and their minimum-norm member unchanged yet moves the network, so training, not the solution set, decides (general response, point 3). We agree that transfer and an optimizer-level account of why are the next steps. Our setting makes such an account testable, because it has to predict which design change moves the extension (p. 10). The Outlook names overcomplete dictionaries and trained sparse autoencoders, where no global inverse exists, as where the paper's recipe goes next: construct candidates that agree on the training data, probe where they disagree, and intervene on training (Sec. 7, pp. 9-10). App. I states what such an account needs: a theory of the feature-learning dynamics of deep Transformers (p. 35).

App. O shows why the single-layer linear-attention models of in-context regression theory (Wu et al., ICLR 2024; Kwon et al., AISTATS 2026) do not reach this setting: without labels, a fixed predictor that uses the context only through its empirical second moment, continuously, which includes a single unnormalized linear-attention layer whose query attends to all context tokens, applied to unlabeled tokens through a zero-bias linear embedding, cannot converge to the correct target for every admissible dictionary-query pair as the context grows. The reported network is outside that class, so an account of what training selects here needs a richer model.

Thanks for the rebuttal. W1 and W2 have been largely addressed, while W3 is partially resolved and W4 remains.

The rebuttal appears to be LLM-generated or polished, which makes it somewhat awkward for human readers. The language feels compressed, with excessive use of ":" and ";". It is hard to follow the logic. I would prefer a rebuttal written in a natural, human style with clear logic.

Reviewer_Ue2EOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies which computation a Transformer learns in in-context dictionary learning. The authors construct three decoders that agree on sparse queries but differ outside the training distribution. Using Jacobian probes and training interventions, they find that sparse-trained Transformers behave closest to a gated-local decoder, while changes in the training objective and query distribution shift the learned function.

Critical strengths
  1. The paper studies an interesting problem: whether a Transformer actually learns a particular algorithm when its predictions match that algorithm. The three decoders provide a clear mathematical setting for studying this problem.
  2. The experiments go beyond prediction accuracy by examining Jacobians and model behavior on dense queries. These measurements help show how sparse-trained models differ from the global inverse.
  3. The authors evaluate how different training objectives and query distributions affect the learned function. The experiments show that models with similar accuracy on sparse queries can behave differently on dense queries.
Critical weaknesses
  1. The main results are not consistent across dictionary sizes. The changes between the three decoders are clear at k=5, but similar changes are not consistently observed at k=8. This raises questions about whether the conclusions hold for larger dictionaries.
  2. Table 8 shows that much of the difference between the model's Jacobian and the proposed decoders remains unexplained, especially at larger k. This makes the interpretation of the learned computation less convincing.
  3. The experiments are limited to small, synthetic complete dictionaries without noise. While this makes the mathematical analysis easier, it is unclear whether the same results would hold for larger dictionaries or more realistic in-context learning tasks.
  4. Support selection becomes less accurate as dictionary size increases, but several Jacobian comparisons focus on queries with correctly selected supports. This may not fully reflect the behavior of the trained models, especially at larger k.
Questions for authors

Please see weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ue2E (1/2)

Thank you for the review and for noting that the Jacobian and dense-query probes go beyond prediction accuracy. Without labels in the context, our task is harder than in-context regression in a precise sense: the single-layer linear-attention models that in-context regression theory analyzes provably cannot solve it for every admissible dictionary, not matter how long the context (App. O; general response).

W1: do the results hold for larger dictionaries? Yes. As you asked about larger dictionaries, we trained additional models during the discussion (n_ctx = 16 x k, i.e. 192 / 256 / 320 at k = 12 / 16 / 20; 400k steps, 600k at k = 20; the paper's models use n_ctx = 128). From k = 5 to k = 20 the network stays closest to gated-local, and this is more clearly at larger k: gated-local is the nearest of the three on 64-67% of queries at k = 5 and on 99% at k = 16 and 20, and the network's reaction to directions that never occur in training falls from 34% of the global inverse's at k = 5 to 7% at k = 20 (table in the general response; App. Q). At every k, the same architecture trained on dense queries reacts 3-8 times as strongly as ours. So we can see is the sparse training, not the architecture, keeps the reaction small.

What k changes is how far one design step, such as adding one atom to every training query, moves the network. Training on wider queries moves it toward the global inverse, and how far depends on how much of the dictionary the training queries cover. One added atom raises that share from 40% to 60% at k = 5, where, with the support-side losses removed, it makes global the nearest reference; from 25% to 37.5% at k = 8, where it moves the network partway toward global under the same objective (Tables 7 and 10); and only from 12.5% to 18.75% at k = 16. With wider training queries at k = 16, and the support-side losses kept, the network turns toward global from 25% coverage and switches to it at half the atoms, at nearly constant sparse-query output cosine (all queries; medians):

s' = atoms per training query (share of the 16)2 (12.5%)3 (18.75%)4 (25%)6 (37.5%)8 (50%)
response to directions leaving the training subspace1.663.776.049.0016.74
queries where gated-local is nearest99%99%94%67%28%
queries where support-masked is nearest1%1%6%19%13%
queries where global is nearest0%0%0%14%59%
sparse-query output cosine0.800.800.800.820.77

So the k = 5 support switch is the large-step end of a pattern that holds at larger k too. Removing the support-side losses moves the network toward support-masked at k = 5 and, less, at k = 8 (Table 10). On dense queries the network's accuracy drops, as gated-local's does, while the dense-trained network decodes them, at every k: the dense-query gap (the drop in output cosine from sparse to dense queries) is 0.113 and 0.259 at k = 5 and 8 (App. G, p. 30) and 0.404, 0.433 and 0.560 at k = 12, 16 and 20, against -0.006 to -0.145 for dense training. At k = 16 the gap falls as the training queries cover more of the dictionary, close to the paper's k = 8 values at the same coverage (App. G, p. 30): 0.198 against 0.259 at 25%, 0.047 against 0.066 at 37.5%, and -0.010 against 0.023 at 50%.

AuthorsOpenReview ↗

Response to Reviewer Ue2E (2/2)

W2: the Table 8 residual. The residual does not change which computation is nearest: it adds the same amount to all three squared distances on every query, so the ranking is the same with or without it (Sec. 3.1, p. 5). At k = 5 it includes drift in directions that sparse training data leave free (App. J, p. 37): two checkpoints 500 steps apart differ by about two thirds of the network's distance to gated-local, and between checkpoints 5,000 steps apart the derivative's part in those directions changes by 0.44-0.71 of its size in sparse-trained networks, against 0.13-0.16 in a dense-trained network, whose data constrain those directions. Under a fixed design the quantities that set the ranking hold still: the size of the network's response to those directions stays at 0.70-0.88 floor units across runs, and alpha varies with a standard deviation of 0.03 over training against design moves of 0.20-0.48.

W3: small, synthetic, noiseless dictionaries. Under observation noise the network still lands nearest gated-local. As you asked about noise, we also trained a model with observation noise (k = 5, sigma = 0.05, signal-to-noise ratio 14.5; at the paper's budget, n_ctx = 128, 60k steps; App. Q); scored on fresh episodes from its own training distribution, it stays nearest gated-local (median distances 0.61, 0.87 and 2.03 to gated-local, support-masked and global) and keeps its dense-query gap (0.114; noiseless seeds 0.111-0.137). The noiseless seeds are the paper's five k = 5 models scored the same way.

Noiseless data are the condition under which the three references are exactly equivalent on the training queries (Proposition 1), but with observation noise they are not. The paper also shows that the network stays far from the global inverse at a second model width, under three non-Gaussian coefficient laws, under momentum SGD and at sparsity s = 3 (p. 6; Apps. G and K).

Larger dictionaries were answered in W1. On size, each episode in the closest controlled studies of previous works hides 8- to 20-dim regression weights, a 7x7 rank-2 matrix or a 10-state Markov transition matrix behind labeled examples. Meanwhile ours hides a dictionary of up to 20 x 20 behind unlabeled ones (general response, point 4). The structure the result rests on, inputs that are sparse combinations of a dictionary's atoms, is the working model that sparse-autoencoder studies adopt for learned representations (p. 1). We agree that more realistic tasks are the natural next step. The Outlook names overcomplete dictionaries and trained sparse autoencoders as where the paper's recipe goes next: construct candidates that agree on the training data, probe where they disagree, and intervene on training (Sec. 7, pp. 9-10).

W4: comparisons on correct-support queries. The conclusions hold on all queries, wrong selections included. Table 9 uses all queries, and the Table 1 caption now points to it: gated-local is the nearest of the three on 64-67%, 90% and 98% of all queries at k = 5, 8 and 12 (p. 5), and on 99% at k = 16 and 20 in the new runs. Table 10 reports the k = 5 and k = 8 training steps on both populations, and Table 7 reports both for every k = 8 arm. Sec. 5 locates the large-k errors in support selection, not in decoding: given the correct support, decoding stays at 0.83 or above.

Thanks for your responses and additional experiments. The new results have addressed my concerns. I will increase my score accordingly.

AuthorsOpenReview ↗

Thank you to Reviewer Ue2E

We thank reviewer Ue2E for your positive feedback and for raising the score. We are glad the new experiments addressed your concerns. Your insightful comments made the paper stronger, and we will keep the larger-dictionary and noise experiments (App. Q) in the final version. We are happy to answer any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2)

We thank all the reviewers. We are glad that they found that the setting makes computational ambiguity directly testable (33Yd), that the probes go beyond prediction accuracy (Ue2E), and that the question is interesting and well-motivated (wX9J). Working through the reviews, we saw that our paper's presentation understated two properties of its setting.

First, in the closest controlled studies of in-context learning, from Garg et al. (NeurIPS 2022) to Park et al. (ICLR 2025), the candidate algorithms already behave differently on the training data. However, ours are provably identical on every training query, and training design still sets which one the network lands nearest (point 4).

Second, those aforementioned cases give the network labeled examples, and ours "dictionary learning" setting does not. That makes the task harder than in-context regression in a precise sense: in-context regression can be solved from averages over the labeled examples, and the single-layer linear-attention models that its theory analyzes see the context only through such averages. But on an unlabeled context they provably cannot reach the correct answer for every admissible dictionary, no matter how long the context given (App. O). It is a harder form of in-context learning, where existing in-context regression theory does not cover. The revised PDF marks changes in blue. In short:

  • From k = 5 to k = 20 (we added experiments for rebuttal), the sparse-trained network stays closest to gated-local, and more clearly at larger k (Ue2E, wX9J; point 1).
  • The claim is about the learned input-output map, not a circuit (wX9J, 33Yd; point 2). "Closest to gated-local" means that the network responds less than either inverse to directions the training data never show, and gated-local does not respond to them at all.
  • Training design (not the task) sets where among the equivalent computations the network lands (point 3).

1. Larger dictionaries request from (Ue2E, wX9J). From k = 5 to k = 20 our network stays closest to gated-local, and more clearly at larger k. As Ue2E and wX9J asked, we trained additional models during the discussion (n_ctx = 16 x k, i.e. 192 / 256 / 320 at k = 12 / 16 / 20, which keeps each atom's expected number of context appearances at its k = 8 value; 400k steps, 600k at k = 20; the paper's models use n_ctx = 128):

k = 5k = 8k = 12k = 16k = 20
queries on which our network is closest to gated-local (higher = more like gated-local)64-67%90%98%99%99%
our network's reaction to an unseen direction, with the global inverse at 100% and gated-local at 0% (lower = more like gated-local)34%19%11%8%7%

Row 1: share of 256 queries on which gated-local is the nearest of the three.

Row 2: each training query mixes two atoms, we then move it slightly in a direction that no training query takes and measure how much the output changes.

The global inverse, which decodes with the inverse of the whole dictionary, reads the move as a little of the other atoms and changes its output; we set its change to 100%. Gated-local solves only for the selected atoms and ignores the move (0%). Both rows say the same thing: of the three references, gated-local is closest to our network, and more clearly so at larger k. At every k, the same architecture trained on dense queries reacts 3-8 times as strongly as ours, so sparse training, not the architecture, keeps the reaction small. k = 5 and 8: the paper's models; k = 12-20: the new runs (the paper's own k = 12 model: 98% and 14%). On dense queries the network's accuracy drops, as gated-local's does, while the dense-trained network decodes them, at every k (reply to Ue2E, W1). App. Q of the revised PDF reports these runs, with the k = 16 coverage series and the noise run (pp. 43-44).

2. What is identified (wX9J, 33Yd). The claim is about the learned function: the network learns a different extension of the same training behavior (abstract), and Sec. 2.2 defines a computation at this level. While all exact on the training queries, gated-local is the one that does not respond to directions outside the training subspace of all the three references. The network responds there less than either other form, so it is closest to gated-local. The ranking also predicts fresh behavior: on new queries moved slightly along a direction where the three references give different answers, gated-local predicts the network's response best at k = 5, 8 and 12 (p. 5; Table 5). No network with this architecture can equal gated-local exactly, because its output is a real-analytic function of the query (App. P), so we report how close it comes: its squared Jacobian norm is 1.16-3.17 times the smallest value any exact solution can have at k = 3-12, against up to 157 times for the global inverse (Fig. 9).

AuthorsOpenReview ↗

General response (2/2)

3. Why training selects this computation (wX9J, 33Yd). The selection is decided by training rather than by the solution set. Removing the support-side losses leaves the set of exact coefficient maps, and its minimum-norm member, unchanged, yet moves the extension (the k = 5 switch; at k = 8, alpha rises by 0.08 on 5 of 5 seeds, App. F.2). The standard model lands much closer to the minimum-norm extension than the global inverse does, but a preference for that extension does not predict when the nearest reference switches (App. I). Changing the training queries moves it too: at k = 5, with the support-side losses removed, one added atom makes the global inverse the nearest reference, and at k = 16, with them kept, training queries that cover half the atoms do so on 59% of queries (reply to Ue2E, W1).

4. How the setting compares with the closest previous works. In the closest controlled papers of in-context learning, the context contains labeled examples, and the candidate algorithms already behave differently on the training data. Here the context is unlabeled, as in classical dictionary learning theory that you need to learn both (two) factor matrixs without label, and the three candidates give identical outputs on every training query:

studywhat each episode hideslabels in the contextcandidates identical on the training data
Garg et al., NeurIPS 202220-dim regression weightsyesno
von Oswald et al., ICML 202310-dim regression weightsyesno
Akyurek et al., ICLR 20238-dim regression weightsyesno
Raventos et al., NeurIPS 20238-dim regression weightsyesno
Gopalani et al., NeurIPS 20247x7 rank-2 matrixyes (observed entries)no
Park et al., ICLR 202510-state Markov transition matrixyes (earlier transitions)no
oursk x k dictionary (k = 3 to 20) and each query's supportnoyes (Proposition 1)

The unlabeled context identifies the dictionary (a classical estimator recovers it from the same contexts, Sec. 5), and the query labels give the exact answer on every training query, but neither fixes the computation (Corollary 1, p. 4). In Raventos et al. and Park et al. the competing algorithms fit the training data differently, so the training loss itself can favor one of them. In our case, all three fit every training query exactly, so the loss cannot favor any, and training design still sets which one the network approaches.

Revised PDF: abstract and introduction (pp. 1-3), including a fourth contribution on the unlabeled context (p. 3); a worked example opening Sec. 3 (p. 4); Sec. 3.1 on the label and the residual (pp. 5-6); plain terms in Secs. 4-7 and in Figs. 3-5, with a sentence in Related work on the unlabeled context (p. 9) and a fuller conclusion (pp. 9-10); new paragraphs on the minimum-norm account (App. I, p. 35) and on what the evidence establishes (App. L, p. 41); two new remarks with proofs, App. O on what the second moments of an unlabeled context can identify and App. P on exact equality with a support-dependent reference (pp. 42-43); and App. Q with the runs added during this discussion (pp. 43-44).

A closing note. We know how much reviewing is asked of everyone, and we are grateful for the care in these reviews. The points are well taken, and working through them made the paper better: the function-level wording and the worked example (wX9J), the statement of what the nearest label measures (all three reviews), the list of what the evidence establishes (33Yd), the larger-dictionary runs and the noise run (Ue2E, wX9J) and the two new remarks (wX9J, 33Yd) all came out of them.

Our paper asks a question that comes before any search for an algorithm inside a network: does matching an algorithm's behavior on the training data show that a network learned it, and if not, which of the computations that agree there does it learn? That question sits where in-context learning theory, identifiability in dictionary learning, and interpretability work such as sparse autoencoders meet. The recipe carries over (Sec. 7): construct candidates that agree on the training data, probe where they disagree, and intervene on training. White-box transformers built on sparse representations (Yu et al., NeurIPS 2023; arXiv:2306.01129) and identifiability in causal representation learning are where it can be used next.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.