acceptodds
Under review as a conference paper at ICLR 2027

Compressed Computation under Loss is likely Computation in Superposition

Abstract

Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an loss (the mean fourth power of the error), instead of the usual , elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, computation in superposition, superposition, compressed computation, reverse engineering neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_83LTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies computation‑in‑superposition in a single‑layer ReLU toy‑model. Training with L4L^4 loss yields genuine superposition, where fewer neurons compute more ReLU functions. Reverse‑engineering reveals spontaneously‑emergent near‑binary sparse codewords decoded via approximate pseudoinverse. The effect holds under sufficient input sparsity.

Critical strengths

It fully reverse‑engineers a toy model for computation‑in‑superposition. With comprehensive control and ablation experiments, it verifies that L4L^4 loss induces genuine computation in superposition and uncovers a clear sparse binary‑code plus pseudoinverse‑decoding mechanism, offering a reusable interpretable testbed for future interpretability research.

Critical weaknesses

All conclusions are drawn only from small‑scale single‑layer toy models on trivial element‑wise ReLU tasks. There is no verification whether this binary‑code mechanism emerges in real large‑scale models, leaving its real‑world generalizability uncertain. Besides, it lacks theoretical account for why L4L^4 loss induces this specific coding solution.

Questions for authors

The near‑binary codewords endogenously learned under L4L^4 loss may be partly understood from the feature‑discrimination perspective proposed in the work "Lu W, Chen M, Li W. Binary and Ternary Quantization Can Enhance Feature Discrimination[J]. arXiv preprint arXiv:2504.13792, 2025."

3 comments

Thank you for the review and for the positive assessment of the reverse-engineering and controls.

"There is no verification whether this binary-code mechanism emerges in real large-scale models." We agree with this assessment, and this is out of scope for our paper. We see the main value of the toy model in that we can fully reverse-engineer its mechanism, which would be infeasible in a large model, and which can then serve as ground truth for interpretability methods. That said, we have now repeated the full mechanistic analysis on a model with ten times as many features and neurons (F=1000F = 1000, N=500N = 500; new App. G) and find the same mechanism. While this is far from a real network, it does indicate that the mechanism persists as the model is scaled up.

"It lacks theoretical account for why L4 loss induces this specific coding solution." We agree that we have no complete theoretical account of this. We think the question splits into two parts:

  1. Why do L4L^4, and exponents above 2 more broadly, lead to superposition? In the new App. F we extend an argument of Hazelden & Linsefors (2026): in a simple setting, the best solutions with and without superposition tie exactly under L2L^2, while under LpL^p superposition is preferred for every p>2p > 2. This matches what we find (Sec. 4.1, Fig. 2), although it is a heuristic rather than a derivation for our model.
  2. Why is the superposition solution specifically a sparse binary code? We agree that we do not have a theoretical account of this. We note, however, that it matches the constructions proposed by Hänni et al. (2024) for computation in superposition, so gradient descent arrives at a solution that theory had already suggested.

Lu, Chen & Li (2025), "Binary and Ternary Quantization Can Enhance Feature Discrimination." Thank you for the pointer. As we understand it, that work quantizes input data to {0,1} or {0,±1} and shows that this can improve class separability. In our network the binary structure is the support of the learned encoder weights in a regression model, and the values on the support stay continuous and non-uniform (Sec. 5). There is a loose connection, in that codewords must be distinguishable from each other, but the settings and mechanisms are different.

Thank you again for your review. Given that the discussion period is coming to an end, we wanted to check whether our response addresses your concerns. If anything is unclear, we'd be happy to clarify before the deadline.

Thank you for your response.

Reviewer_aQZmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes computation under superposition in a toy model, i.e. when there are more features than neurons. An encoder encodes a sparse, high dimensional vector into a low dimensional embedding, and the decoder reconstructs the ReLU of the original input. Rather than the standard L2 loss, the authors use L4 loss to penalize outliers more strongly, thus ensuring that all features are represented in superposition, rather than some features being fully represented without superposition and some features being ignored. The authors find that the encoder weights have many small negative values and few large positive values. Treating the large positive values as the 1’s and small negative values as the 0’s in a binary code allows for the analysis of codes per feature and codes per neuron. Synthetically creating an encoder that shares the connectivity statistics of the learned matrix allows for a new network with only ~1.12x the loss of the learned network.

Critical strengths

The paper is generally well written and easy to follow. The experimental setup is conveyed in an intuitive manner, and the results build from one another.

The ablations were interesting, such as using an ansatz network with the same connectivity metrics but without any notion of exact wiring of the circuit.

Critical weaknesses

As the authors note, the problem and phenomenon are only demonstrated in this constrained system. While toy models are very useful, since L4 loss is also generally not used in training, it is difficult to see how this would extend to more realistic networks or provide broader insights into naturally arising superposition and computation in superposition.

Additionally, while the L4 loss aspect is interesting in promoting superposition, several aspects of the problem setup and some of the mechanistic ablation results, such as replacing learned networks with handcrafted codes, seem to closely resemble works from prior literature. While the experiments in this work provide empirical validation for the phenomenon and mechanism, it is currently unclear what fundamentally new understanding is gained beyond demonstrating that the mechanism arises in this specific training setup.

Questions for authors

My main concerns are highlighted above in the weaknesses section.

3 comments

Thank you for the review, and for the positive comments on clarity and on the ablations.

"L4 loss is also generally not used in training [...] difficult to see how this would extend to more realistic networks." Please see the general response: large networks are trained with neither L4L^4 nor L2L^2 but mostly with cross-entropy, and L2L^2 was already known to be a poor loss for eliciting superposition, for reasons that predict every exponent above 2 to work (new App. F), as we find. In any case, we do not claim that networks are trained with L4L^4. It is that a network trained to compute in superposition can be fully reverse-engineered, and what we find (sparse binary codes, pseudoinverse read-out) is the mechanism proposed by Hänni et al. (2024) for computation in superposition in general.

"Several aspects of the problem setup and [...] mechanistic ablation results, such as replacing learned networks with handcrafted codes, seem to closely resemble works from prior literature." We would be grateful if the reviewer could name the works they have in mind, so that we can compare directly. As far as we know:

  • The setup is deliberately that of Braun et al. (2025), since that is the model whose status Bhagat et al. (2025) disputed. We do not claim the setup as new, beyond the change in the loss function used in training, as discussed extensively in the paper.
  • Handcrafted codes have been proposed as constructions for computation in superposition (Hänni et al., 2024). We know of no prior work that reverse-engineers a trained network computing in superposition and then validates the reverse-engineering by replacing the trained code with designed codes from the same three-parameter family (Fig. 7, Table 1).
  • The closest empirical work we know of is Newgas (2025), which we now cite. It trains a compressed-computation model on Universal-AND and finds a dense, two-valued random weight matrix, whereas our network computes elementwise ReLU and learns a sparse binary code (about 5 of 50 neurons per feature) with a pseudoinverse read-out. Gibson (2026), which we cite as concurrent work, finds binary codes in a different task.

"It is currently unclear what fundamentally new understanding is gained."

  1. Computation in superposition can be elicited in this model by training. This turns the negative result of Bhagat et al. (2025) into a positive one by changing only the loss, and it holds for every exponent above 2.
  2. What the trained network implements. A near-biregular, sparse binary code over neurons with a pseudoinverse read-out, which three scalars capture at about 1.1× the trained loss. This is an instance of the constructions of Hänni et al. (2024), found by gradient descent rather than designed.
  3. Which properties of the code matter. The specific trained code carries little information beyond its family: designed biregular codes with low overlap match it (Fig. 7), while codes without overlap minimization are about 2× worse.

Together these give a trained network computing in superposition whose mechanism is known, which can serve as ground truth for interpretability methods.

Thank you again for your review. Given that the discussion period is coming to an end, we wanted to check whether our response addresses your concerns. If anything is unclear, we'd be happy to clarify before the deadline.

I would like to thank the reviewers for clarifying the intended contribution of the work. The new discussion on the prior literature is also appreciated and helped me determine where the novelty of this work lies. The additional model variants are also helpful in establishing that the original results generalize to different configurations. While I still think that the novelty of these results is still somewhat limited to this controlled setting, I do think the revision has improved the work and so I have raised my informativeness assessment.

Reviewer_jYU3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a toy model of computation in superposition, where the idea is to examine when a neural network can compute more functions than it has neurons. It shows that this is possible in the model when using L4 loss instead of L2 loss, and proposes a mechanism in which the model uses codewords.

Critical strengths

The overall question identified is important and interesting.

Critical weaknesses

The paper only studies a single model with 100 features and 50 hidden layer neurons. Studying a larger class of settings, and identifying shared underlying mechanisms, would significantly improve the insights of the paper. For example, understanding the sensitivity to sparsity is important. The model also studies a specific single function (element wise ReLU). It would be interesting to understand broader sets of computations that could be performed in superposition.

A main result of the paper is that superposition only arises with L4 loss. In practice, given that models are typically trained with L2 loss, it is not clear why we should expect insights of this model to translate. It would be more interesting to identify settings in which superposition can arise with L2 loss.

The general direction is interesting, but the paper's current results and lack of breadth and depth prevent it from making convincing claims.

Questions for authors

None

2 comments

Thank you for the review.

"The paper only studies a single model with 100 features and 50 hidden layer neurons." This is not the case. Already in the submitted version, we vary the width from N=20N = 20 to N=200N = 200 neurons (Sec. 4.4, Fig. 8) and train a model with ten times as many features and neurons (F=1000F = 1000, N=500N = 500; App. A.3). We also study the original embedded model of Braun et al. (2025), with and without its residual connection (App. A), and five loss exponents (Sec. 4.1).

"Understanding the sensitivity to sparsity is important." We agree, which is why the submitted version studies it (Sec. 4.4, Fig. 8): we sweep the feature probability from p=0.005p = 0.005 to p=1p = 1 at every width, across 117 combinations with 5 seeds each. The network computes all 100 features at every width up to 8 expected active features (four times the main operating point), superposition persists up to 48 expected active features at N=80N = 80, and it disappears in the dense regime (p≥0.64p \geq 0.64).

In addition, as described in the general response, we have now repeated the full mechanistic analysis, previously done only at the main operating point, at four further points of this sweep and on an axis-aligned model with F=1000F = 1000 and N=500N = 500. Our results hold (new App. G).

"The model also studies a specific single function (element wise ReLU)." That is correct, and we list it as the main limitation in the Discussion. The question we set out to answer is whether the compressed-computation model of Braun et al. (2025), whose status as computation in superposition Bhagat et al. (2025) disputed, can be trained to compute in superposition, and what it then implements. Tasks with interactions between features, such as the pairwise ANDs of Hänni et al. (2024), are the natural next step, and we name them in the Discussion.

"Superposition only arises with L4 loss [...] models are typically trained with L2 loss." Please see the general response, where we address this point at greater length. To summarize: large networks are trained with neither L4L^4 nor L2L^2 but mostly with cross-entropy. Further, L2L^2 was already known to be a poor loss for eliciting superposition, and the same argument predicts that every exponent above 2 favors superposition (new App. F), which is what we find.

Thank you again for your review. Given that the discussion period is coming to an end, we wanted to check whether our response addresses your concerns. If anything is unclear, we'd be happy to clarify before the deadline.

Comments on the paper

1 comment

We thank the reviewers for their time. All reviews share one concern, namely the scope of the work. We address it once here and answer the remaining points in the individual responses.

Why a toy model. The goal of this work was to find a network where computation in superposition arises through training, being small enough that we can reverse engineer it. Until recently there was no such example: the compressed-computation model of Braun et al. (2025) was introduced with this goal, but Bhagat et al. (2025) showed that it does not compute in superposition. We show that it can be made to, and we explain the solution down to a three-scalar description that we validate by building equivalent networks from designed codes. A fully understood, trained example of computation in superposition is useful as ground truth for interpretability methods, such as the parameter-decomposition methods (APD, SPD) this model was originally built to test. That would not be possible in a large model, where reverse engineering would be much more challenging.

Training with L4L^4. Reviewers note that networks are not trained with L4L^4 loss in practice, so it is unclear how far our results carry over. It is true that L4L^4 is not used to train networks at scale. Neither is L2L^2, however: large networks are mostly trained with cross-entropy. Moreover, L2L^2 was already known to be a poor loss for eliciting superposition. Hazelden & Linsefors (2026) show that, in a simple setting, the best solutions with and without superposition achieve exactly the same squared error, so L2L^2 has no preference for superposition. In the new App. F we extend their calculation to LpL^p losses: superposition is preferred for every p>2p > 2, the two solutions tie at p=2p = 2, and the solution without superposition is preferred for p<2p < 2. This matches what we find: every exponent above 2 that we tried (2.5, 3, 4, 6, 8) elicits superposition, while L2L^2 does not (Sec. 4.1, Fig. 2). We therefore see L4L^4 as a tool for eliciting computation in superposition in a model small enough to reverse-engineer fully. What may carry over to other settings is the mechanism we identified with our reverse engineering: sparse binary codes over neurons, resembling the constructions proposed by Hänni et al. (2024).

Breadth of the results. Reviewers note that the paper focuses on a single model. In fact, the submitted paper already considers many variants of this model:

  • a sweep over 117 combinations of sparsity (pp from 0.005 to 1) and width (NN from 20 to 200) (Sec. 4.4, Fig. 8);
  • a model with ten times as many features and neurons (F=1000F = 1000, N=500N = 500) (App. A.3);
  • the original embedded model of Braun et al. (2025), with and without its residual connection (App. A);
  • five loss exponents above 2 (Sec. 4.1).

However, the full mechanistic analysis was done only at the main operating point, and the other settings were checked through their loss alone. To address the reviewers' concern, we have now repeated the full analysis at four further points of the sweep (10 seeds each) and on an axis-aligned model with F=1000F = 1000 and N=500N = 500 (new App. G, Table 3). Our results hold throughout:

  • the codewords identify their features (swap test 97.5–100%, random controls ≤ 1.9%);
  • the decoder is a pseudoinverse of the encoder (cosine ≥ 0.99);
  • the three-scalar description is always within 2× of the trained network's loss.

Changes in the revised PDF.

  1. New App. F: the LpL^p heuristic above, referred to in Sec. 4.1.
  2. New App. G: the same mechanistic analysis repeated for other model variants, referred to at the end of Sec. 4.4.
  3. Related work: Newgas (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.