- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies computation‑in‑superposition in a single‑layer ReLU toy‑model. Training with loss yields genuine superposition, where fewer neurons compute more ReLU functions. Reverse‑engineering reveals spontaneously‑emergent near‑binary sparse codewords decoded via approximate pseudoinverse. The effect holds under sufficient input sparsity.
Critical strengths
It fully reverse‑engineers a toy model for computation‑in‑superposition. With comprehensive control and ablation experiments, it verifies that loss induces genuine computation in superposition and uncovers a clear sparse binary‑code plus pseudoinverse‑decoding mechanism, offering a reusable interpretable testbed for future interpretability research.
Critical weaknesses
All conclusions are drawn only from small‑scale single‑layer toy models on trivial element‑wise ReLU tasks. There is no verification whether this binary‑code mechanism emerges in real large‑scale models, leaving its real‑world generalizability uncertain. Besides, it lacks theoretical account for why loss induces this specific coding solution.
Questions for authors
The near‑binary codewords endogenously learned under loss may be partly understood from the feature‑discrimination perspective proposed in the work "Lu W, Chen M, Li W. Binary and Ternary Quantization Can Enhance Feature Discrimination[J]. arXiv preprint arXiv:2504.13792, 2025."
3 comments
Thank you again for your review. Given that the discussion period is coming to an end, we wanted to check whether our response addresses your concerns. If anything is unclear, we'd be happy to clarify before the deadline.
Thank you for your response.
Thank you for the review and for the positive assessment of the reverse-engineering and controls.
"There is no verification whether this binary-code mechanism emerges in real large-scale models." We agree with this assessment, and this is out of scope for our paper. We see the main value of the toy model in that we can fully reverse-engineer its mechanism, which would be infeasible in a large model, and which can then serve as ground truth for interpretability methods. That said, we have now repeated the full mechanistic analysis on a model with ten times as many features and neurons (F=1000, N=500; new App. G) and find the same mechanism. While this is far from a real network, it does indicate that the mechanism persists as the model is scaled up.
"It lacks theoretical account for why L4 loss induces this specific coding solution." We agree that we have no complete theoretical account of this. We think the question splits into two parts:
Lu, Chen & Li (2025), "Binary and Ternary Quantization Can Enhance Feature Discrimination." Thank you for the pointer. As we understand it, that work quantizes input data to {0,1} or {0,±1} and shows that this can improve class separability. In our network the binary structure is the support of the learned encoder weights in a regression model, and the values on the support stay continuous and non-uniform (Sec. 5). There is a loose connection, in that codewords must be distinguishable from each other, but the settings and mechanisms are different.