- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper discuss what it costs to read task information from network weights. It builds controlled model zoos: 3-layer MLPs (widths 32–512) on flattened CIFAR-10, trained on 20 binary relabelings of the same 8,000 images from 50 shared seeds, with a constant-task null arm. Two families are compared: a family with a fixed sparse wiring mask shared across all networks, and a dense family. The "reading budget" is the number of first-layer coordinates a seed-split linear probe needs to identify the task at 95% accuracy, measured after per-model standardization.
Critical strengths
The zoos isolate the task as the only varying factor, splits are always by seed, and a null arm measures each instrument's floor.
The paper reports pre-registered predictions with their outcomes, including the failures, and documents controls that changed its own results: structural zeros, matching leakage, the invalid swap control, and the non-invariant features.
task in a linearly readable but order-invariant form; weight-only matching (no images) recovers cheap readability; and removing a permutation symmetry via fixed wiring does not supply correspondence in hidden-to-hidden layers.
Critical weaknesses
Any linear functional of a pixel's weights that is invariant to unit order must read every unit, as the paper itself notes. So "the cost without correspondence grows with the layer" follows mathematically once the reader is restricted to invariant linear aggregates.
The coordinate probe, uses PCA fitted on about 600 training networks (≤599 components) followed by a regularized linear model. Its failure on dense layers (≤0.40) may reflect the sample size and PCA bottleneck: the task signal is a low-variance sum across units, buried under seed-permutation variance. That would make the "infinite budget" a property of finite zoos rather than of dense weights
The networks are tiny 3-layer MLPs on flattened CIFAR-10 with test accuracy of only 0.59–0.79 on different tasks.which makes it not convinced enough.
The paper's interpretation also contains a tension. If fixed wiring gives first-layer units comparable roles across networks (Section 6), the inputs of the masked second layer are those same comparable units, so it is unclear why the second layer "buys nothing". The first-layer matching in the masked family decoding the task at 0.875 suggests unit roles still vary substantially.
The writing is idiosyncratic and dense, several figures render poorly, and the WeightCLIP citation is unresolved ("(?)").
Questions for authors
Does the dense coordinate probe reach the 0.95 bar with more networks per task (e.g. 200 seeds), or without the PCA step (ridge or logistic regression directly on standardized coordinates)? If it does, the "cost" is about sample complexity rather than coordinate count, and the framing should change.
How do standard equivariant weight-space encoders (NFN / DWSNet / NG-GNN) perform on your dense versus masked versus matched zoos, at matched training-set size? This would connect the budget measurements to the methods the paper aims to inform.
Thank you for the detailed review. You were right about the probe: it kept 64 principal components (not up to 599), and we didn't specify in the paper.
PCA and sample size. Without PCA and with the whole layer, the dense probe goes from 0.27–0.39 to 0.71–0.86, and it keeps improving with more training networks. So yes, a good part of the gap is sample size, and the “infinite budget” came from our probe, not from dense weights. What the probe has to learn from data is the unit symmetry. If we build it in by averaging over units first, the same ridge reaches 0.97–0.99 with only 2 training seeds per task.
Tautology. Agreed that the averaging is forced, but training forces it, not our choice of reader. Dense initialisation and the optimiser treat all units alike, so relabelling units doesn’t change the unerlying distribution of trained weights. The best linear reader is then an average over units. This makes a prediction that could have failed: the dense hidden-to-hidden layer should give a linear reader nothing. It stays at chance (0.045 vs 0.05), even with all weights and no PCA.
Second layer and the 0.875. Fair point: the paper was inconsistent here. We now measure correspondence directly. Fixed wiring gives the first layer partial correspondence: same-index units are more alike within a task than across tasks (0.23 vs 0.09), but matching still reshuffles them in a task-dependent way, which is why the permutation decodes the task. That partial correspondence is too weak to give the second layer a fixed input axis: averaging its columns reads 0.15. Matching the first layer fixes that axis, and the same average then reads 0.77 (control 0.13). We’ll correct Sections 4 and 6 accordingly.
NFN / DWSNet / NG-GNN (and WeightCLIP). We couldn’t run them in time and will add them.
Small networks. Agreed on scale. The 4 imbalanced tasks sit at their majority baseline, yet every reader still tells them apart, so what’s read is the labelling, not the accuracy. We’ll report balanced accuracy, and fix the writing, figures and citation.