acceptodds
Under review as a conference paper at ICLR 2027

Correspondence over Content: What Makes Task Information Readable in Neural Network Weights

Abstract

Weight space methods read trained networks to retrieve, compare and edit them. We ask what makes the information in a network's weights readable, and what it costs to read. Studying collections of networks that differ only in how a fixed set of images is labelled, so that the task is the only variable, we find that correspondence between models governs not what a reader can recover but what recovering it costs. When fixed sparse wiring pins each unit to the same inputs in every network, a linear probe identifies the task from a few hundred first layer weights or fewer, with no net growth over a sixteenfold range of width. Dense networks hold the same information in linearly readable form, but only in aggregates over their hidden units: the cheapest reader we found must read every unit, and its cost grows with the layer, with an exponent between and . Matching units to a template, on a few unlabelled images or on the weights alone, brings the dense budget back to a few hundred coordinates, so the premium for missing correspondence itself grows with the network, from threefold to seventeenfold across our widths. In the hidden to hidden layer, removing the permutation symmetry with fixed wiring is not enough: its coordinates stay unreadable, so removing a symmetry is not the same as supplying correspondence. Along the way we document how standard readings mislead: per model scale alone identifies the training dataset on a public model zoo, and an alignment step can itself carry the task.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
weight space learning, permutation symmetry, model zoos, correspondence, model alignment, invariance, probing, sparse networks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_spFgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper discuss what it costs to read task information from network weights. It builds controlled model zoos: 3-layer MLPs (widths 32–512) on flattened CIFAR-10, trained on 20 binary relabelings of the same 8,000 images from 50 shared seeds, with a constant-task null arm. Two families are compared: a family with a fixed sparse wiring mask shared across all networks, and a dense family. The "reading budget" is the number of first-layer coordinates a seed-split linear probe needs to identify the task at 95% accuracy, measured after per-model standardization.

Critical strengths

The zoos isolate the task as the only varying factor, splits are always by seed, and a null arm measures each instrument's floor.

The paper reports pre-registered predictions with their outcomes, including the failures, and documents controls that changed its own results: structural zeros, matching leakage, the invalid swap control, and the non-invariant features.

task in a linearly readable but order-invariant form; weight-only matching (no images) recovers cheap readability; and removing a permutation symmetry via fixed wiring does not supply correspondence in hidden-to-hidden layers.

Critical weaknesses

Any linear functional of a pixel's weights that is invariant to unit order must read every unit, as the paper itself notes. So "the cost without correspondence grows with the layer" follows mathematically once the reader is restricted to invariant linear aggregates.

The coordinate probe, uses PCA fitted on about 600 training networks (≤599 components) followed by a regularized linear model. Its failure on dense layers (≤0.40) may reflect the sample size and PCA bottleneck: the task signal is a low-variance sum across units, buried under seed-permutation variance. That would make the "infinite budget" a property of finite zoos rather than of dense weights

The networks are tiny 3-layer MLPs on flattened CIFAR-10 with test accuracy of only 0.59–0.79 on different tasks.which makes it not convinced enough.

The paper's interpretation also contains a tension. If fixed wiring gives first-layer units comparable roles across networks (Section 6), the inputs of the masked second layer are those same comparable units, so it is unclear why the second layer "buys nothing". The first-layer matching in the masked family decoding the task at 0.875 suggests unit roles still vary substantially.

The writing is idiosyncratic and dense, several figures render poorly, and the WeightCLIP citation is unresolved ("(?)").

Questions for authors

Does the dense coordinate probe reach the 0.95 bar with more networks per task (e.g. 200 seeds), or without the PCA step (ridge or logistic regression directly on standardized coordinates)? If it does, the "cost" is about sample complexity rather than coordinate count, and the framing should change.

How do standard equivariant weight-space encoders (NFN / DWSNet / NG-GNN) perform on your dense versus masked versus matched zoos, at matched training-set size? This would connect the budget measurements to the methods the paper aims to inform.

1 comment

Thank you for the detailed review. You were right about the probe: it kept 64 principal components (not up to 599), and we didn't specify in the paper.

PCA and sample size. Without PCA and with the whole layer, the dense probe goes from 0.27–0.39 to 0.71–0.86, and it keeps improving with more training networks. So yes, a good part of the gap is sample size, and the “infinite budget” came from our probe, not from dense weights. What the probe has to learn from data is the unit symmetry. If we build it in by averaging over units first, the same ridge reaches 0.97–0.99 with only 2 training seeds per task.

Tautology. Agreed that the averaging is forced, but training forces it, not our choice of reader. Dense initialisation and the optimiser treat all units alike, so relabelling units doesn’t change the unerlying distribution of trained weights. The best linear reader is then an average over units. This makes a prediction that could have failed: the dense hidden-to-hidden layer should give a linear reader nothing. It stays at chance (0.045 vs 0.05), even with all weights and no PCA.

Second layer and the 0.875. Fair point: the paper was inconsistent here. We now measure correspondence directly. Fixed wiring gives the first layer partial correspondence: same-index units are more alike within a task than across tasks (0.23 vs 0.09), but matching still reshuffles them in a task-dependent way, which is why the permutation decodes the task. That partial correspondence is too weak to give the second layer a fixed input axis: averaging its columns reads 0.15. Matching the first layer fixes that axis, and the same average then reads 0.77 (control 0.13). We’ll correct Sections 4 and 6 accordingly.

NFN / DWSNet / NG-GNN (and WeightCLIP). We couldn’t run them in time and will add them.

Small networks. Agreed on scale. The 4 imbalanced tasks sit at their majority baseline, yet every reader still tells them apart, so what’s read is the labelling, not the accuracy. We’ll report balanced accuracy, and fix the writing, figures and citation.

Reviewer_wK7JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the accessibility of task information in neural network weights is largely determined by cross-model correspondence rather than by the presence of task-related information alone. The paper introduces a reading-budget perspective and shows theoretically that an exactly permutation-invariant linear reader must aggregate symmetrically over hidden units. The authors validate the idea on controlled MLP model zoos with CIFAR-10 binary relabelling tasks, comparing fixed sparse wiring, dense networks, and dense networks after unit matching. The authors find that correspondence enables task identification from only a few hundred first-layer weights, whereas without it the required reading budget grows with width, and they support these conclusions with extensive controls for scale, structural zeros, matching leakage, and other confounds.

Critical strengths
  1. The paper presents an insightful observation.

  2. The empirical study is carefully controlled.

Critical weaknesses
  1. The central “reading cost” comparison does not fully account for the cost of acquiring correspondence. The reported budget only counts coordinates read after alignment, although matching itself requires access to weights or activations. Thus, the measured benefit is closer to the cost under given correspondence than to a full end-to-end reading cost.

  2. The paper is somewhat limited in technical depth and empirical scope. The theoretical component mainly formalizes a relatively simple consequence of permutation invariance for linear readers, while the empirical evidence is concentrated on small two-layer MLPs, a single dataset, and a narrow family of relabelling tasks. As a result, the paper presents a clear and carefully validated observation, but the overall contribution feels somewhat limited in breadth and depth.

Questions for authors
  1. If all target-model accesses required to acquire correspondence are included in the reading budget, how would the results in Figure 3 and the reported 3.3x–16.9x premium change? In particular, how many target weights/activations are accessed by weight- or activation-based matching, and can these costs be expressed in a unit comparable to the downstream coordinate budget?

  2. Why is the matched dense budget evaluated on only one template split? Can the experiment be repeated over multiple independently constructed template/train-seed splits to provide uncertainty estimates for the matched budget and the correspondence-premium curve?

  3. If entire relabelling tasks, rather than only initialization seeds, are held out, can the reader recover information about unseen relabellings rather than only identify previously observed task classes?

1 comment

Thanks for the precise questions.

Cost of correspondence (Q1): fair, the premium assumed correspondence was given. We now give both readers exactly the same weights, all units at k input pixels, so both cost k × width, for fairness. Matching on those pixels alone, with weights only and no images, needs 8–15 pixels; averaging needs 27–55. Matching stays 2.5–3.7 times cheaper end to end, roughly constant across widths. So there are two numbers: about 3× end to end, and 4–12× per query once correspondence is in hand. Both grow at least linearly with width without correspondence, since every unit has to be touched.

Splits (Q2):we ran a new batch, repeated over 10 train/test splits with a fresh template each time. Matched budgets are 464–858 weights across widths, with about 20% spread. With a finer grid for the averaging reader, the premium goes from 4.4× to 11.9×, growing roughly as N^0.35.

Held-out relabellings (Q3): not run, because of the substantial experiments to add already. The per-pixel mean seems to behave linearly in the class means, which would make unseen labellings recoverable, but hard to say for sure without a proper dedicated setup.

Depth and scope: the theory is simple, agreed. What it changes is that averaging becomes a consequence of training, with a prediction that held (the dense second layer at chance). The scope limits are real; our answer to WcAM describes how we plan to extend it.

Reviewer_WcAMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper is about the interplay between MLP weights after training and the given task solved by the MLP. The paper is mainly experimental, with careful design of experiments - all are variations of 2-layer MLPs, solving simple binary image classification tasks (with limited accuracy). The specific focus is devoted to the complexity of relating weights and task ("reading"). Validation is empirical.

Critical strengths

Experimental design appears clever within the narrow scope Reproducible experiments as far as i can see

Critical weaknesses

The largely empirical study takes place in very limited context of tasks and networks. No theoretical understanding, as far as I can see....

Questions for authors

What are the generalizable insights gained?

1 comment

Thank you. On generalisable insights, multiple things hold beyond MLPs. wherever units are interchangeable under training, the best linear reader averages over them, and a tensor with no fixed axis gives linear readers nothing (dense second layer: 0.045 at chance 0.05). also, what’s cheap to read depends on which axes are fixed. With both fixed, a few hundred weights suffice. With one fixed, you average over the free one at a cost that grows with it. With neither, there’s nothing linear. Matching or fixed wiring is how an axis gets fixed. and, missing correspondence is always paid for,either in weights read, or in training networks (2 vs 30+ per task), or in matching cost.

This gives a recipe that's generalizable accross architectures: find each tensor’s fixed axes. We expect a CNN’s first convolution to be readable by averaging over channels, and deeper convolutions to behave like our second layer. In transformers, only the vocabulary and position axes are fixed. In fine-tunes from a shared base model, the base supplies correspondence for free, which may be why merging works there but needs alignment for independently trained models. Our own nets trained from scratch keep almost none from a shared seed.

Next steps: test this on public CNN and transformer zoos, measure how correspondence decays as fine-tunes move away from a shared base, and compare equivariant encoders on dense, masked and matched zoos. On theory and scope: the revision states the result above formally, and we agree the empirical scope is narrow, by design, so that the task is the only variable.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. They pointed out several things we had missed, and we used the rebuttal period to run new experiments. Some of the results change claims in the paper, so we summarise them here, and answer each point under each review.

Regarding the architecture, we’ll fix the wording mismatch between the limitations section and Appendix A. The networks are 3072 -> w -> w -> 2, with w from 32 to 512.

New experiments and analysis:

(A) A probe without PCA, compared with the PCA-projected one (a 200-seed rerun didn’t fit in the rebuttal window unfortunately). On the same 20,000 stored weights, dropping PCA takes the dense first layer from 0.27–0.39 to 0.59–0.71.(we should have started with that reading instead) Reading the whole layer gets it to 0.71–0.86 with 30 training seeds per task, and it’s still going up with more seeds, so we agree with spFg that part of the gap is sample size.

(B) The same probe on the per-pixel average over hidden units. It gets 0.97–0.99 with only 2 training seeds per task, at every width. So what the probe in (A) is missing is actually the unit symmetry, more than the information.

(C) Both readers get exactly the same weights (all hidden units at k input pixels), so the cost of finding correspondence is counted. Matching units to the template on those pixels needs 8 to 15 pixels, averaging needs 27 to 55, so matching is still about 3 times cheaper with its own cost included.

(D) The matching pipeline over 10 train/test splits, each with a separate template. The matched budget stays between 464 and 858 weights across widths, with about 20% spread between splits.

(E) A direct measurement of correspondence: unit j in one network against unit j in another network trained from a different seed. In the masked first layer, same-index units behave more alike within a task than across tasks. Not in the masked second layer, and not in dense networks. That’s why the masked second layer stays unreadable (more in our answer to spFg).

(F) A finer unit grid for the averaging reader: 62 to 88% of units are enough from width 64 up, not all of them. We also checked each task’s accuracy against its majority class rate.

This changes three things, which we’ll correct in the revision. The probe used 64 principal components, which the paper didn’t say, and the “never reaches the bar” result for dense networks (Figure 3) came from that probe and our training set size, not from dense weights. The averaging reader doesn’t need every unit (Table 8). And on the finer grid, the correspondence premium is 4 to 12 times per query rather than 3.3 to 16.9, and about 3 times once matching is paid for.

In a dense network, the initialisation and the optimiser treat every hidden unit the same way, so relabelling the units doesn’t change the distribution of trained weights. We should have said this in the paper, because it ties the results together. The best linear reader of such a layer is then an average over units, whatever the regularisation. And a standardised hidden-to-hidden layer, with hidden units on both sides, leaves nothing for a linear reader. We checked both: the dense second layer stays at chance (0.045, chance 0.05) with all its weights and no PCA, and the averaged part of the probe from (A) reads 0.99–1.00 on its own.

In the revision we’ll state this with notation in Section 3, add the measurements from (E), fix the abstract and the numbers above, report accuracy against majority baselines, and fix the figures and the missing citation (typo in our .bib file). We didn’t have time for equivariant encoders (NFN, DWSNet), held-out relabellings, or other datasets and architectures; our answer to WcAM says how we plan to go there.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.