acceptodds
Under review as a conference paper at ICLR 2027

Hidden Equivalences in Sheaf Neural Networks: The Share Credited to Matrix-Valued Transport Is Not Intrinsic

Abstract

Sheaf neural networks replace scalar edge weights with learned restriction maps, and their gains are routinely credited to this matrix-valued transport. We show that the share of the gain so credited is not intrinsic to a task or an architecture. The algebra its restriction maps generate constrains the propagation operator of a sheaf layer: by the Wedderburn decomposition it fixes the block structure and which blocks act irreducibly, so scalar maps give a real edge-weighted operator, one-angle rotations a complex-weighted (magnetic) one, and of our six rungs only the dense one acts irreducibly. It does not fix the realizable family, since restricted families can generate the same algebra. On this chain of algebras we build an attribution protocol: an exactly nested six-level ladder audited in function space, a depth-matched no-graph baseline, and a necessity diagnostic that can refuse to answer. The share credited to matrix-valued transport is then 53% and 11% on two heterophilous tasks at two layers and 13% and 17% at four, and it moves with width (46% to 73%), with the operator (11% to 35%) and with the order of credit (13% to 94%). The released operator moves the absolute sheaf-exclusive gain by less than at two layers (a point estimate, not an equivalence test) but reverses the sign of the four-layer depth effect on Minesweeper ( against ), four fifths of which a experiment traces to the normalization alone, acting through the scalar gate as a spectral bound we prove predicts. What the scalar control is allowed to do matters as much as what the sheaf adds: in a preregistered test, a learned degree-3 spectral response on those rungs absorbs 92% of the gain credited to matrix-valued transport on one task, while on the other the interaction runs the other way and is not resolved under a matched training budget. Two later controls on four-layer Minesweeper reduce the gain credited to matrix-valued transport by 98% with learned one-hop filtering and by 71% with fixed three-hop filtering. The larger reduction therefore requires no wider receptive field. On an independently published directed sheaf architecture the capacity effect is points unconditionally, positive conditional on convergence, and not significant once an identity-containing parameterization removes the collapse. Claims about what matrix-valued transport contributes should therefore state the operator, depth and order of credit under which they were measured.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
sheaf neural networks, graph neural networks, cellular sheaves, sheaf Laplacian, spectral graph filters, heterophily, capacity attribution, ablation methodology, evaluation methodology, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_orKPOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether the improvements achieved by sheaf neural networks (SNNs) can be attributed to their use of matrix-valued transport.

The experiments highlighted that the proportion of performance improvement attributed to matrix-valued transport is dependant on datasets, model depths, operator implementations, and attribution orders.

Their final claim is that what matrix-valued transport contributes should state the operator, depth and order of credit under which they were measured.

Critical strengths

I think that this paper is genuily well written and has substantially more strength than weaknesses.

Strength 1: Theoritical foundation and motivation are solid and the contribution to SNN is meaningful.

Strength 2: Experiments are exhaustive.

Critical weaknesses

The weaknesses I found are not critical for the paper in my opinion but I still have to state some that came to my mind while reading the paper:

Weakness 1: I think that the expressivity of the whole network could be an interesting study/

Weakness 2: not really a weakness but I think the authors should rename the discussion section as Conclusion. The reason for this is that when the paper is accepted a lot of people look for these words to read quickly through papers and I feel like Conclusion is more commonly accepted as the end of the main section than discussion. It is a personal test on my end so it is up to the authors.

Questions for authors

I don't have much questions for the authors other than out of curiosity for the SNN that I discovered with this paper:

Q1: What are the potential application field for Sheaf Neural Networks? Q2: Following from Thm 1, can the algebraic structure predict which architecture will perform the best? Q3: As the algebra generated by the restriction maps characterizes the structural capabilities of a layer. Have you considered tracking the algebra generated by the learned restriction maps throughout training? I am wondering if the algebra can change during training.

Congratulation to the author for their paper and good luck with the rebuttal phase.

2 comments
AuthorsOpenReview ↗

Response to Reviewer orKP

Thank you for the careful and encouraging review. Your questions helped us state more clearly what the theory covers and what it leaves open.

W1 (expressivity of the whole network). You are right, and we thank you for raising it. Theorem 1 and Proposition 3 are statements about one propagation operator, and the Limitations (Section 7) say that they concern the operator, not the network. Two of our empirical results bear on the composed network without settling it. First, the fitted dense maps generate all of M_4(R) in every layer and every seed (Appendix F, "Which algebra the fitted maps generate"), so the trained network does realize the irreducible algebra; it simply gains little from it (the dense rung never carries more than 9% of the audited gain). Second, depth changes the allocation of credit in opposite directions on the two tasks (Section 5.1), a network-level effect the operator-level theory does not predict. What a stack of such layers can express with the nonlinearity and the normalization in place is open, and following your comment the revised Limitations now say so.

W2 (Discussion vs. Conclusion). Agreed, thank you: the section is now titled "Conclusion" and opens with the list of what to report.

Q1 (application fields). The settings where sheaf networks have been used are, as far as we know, those where the relation between two nodes is not well described by a single weight: node classification on heterophilous graphs, where neighbours should be transformed rather than averaged (Bodnar et al., 2022; Barbero et al., 2022); opinion dynamics and consensus on structured relations (Hansen & Ghrist, 2021); knowledge graphs with typed relations (Gebhart et al., 2021); recommendation on user–item graphs (Purificato et al., 2025); and hypergraph learning, including the directed sheaf hypergraph model we audit in Section 5.5 (Duta et al., 2023; Mule et al., 2026). Our result bears on all of them in one way: when a sheaf model is reported to help in such a setting, the share of the gain that is due to matrix-valued transport, rather than to edge weighting, gating or a freer spectral response, has to be measured under a stated operator, depth and order of credit, or it is not a property of the setting.

Q2 (can the algebra predict which architecture performs best?). Not as a ranking. Theorem 1 fixes the block structure of the propagation operator: which stalk coordinates are coupled, over which division algebra, and which blocks act irreducibly. That tells you when two nominally different families compute the same functions (scalar maps are edge weights; one-angle rotations are a magnetic Laplacian), and hence which added "capability" cannot add anything because it is already present. It does not fix the realizable family inside a given algebra (Section 3 gives restricted families that generate the same algebra), and it says nothing about optimization or the task. So the algebra is a necessary-condition tool: it rules claims out, it does not pick winners. The data agree that the ordering is not monotone in the algebra: on Minesweeper most of the sheaf-exclusive gain is commutative (a phase suffices), on Roman-empire it is non-commutative, and the irreducible rung is never the main contributor on any task we ran (Section 5.1, "Split by generated algebra").

Q3 (tracking the generated algebra during training). This is a valuable suggestion. We measured it at the end of training only. Appendix F reports the fraction of Frobenius energy in the nested subalgebras RI ⊂ C ⊂ H and C ⊂ M_2(R)^2 ⊂ M_4(R), and the numerical dimension of the algebra the maps generate: full 16-dimensional algebra in every layer and seed, with 67–84% of the energy inside C. So the end state is "full algebra, used sparingly". One thing is known by construction: the dense rung is warm-started at its parent, the rotated-diagonal rung, whose maps lie in the proper subalgebra M_2(R)^2 (two independent 2x2 blocks, dimension 8), and the fitted dense maps generate all of M_4(R); so the algebra does change during the dense stage. What we have not measured is when: whether the generated dimension rises gradually or jumps, and whether the energy leaves C before or after the validation score stops improving. The probe runs on any checkpoint, so it is a matter of saving checkpoints during the dense stage; we have not run this, and the revised Limitations now name it as an open question.

Finally, Reviewer s64g asked whether the matrix-versus-scalar gap could be extra training rather than capacity. We ran the control they proposed, and the gap survives in every cell. The full table and a third-dataset result are in the general response; three cells now have forty seeds instead of the ten we first reported here. Nothing in it changes the per-layer theory you assessed.

Thank you again for your time and your constructive questions; we would be glad to clarify anything further.

Thank you for the answers to my minor weaknesses and questions!

I wish you the best for the end of the discussion period.

Reviewer_CykfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

Sheaf neural networks are graph networks that use learned matrices to transform the messages that are passed along edges, and their performance is usually assigned to these matrices. This paper asks how much of that performance truly comes from the matrices rather than from simpler mechanisms, such as edge weighting. The authors show that some types of matrices are equivalent to simpler models, and they test each capability one step at a time. They find that how much credit the matrices get is not fixed, but depends on choices such as network depth, width and normalisation.

Critical strengths
  1. The experiments are carefully designed, and one key test (Section 5.3) was planned in advance before the results were known. The authors also openly report a prediction that did not hold (Appendix A.7). This matters because the paper is about how to measure gains properly, and this openness makes the results as presented easier to trust.
  2. The paper shows that small implementation choices, such as the type of normalisation that is used, can change the results and even reverse an effect (Section 5.2). Most papers do not report these choices (Appendix G). This is important because it tells researchers what they need to report when comparing sheaf neural networks.
Critical weaknesses
  1. I struggled to follow this paper. A lot is packed into the main text, with much of the important detail left to the appendix. In particular, it was hard to see how the different findings fit together. Because of this, I am not confident in my judgment of the contribution, which is why my confidence score is low. I think the paper needs substantial rewriting in order for it to be accessible to the ICLR audience, and this is the main reason for my overall rating.
  2. Many of the main results are based on only two datasets (Minesweeper and Roman-empire), and some findings differ on other datasets, such as Tolokers (Appendix A.9). I am therefore not sure how widely the stated conclusions apply.
Questions for authors
  1. Could the authors give a short, simple summary of the main takeaway for someone who wants to build or compare sheaf models?
  2. Do the authors expect their findings to hold on other datasets beyond the ones that they tested?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Cykf (1/2): presentation and takeaway (W1, Q1)

Thank you for your careful reading and for explaining the difficulties you encountered with the presentation. We recognize that the submission did not make the connections among the findings sufficiently clear, and we have worked to address this in the revision.

W1 (hard to follow; how the findings fit together). In response to your concern, we have made the following changes to clarify the main message and the connections among the findings. We have retained the original organization so that the changes can be compared directly with the submission. All changes are marked in blue:

  • The abstract is shortened and carries fewer numbers; its claims are unchanged.
  • The Introduction has a new paragraph, "How the parts connect", on how the theory, the protocol and the experiments relate.
  • The Introduction now ends with a paragraph, "Takeaways for practice", that states the one claim the findings share and what each implies for a practitioner, with the section where each is shown.
  • Section 2 now says informally what a sheaf layer is.
  • Section 5 opens with a paragraph of terms (rung, increment, sheaf-exclusive gain, share, our operator and the released one), and its roadmap now includes Section 5.3, which it had omitted.
  • Each of Sections 5.1 to 5.5 starts with its finding in one sentence, and the long paragraph of Section 5.1 is divided into four by topic.
  • Three hard sentences are glossed or reworded, with the same content: Standard 1 in Section 4, the last sentence of Section 5.2, and the budget identity in Section 5.4, which is now displayed.
  • Section 7 is now a Conclusion.

How the findings fit together, in three sentences (the new paragraph in the Introduction says the same more briefly). The theory (Section 3) says which capabilities are in fact the same function, so that a comparison knows what it must keep apart. The protocol (Section 4) measures what each remaining capability adds, one nested step at a time. The experiments (Section 5) then show one thing from several sides: the share credited to matrix-valued transport changes with the order of credit, the depth and the width (5.1), with the operator (5.2) and with what the scalar control is allowed to do (5.3); a data-side intervention is reallocated between rungs by depth and operator in the same way (5.4); and in a published model the sign of the effect depends on which runs are counted (5.5).

We agree that the paper would benefit from the more substantial rewriting you recommend. Our plan for it is to move secondary results (gauge, the quaternion rung, calibration details) out of the main text and to give each section a plainer opening. We have drafted this restructuring and would incorporate it into the final version if the paper is accepted.

Q1 (a short takeaway for someone building or comparing sheaf models). One sentence: the share of a sheaf network's gain that belongs to matrix-valued transport is a property of the measurement protocol as much as of the task, so state the protocol. In practice (this is the new paragraph ending the Introduction):

  1. Before crediting a gain to matrix-valued transport, give the scalar control the same freedoms: learned edge weights, signs, and a learned spectral response. On Minesweeper a learned degree-3 response on the scalar rungs absorbs 92% of the gain that would otherwise be credited to matrices (Section 5.3); on Roman-empire and Tolokers it absorbs no resolved part. The check has to be run, not assumed.
  2. State the normalization and the edge assembly. On Minesweeper the normalization accounts for most of a reversal in the sign of the four-layer depth effect (Section 5.2).
  3. Compare nested, warm-started models, and give the parent the child's training schedule as a control. In our runs the gain survives this in all eight cells, but only the control shows it.
  4. Report the absolute gain with the share, and the order in which capabilities were credited: on Roman-empire the same rotation earns between 13% and 94% of the total depending on whether it is added last or first (Section 5.1).
AuthorsOpenReview ↗

Response to Reviewer Cykf (2/2): how widely the findings apply (W2, Q2)

W2 / Q2 (two datasets; do the findings hold elsewhere?). It depends on which finding, and the revision now says so (last sentence of the takeaways paragraph).

  • The theorems (Theorem 1, Propositions 1 to 3) do not depend on data. A scalar-valued sheaf layer is an edge-weighted operator on any graph.
  • The headline claim is an existence claim: the share credited to matrix-valued transport is not intrinsic. One controlled counterexample establishes it, and we give several (order of credit, width, operator, depth, the spectral freedom of the scalar control). We expect it to hold wherever there is a sheaf-exclusive gain to allocate. Where there is none, as on Amazon-ratings in our runs (+0.0015 [−0.0002, +0.0032] at two layers, Table 1), the question does not arise.
  • Magnitudes and directions are dataset-specific, and we do not expect them to transfer. The paper shows this itself: the share is 53%, 11% and 23% on Minesweeper, Roman-empire and Tolokers at two layers (Table 1), and the effect of the normalization on the depth trend has the opposite sign on Tolokers (Section 5.2; the result you point to in Appendix A.9). We read that as an instance of the claim, not an exception to it: what replicates is that the depth behaviour belongs to the operator, not which normalization is favoured.

We also completed the third dataset where the submission had a single run. The preregistered substitution test on Tolokers now has ten seeds at both depths, and Roman-empire has forty (new Appendix I.2, Table 50). A substitution is found on Minesweeper only. So "most of the credited gain is a denied spectral freedom" is a Minesweeper result, and the paper says so; what generalizes is that the answer depends on what the scalar control is allowed to do.

Thank you again for your time; we would be glad to clarify anything that remains unclear.

Thank you for your responses. I think the new runs, together with the clarification that the main claim does not depend on the specific datasets, address my W2. I also appreciate the additional signposting in the updated paper, so I will increase my presentation to a 2. Overall I have raised my score to a 3. I would still encourage the authors to carry out the planned restructuring for the final version, as I still maintain that the main sections are quite dense. This would help the paper achieve a wider readership.

Reviewer_s64gOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The authors question the fact that sheaf networks benefit from matrix-valued transport. They then show that the answer to this question changes according to design choices such the network depth or normalization.

Critical strengths
  1. The conclusions are convincing. The authors show cases where all the gain from using matrices dissapears, such as with scalar filtering.

 

  1. The empirical methodology is done responsibly. Negative results are reported, nesting checks. The continued-parent controls are important comparisons.
Critical weaknesses
  1. While responsible empirical methodology is one of the strengths I mentioned, the choice of the way the experiments are done is problematic. In parallel the model gets more expressive and being trained more, which doesn't isolate the expressivity change. The apparent matrix improvement is three percentage points, but only one point exceeds what extra training alone achieved. The authors recognize this and run continued parent controls. To be fair, central findings do survive. But most headline attribution percentages still use original increments, including the extra-training. Thus the problem is the interpretation of the percentages, not that all the findings disappear. Correcting them is not a matter of subtraction.

 

  1. On some tasks, much of the apparent matrix benefit can be explained by freedoms denied from scalar baselines. E.g., for Roman-empire there are other possibilities for the reasons: matrix transport could provide a distinct benefit, the chosen scalar filter could be insufficient, or the experiments could lack enough precision.

 

In general, I am open to raise my score. I think the work is good and relevant but these concerns are critical. I hope the authors can show that the main conclusions survive a fair training-budget comparison and show evidence beyond Minesweeper, in which case I will seriously consider raising my score. Concretely, I think the most important thing to test is a signed-scalar baseline with the same total training schedule as the dense ladder.

Questions for authors

Beyond the existing continued-parent controls, does the scalar-versus-matrix performance gap persist when the signed-scalar baseline receives the same total training schedule as the dense ladder?

Which headline attribution claims remain supported?

3 comments
AuthorsOpenReview ↗

Response to Reviewer s64g (1/2): training budget (W1, Q1, Q2)

Thank you for your suggestions; they helped us strengthen the empirical evaluation. Both things you asked for are done. The control you named as the most important, a signed-scalar baseline with the dense ladder's total training schedule, is in this comment: the gap persists in all eight cells. The evidence beyond Minesweeper (Roman-empire with forty seeds, and Tolokers) is in our second comment.

W1 / Q1: does the gap persist at a matched schedule? Yes. You are right that the per-transition control of Appendix A.11 could not answer this, and that its corrected increments cannot be summed. We therefore trained the fitted signed-scalar rung again as itself, with no new parameters, for the three stages the ladder spends between it and the dense rung. The protocol and the full table are in the general response (Appendix I.1 and Table 48 of the revision).

  1. The gap persists. The dense rung's gain over that model excludes zero in all eight cells (two tasks, two operators, two depths).
  2. What the extra training accounts for. It is resolved in two cells, both Minesweeper under our operator: +0.010 [+0.004, +0.016] at two layers and +0.007 [+0.001, +0.012] at four, which is 18% and about a third of the ladder's sheaf-exclusive gain in those cells. In the other six cells it is not resolved (point estimates from −0.000 to +0.005).
  3. The percentages. In the re-run the four headline shares are 52% and 13% at two layers and 12% and 17% at four; budget-matched they are 53%, 13%, 8% and 17%. A share can stay where it was while the gain falls, because the scalar bottom of the ladder gains from continued training too and the total shrinks with it. You wrote that correcting the percentages "is not a matter of subtraction". We agree, and the chain shows why: each per-transition correction subtracts what one more stage buys a rung's own parent, and two of the three parents are themselves matrix-valued models, so the sum of the corrections removes part of the matrix-versus-scalar gap along with the effect of the schedule. That sum is below the budget-matched gain in all eight cells.
  4. Why we read this as "not schedule", and what it does not show. Over its three extra stages the scalar model's training cross-entropy falls by 0.014 or less, at most 27% of the dense rung's fall in the same cell. On Roman-empire it stops improving almost at once (the selected epoch averages 4 to 31, against 96 to 284 for the ladder's matrix-valued stages); on Minesweeper it still improves a little in its first extra stage, which is the effect in point 2. So the schedule was offered and little of it could be used. This does not by itself show that no scalar model could close the gap, because the chain continues the scalar model from the solution it had found. On Minesweeper under our operator the scalar rungs do reach the dense rung's level in a minority of seeds (5 of 40 in the submitted run, 7 of 40 in the re-run), which agrees with the substitution result on that dataset (Section 5.3). Under the released operator on Minesweeper and at four layers on Roman-empire no seed's scalar rung gets there: the smallest per-seed gain is +0.022 and +0.046, +0.062 and +0.043.

Q2: which headline attribution claims remain supported?

  • The shares 53% / 11% at two layers and 13% / 17% at four: supported, and now reported next to their budget-matched values (Section 5.1, paragraph "Training-budget controls"). The largest of them is the one a reader would worry about most, so we re-ran that cell with forty seeds, as in the submitted table. It reproduces it: sheaf-exclusive gain +0.0572 [+0.0478, +0.0666] against +0.0577 [+0.0485, +0.0670], share 52% against 53%.
  • The sheaf-exclusive gain is positive: supported in all eight cells after budget matching.
  • The depth reversal on Minesweeper: supported. It was already computed against continued parents (Table 32: −0.034 [−0.056, −0.012] under our operator, +0.044 [+0.025, +0.063] under the released one).
  • The spectral substitution (92%) and the one-hop and three-hop controls (98%, 71%): supported. Both arms of each were already measured against their own continued parents.
  • Operator (11% to 35%): the control was run under both operators, and the shares stay apart once the budget is matched: 13% under our operator and 36% under the released one on Roman-empire at two layers (Table 48; ten and forty seeds).
  • Width (46% to 73%) and order of credit (13% to 94%): these remain contrasts of ladder shares. We did not re-run them under the new control, and the revision says so (Limitations; Appendix I.1). The budget correction is about four points at most in the four headline cells, against differences of 27 and 81 points.
  • The −29.6-point result on a published architecture: does not involve the ladder and is unaffected.
AuthorsOpenReview ↗

Response to Reviewer s64g (2/2): Roman-empire, and evidence beyond Minesweeper (W2)

We agree that ten seeds could not choose between the three possibilities you list. The submission reported that on Roman-empire no interaction was resolved, in either direction, once the continuation budget was matched; before that control D was resolved positive at four layers and undetermined at two (Table 28). We have now taken the three in turn (two added sentences in Section 5.3, and a new Appendix I.2 with Table 50; the submitted tables are unchanged).

Precision. We re-ran the preregistered substitution test on Roman-empire with 40 seeds per cell, both arms on the same GPUs. G = S(dense) − S(sign); D = G_free − G_restricted, seed-paired; D^cont is the same interaction with both arms measured against their own continued parents. Intervals are 95% t intervals over the ten official split means, as elsewhere in the paper (with 40 seeds, the four seeds of a split are averaged first); † excludes zero.

LnG restrictedG freeDD^cont
240+0.0181 [+0.0145, +0.0218]†+0.0219 [+0.0199, +0.0238]†+0.0037 [+0.0001, +0.0073]†−0.0022 [−0.0057, +0.0012]
440+0.0588 [+0.0541, +0.0634]†+0.0689 [+0.0639, +0.0739]†+0.0102 [+0.0046, +0.0158]†+0.0023 [−0.0046, +0.0092]
2 (submitted, Table 28)10+0.0179+0.0237+0.0059 [−0.0016, +0.0133]−0.0013 [−0.0046, +0.0020]
4 (submitted, Table 28)10+0.0577+0.0743+0.0166 [+0.0045, +0.0287]†+0.0084 [−0.0101, +0.0269]

Every 40-seed point estimate lies inside the submitted ten-seed interval, and the interval of D is about half as wide. Two things are settled that were not. At two layers D is now resolved positive (only just), where ten seeds left its direction undetermined; at four layers it already was. So at both depths a free spectral response helps the matrix rungs at least as much as the scalar ones, the opposite of a substitution. And at four layers the interaction against the continued parents now lies inside the preregistered practical margin of ±0.01, where the ten-seed interval still contained effects of practical size in either direction; at two layers it already did. So under the budget control there is no interaction of practical size on Roman-empire, and precision is no longer what limits the statement.

The scalar filter. A learned degree-3 scalar response does not absorb the Roman-empire gain. The lower ends of the controlled intervals bound what it can take over at 0.006, against a matrix gain over the continued parents of +0.0149 and +0.0419 in the restricted arm; on Minesweeper the same test absorbs 0.055 and 0.085, 94% and 92% of the gain. Within the polynomial family the degree does not change this: the sweep over K ∈ {2, 3, 5} (Table 29) has positive point estimates of D on Roman-empire at every degree. What we have not tested is a filter outside this family; the Limitations now name the family, and we do not claim that no scalar filter could do it.

A distinct benefit of matrix transport. This is the explanation the data now support on Roman-empire, with that qualification. The sheaf-exclusive gain survives the cumulative-budget control (+0.0175 [+0.0146, +0.0205] and +0.0573 [+0.0527, +0.0618] over the scalar rung trained for the same three extra stages), and it is still there when the scalar rungs have a free spectral response (+0.0127 [+0.0097, +0.0157] and +0.0442 [+0.0390, +0.0495] over the continued parents).

Beyond Minesweeper. The same test on Tolokers with ten seeds per cell (the submission had one exploratory run) gives D = +0.0033 [−0.0009, +0.0074] at two layers, inside ±0.01, and +0.0064 [−0.0012, +0.0140] at four, not resolved; against the continued parents +0.0020 [−0.0034, +0.0075] and +0.0038 [−0.0030, +0.0106]. The restricted gain itself is small there (+0.0066 [+0.0018, +0.0113] and +0.0095 [+0.0044, +0.0146]). So of three datasets a substitution is found on one. This is in line with the paper's claim: whether the credited share depends on what the scalar control may do is itself task-specific, now with resolved intervals on two datasets and a third that excludes a substitution of practical size at two layers.

The submitted statement about Roman-empire ("nothing survives that control") still holds with forty seeds, so it stays; Section 5.3 gains two sentences, and the full results are in Appendix I.2 (Table 50). We also corrected the last sentence of Section 5.3, which said that no arm resolved anything on Roman-empire: as Table 3 shows and as stated above, the four-layer interaction was resolved with ten seeds before the budget control, and not after it.

Thank you again for the precise and fair review; both experiments were worth running. If these answers address your concerns, we would be grateful if you would consider updating your assessment.

Thank you for the additional experiments and careful response. The cumulative-budget control substantially resolves W1, and the expanded results clarify the scope and task dependence raised in W2. I am raising my rating to 3. The width/order comparisons still lack the cumulative control, so that caveat remains. My confidence stays at 2, particularly regarding the theoretical analysis.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): what changed in the revision

We thank the three reviewers for their careful and constructive reviews. The revision is better for them: the control Reviewer s64g proposed is now the paper's main robustness check, and the questions of Reviewers Cykf and orKP led to a clearer statement of what the paper claims and what it leaves open. In response to the reviews, we have focused on making the connections among the findings clearer and strengthening the evidence beyond Minesweeper. Reviewer s64g also asked for one specific experiment. This comment says what changed in the revised PDF; the next one reports the two experiments we added, and replies to each reviewer follow separately.

What changed in the PDF. We kept the organization of the submission so that the changes can be compared directly with it. Apart from the abstract, which is shortened, two sentences reworded for clarity (the last sentence of Section 5.2 and the budget identity in Section 5.4, which is now displayed) and two sentences corrected to agree with the tables (the last sentence of Section 5.3 and the description of the experimental scope in the Limitations), the submitted text is not rewritten or moved, and changes are marked in blue. Every table, figure and section keeps its number, because all new tables and the longer new text are in a new last appendix (Appendix I). The main text is ten pages. The supplementary material is unchanged; the code and per-seed records of the two added experiments will be released with the rest of the code.

  • Abstract: shortened for readability, with the same claims; it now also reports the two experiments below (Cykf W1).
  • Introduction: a paragraph on how the theory, the protocol and the experiments connect, and a closing paragraph, "Takeaways for practice" (Cykf W1, Q1, Q2).
  • Section 2: one informal sentence on what a sheaf layer is (Cykf W1).
  • Section 4: one sentence stating Standard 1 in words (Cykf W1).
  • Section 5: a paragraph of terms; one sentence stating the finding at the head of each subsection; Section 5.3 added to the roadmap; the long paragraph of Section 5.1 divided by topic, its sentences unchanged (Cykf W1). Sections 5.1 and 5.3 each gain a few sentences with the new results (s64g W1, W2).
  • Section 7 is now titled "Conclusion" (orKP W2). The Limitations name two open questions (orKP W1, Q3) and the filter family the substitution test uses (s64g W2).
  • Appendix I.1: the cumulative-budget control, Tables 48 and 49. Appendix I.2: the substitution test with forty seeds and a third dataset, Table 50.

Scope of the claims. The paper makes three kinds of statement, and the revision now says so. The results of Section 3 hold on any graph. That the share credited to matrix-valued transport is not intrinsic is an existence claim, which one controlled counterexample establishes. The magnitudes are specific to the datasets tested, and we report them as such.

AuthorsOpenReview ↗

General response (2/2): the two added experiments

Experiment 1: the training-budget control Reviewer s64g asked for. The fitted signed-scalar rung is trained again as itself, with no new parameters, for the three stages the ladder spends between it and the dense rung, each stage under the budget, optimizer reset, loss cap and selection rule of the ladder stage it mirrors. The budget-matched gain is the dense rung's score minus that model's; the budget-matched share divides it by the dense rung's gain over the identity rung continued for all five stages. Intervals are 95% t intervals over the ten official split means; † excludes zero.

DatasetoperatorLnsheaf-exclusive gain, ladderbudget-matched gainshare, laddershare, budget-matched
Minesweeperours240+0.0572 [+0.0478, +0.0666]†+0.0468 [+0.0377, +0.0558]†52%53%
Minesweeperours410+0.0196 [+0.0110, +0.0283]†+0.0129 [+0.0028, +0.0229]†12%8%
Minesweeperreleased210+0.0537 [+0.0388, +0.0686]†+0.0490 [+0.0327, +0.0654]†56%64%
Minesweeperreleased410+0.0928 [+0.0788, +0.1068]†+0.0900 [+0.0744, +0.1056]†55%55%
Roman-empireours210+0.0208 [+0.0165, +0.0251]†+0.0208 [+0.0165, +0.0252]†13%13%
Roman-empireours410+0.0726 [+0.0671, +0.0782]†+0.0722 [+0.0671, +0.0773]†17%17%
Roman-empirereleased240+0.0181 [+0.0145, +0.0218]†+0.0175 [+0.0146, +0.0205]†37%36%
Roman-empirereleased440+0.0588 [+0.0541, +0.0634]†+0.0573 [+0.0527, +0.0618]†28%28%
Tolokersreleased210+0.0066 [+0.0018, +0.0113]†+0.0042 [+0.0001, +0.0083]†14%22%
Tolokersreleased410+0.0095 [+0.0044, +0.0146]†+0.0078 [+0.0021, +0.0134]†16%22%
  • The matrix-versus-scalar gap excludes zero in all eight cells of the two main datasets, and on Tolokers.
  • The extra training is resolved in two cells, both Minesweeper under our operator: +0.010 [+0.004, +0.016] of the sheaf-exclusive gain at two layers and +0.007 [+0.001, +0.012] at four, which is 18% and about a third of the ladder's gain there. It is not resolved in the other six cells. The four headline shares are 52% and 13% at two layers and 12% and 17% at four in this re-run, and 53%, 13%, 8% and 17% once the budget is matched: a change of about four points at most. The submission's forty-unit table has 53%, 11%, 13% and 17%.
  • The continued scalar model uses little of the schedule it is offered: its training cross-entropy falls by 0.014 or less, at most 27% of the dense rung's fall in the same cell. This shows that the gap is not a matter of schedule. It does not by itself show that no scalar model could close it, and we say where that matters: on Minesweeper under our operator the scalar rungs reach the dense rung's level in a minority of seeds (Appendix I.1).

Experiment 2: beyond Minesweeper. We re-ran the preregistered substitution test with forty seeds per cell on Roman-empire, and with ten seeds per cell at both depths on Tolokers, where the submission had one exploratory run. On Roman-empire the interaction D is resolved positive at both depths, the opposite of a substitution: +0.0037 [+0.0001, +0.0073] at two layers, only just and new with forty seeds, and +0.0102 [+0.0046, +0.0158] at four, where ten seeds had resolved it already. Against the continued parents it lies inside the preregistered margin of ±0.01 at both depths (−0.0022 [−0.0057, +0.0012] and +0.0023 [−0.0046, +0.0092]); at four layers the ten-seed interval did not. On Tolokers D is +0.0033 [−0.0009, +0.0074] at two layers, inside the margin, and +0.0064 [−0.0012, +0.0140] at four, not resolved. Of three datasets, a substitution is found on Minesweeper only (Appendix I.2, Table 50).

A note on seeds. Our earlier reply to Reviewer orKP described the budget control as run with ten seeds in every cell. Three cells have since been extended to forty seeds, and the conclusion stated there is unchanged. Roman-empire under the released operator (both depths) was extended because that run is also one arm of the forty-seed substitution test. Minesweeper under our operator at two layers, the cell with the largest share, was extended so that the re-run could be compared on equal terms with the forty-unit table of the submission; it reproduces it (sheaf-exclusive gain +0.0572 [+0.0478, +0.0666] against +0.0577 [+0.0485, +0.0670]; share 52% against 53%). The other cells have ten seeds. Both experiments ran on other hardware than the submitted tables, so every comparison above is within run and seed-paired.

We thank the reviewers again for their time and care.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.