- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether comparing supervision targets in sample-coupled objectives can properly attribute performance differences to the supervised vectors themselves, given that changing a target also changes the candidate weights governing learning updates. The authors present separating these roles by keeping paired and candidate vectors fixed while replacing only the weighting input with reference supervision.
Critical strengths
The paper presents a controlled experimental design that effectively isolates the weighting input from the supervised vectors, providing a clear counterexample to the common practice of discarding supervised vectors based on negative adoption results.
Critical weaknesses
The main positive adoption effect is extremely small, translating to approximately 0.40 correct-concept rank positions among 166 candidates, raising questions about practical significance despite statistical resolution, and the authors themselves acknowledge this is not a cost-adjusted deployment recommendation.
The paper relies heavily on exposed concepts that were used during earlier model development, and the generalization claims to unseen concepts, single-trial EEG, and fixed external decision models are explicitly left unresolved, substantially limiting the scope of the central claim.
Questions for authors
Given that the main adoption effect translates to only 0.40 rank positions among 166 candidates, what concrete practical scenario would justify adopting NR over FULL or reference supervision given the additional computational cost of recomputing weights from reference inputs?
The state-provider interaction test shows that replacing Z by U has a larger benefit under reference outputs, but both C and D conditions have negative adoption relative to U. Does this not suggest that the benefit of the intervention depends on access to a reference learner that itself performs better, which may not be available in practical applications?
The paper tests 20 participants but the TC and X2 experiments use only 19 participants after predictor development on one participant. How was this participant selected, and could the removal of this participant systematically affect the reported preference and adoption results?
The visual classification experiments on CIFAR-100 leave all intervention gains and adoption unresolved. Given that the central claim is about general adoption reversal, what evidence beyond EEG supports the generality of the proposed separation of supervised vectors from candidate weighting?
The paper states that FULL and NR adoption remain unresolved for X2, yet TC adoption is positive. Given that FULL and NR are the primary comparison treatments, does the unresolved status of these contrasts weaken the claim that TC represents a genuine improvement over the tested alternatives?
The matched-length control MC preserves its own FULL direction and AdamW moments while matching NR step length. Could the failure of MC to reproduce the gain be attributed to differences in the direction of updates rather than the weighting input specifically, and how would the authors distinguish these explanations?