acceptodds
Under review as a conference paper at ICLR 2027

What Does Comparing Supervision Targets Identify? Separating Supervised Vectors from Candidate Weighting

Abstract

When one supervision target outperforms another, what does the comparison establish about its supervised vectors? In sample-coupled objectives, changing a target can also change the candidate weights governing its learning updates. We test the adoption consequence of separating these roles: keep paired and candidate vectors fixed and replace their weighting input with reference supervision. In controlled electroencephalography (EEG)-to-visual learning, this substitution turns a coordinate-rescaled target from harmful to beneficial on a cross-image concept-ranking task with model-specific neural templates. A control matching actual AdamW step lengths while retaining its own image-parameter update direction remains harmful. A prospective five-crop test also reproduces a natural target's preference for a reference-informed temperature projection with matched update lengths and positive adoption in the same setting. These interventions retain reference information in weighting; they establish task benefits conditional on that treatment. A negative ordinary comparison therefore need not justify discarding the supervised vectors: their adoption verdict can reverse against the same reference after replacing only the weighting input. Target comparisons support configuration selection, while content attribution requires evidence separating these roles.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, supervision target design, contrastive learning, knowledge distillation, candidate weighting, EEG decoding

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_fJ9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether comparing supervision targets in sample-coupled objectives can properly attribute performance differences to the supervised vectors themselves, given that changing a target also changes the candidate weights governing learning updates. The authors present separating these roles by keeping paired and candidate vectors fixed while replacing only the weighting input with reference supervision.

Critical strengths

The paper presents a controlled experimental design that effectively isolates the weighting input from the supervised vectors, providing a clear counterexample to the common practice of discarding supervised vectors based on negative adoption results.

Critical weaknesses

The main positive adoption effect is extremely small, translating to approximately 0.40 correct-concept rank positions among 166 candidates, raising questions about practical significance despite statistical resolution, and the authors themselves acknowledge this is not a cost-adjusted deployment recommendation.

The paper relies heavily on exposed concepts that were used during earlier model development, and the generalization claims to unseen concepts, single-trial EEG, and fixed external decision models are explicitly left unresolved, substantially limiting the scope of the central claim.

Questions for authors

Given that the main adoption effect translates to only 0.40 rank positions among 166 candidates, what concrete practical scenario would justify adopting NR over FULL or reference supervision given the additional computational cost of recomputing weights from reference inputs?

The state-provider interaction test shows that replacing Z by U has a larger benefit under reference outputs, but both C and D conditions have negative adoption relative to U. Does this not suggest that the benefit of the intervention depends on access to a reference learner that itself performs better, which may not be available in practical applications?

The paper tests 20 participants but the TC and X2 experiments use only 19 participants after predictor development on one participant. How was this participant selected, and could the removal of this participant systematically affect the reported preference and adoption results?

The visual classification experiments on CIFAR-100 leave all intervention gains and adoption unresolved. Given that the central claim is about general adoption reversal, what evidence beyond EEG supports the generality of the proposed separation of supervised vectors from candidate weighting?

The paper states that FULL and NR adoption remain unresolved for X2, yet TC adoption is positive. Given that FULL and NR are the primary comparison treatments, does the unresolved status of these contrasts weaken the claim that TC represents a genuine improvement over the tested alternatives?

The matched-length control MC preserves its own FULL direction and AdamW moments while matching NR step length. Could the failure of MC to reproduce the gain be attributed to differences in the direction of updates rather than the weighting input specifically, and how would the authors distinguish these explanations?

Reviewer_5aJvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When choosing training targets for distillation or contrastive learning, practitioners compare targets by final performance and credit the winner with carrying better information. This paper investigates whether a target's gain or loss comes from what it encodes or from how it reweights examples during training, and separates the two by keeping the new target as what the model learns to match while computing the example weights from the old reference target. On EEG to image decoding (identifying which of 166 object concepts a person was viewing from their brain signals), it shows that a target that hurts under normal training becomes slightly helpful under this change, a gain of about 0.4 rank positions out of 166. It further shows that this gain is not explained by the change in update size alone, and that the best way to set the example weights depends on the target.

Critical strengths

S1. Eq. 6 shows that when a new target Z replaces a reference target U in the contrastive loss, the change in the gradient splits into three terms: (i) the pull toward the target, (ii) the candidate vectors, and (iii) the weights given to each candidate. The NR intervention (keep Z as the target but compute the weights from U) removes only the third term, so what Z encodes can be tested separately from how Z reweights examples. This is what makes the paper's question testable, and it is the main reason I see value in the work.

S2. The step-length control rules out the simplest explanation. MC rescales FULL’s actual AdamW update to NR’s step length at every step while keeping FULL’s direction, and NR still beats it by 1.50 rank positions, positive in 19 of 20 participants (Section 5, Appendix D.1). So NR’s gain on T+ is not just smaller steps.

Critical weaknesses

W1. The harmful-to-beneficial reversal is shown only for a constructed target. The paper’s counterexample, a target that hurts under ordinary training (FULL) but helps once only its weighting input changes, is shown only for T+, which is built by rescaling half of the reference target’s coordinates by a 2:1 ratio. For the natural targets X and X2, FULL adoption is not resolved as harmful (Table 4), so there is no negative verdict to reverse, and NR adoption on X is itself unresolved (Table 3b). TC improves on X and X2 (Section 6), but that is a better weighting treatment, not a reversal. In every other setting of Table 1, NR adoption is unresolved or negative, except Timg with a CNN, which also changes model capacity and whose FULL adoption is unresolved (Appendix G.1).

W2. The effect is small and measured on concepts used during development. NR's adoption gain over the reference target is 0.40 of 166 rank positions (Section 5). The 166 evaluation concepts were used during earlier development (Section 3), and each model supplies its own query outputs and concept templates, so this is not a standard decoding benchmark. Effects this small need confirmation on new concepts.

W3. The study was designed adaptively. Per Table 2, NR and MC were specified after the FULL results, the Timg success criterion was revised after the reference results, the TC advantage on X was discovered after the TC runs, and X2 and the Figure 4 design followed those outcomes. The discovered results carry nominal intervals without correction for multiple comparisons (Section 6.2).

W4. The contribution is narrow and does not support the general claim. Appendix H.1 concedes that separating supervision from weighting is not new (DKD, Zhao et al. 2022; CSD, Kim et al. 2026; Tian 2022) and claims only a same-target counterexample, yet the title and conclusion speak about target comparisons in general. All positive results use one symmetric contrastive loss on one EEG pipeline; the only non-EEG test, CIFAR-100 (Table 1, IV), is unresolved, and relational or contrastive distillation (RKD, cited; CRD, Tian et al. 2020) and CLIP-style training are not tested. Even within this setting, the better treatment depends on the target (NR for T+, TC for X), and the paper's predictor for this choice fails on X, with concept-level error 14.3% worse than predicting no change (Figure 3C), so a practitioner cannot tell in advance how to treat a new target.

W5. The results are not compared with standard EEG to image decoders. There is no comparison with NICE (Song et al. 2024) or ATM (Li et al. 2024), and no result on the standard THINGS-EEG2 zero-shot split of 200 unseen concepts, so it is unclear whether a gain of 0.4 rank positions matters relative to the differences between decoders.

W6. The paper is hard to follow. Section 2 defines eight contrasts (D0, D1, I, R, DNR, S, Dm and H) in two paragraphs, and terms such as "adoption", "receiving", "qualifies" and "state provider" have paper-specific meanings, so the abstract is hard to follow without the body.

Questions for authors

Q1. NR's gain on T+ is not explained by step length (MC) or by temperature: TC, which fits a temperature to the reference weights, has negative adoption on T+ (Appendix E.1). What property of the weights computed from U does NR restore? For example, how do the entropy and ranking of W(R, T+) and W(R, U) differ during training?

Q2. What is the NR gain on concepts never used during development, for example the 200 test concepts of THINGS-EEG2, reported as top-1 and top-5 accuracy?

Q3. Does NR help on a standard distillation or contrastive learning benchmark, for example CRD (Tian et al. 2020) or CLIP-style training?

Q4. Is there a natural target for which FULL adoption is resolved as negative and NR or TC makes it positive? That is the case the paper's argument needs.

A reversal for a natural target (Q4), together with a positive result on unseen concepts (Q2), would raise my score.

Reviewer_ZNnEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies EEG-to-image decoding with a contrastive loss, in which a target embedding serves both as the vector the model output is pulled toward and as the input that determines the loss’s softmax weights. The authors construct a modified target by scaling the coordinates of an original target and show that it lowers performance but slightly improves on the score when the weights are computed from the original target instead. Additional experiments show that the result does not reliably transfer to other targets, architectures, participants, or datasets.

Critical strengths

I did not identify any strengths that support acceptance of the paper. My reasons for this assessment are detailed in the weaknesses section below.

Critical weaknesses

1: Significantly limited conceptual and empirical contribution

Overall, this paper tests a very specific and unlikely case in a single modality, and reports a very small and inconsistent improvement in that specific condition. It therefore provides little or no contribution to science nor to the community.

  • The main experiment is centered around a few scenario where the training target is modified in practically unlikely ways. Specifically, they first consider a case where half of the coordinates of the original target is multiplied by a scalar twice as large as that applied to the other half. Then the author state that resulting drop in performance is recovered when the weights of the contrastive loss is replaced with the original target. This construction has no practical relevance and no generalizability. They also perform the experiment with a few other modifications, including features obtained from "unblurred images" and cropping, but the results do not reproduce the improvement and merely shows the instability of the results.
  • The authors claim an improvement in the scores, but the effect is unusually small. In the main condition, the score (the rank of the correct concept among 166 candidates, rescaled so that chance is 0.5 and the maximum is 1) increases from 0.6056 to 0.6080, a relative improvement of only 0.4%.
  • In addition, the idea of separating the two roles of a target in a contrastive loss is not novel as the authors themselves acknowledge.

2 Poor writing and presentation

The quality of the writing is below the standards of academic writing, and far below the quality expected at this top-tier international conference. The writing problems makes it significantly hard for readers to understand what the authors have done and nearly impossible to precisely assess the paper.

  • Many unique terms and notations appear before they are properly defined, and in some cases the definition is nowhere to be found. In the Introduction alone, "EEG-derived image outputs" and "templates" (L. 081), and "coordinate rescaling" (L. 086) are undefined. The parameter λ\lambda is used in Section 2 (L. 116) but defined only in Section 4 (L. 201–202), and the "nine-variant dynamic targets" from which the baseline target is computed (L. 146–147; L. 701) are never described.
  • Some mathematical notations are used inconsistently. For example, the first subscript of the model symbol ff switches its meaning arbitrarily: sometimes it is a value of λ\lambda in f1,Z0f_{1,Z_0} but sometimes it is the name of a training condition in fNR,Zf_{\mathrm{NR},Z} (L. 115–133). The same issue applies to the score difference DD (L. 120–133). Likewise, ZZ and UU switch between denoting specific targets and serving as placeholders for generic targets (e.g., L. 095–096 vs. L. 380).

3 Questionable research procedure

The authors acknowledge that the success criterion for one experiment was revised after the results were seen. In addition, several main comparisons were designed after earlier results were known, and the evaluation concepts were used during earlier model development (L. 141; Table 2; L. 673–674). This raises serious doubts about the validity of the results.

Questions for authors
  1. Could the authors clarify the significance of this study and its contribution to the field and the broader community?
  2. Could the authors revise the manuscript so that every term, notation, and method is clearly defined and presented in a standard academic writing style?
  3. Could the authors justify why the revision of the success criterion after the results were known does not constitute HARKing?
Reviewer_HbdoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at sample-coupled objectives and argues that the supervision target has two effects in these objectives. The first is supplying the actual supervised vectors, and the second is providing the weighting of how to compare candidates. What this means is that you can get a worse-performing target compared to a reference without the target being uninformative. And so they introduce their technique, which keeps the comparison while using the weights from a reference, and apply this to an EEG-to-image learning problem. They're able to show some positive benefit in that setting, although broader tests are more mixed.

Critical strengths

The way they do the control is incredibly clean, and by using the Adam control they're able to show that the gain they see is not explained by another mechanism. They also report all of their limitations very carefully and acknowledge the limitations of this set of experiments in identifying the value of the supervised vectors in themselves.

Critical weaknesses

The counterexamples effect size is pretty small, and is confounded by the reference. And in fact, they acknowledge that this is the case. This makes it difficult to figure out how important, this is, especially since the paper is quite complex. Furthermore, the target they're talking about is this specifically constructed one, as opposed to showing an effect on new participants in their broader study.

Questions for authors

It'll be useful to understand how the choice of targets and interventions were described in the first place, and therefore whether or not the reported evidence is truly representative, and it would also help to expand this to a broader class of rescalings to see if the gains are comparable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.