acceptodds
Under review as a conference paper at ICLR 2027

Testing Peer Assignment in Selection from Masked Views

Abstract

A classifier can predict different labels from incomplete versions of the same document. Using agreement between these views can improve selection, but an improvement over maximum confidence leaves open whether the supporting views need to be semantically similar. We test that question by fixing the views, class probabilities, and scoring rule, and changing only whether peers are nearest in MiniLM space or sampled uniformly from the other views. Across six sentiment-classification settings, nearest-peer selection reduces error by 2.56–5.66 percentage points relative to maximum confidence. Uniform-minus-nearest error differences are much smaller, from to points, and all six 90% seed intervals lie inside a study-specific -point band. The two selectors often choose different views but return the same label in 97.0–97.9% of cases. Topic-classification and Naive Bayes studies also yield small peer-assignment effects, while several sensitivity settings remain less precise. The matched control separates the value of an agreement score from the incremental value of its particular semantic neighborhood in masked-view classification.

Primary area:
interpretability and explainable AI
Keywords:
text classification, masked-view selection, agreement-based selection, peer assignment, semantic neighborhoods, controlled attribution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_q1GBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies peer assignment in agreement-based selection among word-deleted views of the same document, comparing semantic nearest neighbors with uniformly sampled peers under fixed candidate views, classifier predictions, and scoring rules.

Critical strengths
  1. The controlled comparison isolates the effect of peer assignment.
  2. Distinguishing selected views from predicted labels clarifies why selection changes may not affect predictions.
Critical weaknesses
  1. Limited novelty: The paper lacks a novel algorithm, and the contribution beyond empirical comparison is limited.
  2. Insufficient analysis: The semantic and predictive differences between nearest and random peers are not quantified.
  3. Limited view diversity: The views are generated by randomly deleting words from the same document, resulting in highly correlated and potentially redundant views that may limit the ability to assess the value of semantic neighborhood selection.
Questions for authors
  1. How do nearest and random peer sets differ in semantic similarity and predictive distributions?
  2. How often do all views predict the same label, and how does peer assignment affect cases with label disagreement?
  3. How stable are results under repeated peer sampling, and what justifies the ±1.5 pp margin?
3 comments

Thank you for recognizing that the controlled comparison isolates peer assignment.

W1. A finding about existing methods is a contribution in its own right. To our knowledge, this is the first matched test of whether agreement-based selectors need semantically nearest peers. The answer, that the score matters (2.56–5.66 pp) but the neighborhood adds little (−0.46 to +0.50 pp), is directly useful to anyone designing such selectors. We would welcome a reference if this has been shown before.

W2, Q1. Nearest peers have mean MiniLM cosine 0.82 vs 0.72 for uniform peers, the two sets share only 30% of members (Jaccard 0.30), and Jensen–Shannon divergence is 0.07–0.09 vs 0.09–0.11 nats. The arms therefore differ substantially. Using the classifier's own embedding for neighbors gives contrasts of −0.23 to +0.38 pp, every 90% interval including zero.

Q2. All 8 views agree on 52–56% of documents. On the rest, agreement cuts error from 26.2–29.9% to 16.0–21.2%, while the uniform-minus-nearest contrast stays within −0.96 to +1.10 pp; the only interval excluding zero favors uniform peers.

Q3. Over 30 redraws of uniform peers, mean contrasts are 0.00–0.43 pp (SD 0.15–0.28), all 5th–95th percentiles within [−0.39, 0.66] pp. The paper's one interval excluding zero has a redraw mean of 0.02 pp. The ±1.5 pp band was fixed in advance (Section 3.3) and lies below the smallest agreement gain (2.56 pp).

W3. Both arms draw from the same views, and W2 shows their peer sets still differ substantially, so correlation does not remove the contrast.

Every question is now answered with direct measurements that support the paper's conclusion, and we ask the reviewer to take this into account in the rating.

Thank you again for your review. We hope our response has addressed your concerns. As the discussion period is coming to an end, we would be glad to answer any remaining questions. If our response has resolved your concerns, we would greatly appreciate it if you could reconsider your assessment of our paper in light of it.

I thank the authors for their careful response. After reading the rebuttals and other reviewers comments, I think my rating is fair and I decide to maintain it.

Reviewer_6YCeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

In this manuscript, the authors generate multiple views by randomly deleting words from the original text and compare selection strategies. The selection strategies include maximum confidence, majority voting, semantic nearest-neighbor agreement, and randomly sampled peer agreement. The authors concluded that agreement-based selection improves over maximum-confidence selection, while semantic nearest-neighbor selection provides little additional benefit over random peer selection.

Critical strengths
  1. The paper is generally well organized. The proposed method and experimental procedure are clearly described.

  2. The authors evaluate on multiple datasets with different classifiers and conduct sensitivity analyses.

Critical weaknesses
  1. The paper does not seem to describe why the investigated problem is important. The authors generate multiple incomplete views by randomly deleting words and then develop a selection procedure to identify a reliable prediction among these views. However, it is unclear why such a procedure is useful in practice. I am not sure why this is likely to be beneficial, when the original, complete document is available.

  2. Multiple-view methods has been extensively studied to utilize information from different sources in decision making. The proposed method looks relatively straightforward. It is unclear how the proposed method differed from previous multi-view learning methods. The methodological contribution looks limited.

  3. In the experiment, the baseline that directly classifies the original documents without deleting is not included. I was unable to understand the motivation of the proposed method. Do the authors try to demonstrate that randomly deleting words can improve the prediction performance compare to directly using the original, complete document?

Questions for authors
  1. What is the motivation of studying such a problem? Why is it interesting to generate multiple incomplete document when the original document is available?

  2. What is the methodological contribution on multi-view learning methods?

  3. How does the proposed approach compare with classifying the complete documents without deleting?

2 comments

Thank you for finding the paper well organized and clearly described. The review raises no soundness concern, and we believe the rating of 1 comes from a misreading of the paper's goal.

W1, W3, Q1, Q3. The paper never claims that deleting words improves on the complete document, and it does not propose a selector. It tests an assumption behind self-consistency, semantic entropy and CoCoA: that supporting outputs should be semantically nearest (Sections 1–2). Masked views make this test exact. For completeness, complete-document vs nearest-peer error is 10.13 vs 12.16% (RT/BERT), 10.32 vs 12.32% (RT/DistilBERT), 9.57 vs 12.01% (RT/RoBERTa), 7.57 vs 10.40% (SST-2/BERT), 8.94 vs 11.47% (SST-2/DistilBERT), 5.96 vs 8.45% (SST-2/RoBERTa), 12.75 vs 12.83% (Naive Bayes) and 5.00 vs 5.46% (AG News). Simply averaging the 8 views' probabilities is slightly better than nearest-peer selection in five of six primary settings, which supports the paper's conclusion that the gain comes from pooling support, not from semantic neighborhoods.

W2, Q2. The paper is not a multi-view learning method, so it should not be judged as one. It holds views, outputs and score fixed and changes only peer membership (Section 3.2). The finding: agreement reduces error by 2.56–5.66 pp, while nearest vs uniform peers differ by only −0.46 to +0.50 pp. If prior work has isolated peer assignment this way, we would appreciate the reference.

We respectfully ask the reviewer to reconsider the rating.

Thank you again for your review. We hope our response has addressed your concerns. As the discussion period is coming to an end, we would be glad to answer any remaining questions. If our response has resolved your concerns, we would greatly appreciate it if you could reconsider your assessment of our paper in light of it.

Reviewer_2kjMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

When a classifier predicts different labels from incomplete (masked) views of the same document, simple selectors may ignore disagreement, while agreement-based selectors reward candidates supported by other views. Using agreement between these views can improve selection, but an improvement over maximum confidence leaves open whether the supporting views need to be semantically similar. The paper investigates this question by fixing the views, class probabilities, and scoring rule, and changing only whether peers are nearest in MiniLM space or sampled uniformly from the other views.

Contributions:

  1. The authors propose an approach that keeps candidate views, class probabilities, and scoring rules fixed, but swaps out nearest semantic supporting peers for uniformly sampled peers from the same document.
  2. The paper perform diagnostics across multiple model architectures, tasks, and parameter sensitivities.
Critical strengths

The evaluation spans multiple model families (transformers and Naive Bayes), datasets (Rotten Tomatoes, SST-2, AG News), and sensitivity analyses.

Critical weaknesses

1.The paper is poorly written and organized. Even in the abstract and introductions. it moves quickly into details without a high-level roadmap. This needlessly increases the cognitive load and makes it hard to follow. Rewriting the abstract & intro, plus adding an intuitive running example would help. 2. The paper should also compares its selector against the state-of-the-art (SOTA) models for the considered domains. As a sanity check, the authors should add to the tables the SOTA results for each domain, so that the reader can evaluate the practical impact.
3. The paper relies solely on artificially-created views, which, again, raises the issue of practical applicability. Multi-view learning has been around since at least the 1990s, so the authors should use at least one real-world motivating problem, which would make the paper more compelling. 4. The paper omits metrics regarding runtime, memory overhead, and computational cost.

Questions for authors

n/a

4 comments

Thank you for the review. We respectfully disagree with the main criticisms, which read the paper as a classifier proposal.

W1. The other two reviewers rate the presentation as "Clear and easy to follow", and Reviewer 6YCe finds the paper "generally well organized". The running example the review asks for is already the first paragraph of the paper, and Section 1, paragraphs 2–4, give the roadmap.

W2. The paper does not propose a classifier. Its quantity is a within-pipeline contrast, uniform-peer minus nearest-peer error with the same classifier, views and score (Equation 1). A stronger classifier changes both arms equally, so SOTA results do not bear on it. For scale, complete-document error is 5.96–10.32% in the six primary settings, versus 8.45–12.83% for the nearest-peer selector (Table 2).

W3. Constructed views are what make the test exact: all predictions are computed once, and only the peer set changes. Multi-view learning combines sources during training; here nothing is trained, and the question is which outputs an agreement-based selector should use at inference time (Section 2). If the reviewer knows a natural setting that allows this intervention, we would welcome the reference.

W4. Measured on RT/RoBERTa (L40S, per document): classifying 8 views takes 2.24 ms, MiniLM embedding 0.76 ms, nearest-peer search 0.02 ms, uniform sampling 0.14 ms. Semantic neighborhoods add about one third to the cost while, as the paper shows, uniform peers perform about as well.

We ask the reviewer to reconsider whether a rating of 1 reflects the paper's actual question.

Reviewer_2kjMOpenReview ↗

Thank you for your comments. After carefully reading all reviews and rebuttals, I maintain may original rating.

Thank you for your comments. After carefully reading all reviews and rebuttals, I maintain may original rating.

Thank you for your prompt reply. Our response addressed each of the four points you raised (writing, SOTA comparison, constructed views, and computational cost) with clarifications and new measurements. Could you let us know which specific concerns remain unresolved? We would be glad to continue the discussion during the remaining period. If these concerns have been addressed, we would kindly ask you to reconsider your assessment in light of our response.

Thank you again for engaging with our response. As the discussion period is coming to an end, we would still be glad to address any specific concern that remains. If our response has resolved some of your concerns, we would appreciate it if you could take this into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.