- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies peer assignment in agreement-based selection among word-deleted views of the same document, comparing semantic nearest neighbors with uniformly sampled peers under fixed candidate views, classifier predictions, and scoring rules.
Critical strengths
- The controlled comparison isolates the effect of peer assignment.
- Distinguishing selected views from predicted labels clarifies why selection changes may not affect predictions.
Critical weaknesses
- Limited novelty: The paper lacks a novel algorithm, and the contribution beyond empirical comparison is limited.
- Insufficient analysis: The semantic and predictive differences between nearest and random peers are not quantified.
- Limited view diversity: The views are generated by randomly deleting words from the same document, resulting in highly correlated and potentially redundant views that may limit the ability to assess the value of semantic neighborhood selection.
Questions for authors
- How do nearest and random peer sets differ in semantic similarity and predictive distributions?
- How often do all views predict the same label, and how does peer assignment affect cases with label disagreement?
- How stable are results under repeated peer sampling, and what justifies the ±1.5 pp margin?
3 comments
Thank you again for your review. We hope our response has addressed your concerns. As the discussion period is coming to an end, we would be glad to answer any remaining questions. If our response has resolved your concerns, we would greatly appreciate it if you could reconsider your assessment of our paper in light of it.
I thank the authors for their careful response. After reading the rebuttals and other reviewers comments, I think my rating is fair and I decide to maintain it.
Thank you for recognizing that the controlled comparison isolates peer assignment.
W1. A finding about existing methods is a contribution in its own right. To our knowledge, this is the first matched test of whether agreement-based selectors need semantically nearest peers. The answer, that the score matters (2.56–5.66 pp) but the neighborhood adds little (−0.46 to +0.50 pp), is directly useful to anyone designing such selectors. We would welcome a reference if this has been shown before.
W2, Q1. Nearest peers have mean MiniLM cosine 0.82 vs 0.72 for uniform peers, the two sets share only 30% of members (Jaccard 0.30), and Jensen–Shannon divergence is 0.07–0.09 vs 0.09–0.11 nats. The arms therefore differ substantially. Using the classifier's own embedding for neighbors gives contrasts of −0.23 to +0.38 pp, every 90% interval including zero.
Q2. All 8 views agree on 52–56% of documents. On the rest, agreement cuts error from 26.2–29.9% to 16.0–21.2%, while the uniform-minus-nearest contrast stays within −0.96 to +1.10 pp; the only interval excluding zero favors uniform peers.
Q3. Over 30 redraws of uniform peers, mean contrasts are 0.00–0.43 pp (SD 0.15–0.28), all 5th–95th percentiles within [−0.39, 0.66] pp. The paper's one interval excluding zero has a redraw mean of 0.02 pp. The ±1.5 pp band was fixed in advance (Section 3.3) and lies below the smallest agreement gain (2.56 pp).
W3. Both arms draw from the same views, and W2 shows their peer sets still differ substantially, so correlation does not remove the contrast.
Every question is now answered with direct measurements that support the paper's conclusion, and we ask the reviewer to take this into account in the rating.