- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ProbDPP, a reliability-aware data selection method that combines geometric diversity with source availability under stochastic failures. It derives a regularized objective with an additive reliability term and uses a KL-UCB semi-bandit algorithm to learn unknown availability probabilities.
Critical strengths
- The paper addresses a practically relevant selection problem by considering both data diversity and source availability under limited selection budgets.
- The evaluation spans QA, RAG, summarization, and visual reconstruction, providing evidence that combining reliability and diversity can benefit multiple downstream applications.
Critical weaknesses
- The objective’s connection to received information value is insufficiently justified. Regularization preserves nonzero features for dropped items, retaining full-set geometry while reducing failures to an additive penalty. It therefore neither directly measures surviving information nor distinguishes failure structures with identical marginals.
- Each online decision requires a generally intractable subset maximization, yet the regret bound assumes exact optimization. Without a scalable approximation and an analysis of its optimization error, the gap between practical implementation and theoretical guarantees remains unresolved.
- The regret proof derives bounds under optimistic confidence estimates but does not explicitly account for confidence-failure events when taking expectations. The required exploration-constant conditions are also unclear, leaving the expected regret guarantee incompletely justified.
- The baselines lack joint alternatives such as reliability-aware MMR or expected-surviving-information maximization. Since fixed-cardinality ProbDPP is also equivalent to quality-weighted DPP, the results do not sufficiently isolate the proposed regularization’s added value.
- Real failure traces and systematic tests linking availability to content difficulty or source quality are missing. The evaluation also assumes candidate embeddings are available upfront, without accounting for their acquisition costs or how they are obtained when sources are inaccessible.
Questions for authors
See Weakness