- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper reformulate the problem of MPU as a retrieval problem, to find the closely related data for negative classes. The solution is to combine the advatages of MPU and TabPFN. The evaluation across 25 datasets shows that the method is competitive.
Critical strengths
- The problem reformulation is reasonable and seems to be effective to deal with the problem
- The paper is well written and easy to follow
- The experiment includes 25 datasets with each results reported in Appendix.
Critical weaknesses
- The experimental evaluation is restricted to a single MPU construction protocol, with one held-out negative class per dataset. It remains unclear whether the method generalizes to more unlabeled negative classes. If there are multiple classes included in the negative set, I am not sure whether the designed method can be robust enough, especially for those unlabeled negative classes similar to labeled classes.
- A central contribution of the paper is the Feedback-Guided Context Retriever (FGCR), which distills predictions from a frozen TabPFN and iteratively refines the pseudo-labeled context. However, it remains unclear whether training an additional retriever is necessary, since TabPFN already provides class probabilities that could be used directly for pseudo-label selection and iterative context refinement. A natural baseline would initialize the context using the same GeoDNI procedure, obtain out-of-fold predictions from frozen TabPFN, and directly select high-confidence pseudo-labeled examples for each class to update the context over multiple rounds, without training FGCR. This baseline should use the same context budgets and refinement iterations as the proposed method. Although the authors include a w/o FGCR ablation (Table 5(b)), it is unclear whether this variant implements such a direct iterative self-training strategy. A more explicit comparison would help establish whether the performance improvements arise from the proposed learned retriever or could be achieved through simpler iterative context refinement.
Questions for authors
Please refer to weaknesses.
2 comments
Thanks for running the additional experiments. The results on the ten multiclass datasets address most of my concern about using a single hidden negative class. I’d still be interested in how the method performs when a hidden class closely resembles a labeled class. The teacher-only comparison answers my second question. I keep my original positive rating.
Rebuttal Answer
Thank you for carefully reading our paper and for your insightful questions and valuable comments.
W1: Additional Experiments with Multiple Hidden Negative Classes
We thank the reviewer for highlighting this important limitation. In our original experiments on multiclass datasets, each task designated only one original class as the unlabeled negative class. To address this concern, we conducted additional experiments under the most challenging 5-shot setting. For each multiclass dataset, we designated multiple original classes as hidden negative classes: two for datasets with four classes, three for datasets with five to seven classes, and four for datasets with eight or more classes. All other experimental settings remained unchanged. We compared DRIP-MPU with the four strongest baselines under the same multiple-hidden-negative protocol.
Table 1 presents the results. DRIP-MPU achieved an average AUC of 81.25 and an average Pos.-F1 of 52.11, outperforming the four baselines. These results support its applicability to the multiple-hidden-negative tasks tested here.
We also acknowledge that introducing multiple hidden negative classes affects performance. While the average AUC remains close to the original result, Pos.-F1 decreases by 5.52 percentage points. Addressing this decline is a direction for future work.
Table 1. Average AUC and Pos.-F1 (%) over the ten multiclass datasets in the 5-shot setting with ten seeds. All rows use multiple hidden negative classes, except DRIP-MPU (single negative class).
W2: Investigating the Performance Gains from the Learned Retriever
We thank the reviewer for suggesting this meaningful comparison. First, we clarify that the “w/o FGCR” ablation in the original manuscript did not perform iterative self-training and therefore does not directly answer this question. To address it, we implemented the suggested direct teacher-only self-training variant. It initializes the context using the same GeoDNI results, then iteratively selects pseudo-labeled examples and updates the context based on class predictions from the frozen TabPFN, without training an FGCR retriever.
As shown in Table 2, under the 5-shot setting, this variant achieves an average AUC of 73.99% and an average Pos.-F1 of 54.98%. This shows that direct self-training with TabPFN feedback is a competitive simple alternative. DRIP-MPU further improves the average AUC and Pos.-F1 by 1.42 and 1.77 percentage points, respectively. The learned retriever in the full method thus provides additional average gains on both metrics.
Table 2. Performance comparison between DRIP-MPU and the teacher-only self-training variant in the 5-shot setting.