- Rating
- 3
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies what minimum uncertainty information a predictor needs to retain for optimal decision-making. Using a mathematical decomposition, it identifies the necessary information components and their minimum dimension.
The theoretical and experimental results show that predicting only the information relevant to the loss function, rather than the full probability distribution, is sufficient for optimal decisions. The paper also examines how missing information, estimation errors, and decoding errors affect decision performance.
Critical strengths
-
The theoretical analysis is solid. The non-prefix result is particularly interesting and somewhat novel. It challenges the common intuition that higher-order information necessarily includes lower-order information. For example, some loss functions require third-order information but do not need any second-order Johnson components.
-
The experimental results are consistent with the theoretical findings.
Critical weaknesses
-
Constructing the decision-relevant subspace from differences between loss vectors is a fairly standard idea.
-
As the authors acknowledge, the number of correct labels or relevant documents is often not fixed in real-world multi-label classification, recommendation, and retrieval tasks. However, the theoretical results rely on fixed set cardinalities and symmetric overlap losses, which limits their applicability.
-
The experiments are relatively small in scale, making it difficult to assess the practical value of the proposed approach in real-world applications.
Questions for authors
- I would like the authors to clarify how this work relates to existing research, particularly which theoretical tools and ideas are already established and which key results are genuinely new and cannot be directly derived from prior work. Also, since the analysis assumes fixed set cardinalities and symmetric overlap losses, it would be helpful to discuss how much these assumptions limit the applicability of the method to real-world tasks.
2.Although the paper includes experiments using real Wikipedia data, the candidate sets and decision spaces are still relatively small. Could the authors provide larger-scale retrieval experiments with an open candidate space to better demonstrate the scalability and practical value of the proposed approach?
5 comments
To Reviewer NbBA (2/2)
W3/Q2. Computational benefits at larger candidate scales.
Thank you for your valuable comment. Our added WikiSeeAlso experiments show substantial computational savings at larger candidate scales (Section 6.3 and Appendix I.1). With 128 candidate labels, directly predicting pair scores (minimum representation) achieves 26.594× faster complete fitting and 34.178× faster inference than Full CE. Mean success is 16.835% versus 17.245%, with a paired difference of −0.410% (95% CI [−1.243%, 0.389%]). These results quantify the computational benefit and the accompanying decision-quality tradeoff.
We evaluate candidate sets of 32, 64, and 128 labels. For each article, a retrieval model fitted using only training data selects the highest-ranked candidate labels from the label pool available in training. At 128 candidates, the evaluation cohort contains 27,155 training articles and 9,605 test articles. Each example has three relevant labels, and the task is to select two labels that are both relevant. Our method predicts a success score for each candidate pair, whereas Full CE predicts the probability of each possible triple of relevant labels. At 128 candidates, this means 8,128 pair scores instead of 341,376 triple probabilities. Both models use the same 61,953-parameter scoring architecture.
Both methods use five paired seeds, 100 training epochs, and checkpoints selected on validation data. Success below is reported as mean ± standard deviation across seeds. Speedups are Full CE time divided by direct-prediction time. Complete fitting includes all 100 epochs, validation, and checkpointing. Inference is measured on 512 articles and includes batching cached features, prediction, score export, and exact decoding.
Candidate labels Direct pair prediction success Full CE success Complete-fit speedup Inference speedup 32 30.478% ± 0.807% 31.036% ± 0.859% 2.663× 3.034× 64 22.530% ± 0.514% 22.878% ± 0.877% 8.541× 16.946× 128 16.835% ± 0.167% 17.245% ± 0.804% 26.594× 34.178×
We appreciate your thoughtful questions and would be happy to clarify any remaining points about our work.
Kind Follow-up on Our Rebuttal Submission
Dear Reviewer NbBA,
We hope you are doing well. We would like to follow up on our rebuttal, which addresses your questions about the contribution, assumptions, and experimental scale.
We are particularly grateful for your comment on the fixed outcome cardinality assumption. It motivated us to develop a new theoretical extension to variable outcome cardinality, leading to additional sufficiency and minimum-dimension results that broaden the scope of our theory.
Our responses and revisions include:
-
A clearer contribution: We distinguish established tools from our new results, which identify the information needed for optimal decisions and its minimum dimension.
-
Broader theoretical scope: Following your suggestion, we extended the theory to variable outcome sizes under a fixed decision budget. The extension provides sufficient prediction targets and their exact minimum dimension. We also clarify that symmetry applies to the loss.
-
Larger-scale experiments: We added WikiSeeAlso comparisons with 32, 64, and 128 candidates, showing the training and inference savings alongside the difference in decision accuracy.
If you have a moment, we would be grateful if you could review our responses and let us know whether any questions remain about the theoretical extension or the additional experiments.
We sincerely appreciate the time and care you have devoted to reviewing our work, and thank you for the suggestion that helped us strengthen its theoretical scope.
Warm regards,
The Authors
To Reviewer NbBA (1/2)
W1/Q1. The loss-specific characterization and its contribution.
W2/Q1. How restrictive are fixed cardinalities and symmetry?