- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies an interesting question: how to allocate a limited annotation budget between expensive fine-grained supervision, which provides the full multi-dimensional response, and cheaper coarse supervision, which gives an aggregated scalar signal based on an unknown aggregation rule. The paper first shows that coarse labels become less useful when the aggregation rule is unknown. In that case, coarse labels are worth using only if they are cheap enough or informative enough. It then proposes an estimate-and-track method that learns the unknown aggregation rule online and adjusts how much budget is spent on fine and coarse labels. The authors show that this adaptive strategy can eventually match the leading-order performance of the ideal allocation. Synthetic experiments are used to support these theoretical results and show that the learned allocation moves toward the oracle allocation.
Critical strengths
- This paper clearly explains why coarse labels become less useful when the aggregation rule is unknown. By showing what information is lost and how this affects when coarse supervision is worth using, it gives a simple and interpretable answer to the main question of the paper.
Critical weaknesses
- The setting of the paper is quite idealized, which makes its practical relevance somewhat unclear. The analysis relies on several strong assumptions, so it is unclear whether the main conclusions can carry over to more realistic settings.
- The experiments are mainly conducted on synthetic data under settings that closely follow the theoretical assumptions.
Questions for authors
- Eq. (83) seems to give a stronger bound than what directly follows from Eq. (81), especially when the number of coarse samples grows sublinearly.
- I would like to better understand which parts of the theoretical results go beyond a direct specialization of these existing frameworks?
- If the learner could observe the current sample before choosing fine or coarse supervision, would the main conclusions still hold?
Dear Reviewer Gjho,
We would like to kindly follow up on our detailed response, especially the derivation of Equation (83) directly from the block-inverse expression in Equation (82), including the sublinear coarse-sample regime. We believe this resolves the specific technical concern raised in your review.
With the discussion deadline approaching, we would appreciate knowing whether you find the explanation satisfactory and whether any substantive concerns remain. If the clarification resolves your soundness concern, we would be grateful if you could reflect this in your final assessment. We are happy to provide further details if needed.
Thank you for your time and consideration.
Best,
Submission 13557 Authors