acceptodds
Under review as a conference paper at ICLR 2027

When Is Coarse Supervision Worth It? Cost-Aware Learning under Unknown Aggregation

Abstract

Modern learning systems often acquire supervision at multiple resolutions, trading annotation cost against information content. We study cost-aware two-resolution learning, where expensive fine labels reveal a vector response and cheaper coarse labels reveal a scalar aggregate formed with unknown weights, while the target remains the full response. The challenge is that unknown aggregation changes which directions coarse data can identify, so the value of coarse supervision depends jointly on cost, noise, and identification. We characterize this information geometry and develop an estimate-and-track policy that learns the aggregation rule and tracks the optimal resolution mix. We derive a closed-form break-even condition for coarse supervision and prove that the online policy attains the optimal leading cumulative-risk coefficient, with a matching local asymptotic minimax lower bound. Synthetic experiments support the predicted all-fine/mixed transition, show the online learner approaching the oracle-share benchmark, and demonstrate a finite-budget gain over all-fine acquisition when coarse supervision is sufficiently favorable. Our results provide a principled way to balance information and annotation cost across supervision resolutions.

Primary area:
learning on time series and dynamical systems
Keywords:
cost-aware learning, active learning, optimal experimental design

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GjhoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies an interesting question: how to allocate a limited annotation budget between expensive fine-grained supervision, which provides the full multi-dimensional response, and cheaper coarse supervision, which gives an aggregated scalar signal based on an unknown aggregation rule. The paper first shows that coarse labels become less useful when the aggregation rule is unknown. In that case, coarse labels are worth using only if they are cheap enough or informative enough. It then proposes an estimate-and-track method that learns the unknown aggregation rule online and adjusts how much budget is spent on fine and coarse labels. The authors show that this adaptive strategy can eventually match the leading-order performance of the ideal allocation. Synthetic experiments are used to support these theoretical results and show that the learned allocation moves toward the oracle allocation.

Critical strengths
  1. This paper clearly explains why coarse labels become less useful when the aggregation rule is unknown. By showing what information is lost and how this affects when coarse supervision is worth using, it gives a simple and interpretable answer to the main question of the paper.
Critical weaknesses
  1. The setting of the paper is quite idealized, which makes its practical relevance somewhat unclear. The analysis relies on several strong assumptions, so it is unclear whether the main conclusions can carry over to more realistic settings.
  2. The experiments are mainly conducted on synthetic data under settings that closely follow the theoretical assumptions.
Questions for authors
  1. Eq. (83) seems to give a stronger bound than what directly follows from Eq. (81), especially when the number of coarse samples grows sublinearly.
  2. I would like to better understand which parts of the theoretical results go beyond a direct specialization of these existing frameworks?
  3. If the learner could observe the current sample before choosing fine or coarse supervision, would the main conclusions still hold?
1 comment

Dear Reviewer Gjho,

We would like to kindly follow up on our detailed response, especially the derivation of Equation (83) directly from the block-inverse expression in Equation (82), including the sublinear coarse-sample regime. We believe this resolves the specific technical concern raised in your review.

With the discussion deadline approaching, we would appreciate knowing whether you find the explanation satisfactory and whether any substantive concerns remain. If the clarification resolves your soundness concern, we would be grateful if you could reflect this in your final assessment. We are happy to provide further details if needed.

Thank you for your time and consideration.

Best,

Submission 13557 Authors

Reviewer_k13vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate labeling budgets between fine-grained and coarse-grained queries. Fine-grained queries provide full labels at a higher cost, while coarse-grained queries return a weighted aggregation at a lower cost, with unknown aggregation weights. The goal is to recover the full fine-grained representation. When the aggregation weights are known, coarse labels become useful once the cost ratio exceeds KK. When the weights are unknown, estimating them reduces the information gained from coarse labels, leading to a higher threshold of dKd−K+1\frac{dK}{d-K+1} and a smaller maximum benefit. The authors also propose an epoch-based adaptive allocation strategy that estimates the unknown weights while tracking the optimal allocation. They establish matching leading-order upper and local asymptotic minimax lower bounds. Experiments are limited to two synthetic problem instances, with performance measured using exact population risk.

Critical strengths
  1. The profiling‑based analysis is the strongest contribution of this paper. The fact that unknown aggregation effectively discards K-1 directions usable by coarse observations shifts the threshold from K to dK/(d−K+1) and reduces maximum possible gain from 1/K to (d−K+1)/(dK). To my knowledge this geometric observation is new. It translates a high‑level conceptual question — when can cheap aggregated labels help if we do not know how aggregation works — into concrete closed‑form quantities.
  2. The online learning results are also solid. The upper bound matches a local minimax lower bound, so the claim that “learning the allocation incurs no first‑order cost” is supported, rather than merely speculative.
Critical weaknesses

(1) The paper does not clearly demarcate its own novelty. The analysis under known w* essentially reduces to standard A‑optimal experimental design calculations. The profiling argument itself boils down to a fairly short Schur‑complement computation. Moreover, the high‑level template of “reserve‑estimate‑track” used for the online policy has appeared in prior work on sequential design and transductive bandits, and related lines of work on label proportions and aggregate observations are also not cited or properly contrasted. It remains ambiguous which parts constitute the core contribution: is it the geometric profiling analysis, the specific policy construction, or the accompanying lower bound? For a theory‑focused submission this ambiguity is problematic.

(2) I also have reservations about how brittle the overall setup is. A critical modelling choice assumes a single global w* shared across every data item. In many realistic scenarios annotator or aggregation weights would change per‑example or per‑rater, yet the paper provides no sensitivity analysis for this. All experiments are synthetic; competing approaches are given oracle access to quantities they would not have in practice. There are no realistic practical baselines. In the 5× regime (λ = 5λcrit_U) the final improvement is only 1.08%, with confidence intervals that include zero; even under uniform weighting Figure 7 shows mean gain of only 0.15% for ρ=20. The authors report these numbers honestly, yet they never explicitly discuss that meaningful improvements are only available within a very narrow parameter regime.

Questions for authors
  1. What happens if w varies across individual items, either via small perturbations or independent draws per sample? Does the K-1 directional degradation degrade smoothly, or does the whole structure break down? A simple numerical sweep would suffice, and this will weigh heavily on my final assessment.
  2. Given the modest gains observed in the experiments, for which choices of (ρ, K, d) would mixed label acquisition actually be practically recommended? Can you characterise crossover time as a function of problem parameters beyond a handful of test instances?
  3. Is the absence of first‑order penalty an inherent feature of this specific model, or a more general outcome when working with expected risk together with a cheap design stream? How should we situate this result relative to existing adaptive design and bandit literature? Note also that the no-first-order property is currently stated for cumulative anytime risk; it would be helpful to clarify whether the constant term changes if switching to terminal-time risk.
  4. Could you include a simple one‑shot plug‑in baseline? This would help isolate whether the epoch schedule and cross‑fitting components of EET are actually necessary.
1 comment

Dear Reviewer k13v,

Thank you again for your constructive review and for recognizing the novelty of our information-geometric result and the matching online guarantees. We wanted to briefly follow up on our detailed response.

As the discussion period is approaching its end, we would appreciate any further thoughts on whether our explanations have addressed your concerns. We are happy to clarify any remaining questions.

Thanks for your support.

Best,

Submission 13557 Authors

Reviewer_LaFqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the optimal allocation of fine-grained and coarse supervision under a finite annotation budget. For linear prediction models, it characterizes the static cost–information frontier under both known and unknown aggregation rules, deriving the break-even condition for coarse supervision and the optimal resolution mix. Building on these theoretical results, the authors propose EET, which estimates the aggregation rule online while tracking the optimal coarse-budget share. The theoretical analysis is clean and thorough, and shows that EET achieves the optimal first-order cumulative-risk coefficient.

Critical strengths
  1. This paper studies how to allocate supervision at different resolutions under a limited annotation budget. In many practical annotation tasks, fine-grained labels are typically more expensive, while coarse labels are cheaper but provide less information, making this problem practically relevant.

  2. The theoretical results are clear and thorough. The authors characterize the static cost–information frontier, break-even condition, and optimal resolution mix under both known and unknown aggregation rules, and further analyze online adaptation, resulting in a fairly complete theoretical framework.

Critical weaknesses
  1. The main results rely on the assumptions of a linear prediction model and fixed linear aggregation. Since prediction models in real-world settings are often substantially more complex, it remains unclear to what extent the theoretical results extend to more general settings.

  2. All experiments are conducted on synthetic data, with no validation in realistic settings. Including at least one real or semi-realistic multi-resolution supervision experiment would substantially strengthen the evidence for the practical applicability of the proposed method.

Questions for authors
  1. How reasonable is the fixed linear aggregation assumption in real-world settings? For example, can the authors fit ycoarse≈ω⊤yfiney_{\mathrm{coarse}} \approx \omega^{\top} y_{\mathrm{fine}} in at least one real multi-resolution supervision setting and report the fitting error to validate this assumption?

  2. In practical applications, what levels of annotation cost and noise are typically required for coarse supervision to provide meaningful gains?

  3. Could the authors include at least one real or semi-realistic multi-resolution supervision experiment to validate the theoretically predicted allocation behavior and practical gains? See Weaknesses 2.

3 comments

Dear Reviewer LaFq,

We would like to kindly follow up on our response, particularly the new real-data TripAdvisor experiment conducted to directly address your question about the plausibility of fixed linear aggregation.

With the discussion deadline approaching, we would greatly appreciate your feedback on whether the additional evidence and theoretical clarifications address your main concerns. If they do, we hope you might consider updating your assessment accordingly. We would also be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Best,

Submission 13557 Authors

Thank you for clarifying that the fixed linear aggregation assumption applies at the conditional-mean level and for providing additional validation on real-world rating data. These results substantially alleviate my concerns about the practical plausibility of this assumption. Although the additional experiments do not validate the proposed budget allocation algorithm or its practical gains in real or semi-realistic settings, I appreciate the paper's theoretical contributions. Therefore, I will raise my score.

Thank you for reconsidering the assessment after taking another look. We appreciate your recognition of the additional real-data evidence and the theoretical contributions of our work. Thank you again for your constructive feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updates on Paper Submission

Dear Reviewers and Area Chairs,

Thank you for your constructive feedback and discussions. We have uploaded a revised manuscript incorporating the clarifications and additional evidence discussed during the rebuttal. The main updates include a real-data validation of the aggregation assumption, a clearer comparison with related work, and additional explanations of the theoretical results and modeling choices. All changes are highlighted in blue for ease of reference.

We appreciate your time and consideration!

Best,

Submission 13557 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.