acceptodds
Under review as a conference paper at ICLR 2027

When Global Grades Fail Locally: Learning Local Supervision from Aggregate Feedback and Hindsight

Abstract

Agentic systems often receive one grade for an entire trajectory, even though learning requires feedback on individual steps. Copying the final grade to every step can therefore create wrong local labels: a failed trajectory may contain correct steps, and a successful trajectory may contain mistakes. We study how to recover local correctness without densely labeling every step. Our method combines aggregate feedback with a small number of local labels to train a local critic. Hindsight then allows the critic to reassess earlier claims using evidence that becomes available later in the trajectory. Our theory explains what aggregate feedback leaves unresolved and when hindsight provides additional information. It also shows how local correctness relates to errors in inherited labels. We evaluate the method in a controlled multimodal forecast-and-recount setting with independent simulator ground truth. On a frozen final set, the retrospective critic reduces local-correctness log loss by 32% relative to a matched static critic, from 0.5781 to 0.3943. A separate frozen validation set shows the same pattern. These results show that aggregate feedback, sparse local supervision, and hindsight can recover useful local supervision without dense step-level labels.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Hierarchical supervision, aggregate supervision, weak supervision, hindsight, process supervision, agentic systems, local supervision, multimodal reasoning.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VafMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to provide reliable local supervision for agent trajectories, which is an important topic in agent evaluation. The main motivation is that a trajectory can fail even when some intermediate predictions are correct, which makes the final trajectory label insufficient for agent evaluation. Based on this, the authors propose a local critic trained with aggregate counts and a small number of local count labels. The critic uses hindsight evidence to judge earlier predictions and help identify wrong labels inherited from the final grade. The authors provide theoretical results on aggregate feedback and the value of hindsight. Finally, they use a controlled video-counting task to demonstrate improvements in local-correctness prediction over baselines.

Critical strengths
  1. This paper studies an important problem in AI agent evaluation. The motivation is reasonable and easy to understand, since the final trajectory label may not correctly reflect the performance of each intermediate step.

  2. This paper provides a clean framework for combining aggregate feedback with a small number of local labels. The overall idea of the methodology is easy to understand, with clear illustrative figures.

  3. The experiments provide evidence for the effectiveness of the proposed method, and the methods are compared in a fair way.

Critical weaknesses
  1. The novelty of this paper is limited. There is already substantial related work on learning from aggregate observations, sparse supervision, and hindsight information (Zhang et al., 2020; Shi et al., 2020; Harutyunyan et al., 2019). The proposed method mainly combines these existing ideas in a relatively straightforward way. It remains unclear whether this combination provides a sufficiently new contribution to meet the ICLR publication standard.

  2. The experiments are very limited. All experiments are conducted on a controlled video event-counting task (CLEVRER), and it is still unclear how the method performs on other video or multimodal tasks. The structure of agent trajectories used in the experiments is also overly simple, compared with those of typical interactive agents.

  3. There is a significant gap between the theoretical analysis and the implemented method. For example, Proposition 2 requires that the local correctness probability is conditioned on the final grade. However, the primary critic does not use the final grade when estimating local correctness. It appears that many theoretical results mainly describe general statistical properties, rather than providing direct guarantees for the learned critic.

  4. The presentation and organization of the paper need substantial improvement. Most of the mathematical results are placed in the appendix, while the main text provides only limited technical explanations and well below the ICLR page limit. The sentences are not well connected, and the paper reads like an unpolished AI-written draft rather than a well-polished research paper.

Reference

  1. Zhang, Y., Charoenphakdee, N., Wu, Z., & Sugiyama, M. (2020). Learning from aggregate observations. Advances in Neural Information Processing Systems, 33, 7993-8005.
  2. Shi, Y., Liu, J., Wang, B., Qi, Z., & Tian, Y. (2020). Deep learning from label proportions with labeled samples. Neural Networks, 128, 73–81.
  3. Harutyunyan, A., Dabney, W., Mesnard, T., Azar, M. G., Piot, B., Heess, N., van Hasselt, H., Wayne, G., Singh, S., Precup, D., & Munos, R. (2019). Hindsight credit assignment. Advances in Neural Information Processing Systems, 32, 12467–12476.
Questions for authors
  1. The experiments are only conducted on a controlled synthetic video event-counting task. Can the authors provide more experiments on other tasks and more realistic agentic trajectories to further demonstrate the effectiveness of the proposed method?

  2. Is it possible to provide more theoretical analysis that directly supports the proposed method, rather than general statistical results?

1 comment
AuthorsOpenReview ↗

Response to Reviewer VafM

Thank you for your detailed feedback on the contribution, theory, and presentation.

We have rewritten the introduction and method around the scientific question, construction, and evidence, and moved the central theoretical explanations into the main text. Section 1 now compares directly with Zhang et al. (2020), Shi et al. (2020), and Harutyunyan et al. (2019).

Our contribution is the construction and evaluation of local correctness supervision for checking labels inherited from a final grade. The observed aggregate sums event counts; it does not specify how many claims are correct. Hindsight helps assess an earlier claim’s truth, which is a different target from action credit assignment.

Section 2 states the grade-conditioning requirement and treats the implemented output as a score. Section 3 and Appendix A.5 (Proposition 5) bound the expected shortfall in errors found under a fixed review budget in terms of the fitted model’s expected count KL divergence. The reference count law conditions on the final grade, and we present the result as a comparison bound.

Section 5.2 and Appendix G add binary-feedback, student-distillation, and count-prediction-policy experiments. These controlled simulations extend the evaluation beyond critic accuracy.

Thank you again for helping us improve the paper. The student and policy results give us reason to expect that recovered local supervision can also help learning on more realistic agent trajectories. We would be glad to share further results as they become available.

Reviewer_JPLwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to estimate whether individual steps are correct when only coarse trajectory-level supervision is available. It trains a local critic using exact event-count subtotals, a small number of local count labels, and video evidence observed after the prediction. The problem is important, and the controlled evaluation is careful. However, the proposed method does not actually learn local supervision from the global grade that motivates the paper. It relies on a different and much stronger source of supervision. Several probability and modeling claims are also not justified.

Critical strengths
  • The paper identifies a real problem with assigning a final outcome label to every intermediate step.
  • The experimental setting provides independent ground truth.
  • The frozen validation and final evaluation protocol is careful.
  • The presentation is generally clear.
Critical weaknesses
  • The paper is motivated by the situation where an agent receives one binary grade (L) for the complete trajectory. However, this grade is not used to train the local critic. Instead, training requires the exact subtotal T=N1+N2+N3T=N_1+N_2+N_3 from the simulator, together with some true local counts. This subtotal is much more informative than a binary success or failure grade and may not exist in general agentic tasks. Therefore, the paper does not solve the problem suggested by its title. It solves a narrower count-decomposition problem using privileged aggregate ground truth. The final grade is used only after local correctness has already been estimated.
  • The subtotal likelihood is computed as ∑n1+n2+n3=t∏jfθ(nj∣Xj)\sum_{n_1+n_2+n_3=t} \prod_j f_\theta(n_j\mid X_j). This assumes that the three interval counts are conditionally independent under the critic. But the intervals belong to the same video and are generated by shared objects and dynamics, so their event counts can be strongly dependent.
  • The static critic sees the interval before the event, while the hindsight critic sees the target interval after the event has occurred. The latter therefore receives direct visual evidence of the quantity it must estimate. It is unsurprising that observing the realized event improves prediction over forecasting it from an earlier interval. This demonstrates the value of additional information, but it does not show that aggregate feedback successfully resolves local supervision. The gain may primarily come from turning a forecasting problem into a retrospective counting problem.
Questions for authors

The paper defines q=L(1−p)+(1−L)pq=L(1-p)+(1-L)p and describes (q) as the probability that the inherited label is wrong. This is correct only if (p) is the probability of local correctness conditioned on all relevant information, including (L). However, the critic produces (p=P(Y=1\mid X,a)) without conditioning on the final grade. In general, P(Y=1∣X,a,L)≠P(Y=1∣X,a).P(Y=1\mid X,a,L)\neq P(Y=1\mid X,a).

The paper itself acknowledges this issue in the appendix. Consequently, (q) is merely a transformed model score. This undermines its interpretation as a calibrated probability for detecting incorrect inherited labels.

1 comment
AuthorsOpenReview ↗

Response to Reviewer JPLw

Thank you for your careful reading and for pointing out where the feedback signals and probability interpretation needed clarification.

The abstract and Section 2 now distinguish the signals: the video critic uses exact subtotals and 77 local counts; the final grade is used afterward for label review.

Appendix G.3 (Table 12) tests binary grades plus sparse counts under the specified recount mechanism, without observed subtotals or recount values. With 77 counts, log loss improves over copied grades by 0.1227 for independent counts and 0.2195 under shared dependence. Matched comparisons do not establish an additional gain over counts alone.

Section 2 corrects the interpretation of qq: its posterior identity requires p=P(Y=1∣X,a,L)p=P(Y=1\mid X,a,L). Since the implemented critic omits LL, we describe its output after reflection as a score. Section 3 identifies the product likelihood as a conditional-independence approximation; Appendix G.2 examines shared dependence.

Table 1 holds retrospective evidence fixed: subtotal-only training gives log loss 0.5440, while adding 77 local labels gives 0.3943. This comparison separates the supervision change from the static-versus-hindsight contrast, which measures the value of later evidence for retrospective verification.

Thank you again for these helpful comments. We would be happy to clarify any remaining questions about the assumptions or comparisons.

Reviewer_3P6eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how local correctness can be estimated when supervision consists of aggregate counts and sparse local labels. The authors train an auxiliary critic using video representations and hindsight evidence, then use its predictions to identify local claims that would receive incorrect labels if they inherited the final trajectory grade. Controlled experiments show improved local-correctness estimation and error discovery. However, the demonstrated benefits remain confined to an external evaluator for a specialized counting task; the paper does not establish whether the recovered supervision improves the underlying model or transfers to broader agentic settings.

Critical strengths

A clear supervision problem. The paper distinguishes local correctness from overall trajectory success and provides a concrete framework for combining aggregate observations, sparse labels, and hindsight.

Careful controlled evaluation. Matched static critics, aggregate-only training, alternative objectives, and fully supervised references help identify the contributions of supervision and available evidence. Separate frozen evaluation cohorts and group-level uncertainty estimates strengthen the empirical analysis.

Critical weaknesses

The auxiliary critic’s benefit to the underlying model remains unestablished. The method trains a small external network for specialized count-based verification while keeping the original model frozen. Consequently, the reported improvements concern the evaluator’s predictions and label-screening performance, rather than improved reasoning or task completion by the original model. It remains unclear whether the learned signal could be incorporated into the model through lightweight fine-tuning or distillation, and whether retaining a separate critic offers an advantage over such integration. Full-scale model training may require substantial additional resources, but a small-scale adaptation experiment would help establish whether the recovered supervision provides a useful learning signal beyond external diagnosis.

The empirical task is too narrow to establish broader applicability. The evaluation is restricted to event-count forecasts in a synthetic video environment, with exact simulator-derived counts and direct observation of the realized target interval. These conditions provide unusually clear supervision and verification opportunities. The paper does not demonstrate that the approach extends to heterogeneous local claims, natural-language reasoning steps, or interactive agent actions, where correctness may be ambiguous and future evidence may not directly reveal the answer. Broader task coverage is needed to support the proposed connection to general process supervision and agentic learning.

Questions for authors

weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3P6e

Thank you for the constructive suggestion to test whether the recovered supervision helps learning. We added a controlled count-model experiment that transfers probabilities from a critic trained on exact subtotals to a logistic student and a count-prediction policy. Both learners use only forecast-time inputs at evaluation.

Section 5.2 and Appendix G.4 (Table 13) report student log loss of 0.6113, compared with 0.7058 for copied final grades: a paired reduction of 0.0945 (95% interval [0.0813, 0.1076]) over 20 seeds. The policy reaches 46.22% step success, compared with 37.11% for copied trajectory outcomes. These results demonstrate that recovered local probabilities can provide useful training targets.

Sections 4–5 now distinguish this learning experiment from the video study, which evaluates retrospective verification while keeping the original language model frozen. The student demonstrates one way to integrate the critic’s supervision into a forecast-time learner.

Thank you again for suggesting this experiment. The student and policy results give us reason to expect that recovered local supervision can also help learning on more realistic agent trajectories. We would be glad to share further results as they become available.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification of the video results and error scores

Thank you for the careful follow-up. Our video critic is trained on 256 exact subtotals and 77 local count labels, with later frames providing visual evidence. The final grade enters afterward, when the local scores are used to check inherited labels (Sections 2–3).

Table 1 measures the benefit of adding local labels with hindsight held fixed. The matched static comparison measures the value of later observations with supervision held fixed. A subtotal constrains the sum of event counts rather than specifying each forecast’s correctness. The critic learns to assess individual forecasts from these constraints and the visual evidence.

Section 2 presents the implemented qq as a score for ranking inherited labels and states the conditions for a probability interpretation. Appendix E.2 and Figure 3 evaluate this ranking directly: the 154 highest-ranked labels on the final video set contain 149 errors, compared with 136 for the matched static critic (paired gain 13, 95% interval [4.98, 22]).

Section 3 treats factorization as a modeling approximation. Appendix G.2 examines its behavior under a specified form of dependence. The held-out video results evaluate the fitted model’s prediction and label-selection performance under this approximation.

Thank you again for your careful consideration of the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.