- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how to provide reliable local supervision for agent trajectories, which is an important topic in agent evaluation. The main motivation is that a trajectory can fail even when some intermediate predictions are correct, which makes the final trajectory label insufficient for agent evaluation. Based on this, the authors propose a local critic trained with aggregate counts and a small number of local count labels. The critic uses hindsight evidence to judge earlier predictions and help identify wrong labels inherited from the final grade. The authors provide theoretical results on aggregate feedback and the value of hindsight. Finally, they use a controlled video-counting task to demonstrate improvements in local-correctness prediction over baselines.
Critical strengths
-
This paper studies an important problem in AI agent evaluation. The motivation is reasonable and easy to understand, since the final trajectory label may not correctly reflect the performance of each intermediate step.
-
This paper provides a clean framework for combining aggregate feedback with a small number of local labels. The overall idea of the methodology is easy to understand, with clear illustrative figures.
-
The experiments provide evidence for the effectiveness of the proposed method, and the methods are compared in a fair way.
Critical weaknesses
-
The novelty of this paper is limited. There is already substantial related work on learning from aggregate observations, sparse supervision, and hindsight information (Zhang et al., 2020; Shi et al., 2020; Harutyunyan et al., 2019). The proposed method mainly combines these existing ideas in a relatively straightforward way. It remains unclear whether this combination provides a sufficiently new contribution to meet the ICLR publication standard.
-
The experiments are very limited. All experiments are conducted on a controlled video event-counting task (CLEVRER), and it is still unclear how the method performs on other video or multimodal tasks. The structure of agent trajectories used in the experiments is also overly simple, compared with those of typical interactive agents.
-
There is a significant gap between the theoretical analysis and the implemented method. For example, Proposition 2 requires that the local correctness probability is conditioned on the final grade. However, the primary critic does not use the final grade when estimating local correctness. It appears that many theoretical results mainly describe general statistical properties, rather than providing direct guarantees for the learned critic.
-
The presentation and organization of the paper need substantial improvement. Most of the mathematical results are placed in the appendix, while the main text provides only limited technical explanations and well below the ICLR page limit. The sentences are not well connected, and the paper reads like an unpolished AI-written draft rather than a well-polished research paper.
Reference
- Zhang, Y., Charoenphakdee, N., Wu, Z., & Sugiyama, M. (2020). Learning from aggregate observations. Advances in Neural Information Processing Systems, 33, 7993-8005.
- Shi, Y., Liu, J., Wang, B., Qi, Z., & Tian, Y. (2020). Deep learning from label proportions with labeled samples. Neural Networks, 128, 73–81.
- Harutyunyan, A., Dabney, W., Mesnard, T., Azar, M. G., Piot, B., Heess, N., van Hasselt, H., Wayne, G., Singh, S., Precup, D., & Munos, R. (2019). Hindsight credit assignment. Advances in Neural Information Processing Systems, 32, 12467–12476.
Questions for authors
-
The experiments are only conducted on a controlled synthetic video event-counting task. Can the authors provide more experiments on other tasks and more realistic agentic trajectories to further demonstrate the effectiveness of the proposed method?
-
Is it possible to provide more theoretical analysis that directly supports the proposed method, rather than general statistical results?
Response to Reviewer VafM
Thank you for your detailed feedback on the contribution, theory, and presentation.
We have rewritten the introduction and method around the scientific question, construction, and evidence, and moved the central theoretical explanations into the main text. Section 1 now compares directly with Zhang et al. (2020), Shi et al. (2020), and Harutyunyan et al. (2019).
Our contribution is the construction and evaluation of local correctness supervision for checking labels inherited from a final grade. The observed aggregate sums event counts; it does not specify how many claims are correct. Hindsight helps assess an earlier claim’s truth, which is a different target from action credit assignment.
Section 2 states the grade-conditioning requirement and treats the implemented output as a score. Section 3 and Appendix A.5 (Proposition 5) bound the expected shortfall in errors found under a fixed review budget in terms of the fitted model’s expected count KL divergence. The reference count law conditions on the final grade, and we present the result as a comparison bound.
Section 5.2 and Appendix G add binary-feedback, student-distillation, and count-prediction-policy experiments. These controlled simulations extend the evaluation beyond critic accuracy.
Thank you again for helping us improve the paper. The student and policy results give us reason to expect that recovered local supervision can also help learning on more realistic agent trajectories. We would be glad to share further results as they become available.