- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper looks at credit assignment in multi-agent video reasoning: when the final answer is wrong, which stage of evidence delivery failed, and which failures should become training targets? The authors build SKATEEG, a figure skating benchmark with 7,200 video questions and 2,400 text-only rule controls. Each question comes with record-level evidence annotations and AND/OR support structures describing what a fully supported answer requires. On top of this, their EDGE framework traces each reference record through four stages (request, acquisition, description, use), finds the first confirmed failure, and selects sufficient paths so that negative task credit and hindsight signals go to the actions responsible for those failures. With a 7B backbone, EDGE reaches 63.2% accuracy and 53.2% GroundedAcc, i.e., +2.9 and +8.3 points over MATPO. The diagnosis itself is validated against repair replay and blind link annotation, and the training choices are tested with matched controls.
Critical strengths
The experiments go beyond end-task scores and probe the mechanism directly. The diagnosis is checked against an independent repair-replay reference and blind link annotation: first break hits 86.0% where a same-information LLM judge gets 62.5% (Table 3), and the diagnosis costs no extra trajectory executions. On the training side, the negative-only shuffle and dependency-ranked controls keep target counts and weight multisets fixed, so the gains are tied to where the signal goes, with the amount of signal held fixed. Fig3 adds a direct check that the trained reasoner depends on the critical evidence itself. Papers in this space often assert that their credit signal is correct; this one actually measures it, and that is the main reason I lean positive.
SKATEEG is also a real contribution on its own. The record-level annotations and AND/OR support structures let you separate answer accuracy, evidence localization, and complete support, and they make handoff failures between agents directly measurable. The annotation pipeline involves expert review and adjudication, and the authors ran a blind stratified audit over the questions (reporting a residual error rate below 1%). Even if EDGE itself turned out not to transfer, I would expect this benchmark and its evaluation protocol to be used by others.
Critical weaknesses
The cost side of the annotation story is largely missing. EDGE needs question-level evidence records with AND/OR labels at training time, and producing them takes domain experts plus adjudication. Appendix B.4 describes the pipeline and states that annotators were paid above the local minimum wage, but I found no numbers from which I could estimate the effort per question or for the full dataset, and the cost analysis in Appendix D.5 covers tokens and GPU-hours only. So the paper shows convincingly that the method works when these annotations exist, and much less about what it takes to get them for a new task. This concern bears on the barrier to adoption; the reported results themselves stand.
The evidence for generality is thin. The end-to-end comparisons all use figure skating, one shared 7B backbone, and a single-pass three-agent pipeline. Table 28 does add a configuration with three independent 3B backbones, and it is encouraging that role-specific hindsight helps there as well, but that table probes the hindsight piece alone, and the full diagnose-then-select pipeline remains untested there. The authors state this scope clearly, which I take as a plus. My reading is that the contribution is a careful empirical study within this particular setting; claims beyond that setting would need more evidence.
Questions for authors
Q1. Can you give some sense of the human effort behind the evidence annotations? Appendix B.4 explains the proposer-generator-verifier pipeline and the adjudication step, but approximate time per question, or the total annotator-hours for the dataset, would let readers judge what it costs to apply this approach to a new task. Existing logs or budget records would be enough; no new annotation work is needed. A concrete number here would largely resolve my concern about annotation costs in the weaknesses section.
Q2. Could you clarify what the existing diagnostic audits imply when evidence is harder to verify than in figure skating? Table 24 and Appendix F.2 check target selection and positive-credit gating, and Appendix E.2 analyzes diagnostic errors. What would help is a clear split between the failure modes these analyses already cover and the ones whose effect on training is still unmeasured. A short discussion based on your existing results would be enough; no new experiment is needed. This bears on how far the method applies beyond the current setting.
Q3. How binding is the single-pass assumption? The pipeline in Section 3.4 never revisits earlier actions, which keeps the handoff graph acyclic, and Proposition 1 carries its own structural conditions. If a system allowed revisits or multiple interaction rounds, which pieces would have to be redefined: first-break localization, action ownership, or the sufficient-path filter? This is a scope question; no new experiment is needed. If the definitions extend naturally to somewhat richer interaction patterns, that would raise my assessment of the method's transferable value.