acceptodds
Withdrawn from ICLR 2027

Who Missed the Jump? Evidence-Grounded Credit Assignment for Multi-Agent Figure Skating Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable video reasoning requires both a correct answer and the evidence that supports it. In figure skating, a brief takeoff or landing interval can determine a judgment, yet answer accuracy and evidence localization need not improve together. Dividing the task among cooperating agents introduces a second ambiguity: an outcome reward does not identify where evidence delivery failed or which failures need correction. Our approach makes explicit evidence requirements a common reference for evaluation, diagnosis, and training. We introduce **SkateEG**, with 7,200 video-grounded questions over 1,194 programs and 2,400 text-only controls. Its annotated records specify where to find the evidence and whether records are jointly required or sufficient alternatives. Our **EDGE** framework first diagnoses each record's earliest confirmed failure in request, acquisition, description, or use, then selects sufficient-path targets for negative task credit and localized hindsight. This selection avoids correcting unused alternatives when another path already supplies complete support. Positive task credit reinforces supported delivery, while unresolved targets receive zero task weight. On 600 independent failures, the first-break nominee matches repair-replay point rankings in 86.0% of cases, versus 62.5% for a same-information judge, without additional trajectory executions for diagnosis. Across three training seeds, the 7B system reaches 63.2% accuracy and 53.2% GroundedAcc, the proportion of test questions answered correctly with complete evidence support, exceeding MATPO by 2.9 and 8.3 percentage points, respectively. Matched negative-credit and alternative-path controls support the value of the target-selection rules. Together, these results connect explicit evidence requirements to learning from failed handoffs in a three-agent pipeline with annotated training evidence. Code and data will be released.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Action Quality Assessment, Figure Skating, Multi-Agent System, On-Policy Distillation, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_CLGyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at credit assignment in multi-agent video reasoning: when the final answer is wrong, which stage of evidence delivery failed, and which failures should become training targets? The authors build SKATEEG, a figure skating benchmark with 7,200 video questions and 2,400 text-only rule controls. Each question comes with record-level evidence annotations and AND/OR support structures describing what a fully supported answer requires. On top of this, their EDGE framework traces each reference record through four stages (request, acquisition, description, use), finds the first confirmed failure, and selects sufficient paths so that negative task credit and hindsight signals go to the actions responsible for those failures. With a 7B backbone, EDGE reaches 63.2% accuracy and 53.2% GroundedAcc, i.e., +2.9 and +8.3 points over MATPO. The diagnosis itself is validated against repair replay and blind link annotation, and the training choices are tested with matched controls.

Critical strengths

The experiments go beyond end-task scores and probe the mechanism directly. The diagnosis is checked against an independent repair-replay reference and blind link annotation: first break hits 86.0% where a same-information LLM judge gets 62.5% (Table 3), and the diagnosis costs no extra trajectory executions. On the training side, the negative-only shuffle and dependency-ranked controls keep target counts and weight multisets fixed, so the gains are tied to where the signal goes, with the amount of signal held fixed. Fig3 adds a direct check that the trained reasoner depends on the critical evidence itself. Papers in this space often assert that their credit signal is correct; this one actually measures it, and that is the main reason I lean positive.

SKATEEG is also a real contribution on its own. The record-level annotations and AND/OR support structures let you separate answer accuracy, evidence localization, and complete support, and they make handoff failures between agents directly measurable. The annotation pipeline involves expert review and adjudication, and the authors ran a blind stratified audit over the questions (reporting a residual error rate below 1%). Even if EDGE itself turned out not to transfer, I would expect this benchmark and its evaluation protocol to be used by others.

Critical weaknesses

The cost side of the annotation story is largely missing. EDGE needs question-level evidence records with AND/OR labels at training time, and producing them takes domain experts plus adjudication. Appendix B.4 describes the pipeline and states that annotators were paid above the local minimum wage, but I found no numbers from which I could estimate the effort per question or for the full dataset, and the cost analysis in Appendix D.5 covers tokens and GPU-hours only. So the paper shows convincingly that the method works when these annotations exist, and much less about what it takes to get them for a new task. This concern bears on the barrier to adoption; the reported results themselves stand.

The evidence for generality is thin. The end-to-end comparisons all use figure skating, one shared 7B backbone, and a single-pass three-agent pipeline. Table 28 does add a configuration with three independent 3B backbones, and it is encouraging that role-specific hindsight helps there as well, but that table probes the hindsight piece alone, and the full diagnose-then-select pipeline remains untested there. The authors state this scope clearly, which I take as a plus. My reading is that the contribution is a careful empirical study within this particular setting; claims beyond that setting would need more evidence.

Questions for authors

Q1. Can you give some sense of the human effort behind the evidence annotations? Appendix B.4 explains the proposer-generator-verifier pipeline and the adjudication step, but approximate time per question, or the total annotator-hours for the dataset, would let readers judge what it costs to apply this approach to a new task. Existing logs or budget records would be enough; no new annotation work is needed. A concrete number here would largely resolve my concern about annotation costs in the weaknesses section.

Q2. Could you clarify what the existing diagnostic audits imply when evidence is harder to verify than in figure skating? Table 24 and Appendix F.2 check target selection and positive-credit gating, and Appendix E.2 analyzes diagnostic errors. What would help is a clear split between the failure modes these analyses already cover and the ones whose effect on training is still unmeasured. A short discussion based on your existing results would be enough; no new experiment is needed. This bears on how far the method applies beyond the current setting.

Q3. How binding is the single-pass assumption? The pipeline in Section 3.4 never revisits earlier actions, which keeps the handoff graph acyclic, and Proposition 1 carries its own structural conditions. If a system allowed revisits or multiple interaction rounds, which pieces would have to be redefined: first-break localization, action ownership, or the sufficient-path filter? This is a scope question; no new experiment is needed. If the definitions extend naturally to somewhat richer interaction patterns, that would raise my assessment of the method's transferable value.

Reviewer_KtWYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the skating benchmark SKATEEG, which distinguishes the correctness of the answer from the completeness of the evidence through explicit evidence recording and AND/OR support relationship annotation. At the same time, it presents the three-agent training framework EDGE. This framework identifies the first failure point in the evidence transmission chain, selects the correction target on the path of sufficient evidence, and combines breakpoint routing strategy optimization (BRPO) and link localization post-distillation (LHD) for training. The trained system achieves an accuracy rate of 63.2% and a GroundedAcc of 53.2%, and the results of the control experiments also support that the negative credit allocation of evidence alignment has practical value.

Critical strengths
  1. Valuable definition and evaluation of complete evidence support in the benchmark. SKATEEG explicitly distinguishes between the mutually necessary AND evidence paths and the alternative OR evidence paths, avoiding treating any relevant references directly as complete support. Appendix F.1 conducts independent manual evaluation of the complete test set instead of relying on the prediction results of the training analyzer. This makes the improvement in GroundedAcc more convincing than a simple increase in the reference rate, and it is an important contribution of the paper.
  2. Rigorous matched controls for negative credit assignment. The Negative-only shuffle baseline preserves positive gating, the LHD objective, and the associated weight statistics, while Dependency-ranked targets matches access to reference information, the analyzer checkpoint, as well as the target count and weight set. EDGE improves GroundedAcc by 2.8 and 1.9 percentage points over these two controls, respectively, and this advantage holds under a shared generation token budget. These results support the claim that the performance gains stem at least in part from properly aligning negative credit with the exact locations of evidence failures.
Critical weaknesses
  1. An explanatory gap remains between the error-correction mechanism of LHD and its optimization objective. Equation (6) applies a stop-gradient to both the privileged branch and the gating mechanism, making the LHD gradient for each sampled token equivalent to a negative log-likelihood gradient with a non-negative weight. Even if a token's probability decreases after incorporating Hindsight, it does not receive an explicit negative penalty from this loss, but merely a smaller reinforcement weight. This behaves more like confidence-weighted self-imitation and does not directly guarantee learning the corrected output. This issue is particularly critical in all-incorrect groups where the task advantage is zero. While existing ablations demonstrate that LHD is effective, they do not yet disentangle the respective contributions of error-correcting information, confidence weighting, and self-imitation itself.
  2. The robustness of the primary repair attribution results to the Oracle abstention policy remains unclear. Appendix E.1 stipulates that when necessary evidence is missing, the repaired Reasoner must abstain from answering, whereas the standard Reasoner may still attempt an answer using partial evidence or prior knowledge. The independent AND audit in Appendix E.3 further shows that the strict Oracle abstains across all tested insufficient-evidence conditions. Consequently, the relative repair value attributed to the Reasoner may be partially driven by this intervention policy rather than solely reflecting the underlying evidence bottleneck. Although independent chain annotations and minimal-repair controls provide supplementary evidence, the paper has not demonstrated whether the repair rankings across the primary 600 cases are robust to alternative abstention policies, which limits the interpretation of the 86.0% attribution agreement rate.
Questions for authors
  1. What actual updates does LHD produce for incorrect tokens whose probabilities decrease after incorporating hindsight information, and how do the authors explain the error-correcting effect of this mechanism in groups where all responses are incorrect and the task advantage is zero?
  2. Under consistent target masking, loss normalization, and training budgets, does LHD still maintain an advantage over baselines using constant gating or shuffled hindsight information?
  3. If the Reasoner repair permits calibrated responses based on incomplete evidence rather than enforcing strict abstention, do the primary agent rankings and EDGE's attribution agreement rate remain stable?
Reviewer_rAqSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SKATEEG, a figure-skating video reasoning benchmark. The paper includes 'explicit evidence requirements', and EDGE, a framework for assigning training credit across planning, perception, and reasoning agents. EDGE identifies the first failed handoff for each evidence record, then selects which failures need correction based on whether the records are jointly necessary or sufficient alternatives. The selected actions receive negative task credit and localized hindsight, while positive credit requires complete evidence support.

Critical strengths
  1. The distinction between record-level failure and answer-level support appears useful. I.e., when one evidence path already supports the answer, correcting an unused alternative creates an unnecessary training target. The method handles this distinction explicitly.

  2. The ablations provide good evidence for the training choices.E.g., shuffling only negative-credit assignments reduces GroundedAcc by 2.8pp, but maintains positive gating and hindsight.

  3. Evaluation includes independent human annotations, repair replay, and interventions to understand different parts of the proposed mechanism. Considerably more useful than if this was to only focus on accuracy, which is great.

Critical weaknesses
  1. The attribution evaluation does not directly test the alternative-path selection rule used during training. Appendix E.1 evaluates the agent associated with the chronologically earliest predicted break. For OR questions, however, training selects the failed path with the longest confirmed passing prefix. The reported 86.0% attribution agreement therefore leaves open whether the path actually selected for training has the greatest repair value. The results compare sufficient-path filtering against targeting all records, but do not isolate the choice among failed alternatives. A comparison with random path selection and other ordering rules would help establish the value of the longest-prefix rule.

  2. A minor point concerns the interpretation of the repair rankings. The main attribution evaluation uses full replacement, including an expert Reasoner that refuses when required facts are missing. A brief summary of what the auxiliary results (App. E3) suggest about ranking stability would help readers interpret the main attribution score.

  3. I found the paper quite difficult to read in terms of its prose. Concepts/names were introduced without being defined, and earlier sections of the paper seemed to make the assumption that I already understood later parts in the paper.

Questions for authors
  1. OR questions with several failed alternatives - how does the repair value of the selected longest-prefix path compare with random selection or the best path identified through replay?

  2. From the existing auxiliary comparisons, do minimal repair and full replacement generally agree on which agents are most useful to repair? Were there any notable differences associated with the expert Reasoner's refusal behavior?

Reviewer_22erOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses evidence-grounded video QA for figure skating, where a wrong answer from a Planner, Perception, Reasoner pipeline cannot be traced to the agent that failed under an outcome reward alone. It introduces SKATEEG\textit{SKATEEG}, 7,200 video questions annotated with gold evidence records and AND/OR support labels, and EDGE\textit{EDGE}, which locates the first failed handoff per record, selects correction targets along sufficient evidence paths, routes the GRPO advantage to those targets via BRPO\textit{BRPO}, and distills link-specific hindsight via LHD\textit{LHD}. A shared 7B backbone reaches 63.2 accuracy and 53.2 GroundedAcc, 2.9 and 8.3 points above MATPO, and first-break attribution agrees with oracle repair replay on 86.0 percent of 600 failures without extra executions.

Critical strengths
  1. $\textit{SKATEEG} $ attaches record-level evidence (window, phase, modality, claim) with audited AND/OR support to every question, anchors windows to takeoff and landing frames with 1.3 and 1.6 frame agreement, and uses annotators with judging or coaching experience (κ\kappa of 0.87 and 0.82). Prior grounded video QA datasets give one window per question.
  2. The sufficient-path rule separates record-level diagnosis from task-level correction, and its effect appears exactly where it should, 4.9 accuracy and 7.4 GroundedAcc points on multi-path OR questions against 0.0 and 0.3 on AND questions.
Critical weaknesses
  1. Non-standard terms are used without being defined. The title phrase figure skating reasoning is never defined, and evidence record, handoff, link, first break, sufficient path, support gate, and the metrics CompleteSupport and GroundedAcc are used in Sections 2 and 3 and Tables 1, 2, and 4 but defined only in Appendices B.7, C.2, and C.3. Please add a terminology section to Section 2 that defines each term once and use it consistently, since the main text cannot be evaluated as written.

  2. The method is evaluated only on figure skating QA. Without a test of a SKATEEG\textit{SKATEEG}-trained policy on another video QA domain, the contribution is limited to one domain and its generality is unsupported.

Questions for authors
  1. Please add a baseline that fine-tunes requests and citations on the training E∗(q)E^*(q) and then trains with GRPO, and report accuracy and GroundedAcc with the same seeds.

  2. Please evaluate the SKATEEG\textit{SKATEEG}-trained EDGE\textit{EDGE} and MATPO checkpoints, together with the pre-RL checkpoint, zero-shot on out-of-domain video QA. NExT-GQA gives accuracy and evidence IoU with existing grounding labels, and SPORTU or SportR gives accuracy on other sports. Report whether the evidence-citing behavior persists and whether accuracy moves relative to the pre-RL checkpoint. This requires inference only.

  3. Is the program-level split skater-disjoint, and how were the exactly proportional test counts in Table 6 obtained?

  4. Did the GPT-5 judge receive the code-computed L1 and L2 states? If not, please report a judge variant that does, so that the 86.0 against 62.5 gap can be attributed to the first-break rule rather than to deterministic parsing of the 360 L1 and L2 breaks.

  5. Please analyze the analyzer improvement in Figure 7a rather than only reporting it. Score the refreshed and the frozen analyzer on one fixed set of stored trajectories with one fixed reference at every checkpoint, so that policy drift and reference noise are removed from the curve. Break the agent-hit change down by link, since L1 and L2 are computed by code and should not move, and report L3 and L4 precision and recall per checkpoint together with the gate false-admission rate on the same trajectories. This needs only re-scoring of stored rollouts, not new training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.