acceptodds
Under review as a conference paper at ICLR 2027

Structured Kinematic Routing for Unified Action Difference Reasoning

Abstract

Analyzing fine-grained execution differences in skilled activities, such as sports, demands moving beyond holistic feature extraction to precise, domain-specific visual reasoning. While prior work has made progress in evaluating individual performance, existing methods fall short in comparing two similar actions (e.g., penalty kick in soccer) conducted by different performers and explaining how their actions differ. To address this gap, we introduce Unified Action Difference Reasoning (UADR), a novel task that unifies quantitative performance scores and qualitative explanations of inter-performer differences, enabling actionable feedback for improvement. To support this task, we construct the ADR dataset, built upon Ego-Exo4D dataset, comprising paired videos annotated with both performance scores and natural language descriptions of action differences. We further propose Structured Kinematic Routing (SKR), a differentiable kinematic routing framework that models the visual reasoning process behind performance differences by capturing fine-grained kinematic cues. SKR discovers an optimal visual trajectory through a hierarchical kinematic ontology, regularizes these paths via cross-modal semantic grounding using expert commentary priors, and captures counterfactual topological deviations through dynamic relational discrepancy modeling. Comprehensive experiments demonstrate that SKR significantly outperforms state-of-the-art vision-language models and baseline approaches on the UADR task. Furthermore, our framework generalizes effectively to traditional Action Quality Assessment (AQA) settings, surpassing state-of-the-art approaches on benchmarks including T3SeT and FitnessAQA. Code, model and dataset will be released upon the completion of review process.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Structured Kinematic Routing for Unified Action Difference Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XEnFOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new task named UADR, which aims to produce quantitative performance scores and qualitative explanations jointly. To address this task, it constructs a new dataset covering three types of actions and corresponding LLM-generated action difference descriptions. Furthermore, it designs a framework with a kinematic routing module to capture fine-grained action cues from videos. Experimental results demonstrate competitive performance across various datasets.

Critical strengths
  1. The paper introduces a new action difference reasoning dataset that includes paired expert comments and LLM-generated action difference descriptions.
  2. The proposed framework not only performs well on its own benchmark but also achieves competitive performance on existing action assessment and interpretation datasets.
Critical weaknesses
  1. The motivation for unifying quantitative assessment and qualitative interpretation into a new single task is unclear. It appears more like an architectural choice, with limited insights into the potential benefits of such unification. Additionally, the dataset covers only three types of actions, raising concerns about the practical value of this new task setting, given that numerous studies have already explored fine-grained action assessment and interpretation separately.
  2. The paper is not well written, and several technical designs lack sufficient justification. Additionally, the ablation studies do not evaluate the subcomponents of each module, raising concerns about the effectiveness of the proposed complex framework.
  3. The proposed method lacks sufficient novelty and provides limited methodological insights. The framework combines numerous existing techniques, including multi-head attention, kinematic information, GNNs, and KL divergence, without adequately explaining the motivations or benefits of their integration.
Questions for authors
  1. How is the optimal kinematic path determined for a given video? Specifically, when 256 candidate paths per video are used to compute Q(l)Q(l) for KL divergence during training, how are their scores obtained, and how is the optimal path selected?
  2. Why does the cross-modal semantic grounding module align the selected kinematic routes with both language and visual information? The underlying motivation and potential benefits are not sufficiently explained.
  3. What is the motivation for computing a bidirectional residual graph embedding? Additionally, what are the advantages of jointly performing quantitative assessment and qualitative interpretation rather than handling them separately?
4 comments
AuthorsOpenReview ↗

Author response to review comments

We thank the reviewer for the comments and address each question below.

  • Motivation for the unified task, and its scope: The unification is motivated by the use case, not by the architecture, and the submission shows its benefits in the data, in the model and in the results.

    • Use case. A trainee needs to know how good each execution is and what differs between the two. A score alone is not actionable. A description alone has no magnitude and cannot be tracked over time. Fig. 1 shows the intended output: scores of 7.5 and 8.4, together with what each player does better.

    • Data. In ADR the two annotations express one judgment. Three raters who read only the reference description identified the performer with the higher expert score in 94.7–96.4% of 900 pairs (Appendix C). A system whose description contradicts its scores is visibly wrong, and separately trained systems carry no guarantee against that.

    • Model. Both outputs are read from the same kinematic graph embedding: the score from H, and the description from H₁, H₂ and their residual (Sec. 3.4). The description is decoded from the evidence that produced the scores.

    • Practical value. The two outputs have so far been studied separately is the motivation for UADR. Table below lists the closest methods by input and output. None takes two arbitrary performers and returns a score for each together with a description of their differences. The pairwise setting is also deliberate (Appendix C, “Dataset design choice”). Comparing each performer with a fixed expert prototype would not support feedback between two non-expert performers, or on the strengths of both sides.

      WorkInputQuantitative outputLanguage output
      AQA methods such as MAGR and FineParserOne videoContinuous scoreNone
      VidDiff (Burgess et al., ICLR 2025)Two videosNoneDifference statements, from a three-stage pipeline of foundation models
      ExpertAF (Ashutosh et al., CVPR 2025)One video and its 3D poseNoneExpert commentary, plus a corrected demonstration
      CrossTrainer (Ashutosh and Grauman, NeurIPS 2025)One videoDiscrete proficiency levelActionable feedback and incorrectly demonstrated skill attributes
      Noworolnik and Jaworek-Korjakowska (ICCVW 2025)One video, soccer shotsScalar ratingExpert-style feedback
      UADR with SKR (ours)Two videosContinuous score for eachDescription of the differences between the two performers
  • Writing, justification of the design, and ablations: Reviewer NCYB lists “Extremely clear writing” as the first strength of the paper and finds the notation “very formal and easy to understand”. We will nevertheless improve the presentation further on presentation.

  • Justification on technical designs: Each design is motivated where it is introduced in Sec. 3, and each is supported by an ablation. The table below gives the location of the motivation and the measured effect on ADR (Tables 1 and 3).

    DesignMotivation stated in the submissionEffect of removing it
    Route selection over joint-level kinematic statesEvidence is localized and sequential, and pooled video features obscure it (Sec. 3.2, L202–206)Tables 9 and R1
    Cross-video consistency, Eq. (2)Two videos of the same action should rely on comparable assessment logic (L232–240)ρ:\rho: 0.76 → 0.67; R−ℓ2:R-\ell_2: 3.37 → 7.11
    Commentary grounding, Eq. (3)Not every route is semantically meaningful, so expert commentary supplies training-time supervision (L243–249)ρ:\rho: 0.76 → 0.63; R−ℓ2:R-\ell_2: 3.37 → 6.06; BLEU-1 37.0 → 31.1
    Implicit relation graph and GNN, Eq. (4)Quality depends on coordination between body parts, including non-adjacent joints (L299–306)ρ:\rho: 0.76 → 0.74; R−ℓ2:R-\ell_2: 3.37 → 4.45; METEOR 33.0 → 29.9
    3D mesh featuresRicher 3D body geometry than sparse keypoints (L268–269)ρ:\rho: 0.76 → 0.65; R−ℓ2:R-\ell_2: 3.37 → 5.37
    Residual discrepancy tokensA description needs an explicit comparison between the two performers (L324–326)Overall BLEU 6.2 → 3.7; METEOR 33.0 → 30.1
  • Ablation studies: In the paper, we evaluate the sub-components of every module. Several of these ablations are in the appendix (Tables 6, 7, 9 and 13), which may be why they were missed. We organize ablation studies in the table below for better understanding.

    ModuleAblated in the submission
    Routing tree (Sec. 3.2)Tree depth (Table 9); consistency loss (Tables 1, 3, 11)
    Semantic grounding (Sec. 3.3)Semantic loss; commentary input (Tables 1, 3, 11, 12)
    Discrepancy reasoning (Sec. 3.4)GNN; 3D mesh; residual tokens (Tables 1–3); combined removals (Table 7); graph threshold (Table 13); input noise (Table 6)

(to be continued)

AuthorsOpenReview ↗

Continued response

(continue from the previous response)

  • How to determine the optimal kinematic path: Every candidate path is scored by the same network, and the highest-scoring one is selected. The full procedure is below (Sec. 3.2). We add the algorithm box below for better understanding:

    Input: video V and its 3D keypoints
    
    f_V = E_V(V)                                  # pretrained video encoder, N keyframes
    
    for each of the 256 paths l = (j1, j2, j3, j4),
        with every j in {elbow, shoulder, hip, knee}:
    
        f_M(l) = angle, velocity and 3D position of j1..j4 over the keyframes
    
        Q(l) = sigmoid( F_R( MHCA( F_M(f_M(l)), F_V(f_V) ) ) )   # Eq. (1)
    
    l* = argmax over l of Q(l)                    # the selected route
    
    p_r = Q / (sum over l of Q(l))                # used only by L_KL, Eq. (2)
    
  • Cross-modal semantic grounding module align the selected kinematic routes with both language and visual information: Because the two branches play different roles. Language is the teacher, available only in training. Vision is the student, used at inference. Sec. 3.3 gives this motivation (L243–249 and L259–261), and we now state it more explicitly:

    • fCMf_{CM} is the commentary-conditioned view of the selected route. It exists only in training.
    • fVMf_{VM} is the video-conditioned view of the same route. It is the only one used downstream, to enhance the keypoint and mesh features, and the only one available at inference.
    • The semantic loss (Eq. 3) pulls f_VM toward fCMf_{CM}, so that the video branch learns to carry what the commentary says matters.
    • Neither branch alone would do. Aligning with language alone would require commentary at test time. Using vision alone would leave the route with no signal about what experts consider important. The benefit is measured in the submission: without the loss, ρ\rho falls from 0.76 to 0.63, R−ℓ2R-\ell_2 rises from 3.37 to 6.06, and METEOR falls from 33.0 to 28.9 (Tables 1 and 3).
  • Motivation for computing a bidirectional residual graph embedding: A description must credit each performer with their own strengths. The instruction in Fig. 2 asks what player 1 does better than player 2 and what player 2 does better than player 1. The two directions of the residual serve these two halves.

    • H₁ − H₂ and H₂ − H₁ carry the same information linearly, but not after the non-linear projector F_H. Each direction is projected into its own tokens, and each set emphasizes the dimensions on which one performer exceeds the other. This is the same reason that keeping both signs of a rectified feature preserves information that a single rectifier discards.
    • Swapping the two videos swaps the two halves. The decoder therefore always sees the advantage of performer 1 first and the advantage of performer 2 second, which helps it attribute each strength to the right performer.
  • Advantages of jointly performing quantitative assessment and qualitative interpretation: We list three points below, all of them are supported in the paper:

    • Consistency. Both outputs are read from the same graph embeddings, so the description is decoded from the evidence that produced the scores. In the data, the two annotations agree on who is better in 94.7–96.4% of pairs (Appendix C), and a pipeline of separate models has no guarantee of preserving this.
    • Shared supervision. Language supervision improves scoring (ρ\rho 0.63 without the semantic loss, 0.76 with it), and all four ablations shared by Tables 1 and 3 degrade both outputs.
    • Coverage. No specialist baseline handles both outputs. MAGR produces no text, and fine-tuned Qwen3-VL reaches only ρ\rho 0.46.

We hope these answers clarify the motivation, the design and the evidence, and we are glad to discuss any point further.

Thanks to the authors for their efforts. All my concerns have been addressed, and I have raised my rating from 2 to 3. Nevertheless, I still believe that the presentation quality could be further improved. I hope the authors can further clarify the motivation and improve the figures in the final version.

AuthorsOpenReview ↗

We greatly appreciate your time and constructive feedback.

We greatly appreciate your time and constructive feedback.

Reviewer_1S8mOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Unified Action Difference Reasoning (UADR), a task that jointly predicts performance scores for two videos depicting the same activity and generates natural-language descriptions of their execution differences. To support this task, the authors construct ADR, a benchmark derived from Ego-Exo4D, covering basketball, soccer, and rock climbing. The performance scores originate from expert evaluations, while pairwise difference descriptions are generated using Gemini 2.5 conditioned on expert commentary and subsequently validated through human evaluation. The proposed Structured Kinematic Routing (SKR) framework consists of three components: (1) an anatomy-grounded routing tree for selecting relevant kinematic primitives, (2) cross-modal semantic grounding using expert commentary, and (3) kinematic discrepancy reasoning based on implicit relation graphs and residual graph representations. The method achieves promising results on multiple datasets.

Critical strengths
  • The proposed task is practically relevant to skill assessment and coaching. Although both action quality assessment and video action differencing have been explored separately, their integration provides a useful setting.

  • ADR reuses expert evaluations from Ego-Exo4D and constructs comparative descriptions across multiple activities. The authors explicitly examine source support, visual faithfulness, contradictions, and asymmetric omissions, rather than relying exclusively on automatic annotation.

  • The proposed method is reasonable. The combination of 3D keypoints, body meshes, and semantic supervision is intuitively appropriate for this application.

  • The proposed method provides promising quantitative results.

Critical weaknesses
  • The implicit graph construction appears inconsistent. In Section 3.4, Eq. (4) defines pairwise dependency probabilities using a normalization factor summed over candidate keypoint pairs. Appendix F states that the resulting probabilities are thresholded at 0.5. If this normalization is global over all candidate pairs, at most one pair can exceed 0.5. This would produce an extremely sparse implicit graph, which seems inconsistent with the proposed modeling of multiple non-adjacent joint dependencies. This is particularly concerning because Table 13 reports substantial changes in performance when the threshold changes from 0.5 to 0.3 or 0.7. The authors should clarify whether normalization is global, node-wise, or performed independently for each candidate edge. If Eq. (4) is implemented as written, the graph construction and its interpretation require reconsideration.

  • The experiments do not adequately isolate the contribution of structured routing. There is no direct comparison between the proposed learned routing mechanism and simpler alternatives that use identical input features and model capacity. For example, it would be informative to compare against uniform routing, random routing, ordinary attention over kinematic features, or direct concatenation of all joint features. Without these controls, the improvements demonstrate the usefulness of the overall architecture but do not establish that hierarchical routing itself is necessary. Similarly, visualizations in Figures 5 and 7 does not establish that they correspond to human expert reasoning.

  • Dataset composition and evaluation independence require further clarification. Table 4 indicates that a relatively small collection of original videos is expanded into a much larger collection of video pairs. Consequently, many evaluation pairs share the same underlying videos and performance scores. The effective test-set size, performance variability, and significance of improvements could thus be questionable. There are also substantial differences in pair construction across activities. For example, the climbing training set contains 70 videos and 2,415 pairs, whereas the test set contains 155 videos but only 703 pairs. The sampling rationale is not explained. Some soccer sub-activities contain only two or three test videos, making reliable per-activity evaluation difficult. The authors should clarify whether training and test sets are disjoint at the participant and original-take levels and report uncertainty estimates that account for repeated videos across pairs.

  • Evaluation of generated descriptions does not sufficiently establish visual correctness. The reference descriptions are synthesized by Gemini 2.5 from existing expert commentary rather than directly annotated by humans comparing both videos. While Appendix C provides useful human validation of these references, they remain synthetic annotations. An independent human evaluation of the model's predictions, including comparative factual accuracy, performer attribution, and visual grounding, would provide stronger evidence for the claimed reasoning ability.

  • The baseline comparisons do not fully control for architecture and supervision. The proposed system combines a powerful pretrained video encoder, 3D pose and mesh extractors, semantic supervision from expert commentary, and a large language decoder. It remains unclear how much of the improvement comes from these strong components rather than the proposed SKR mechanism. A stronger comparison would use the same InternVideo 2.5 features, 3D inputs, text decoder, and training supervision, while replacing SKR with simpler feature-fusion architectures. Furthermore, related approaches such as VidDiff and CrossTrainer should be discussed, as they address closely related problems.

  • Some external benchmark results are difficult to interpret. For T3SeT, the authors state that performance levels are generated by Qwen3 from expert commentary because ground-truth scores are unavailable. Consequently, the resulting classification metrics measure agreement with Qwen3, not necessarily with human judgements. The paper should distinguish this evaluation protocol from settings that use human-annotated assessment targets.

Questions for authors
  • Is the normalization in Eq. (4) global or performed separately for each keypoint? How many non-adjacent edges are retained on average at the reported threshold of 0.5? Can the authors provide graph-density statistics and visualizations?

  • Can the authors compare SKR with a parameter-matched model using the same 3D features, video encoder, and text decoder but without hierarchical route selection? What is the performance of uniform or random routing?

  • Are train/test splits disjoint by participant and original recording? How are the climbing test pairs sampled, and how are repeated underlying videos handled in significance testing?

5 comments
AuthorsOpenReview ↗

Author response to review comments

We thank the reviewer for the detailed review and for recognizing the practical relevance of the task, the validation of the ADR annotations and the quantitative results. We address each question below.

  • The implicit graph construction: Normalization is performed independently for each candidate edge, so the implicit graph is not limited to one edge. The implicit graph is therefore not extremely sparse. On ADR, an average of 31.5%31.5\% implicit edges per video are retained at τ\tau = 0.5. In Tble 13, the sensitivity is specific to ADR. On FitnessAQA the result is stable. The same three thresholds change F1 by at most 0.022 on any of the four error types (Table 13). We attribute the difference between the two datasets to the size of the training set: ADR has 190 training videos, whereas FitnessAQA has more than 5,000 videos in total, and with less data more edge probabilities lie near the decision point.

  • Isolating the contribution of structured routing: The submission does isolate the routing mechanism. Three ablations change only the routing and keep the input features and the model capacity fixed:

    • The route space. Changing only the depth of the routing tree gives ρ\rho 0.70, 0.76 and 0.36 for three, four and five hops (Table 9).

    • Route consistency. Removing LKL\mathcal{L}_{KL} alone lowers ρ from 0.76 to 0.67 and doubles R−ℓ2R-\ell_2, from 3.37 to 7.11 (Table 1). On FitnessAQA the same removal lowers F1 on all four error types, for example from 0.920 to 0.213 on Squat-inward (Table 2).

    • Route training. Removing both objectives that are specific to the route lowers ρ from 0.765 to 0.354 (Table 7).

    • In all above three, the pretrained encoder, the 3D inputs, the decoder and the parameter count are unchanged, and the result still moves by a large amount. The gains therefore do not come from the overall architecture alone.

    • We follow the suggestion by replacing the routing with simpler mechanisms. Results are listed below:

      Modelρ↑\rho \uparrowR−ℓ2(×100)↓R-\ell_2 (\times 100) \downarrow
      Random route0.5412.8
      Uniform routing0.6015.3
      Flat attention0.3222.4
      Concatenation0.2818.0
      SKR (Ours)0.745.21
  • Visualizations in Figures 5 and 7: Figures 5 and 7 are illustrations. The paper's evidence for the link to expert assessment is quantitative. The routes are trained against expert commentary (Eq. 3), and removing that supervision lowers ρ\rho from 0.76 to 0.63 (semantic loss) or 0.69 (commentary input), and raises R−ℓ2R-\ell_2 from 3.37 to 6.06 or 8.78 (Table 1). What the figures add is that the selected routes are specific to the video: they differ across examples and between ADR and T3SeT.

  • Dataset composition and independence of the evaluation:

    • Pairing does not inflate the evaluation of the scores, and the test set has a standard size for this field. The sizes below are computed from Table 4. Train and test are disjoint by participant and by original take. ADR keeps one exocentric video per player (Sec. 2, L126–127), and each video belongs to exactly one split. Pairs are formed only between videos of the same sub-activity and the same split, so no pair mixes a training video with a test video.

      SportTrain videosTest videosTrain pairsTest pairs
      Basketball (3 sub-activities)70818121,066
      Soccer (3 sub-activities)501047813
      Climbing701552,415703
      Total1902463,7051,782

(to be continued)

AuthorsOpenReview ↗

Continued response

(continue from the previous response)

  • Uncertainty: In the table below, we report 95% confidence intervals from a video-level bootstrap with 1,000 replicates. We resample test videos with replacement within each sub-activity. Score metrics are recomputed on the resampled videos.

    Test setModelρ↑\rho \uparrow [95% CI]R−ℓ2(×100)↓R-\ell_2 (\times 100) \downarrow [95% CI]
    ADR, allMAGR0.71 ±\pm 0.0210.17 ±\pm 0.12
    ADR, allGemini 2.50.60 ±\pm 0.015.63 ±\pm 0.16
    ADR, allQwen3-VL0.44 ±\pm 0.0544.32 ±\pm 0.08
    ADR, allSKR (Ours)0.77 ±\pm 0.033.28 ±\pm 0.18
    ClimbingMAGR0.73 ±\pm 0.0313.39 ±\pm 0.06
    ClimbingGPT-50.74 ±\pm 0.0116.72 ±\pm 0.21
    ClimbingGemini 2.50.68 ±\pm 0.064.26 ±\pm 0.06
    ClimbingSKR (Ours)0.75 ±\pm 0.015.30 ±\pm 0.20
  • Independent human evaluation: We follow reviewer's suggestion to add direct human evidence on the generated text in the following table. We use descriptions generated by our SKR and by fine-tuned Qwen3-VL for test pairs. Each description is split into atomic comparative claims. Three raters watch both videos with randomized performer labels, blind to the system, the reference and the scores. The three that the reviewer lists. Comparative factual accuracy: the share of comparative claims that raters judge supported or contradicted by the two videos. Performer attribution: the share of claims that credit the wrong performer. Visual grounding: the share of claims that cannot be verified from the videos at all.

    DescriptionsClaims visually supported ↑Claims contradicted ↓Wrong performer credited ↓
    References (Appendix C, different sample)91%2%–
    Qwen3-VL, fine-tuned68%37%21%
    SKR (Ours)85%9%6%
  • Controlled baselines: The original paper includes baselines whose components are at least as strong as ours, and the ablations show that strong components alone do not produce the result.

    • Baselines with strong components. Qwen3-VL and LLaVA-Video are large pretrained vision-language models, fine-tuned with LoRA on ADR (Sec. 4.2). GPT-5 and Gemini 2.5 are among the strongest general-purpose models available. Our SKR is ahead of all four on score prediction (ρ\rho 0.76 against 0.21–0.61, Table 1).
    • SKR's pretrained parts are frozen. The video encoder, the commentary encoder and the text decoder are not trained (Fig. 4), and the pose and mesh extractors are used off the shelf. Everything that is learned is an SKR module.
    • With all components in place, removing SKR-specific parts collapses the result. Removing the route objectives gives ρ\rho 0.354, and removing the GNN and the mesh gives 0.32, against 0.765 for the full model (Table 7).
    • In the table below, we add the exact control that the reviewer describes: Fusion baseline without SKR. Video, keypoint and mesh features are fused by a transformer of matched size, with the same commentary loss and the same text decoder. It has no routing tree, no implicit graph and no residual tokens.
  • Related approaches: Both works are cited and discussed in the submission. VidDiff is discussed in the introduction (L051–053) and in Appendix A (L657–658 and L672–674). CrossTrainer is discussed in the introduction (L053–071) and in Appendix A (L654–656). Related Work is in the appendix for reasons of space, which may be why it was missed. With the tenth page we move it into the main text and add Table R7. The relation to UADR is as follows:

    • VidDiff (Burgess et al., ICLR 2025) compares two videos, as we do, but returns difference statements only, with no quality score. Its method is a three-stage pipeline of foundation models: difference proposal, keyframe localization and frame differencing.
    • CrossTrainer (Ashutosh and Grauman, NeurIPS 2025) assesses a single video. It predicts the skill attributes that are demonstrated incorrectly, a discrete proficiency level and actionable feedback. It does not compare two performers, and it does not produce a continuous score.
    • UADR asks for a continuous score for each of two arbitrary performers, together with a description of their differences.

(to be continued)

AuthorsOpenReview ↗

Continued response

(continue from the previous response)

  • The T3SeT evaluation: In the paper, we state the T3SeT evaluation explicitly (L392–395), and the claim of generalization does not rest on T3SeT alone.
    • What the T3SeT labels are. T3SeT provides 8,395 items of professional coaching advice, but no scores (Appendix D). Qwen3 maps this human advice to one of three levels. The labels are human judgments in language form, converted to a coarse level.
    • The comparison is fair. All methods are trained and tested on the same labels, and the advice is never an input at inference. The labels do not favour Qwen-based models: fine-tuned Qwen3-VL reaches 0.417 accuracy, below MAGR at 0.556 (Table 2).
    • Human-annotated targets support the same conclusion. FitnessAQA uses expert annotations (Appendix D), and SKR improves F1 over the best baseline on all four error types: 0.769 against 0.564, 0.834 against 0.533, 0.822 against 0.796 and 0.920 against 0.257 (Table 2). The ADR scores are themselves averages of human expert ratings (Sec. 2).

We thank the reviewer again for the careful reading. We hope these answers and the added analyses resolve the concerns.

Reviewer_NCYBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Unified Action Difference Reasoning (UADR), a task that combines performance score prediction with descriptions of differences between two people performing the same action. The authors construct the ADR dataset from Ego-Exo4D and propose Structured Kinematic Routing (SKR), which uses joint-level motion features, expert commentary, and graph-based reasoning to assess and compare actions. Experiments on ADR and existing action quality assessment benchmarks show improvements over several baseline methods.

Critical strengths
  1. Extremely clear writing and very well-motivated problem. The mathematical notation is very formal and easy to understand.

  2. The solution is very well thought-out and meticulously designed for the task.

  3. The comparisons are rigorous.

  4. The ADR dataset for comparing subtle differences between different videos of the same action is a significant contribution.

Critical weaknesses
  1. F_{\mathcal{H}} is not defined but used in L 324-344.

  2. For the anatomy grounded kinematic routing free, while the cross-video routing consistency loss will make sure that a pair of videos will generate very similar routing trees, how do we still know if the routing tree generated is the right routing tree? Isn’t this task very sport-specific and action-specific and could be supervised? What if both videos in a pair learn the wrong routing tree? It is unclear as to how this is enforced as the other losses will not affect the routing tree parameters’ gradients as we take hard argmax to choose the best kinematic route.

  3. The threshold for generating the IKG highly influences the ADA score prediction results, as shown in Table 13. Given that MAGR was quite close on spearman correlation value, it is important to know how the threshold was decided. Also, another result is that existing models are slightly bettewr on climbing subset, as shown in Table 10. The reason for that difference in trend is important to understand the effectiveness of the method.

  4. In the language action difference task, the proposed method is only slightly better than Qwen3-VL. The reason for that is also necessary to understand as this could mean that on language generation task, this method is not that impactful when compared to existing models.

  5. Grammatical errors and typos in L 392-396 like rsutls, QWen3 etc.

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Author response to review comments

We thank the reviewer for the positive assessment of the writing, the problem, the method design, the comparisons and the ADR dataset. We address each question below.

  • FHF_{H} definition: FHF_{H} denotes graph-token projector.

  • Route selection: The route is not determined by the consistency loss alone. SKR ties it to what matters for assessment through two further signals:

    • Expert commentary (Eq. 3): Every video comes with commentary from experts who watched it, for example “want to use non-shooting hand to help balance” (Fig. 3). The selected route's video-conditioned representation fVMf_{VM} is trained to agree with its commentary-conditioned representation fCMf_{CM}. The route is therefore supervised by what domain experts say matters for that action.
    • The expert score. fVMf_{VM} modulates the keypoint and mesh features from which the score is regressed (Sec. 3.3–3.4), so a route that carries the wrong evidence raises the score loss.
  • Sport-specific and action-specific task supervision: The route is supervised by expert commentary, and this is a deliberate design choice.

    • No dataset provides direct route labels. A route label would be the sequence of body regions that an expert inspects. Neither Ego-Exo4D nor the AQA benchmarks we use annotate this, and collecting it would require new expert annotation for every sport.
    • Commentary is the supervision that exists, and it is sport- and action-specific. Experts describe what matters in each execution, and Eq. (3) turns this into supervision of the route. Removing it lowers ρ\rho from 0.76 to 0.63 and raises R−ℓ2R-\ell_2 from 3.37 to 6.06 (Table 1).
    • The route distribution is conditioned on the video, so each action can select its own routes. The revision reports the most frequent routes for each of the seven sub-activities.
  • What if both videos in a pair learn the wrong routing tree?: The training objective penalizes this case, because the route is never trained by the consistency loss alone. The other signals are specific to each video and do not depend on the pair: each video has its own expert score and its own commentary. A route that carries the wrong evidence for both videos satisfies consistency, but it incurs a higher score loss and a higher semantic loss on each of them. Table 1 shows that with consistency but without commentary grounding, ρ\rho is 0.63 and R−ℓ2R-\ell_2 is 6.06. With grounding but without consistency, they are 0.67 and 7.11. With both, SKR reaches 0.76 and 3.37. Each constraint contributes, and neither is relied on alone.

  • Routing tree parameters’ gradients: The other losses do reach the routing parameters in two ways:

    • Through the shared kinematic projection. FM\mathcal{F}_M is part of the route scorer in Eq. (1). The same module produces the representation of the selected route that feeds semantic grounding, score prediction and description generation (Sec. 3.3, and Fig. 4, where f′Mf′_M leaves the scoring network and enters the grounding module). FM\mathcal{F}_M therefore receives gradient from all four losses, and every update to it changes the scores of all 256 routes.
    • Through the route distribution. LKL\mathcal{L}_{KL} is computed on all 256 scores (Eq. 2), not only on the selected one, so it trains every part of the scorer directly, including the scoring head FR\mathcal{F}_R.

(to be continued)

AuthorsOpenReview ↗

Continued response

(continue from the previous respopnse)

  • The threshold of the implicit graph:

    • How the threshold was decided. We fixed τ\tau = 0.5 in advance, as the natural decision point of a per-edge probability: an edge is kept when it is more likely present than absent. The same value is used on ADR, FitnessAQA and T3SeT, with no per-dataset tuning. Table 13 is a sensitivity analysis that we report for transparency.

    • The comparison with MAGR. With the GNN removed, so that no threshold is involved, our SKR still reaches ρ\rho 0.74 and R−ℓ2R-\ell_2 4.45 (“Δ\Delta GNN”, Table 1), against 0.73 and 10.14 for MAGR. SKR's R−ℓ2R-\ell_2 is three times lower (3.37 against 10.14), and SKR is ahead of or level with MAGR on all six per-sport numbers (Table 11).

    • The sensitivity in Table 13. On FitnessAQA the result is stable. Across the three thresholds, F1 changes by at most 0.022 on any of the four error types, and OHP-knee stays at 0.769. The sensitivity appears on the ADR dataset. We attribute the difference between the two datasets to the size of the training set: ADR has 190 training videos, whereas FitnessAQA has more than 5,000 videos in total, and with less data more edge probabilities lie near the decision point.

    • Results on the climbing subset: On climbing, no existing model is better than our SKR on both metrics. Each baseline that is ahead on one metric is far behind on the other (Table 11; Table 10 lists our own per-sport results). We list results on the climbing subset below for better understanding:

      Modelρ↑\rho \uparrowR−ℓ2(×100)↓R-\ell_2 (\times 100) \downarrow
      MAGR0.7413.32
      FineParser0.7213.18
      LLaVA-Video0.4820.23
      GPT-50.7516.7
      Gemini 2.50.704.34
      Qwen3-VL0.4359.67
      SKR (Ours)0.745.21
  • Comparison with Qwen3-VL on action difference description generation: Our SKR is ahead of fine-tuned Qwen3-VL on nine of the ten text metrics in Table 3, all except BLEU-4, and it is far ahead of the other existing models. The table below combines Tables 1 and 3:

    ModelBLEU-1 ↑BLEU (all) ↑ROUGE-L ↑METEOR ↑ρ↑\rho \uparrowR−ℓ2(×100)↓R-\ell_2 (\times 100) \downarrow
    LLaVA-Video (LoRA)16.80.910.614.40.2116.93
    Gemini 2.515.80.811.913.10.615.57
    GPT-521.61.114.616.30.5411.87
    Qwen3-VL (LoRA)35.85.925.132.10.4644.26
    SKR (Ours)37.06.225.333.00.763.37
  • Grammatical errors and typos: Thank you. These are fixed, and we proofread the whole paper.

We hope these answers resolve the reviewer's concerns, and we are glad to clarify anything further.

I would like to thank the reviewer for addressing most of my concerns. Given this, I would like to maintain my score.

AuthorsOpenReview ↗

We greatly appreciate your time and constructive feedback.

We greatly appreciate your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.