- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a neuro-symbolic framework for Action Quality Assessment (AQA), which is named CriScore. The core idea of CriScore is to treat scoring criteria as learnable executable structures. The paper evaluates CriScore on MTL-AQA, TASD-2, FineDiving, and UI-PRMD, and experiments show that CriScore achieves the strongest reported results. I find the general idea of making the scoring structure explicit and trainable interesting. However, I have several concerns about the central methodological claim.
Critical strengths
- The paper raises an interesting question about the scoring structure in AQA.
- The experimental analysis is extensive.
- The proposed method achieves the best results.
Critical weaknesses
-
My main conceptual concern is the interpretation of what is actually being learned. The claimed learnable scoring rules appear closer to structured fusion over learned representations than to semantic rule learning. In detail, the proposed method mainly consists of source selection, weighted rank-based composition, blending with a base scorer, monotonic score mapping, and optional bounded residual correction. These operations are interpretable and executable, but they are closer to a constrained structured ensemble than to learning semantic scoring rules such as conditional expert criteria over observable action properties. I therefore think the paper should either substantially weaken the learnable scoring rule claim or provide stronger evidence that the learned bindings and concepts have semantic validity, e.g., through concept-level annotations, official sub-scores, concept probing, or expert validation.
-
The necessity of using an LLM for structure learning is not sufficiently motivated. The paper motivates CriScore by arguing that fixed scoring structures are limiting and that scoring criteria should instead be learnable. I agree with this motivation. However, this does not by itself justify the use of an LLM. At present, the role of the LLM seems better characterized as a semantic search prior / candidate proposer than as a necessary component for learning scoring criteria.
-
The concept-evidence binding is only weakly grounded. The paper places substantial emphasis on binding evaluation concepts to evidence sources. However, the empirical evidence for these bindings is limited. For example, showing that removing VideoMAE evidence reduces final SRCC demonstrates that VideoMAE is useful for quality prediction; it does not demonstrate that VideoMAE specifically represents the evaluation concept assigned to it. The TASD-2 replay and source-deletion experiments are informative about predictive importance, but not about semantic grounding. This is an important distinction because the interpretability claim of the framework depends heavily on the meaning of these bindings.
-
Table 1 does not appear to provide a fully feature-controlled comparison. The broad comparison against 33 baselines is impressive at first glance, but the methods do not appear to operate under a common information budget. I strongly recommend that Table 1 explicitly report the input modality, backbone, feature source, metadata, and auxiliary information available to each method. More importantly, the paper should include a same-feature controlled comparison in which competing fusion/gating/structure-learning methods receive exactly the same evidence bank as CriScore.
-
The current analysis does not clearly distinguish CriScore from gating or learned fusion. CriScore mainly selects evidence sources, assigns weights, combines them with a base scorer, and optionally applies a residual correction. This is similar to a structured gating or mixture-of-experts mechanism. The paper should compare against a strong learned gating/fusion baseline using the same evidence sources and calibration modules. Otherwise, it is unclear whether the gains come from genuinely neuro-symbolic reasoning or simply from better source selection and fusion.
-
Important search and calibration hyperparameters lack sensitivity analysis. The method depends heavily on a number of discrete search and calibration choices, but most are fixed without systematic sensitivity analysis.
-
The program design itself is under-justified. Components such as base-score blending, temperature scaling, and residual correction appear manually chosen, and it is unclear why they are necessary for learning scoring criteria rather than simply increasing model flexibility.
-
Minors
(1) The paper would benefit from distinguishing more carefully between semantic criteria, evidence-source selection, and numerical calibration.
(2) Several ablations remove multiple mechanisms simultaneously. For example, bypassing execution also removes composition and calibration, making the causal attribution of the resulting performance drop less precise.
(3) The selected program is task-level and frozen across observations, limiting adaptation to sample-specific quality cues.
(4) The paper should report the actual selected programs for all four datasets in a compact table. This would make the claimed interpretability easier to assess.
Questions for authors
See above.
I am currently leaning toward acceptance because the paper presents an interesting and potentially useful framework, and the empirical results are strong. However, my positive assessment depends on the authors being able to clearly address the main conceptual concerns above, especially the distinction between semantic rule learning and structured fusion/calibration.
I would also like the authors to more carefully position the contribution. If these concerns cannot be convincingly clarified during rebuttal, I would likely lower my score.
4 comments
W4. Feature-controlled comparison.
a. Baseline information. To clarify the information used by the baselines in Table 1, we summarize their input features, model architectures, and auxiliary information below. A detailed comparison of the evaluated baselines is provided in Table 3 of the revised manuscript.
- Feature Regression methods: Ridge and SVR use cached global sequence statistics with L2-regularized linear regression or RBF-kernel regression, without neural encoders.
- General Temporal methods: Use cached temporal features, including descriptors for FineDiving and joint sequences for UI-PRMD, with Transformer, Inception, TCN, and state-space architectures. Action indicators and difficulty metadata are used where applicable.
- Graph Motion methods: Use 2D/3D skeletal trajectories and kinetic graphs with spatial-temporal GCNs, including topology-refinement and hierarchical variants. Auxiliary inputs include joint connectivity and action indicators.
- AQA-Specific methods: Use video, optical flow, pose heatmaps, splash masks, and multimodal features with specialized AQA architectures, including contrastive regression, hierarchical parsing, and graph-based models. Reference scores, action-stage annotations, and difficulty metadata are used according to the respective protocols.
b.Same-evidence controlled comparison. Table 4 compares different fusion and program-selection methods using the same evidence predictions, validation folds, and score-mapping setup. All search-based methods are allowed to evaluate up to 64 candidates. This ensures that the methods are compared using the same evidence and under the same search budget. Under this controlled setting, CriScore achieves the best SRCC across all four datasets, outperforming non-language program selection by an average of 0.613 percentage points.
W5.Comparisons with learned fusion and gating methods.
To provide a stronger comparison, we have added a learned gating baseline that adjusts the weights of different evidence sources for each video. This baseline uses the same evidence predictions as CriScore, with an MLP to generate the weights and the same score-mapping setup. All methods use the same evidence predictions and score-mapping setup, allowing a direct comparison of their scoring strategies. We have also updated Table 4 in the revised manuscript to include the new baseline and revised the corresponding discussion in Appendix C.2 accordingly.
| Method | MTL-AQA | TASD-2 | FineDiving | UI-PRMD |
|---|---|---|---|---|
| Learned Gating Fusion | 85.22 | 85.74 | 80.78 | 73.99 |
| CriScore | 86.12 | 86.96 | 81.34 | 74.67 |
W6. Sensitivity analysis of search and calibration hyperparameters.
We have conducted additional experiments to examine the sensitivity of CriScore to the candidate evaluation budget and residual-correction strength. The results are shown in below, and the corresponding experiments and analysis have been added to Appendix C.8 of the revised manuscript.
a. Candidate search budget. The table below reports SRCC (%) at candidate evaluation budgets of 24, 32, 48, and 64, with temperature fixed at T = 1.0 and all other settings unchanged. These results show that increasing the search budget improves prediction performance, with diminishing returns at larger budgets.
| Candidate budget K | MTL-AQA | TASD-2 | FineDiving | UI-PRMD |
|---|---|---|---|---|
| 24 | 85.34 | 86.02 | 80.42 | 73.81 |
| 32 | 85.78 | 86.48 | 80.90 | 74.24 |
| 48 | 86.05 | 86.82 | 81.18 | 74.54 |
| 64 (default) | 86.12 | 86.96 | 81.34 | 74.67 |
b. Residual-calibration strength. The table below reports the sensitivity of residual calibration on UI-PRMD using training-side grouped cross-validation with three seeds and five folds. We vary the correction strength while keeping the fitted residual predictors, score mappings, and clipping bounds unchanged. The clipping bound is fixed at half the training-side residual standard deviation. These results demonstrate the importance of controlling residual-correction strength rather than applying the full predicted adjustment.
| Residual correction strength | UI-PRMD SRCC (%) |
|---|---|
| 0 | 74.023 |
| 0.25 | 74.340 |
| 0.50 (default) | 74.667 |
| 0.75 | 74.417 |
| 1.00 | 73.907 |
c. Temperature and blending-strength sensitivity. We examine the effects of temperature and blending strength using the FineDiving training-side validation results. Each setting is evaluated across three seeds and five outer folds, with the selected evidence sources and other scoring operations unchanged. The table reports mean inner-validation SRCC (%). Results show that SRCC reaches its highest value at for all tested temperatures, with the best result obtained at and . Performance varies by at most 0.12 percentage points across temperatures for a fixed blending strength, indicating limited sensitivity to within the evaluated range.
| T \ β | 0.25 | 0.50 | 0.75 | 1.00 |
|---|---|---|---|---|
| 0.5 | 80.27 | 80.58 | 80.72 | 80.68 |
| 1.0 | 80.22 | 80.52 | 80.67 | 80.66 |
| 2.0 | 80.19 | 80.46 | 80.60 | 80.59 |
W7. Motivation and contribution of the scoring-program operators.
The operators in CriScore serve different roles. Base-score blending controls how much additional evidence contributes to the base prediction. Temperature scaling adjusts the relative weights of different evidence sources. Residual correction provides an optional and bounded adjustment to the final prediction. These settings are selected or fitted using training data and remain fixed during evaluation.
To further evaluate the contribution of base-score blending, we set to remove the base prediction from the composition stage while preserving the remaining scoring operations. The results are shown below.
| Dataset | Default SRCC (%) | w/o Base () | Retained Evidence |
|---|---|---|---|
| MTL-AQA | 86.12 | 81.43 | Swin only |
| TASD-2 | 86.96 | 83.19 | I3D, VideoMAE |
| FineDiving | 81.34 | 78.65 | Difficulty, bounding-box compactness, splash-presence evidence |
| UI-PRMD | 74.67 | 70.22 | Action-type and phase evidence |
The results show that removing the base contribution consistently reduces SRCC across all four datasets. This intervention preserves the remaining scoring operations, whereas No Exec. bypasses the entire scoring program. Table 2 further shows that replacing the selected source weights with equal weights or disabling residual correction also leads to performance degradation. Together, these results demonstrate the contributions of base-score blending, source weighting, and bounded residual correction to CriScore's performance.
W8. Distinguishing semantic criteria, source selection, and numerical fitting.
- Semantic criteria define what aspects of a performance matter from domain knowledge (e.g., body posture, entry angle).
- Source selection (evidence binding) determines which available evidence sources are associated with these aspects.
- Numerical fitting resolves how these sources are weighted, aggregated, and calibrated under overall-score supervision. We have added a brief explanation in Section 3.1 to clarify these distinctions.
W9. Several ablations remove multiple mechanisms simultaneously.
Our ablation experiments include both individual and combined removals. No Diag. and No Sem. separately examine the effects of training diagnostics and domain knowledge, while No S.+D. removes both. Similarly, No Rel., No Corr., and No Cal. each remove one scoring component. In contrast, No Exec. bypasses the entire scoring program, removing composition and its associated calibration together. We have clarified what each ablation removes in Section 4.3 and Table 3 to distinguish the effects of individual components from those of removing multiple components together.
W10. Task-level programs and the scope of sample-specific adaptation.
We clarify that not all components in CriScore remain frozen across observations. After training, the scoring program, including the selected evidence sources, scoring relations, and numerical parameters, remains frozen for the same task. However, different videos produce different evidence predictions and base scores. Their normalized evidence also varies according to their relative rankings within the evaluation population. When residual correction is enabled, the correction can also differ across videos. Thus, the same scoring program processes different evidence from individual videos to produce their respective quality scores. This design aims to maintain consistent scoring criteria across videos within the same task while allowing their individual quality differences to be reflected in the assessment. Extending CriScore to support sample-specific adaptation of scoring criteria represents a promising direction for our future work.
W11. Selected programs for all four datasets.
To provide a clearer view of the scoring programs selected by CriScore, we summarize their evaluation concepts, evidence sources, composition operations, and calibration settings across all four datasets. The results are presented in the table below. We have also refined Appendix C.6 by incorporating this table to provide a clearer and more systematic presentation of the selected programs.
| Dataset | Evaluation Concepts | Evidence and Composition | Calibration / Correction |
|---|---|---|---|
| MTL-AQA | Action quality; visual cues | Base/Swin: 0.75/0.25; rank fusion | Quantile mapping; no correction |
| TASD-2 | Execution; coordination | Base/I3D/VideoMAE: 1:1:1; rank fusion | Quantile mapping; no correction |
| FineDiving | Difficulty; compactness; splash | Difficulty/bbox/splash: priors 0.60/0.20/0.20; base blending | Monotone calibration; via inner cross-validation |
| UI-PRMD | Exercise type; phases | Type/phase: equal weights; Base/Auxiliary: 0.50/0.50 | Affine mapping; bounded residual, , |
Thank you for your careful reading and constructive feedback. In the revised manuscript, we have clarified the methodology and design of CriScore, strengthened the experimental comparisons through additional baselines and sensitivity analyses, and improved the presentation of the selected scoring programs and ablation studies. The corresponding changes are highlighted in green in the revised manuscript. The specific responses are detailed below:
W1.The distinction between structured fusion and semantic scoring-rule learning.
We would like to clarify what CriScore actually learns. The task description specifies the aspects that matter for evaluating an action. CriScore then associates these aspects with available evidence sources, determines which sources to use and how to combine them, and selects the scoring program based on overall-score supervision. Therefore, what is learned is not simply a set of fusion weights, but also how the available evidence is selected and organized into an executable scoring structure.
Our existing experiments support this distinction. Table 2 shows that removing domain knowledge, shuffling evidence bindings, or replacing structured reasoning with weighted fusion reduces performance. Table 4 further demonstrates CriScore's advantages over optimized weighted fusion and non-language program selection under the same evidence and calibration settings. We have also refined Section 2.2 and Definition 1 to clarify the AQA prediction objective, the role of task-level criterion structures, and the meaning of evidence bindings. Further validating these associations through concept-level supervision or expert assessment represents a promising direction for our future work.
W2.Motivation for the LLM and its contribution to candidate construction.
An LLM is not a prerequisite for constructing scoring programs, as candidate programs can also be generated and selected through non-language search within the predefined program space. In CriScore, the LLM serves as a task-informed candidate proposer, using task descriptions and training diagnostics to construct candidate scoring structures. Score supervision then guides program selection and numerical calibration. Table 4 compares CriScore with uniform fusion, optimized weighted fusion, and non-language program selection using the same evidence predictions and calibration settings. CriScore outperforms all alternatives across four datasets, with an average SRCC gain of 0.613 percentage points over non-language program selection under the same search budget.
W3.The concept-evidence binding is only weakly grounded.
Not all AQA datasets provide fine-grained quality sub-scores. While TASD-2 includes execution and synchronization scores, the other datasets mainly provide overall quality scores and other annotations rather than separate scores for specific evaluation aspects. We therefore train all evidence predictors using overall-quality scores without requiring concept-specific annotations. Although they share the same training target, different backbones and input features allow them to capture different information about action quality.
In CriScore, task descriptions, source descriptions, and training diagnostics guide the association between evaluation concepts and evidence sources. Our source-removal experiments show that the selected sources help predict overall quality, rather than directly confirming that each source measures its assigned concept. Further validating these associations through concept-level annotations or expert assessments represents a promising direction for our future work.