acceptodds
Withdrawn from ICLR 2027

Learning Executable Scoring Criteria for Action Quality Assessment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action quality assessment (AQA) relies on scoring criteria that determine which evidence matters and how it contributes to a quality judgment. Existing methods typically encode these criteria implicitly in learned mappings or specify them explicitly through hand-designed evaluation structures. The latter fix how evidence is organized and combined, preventing score supervision from revising these structural choices. Yet translating domain knowledge into an effective scoring structure requires accounting for the reliability and complementarity of model-derived evidence, which must be established from data. We therefore formulate scoring criteria themselves as learnable structures: domain knowledge guides the construction of candidate criteria, while score supervision selects among these alternatives and calibrates their numerical realization. To this end, we propose Learning Executable Scoring Criteria for Action Quality Assessment (CriScore), a neuro-symbolic framework that represents learned scoring criteria as executable task-level programs. Domain knowledge guides the induction of evaluation concepts, evidence bindings, and scoring relations within a predefined evidence and operation space. Overall-score supervision trains the evidence predictors and supports program selection and numerical calibration, adapting the induced criteria to observed assessment performance. The resulting program is reused across new observations through neuro-symbolic quality reasoning, producing quality scores together with execution traces. Experiments across four AQA datasets demonstrate the effectiveness of CriScore, supporting the value of learnable scoring criteria for action quality assessment. The code and data are available at https://anonymous.4open.science/r/CriScore/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Action Quality Assessment, Neuro-Symbolic Reasoning, Adaptive Scoring Rules

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yBoqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a neuro-symbolic framework for Action Quality Assessment (AQA), which is named CriScore. The core idea of CriScore is to treat scoring criteria as learnable executable structures. The paper evaluates CriScore on MTL-AQA, TASD-2, FineDiving, and UI-PRMD, and experiments show that CriScore achieves the strongest reported results. I find the general idea of making the scoring structure explicit and trainable interesting. However, I have several concerns about the central methodological claim.

Critical strengths
  • The paper raises an interesting question about the scoring structure in AQA.
  • The experimental analysis is extensive.
  • The proposed method achieves the best results.
Critical weaknesses
  • My main conceptual concern is the interpretation of what is actually being learned. The claimed learnable scoring rules appear closer to structured fusion over learned representations than to semantic rule learning. In detail, the proposed method mainly consists of source selection, weighted rank-based composition, blending with a base scorer, monotonic score mapping, and optional bounded residual correction. These operations are interpretable and executable, but they are closer to a constrained structured ensemble than to learning semantic scoring rules such as conditional expert criteria over observable action properties. I therefore think the paper should either substantially weaken the learnable scoring rule claim or provide stronger evidence that the learned bindings and concepts have semantic validity, e.g., through concept-level annotations, official sub-scores, concept probing, or expert validation.

  • The necessity of using an LLM for structure learning is not sufficiently motivated. The paper motivates CriScore by arguing that fixed scoring structures are limiting and that scoring criteria should instead be learnable. I agree with this motivation. However, this does not by itself justify the use of an LLM. At present, the role of the LLM seems better characterized as a semantic search prior / candidate proposer than as a necessary component for learning scoring criteria.

  • The concept-evidence binding is only weakly grounded. The paper places substantial emphasis on binding evaluation concepts to evidence sources. However, the empirical evidence for these bindings is limited. For example, showing that removing VideoMAE evidence reduces final SRCC demonstrates that VideoMAE is useful for quality prediction; it does not demonstrate that VideoMAE specifically represents the evaluation concept assigned to it. The TASD-2 replay and source-deletion experiments are informative about predictive importance, but not about semantic grounding. This is an important distinction because the interpretability claim of the framework depends heavily on the meaning of these bindings.

  • Table 1 does not appear to provide a fully feature-controlled comparison. The broad comparison against 33 baselines is impressive at first glance, but the methods do not appear to operate under a common information budget. I strongly recommend that Table 1 explicitly report the input modality, backbone, feature source, metadata, and auxiliary information available to each method. More importantly, the paper should include a same-feature controlled comparison in which competing fusion/gating/structure-learning methods receive exactly the same evidence bank as CriScore.

  • The current analysis does not clearly distinguish CriScore from gating or learned fusion. CriScore mainly selects evidence sources, assigns weights, combines them with a base scorer, and optionally applies a residual correction. This is similar to a structured gating or mixture-of-experts mechanism. The paper should compare against a strong learned gating/fusion baseline using the same evidence sources and calibration modules. Otherwise, it is unclear whether the gains come from genuinely neuro-symbolic reasoning or simply from better source selection and fusion.

  • Important search and calibration hyperparameters lack sensitivity analysis. The method depends heavily on a number of discrete search and calibration choices, but most are fixed without systematic sensitivity analysis.

  • The program design itself is under-justified. Components such as base-score blending, temperature scaling, and residual correction appear manually chosen, and it is unclear why they are necessary for learning scoring criteria rather than simply increasing model flexibility.

  • Minors

(1) The paper would benefit from distinguishing more carefully between semantic criteria, evidence-source selection, and numerical calibration.

(2) Several ablations remove multiple mechanisms simultaneously. For example, bypassing execution also removes composition and calibration, making the causal attribution of the resulting performance drop less precise.

(3) The selected program is task-level and frozen across observations, limiting adaptation to sample-specific quality cues.

(4) The paper should report the actual selected programs for all four datasets in a compact table. This would make the claimed interpretability easier to assess.

Questions for authors

See above.

I am currently leaning toward acceptance because the paper presents an interesting and potentially useful framework, and the empirical results are strong. However, my positive assessment depends on the authors being able to clearly address the main conceptual concerns above, especially the distinction between semantic rule learning and structured fusion/calibration.

I would also like the authors to more carefully position the contribution. If these concerns cannot be convincingly clarified during rebuttal, I would likely lower my score.

4 comments

Thank you for your careful reading and constructive feedback. In the revised manuscript, we have clarified the methodology and design of CriScore, strengthened the experimental comparisons through additional baselines and sensitivity analyses, and improved the presentation of the selected scoring programs and ablation studies. The corresponding changes are highlighted in green in the revised manuscript. The specific responses are detailed below:

W1.The distinction between structured fusion and semantic scoring-rule learning.

We would like to clarify what CriScore actually learns. The task description specifies the aspects that matter for evaluating an action. CriScore then associates these aspects with available evidence sources, determines which sources to use and how to combine them, and selects the scoring program based on overall-score supervision. Therefore, what is learned is not simply a set of fusion weights, but also how the available evidence is selected and organized into an executable scoring structure.

Our existing experiments support this distinction. Table 2 shows that removing domain knowledge, shuffling evidence bindings, or replacing structured reasoning with weighted fusion reduces performance. Table 4 further demonstrates CriScore's advantages over optimized weighted fusion and non-language program selection under the same evidence and calibration settings. We have also refined Section 2.2 and Definition 1 to clarify the AQA prediction objective, the role of task-level criterion structures, and the meaning of evidence bindings. Further validating these associations through concept-level supervision or expert assessment represents a promising direction for our future work.

W2.Motivation for the LLM and its contribution to candidate construction.

An LLM is not a prerequisite for constructing scoring programs, as candidate programs can also be generated and selected through non-language search within the predefined program space. In CriScore, the LLM serves as a task-informed candidate proposer, using task descriptions and training diagnostics to construct candidate scoring structures. Score supervision then guides program selection and numerical calibration. Table 4 compares CriScore with uniform fusion, optimized weighted fusion, and non-language program selection using the same evidence predictions and calibration settings. CriScore outperforms all alternatives across four datasets, with an average SRCC gain of 0.613 percentage points over non-language program selection under the same search budget.

W3.The concept-evidence binding is only weakly grounded.

Not all AQA datasets provide fine-grained quality sub-scores. While TASD-2 includes execution and synchronization scores, the other datasets mainly provide overall quality scores and other annotations rather than separate scores for specific evaluation aspects. We therefore train all evidence predictors using overall-quality scores without requiring concept-specific annotations. Although they share the same training target, different backbones and input features allow them to capture different information about action quality.

In CriScore, task descriptions, source descriptions, and training diagnostics guide the association between evaluation concepts and evidence sources. Our source-removal experiments show that the selected sources help predict overall quality, rather than directly confirming that each source measures its assigned concept. Further validating these associations through concept-level annotations or expert assessments represents a promising direction for our future work.

W4. Feature-controlled comparison.

a. Baseline information. To clarify the information used by the baselines in Table 1, we summarize their input features, model architectures, and auxiliary information below. A detailed comparison of the evaluated baselines is provided in Table 3 of the revised manuscript.

  • Feature Regression methods: Ridge and SVR use cached global sequence statistics with L2-regularized linear regression or RBF-kernel regression, without neural encoders.
  • General Temporal methods: Use cached temporal features, including 32×6032\times60 descriptors for FineDiving and 32×7632\times76 joint sequences for UI-PRMD, with Transformer, Inception, TCN, and state-space architectures. Action indicators and difficulty metadata are used where applicable.
  • Graph Motion methods: Use 2D/3D skeletal trajectories and kinetic graphs with spatial-temporal GCNs, including topology-refinement and hierarchical variants. Auxiliary inputs include joint connectivity and action indicators.
  • AQA-Specific methods: Use video, optical flow, pose heatmaps, splash masks, and multimodal features with specialized AQA architectures, including contrastive regression, hierarchical parsing, and graph-based models. Reference scores, action-stage annotations, and difficulty metadata are used according to the respective protocols.

b.Same-evidence controlled comparison. Table 4 compares different fusion and program-selection methods using the same evidence predictions, validation folds, and score-mapping setup. All search-based methods are allowed to evaluate up to 64 candidates. This ensures that the methods are compared using the same evidence and under the same search budget. Under this controlled setting, CriScore achieves the best SRCC across all four datasets, outperforming non-language program selection by an average of 0.613 percentage points.

W5.Comparisons with learned fusion and gating methods.

To provide a stronger comparison, we have added a learned gating baseline that adjusts the weights of different evidence sources for each video. This baseline uses the same evidence predictions as CriScore, with an MLP to generate the weights and the same score-mapping setup. All methods use the same evidence predictions and score-mapping setup, allowing a direct comparison of their scoring strategies. We have also updated Table 4 in the revised manuscript to include the new baseline and revised the corresponding discussion in Appendix C.2 accordingly.

MethodMTL-AQATASD-2FineDivingUI-PRMD
Learned Gating Fusion85.2285.7480.7873.99
CriScore86.1286.9681.3474.67

W6. Sensitivity analysis of search and calibration hyperparameters.

We have conducted additional experiments to examine the sensitivity of CriScore to the candidate evaluation budget and residual-correction strength. The results are shown in below, and the corresponding experiments and analysis have been added to Appendix C.8 of the revised manuscript.

a. Candidate search budget. The table below reports SRCC (%) at candidate evaluation budgets of 24, 32, 48, and 64, with temperature fixed at T = 1.0 and all other settings unchanged. These results show that increasing the search budget improves prediction performance, with diminishing returns at larger budgets.

Candidate budget KMTL-AQATASD-2FineDivingUI-PRMD
2485.3486.0280.4273.81
3285.7886.4880.9074.24
4886.0586.8281.1874.54
64 (default)86.1286.9681.3474.67

b. Residual-calibration strength. The table below reports the sensitivity of residual calibration on UI-PRMD using training-side grouped cross-validation with three seeds and five folds. We vary the correction strength while keeping the fitted residual predictors, score mappings, and clipping bounds unchanged. The clipping bound is fixed at half the training-side residual standard deviation. These results demonstrate the importance of controlling residual-correction strength rather than applying the full predicted adjustment.

Residual correction strengthUI-PRMD SRCC (%)
074.023
0.2574.340
0.50 (default)74.667
0.7574.417
1.0073.907

c. Temperature and blending-strength sensitivity. We examine the effects of temperature TT and blending strength β\beta using the FineDiving training-side validation results. Each setting is evaluated across three seeds and five outer folds, with the selected evidence sources and other scoring operations unchanged. The table reports mean inner-validation SRCC (%). Results show that SRCC reaches its highest value at β=0.75\beta=0.75 for all tested temperatures, with the best result obtained at T=0.50T=0.50 and β=0.75\beta=0.75. Performance varies by at most 0.12 percentage points across temperatures for a fixed blending strength, indicating limited sensitivity to TT within the evaluated range.

T \ β0.250.500.751.00
0.580.2780.5880.7280.68
1.080.2280.5280.6780.66
2.080.1980.4680.6080.59

W7. Motivation and contribution of the scoring-program operators.

The operators in CriScore serve different roles. Base-score blending β\beta controls how much additional evidence contributes to the base prediction. Temperature scaling TT adjusts the relative weights of different evidence sources. Residual correction δ\delta provides an optional and bounded adjustment to the final prediction. These settings are selected or fitted using training data and remain fixed during evaluation.

To further evaluate the contribution of base-score blending, we set β=1.0\beta=1.0 to remove the base prediction from the composition stage while preserving the remaining scoring operations. The results are shown below.

DatasetDefault SRCC (%)w/o Base (β=1.0\beta=1.0)Retained Evidence
MTL-AQA86.1281.43Swin only
TASD-286.9683.19I3D, VideoMAE
FineDiving81.3478.65Difficulty, bounding-box compactness, splash-presence evidence
UI-PRMD74.6770.22Action-type and phase evidence

The results show that removing the base contribution consistently reduces SRCC across all four datasets. This intervention preserves the remaining scoring operations, whereas No Exec. bypasses the entire scoring program. Table 2 further shows that replacing the selected source weights with equal weights or disabling residual correction also leads to performance degradation. Together, these results demonstrate the contributions of base-score blending, source weighting, and bounded residual correction to CriScore's performance.

W8. Distinguishing semantic criteria, source selection, and numerical fitting.

  • Semantic criteria define what aspects of a performance matter from domain knowledge (e.g., body posture, entry angle).
  • Source selection (evidence binding) determines which available evidence sources are associated with these aspects.
  • Numerical fitting resolves how these sources are weighted, aggregated, and calibrated under overall-score supervision. We have added a brief explanation in Section 3.1 to clarify these distinctions.

W9. Several ablations remove multiple mechanisms simultaneously.

Our ablation experiments include both individual and combined removals. No Diag. and No Sem. separately examine the effects of training diagnostics and domain knowledge, while No S.+D. removes both. Similarly, No Rel., No Corr., and No Cal. each remove one scoring component. In contrast, No Exec. bypasses the entire scoring program, removing composition and its associated calibration together. We have clarified what each ablation removes in Section 4.3 and Table 3 to distinguish the effects of individual components from those of removing multiple components together.

W10. Task-level programs and the scope of sample-specific adaptation.

We clarify that not all components in CriScore remain frozen across observations. After training, the scoring program, including the selected evidence sources, scoring relations, and numerical parameters, remains frozen for the same task. However, different videos produce different evidence predictions and base scores. Their normalized evidence also varies according to their relative rankings within the evaluation population. When residual correction is enabled, the correction can also differ across videos. Thus, the same scoring program processes different evidence from individual videos to produce their respective quality scores. This design aims to maintain consistent scoring criteria across videos within the same task while allowing their individual quality differences to be reflected in the assessment. Extending CriScore to support sample-specific adaptation of scoring criteria represents a promising direction for our future work.

W11. Selected programs for all four datasets.

To provide a clearer view of the scoring programs selected by CriScore, we summarize their evaluation concepts, evidence sources, composition operations, and calibration settings across all four datasets. The results are presented in the table below. We have also refined Appendix C.6 by incorporating this table to provide a clearer and more systematic presentation of the selected programs.

DatasetEvaluation ConceptsEvidence and CompositionCalibration / Correction
MTL-AQAAction quality; visual cuesBase/Swin: 0.75/0.25; rank fusionQuantile mapping; no correction
TASD-2Execution; coordinationBase/I3D/VideoMAE: 1:1:1; rank fusionQuantile mapping; no correction
FineDivingDifficulty; compactness; splashDifficulty/bbox/splash: priors 0.60/0.20/0.20; base blendingMonotone calibration; β\beta via inner cross-validation
UI-PRMDExercise type; phasesType/phase: equal weights; Base/Auxiliary: 0.50/0.50Affine mapping; bounded residual, η=0.50\eta=0.50, B=0.50σres,trB=0.50\sigma_{\mathrm{res,tr}}
Reviewer_uKmBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It is hard to provide a summary of this paper, as it seems to only present an high level idea of the pipeline. I will try to do my best, regardless.

The paper focuses around action quality assessment, that is, quantifying how well an observed action is performed according to domain knowledge. This involves perception, providing evidence for the different actions and then a scoring criteria that aims to connect this evidence with a quantitative score. The author propose CriScore, a neuro-symbolic framework that makes this scoring criteria a learnable structure, i.e. a learnable program. This works by providing an LLM prior knowledge (probably via a prompt?), which then returns some candidate program. A supervision, that is not clear what it is and where it comes from, helps removing program candidates from the set. Then this program is evaluated to get the final answer.

Critical strengths

I was not able to understand the paper, even after reading it multiple times. As such I could not find strengths.

Critical weaknesses

By far, the main weakness of the paper is clarity.

I find it difficult to understand how CriScore differs from existing methods for action quality assessment and, more generally, what the main contribution of the proposed approach is.

The introduction is repetitive. For example, the fact that "CriScore uses domain knowledge to construct candidate criteria" and that "the final scoring function is an executable program" is mentioned several times, without giving much more explanation. In particular, it is not clear how these ideas are actually implemented. This is the most important part for understanding what the paper really does.

The abstract also raises several questions that are not sufficiently clarified in the paper. For example, it is not clear what the authors mean by "calibrating the numerical realization of the criteria". As I understand it, CriScore uses some form of supervision (although it is unclear what this supervision is and where it comes from, and especially why is it calibrated) only for the final score, i.e. the output of the program. Does this mean that CriScore proposes a set of programs and then chooses the closest to the supervised score?

More generally, the use of the term "calibrated" is unclear. What exactly is being calibrated? Who provides the calibrated parameters? Where do the calibrated scores come from? I am not convinced that the proposed score is calibrated in the usual sense of calibration (e.g., local or global calibration). The paper should clearly define what it means by calibration and explain how it is measured.

The paper also frequently uses terms such as "quality evidence" and "execution traces" without clearly defining them. For example, what exactly is an execution trace, and how is it evaluated?

The lack of clarity persists throughout all the paper.


The second most important problem is formulation and methodology. Here I write some notes.

The preliminaries are difficult to follow. The section introduces concepts before they have been properly motivated or defined for the reader. Some of the definitions also seem unnecessarily formal given the mathematical framework they describe. The authors should explain why each definition is needed and, more importantly, where it is used later in the methodology. For instance, the equations are presented without enough intuition about what the different quantities represent or how they relate to the actual action-quality assessment problem.

Similarly, the construction of the scoring criteria is not sufficiently explained. It is unclear how the criteria are generated and how the underlying concepts are grounded. It is also unclear what the scoring function is built from. Is there a predefined vocabulary of concepts, or are the concepts generated dynamically? In this context, what exactly is a "criterion field"?

The role of the LLM is unclear. The paper seems to use an LLM to provide domain knowledge and to propose the concepts used by CriScore, but this is not made sufficiently clear in either the abstract or the main text. The specific LLM being used is not even clearly identified in the main paper.

There are also several important methodological details that are missing. For example, the paper does not clearly explain what the "H_{diag} training diagnostics" are.

For "quality evidence learning", what exactly is the prediction loss? What is the regularization term?

The term "rule compilation" is also unclear. What is being compiled, and into what representation? Is this related to knowledge compilation, for example into a circuit, or does "program" refer to a different executable representation like ProLog? The paper should define this clearly.

Overall, despite the large amount of formalism, I still find it difficult to understand what was actually implemented. The paper contains a lot of mathematical definitions, but not enough concrete explanation of the actual procedure.


Several important parts of the experimental evaluation are not sufficiently explained.

For example, the metrics SRCC, NMSE, MAE, and RMSE are not defined in the main paper and are only explained in the appendix. Although these are standard metrics, they should still be briefly defined when they are first introduced, especially since the reader needs to understand what each metric measures in this particular task.

The paper also introduces several abbreviations and experimental conditions that are difficult to understand:

No Diag., No Sem., No LLM, No S.+D., Equal W., Shuffle B., Fusion, No Rel., No Corr., No Constr., No Fall., Alt. Comp., No Cal., No Exec.

These settings should be clearly defined.

The notions of calibration and robustness also need to be defined more clearly. It is unclear how these quantities are measured and how the results should be interpreted. For example, Figure 2(b) is difficult to understand without a clearer explanation of what is being plotted and what conclusion the authors want the reader to take from it.

The baselines are another concern. They are described in Appendix A.2 and are not discussed in the main paper. In particular, the authors should explain what each baseline does, what its input is, and how it processes the information available to CriScore. Without this information, it is difficult to assess whether the comparisons are fair and meaningful.

The research questions are also somewhat loosely formulated. They could either be rewritten to be more precise and clearly connected to the experiments, or removed in favor of a clearer description of the hypotheses and goals of the experiments.

These are only some of the inconsistencies and issues I found while reading the manuscript. More generally, I found it difficult to reconstruct the full methodology and experimental pipeline from the paper. There is a lot of formal notation, but relatively little concrete explanation of what the authors actually do.


Lastly related works.

The discussion of Neuro-Symbolic AI, seems incomplete. In particular, work on Inductive Logic Programming seems highly relevant to the proposed approach but are not discussed. As well as classical Neuro-Symbolic methods such as DeepProbLog and Logic Tensor Networks. Usually, review such as De Raedt et al., "From Statistical Relational to Neuro-Symbolic Artificial Intelligence," IJCAI 2021, and d’Avila Garcez et al., "Neural-Symbolic Learning and Reasoning: A Survey and Interpretation," in *Neuro-Symbolic Artificial Intelligence: The State of the Art, 2022 should be mentioned.

Questions for authors

Here are my questions.

  1. What do you mean by a "calibrated score"? What notion of calibration are you referring to? The paper does not make this clear. Please define calibration precisely.

  2. To what extent does CriScore differ from existing methods for action-quality assessment? Please clearly identify the main differences from existing approaches and explain what is novel about CriScore.

  3. Could you explain the proposed approach in simpler terms? In particular, could you describe the complete pipeline from the input action to the final score, including how the concepts, criteria, programs, and supervision are used? Ground this to the technology you used (e.g. what language you chose for the program?)

  4. What supervision is used to construct CriScore? Is supervision provided only for the final score, or is there also supervision for the candidate concepts and criteria? If only the final score is supervised, how does the method select or calibrate the candidate programs?

  5. How are the concepts and scoring criteria generated and grounded? Is there a predefined vocabulary, or are they generated by the LLM? How is the LLM used?

5 comments

Thank you for your careful reading and constructive feedback. In the revised manuscript, we have clarified the motivation and methodology of CriScore, provided further details on scoring criteria construction, program calibration, and execution, and improved the descriptions of experimental settings, baseline comparisons, and evaluation protocols. We have also expanded the related work on neuro-symbolic learning and reasoning. The corresponding changes are highlighted in blue in the revised manuscript. The specific responses are detailed below:

W1&Q2. Main contribution and novelty.

Action quality assessment (AQA) aims to evaluate how well an action is performed. For example, judging a diving performance involves considering multiple aspects, such as take-off, aerial movements, and entry. A key challenge is determining which evidence from available models should be used and how to combine it into an overall score.

Our motivation is that domain knowledge tells us what aspects of an action matter, but does not tell us which model predictions are most useful or how they should be combined. Existing deep learning methods mainly learn to predict quality scores directly, leaving the scoring process largely implicit. Rule-based methods make this process explicit but usually rely on predefined scoring structures.

The main contribution of CriScore is to make scoring criteria learnable rather than predefined. Instead of simply combining model predictions using fixed rules or learned weights, CriScore constructs different candidate scoring structures based on domain knowledge and training diagnostics. These structures specify which evidence sources to use, how they are connected to evaluation concepts, and how their predictions are combined. Overall-score supervision is then used to select and calibrate a suitable scoring program. This allows CriScore to learn the organization and combination of evidence while keeping the resulting scoring process explicit and executable.

Our experimental analyses further support the contribution of learned scoring criteria, as demonstrated by the program comparisons in Figure 2b, component ablations in Table 2, and controlled comparisons in Appendix C.2.

W12&Q3. Overall methodology and pipeline.

CriScore is designed for action quality assessment (AQA), where the goal is to evaluate how well an action is performed based on video observations. It uses evidence from different models to produce an overall quality score through a task-specific scoring program. As illustrated in Figure 1, CriScore follows a task-level preparation and inference pipeline. During preparation, CriScore first constructs candidate scoring criteria using domain evaluation knowledge and training diagnostics. It then trains the evidence predictors using overall quality scores to provide the quality evidence used by these criteria. These predictions and the reference scores are then used to calibrate and select a scoring program. During inference, the frozen assessor is applied to new action observations to produce quality scores and execution traces.

Specifically, an LLM uses domain knowledge and training diagnostics to propose candidate scoring criteria. These criteria specify which evidence sources to use and how their predictions should be combined. The candidates are compiled into executable scoring programs within a predefined evidence and operation space. The programs are represented as structured JSON configurations and executed using Python functions. SciPy and NumPy are used for numerical calculations.

The LLM only proposes candidate programs and does not make the final selection. Figure 3 provides an instance-level example from TASD-2, illustrating how the selected program converts predictions from the base scorer, I3D, and VideoMAE into normalized ranks and computes their average. The resulting rank of 0.47826 is then mapped to a final quality score of 56.70. The calculation steps are recorded in an execution trace.

To improve the clarity of the overall workflow, we have refined the Method Overview in Section 3 and clarified the roles of its three stages.

W5&Q5. Concept and criteria construction.

In CriScore, evaluation concepts are identified from the task description rather than selected from a predefined vocabulary. The available evidence sources and scoring operations are specified in advance through the task specification (V). We use DeepSeek-V4-Pro, as specified in Section 4.1, to identify relevant concepts, associate them with available evidence sources, and specify how the evidence contributes to the final score. These components form candidate scoring criteria, which are compiled into executable programs within the predefined space. Criterion fields refer to specific elements of a scoring structure that are assigned fixed values during induction. These values restrict which candidate scoring structures can be generated. Eq.(5) formalizes how these fields constrain the candidate space.

W6&Q5. Role of the language model.

In CriScore, the language model serves as an auxiliary tool for knowledge-guided criteria induction rather than a quality predictor. It uses the task description and training diagnostics to identify evaluation concepts, establish evidence bindings, and induce scoring relations within the predefined evidence and operation space. Its output is a structured candidate representation that is subsequently compiled into executable scoring programs. The LLM does not directly predict quality scores or execute the final scoring programs. DeepSeek-V4-Pro is used as the default model for criteria induction, as specified in Section 4.1.

W2&Q1.Supervision and Calibration.

In CriScore, calibration refers to fitting the numerical parameters of an induced scoring program using overall quality scores from the training data. It does not refer to statistical calibration of predicted probabilities. Score supervision serves two roles. It trains the evidence predictors and fits the numerical parameters left unspecified by the induced criteria within the candidate space Aξ\mathcal{A}_{\xi}. Program selection then identifies the candidate scoring structure to be retained. Figure 2a examines the contribution of supervised calibration by comparing performance with and without this process.

Q4.Supervision for Criteria and Program Selection.

CriScore uses overall quality scores as its supervision signal and does not require additional concept-specific or criterion-level annotations. Evaluation concepts and candidate criteria are constructed from domain evaluation knowledge and training diagnostics within the predefined evidence and operation space. Score supervision serves two roles. It trains the evidence predictors and supports the calibration and selection of candidate scoring programs. Specifically, the evidence predictors are trained to estimate overall quality scores. Cross-fitted predictions are then used to compare candidate programs, while reference scores are used to determine their numerical parameters. The selected program and its parameters are fixed before inference. Further details are provided in Sections 2.1 and 3.2.

W3. Quality Evidence and Execution Traces.

In CriScore, quality evidence refers to the numerical quality estimates produced by the available evidence predictors, which serve as inputs to the scoring program. An execution trace records how these estimates are processed and combined by the selected program to produce the final score. Figure 3 illustrates the execution process for a TASD-2 dive, showing how evidence from different sources contributes to the final prediction. Figure 8 further examines the predictive contributions of individual evidence sources through source-removal experiments. We have clarified the definitions of quality evidence and execution traces in the revised manuscript.

W7. Training Diagnostics and Predictor Learning.

The training diagnostics HdiagH_{\mathrm{diag}} summarize the predictive performance of available evidence sources and the related training information. They contain two parts. The first consists of three statistics for each evidence source: (1) fold-wise Spearman correlations with the overall quality scores, (2) the mean correlation across folds, and (3) the minimum fold-wise correlation gain over the base scorer. The gain is the smallest difference between the source's correlation and the base scorer's correlation across validation folds. The second part includes the base scorer's fold-wise correlations, training-set size, number of folds, and selected source-head configurations. All statistics are computed using out-of-fold predictions from the training partition. These diagnostics are provided to the LLM during criteria induction.

For quality evidence learning, each evidence predictor is trained to estimate the overall quality score. The prediction loss is the squared error between the predicted and ground-truth scores. The regularization term Ωj(θj)\Omega_j(\theta_j) is an L2 penalty on the regression coefficients, excluding the intercept. The regularization strength and temporal feature representation are selected using training-side out-of-fold validation. The resulting predictors provide quality evidence for the induced scoring criteria without requiring concept-specific labels.

We have clarified the composition of HdiagH_{\mathrm{diag}}, the prediction loss, and the regularization term in Sections 3.1 and 3.2.

W4. Preliminary Definitions and Mathematical Formulation.

The preliminary definitions describe how domain evaluation requirements are connected to the evidence used for quality assessment. Specifically, Definition 1 explains how evaluation concepts are associated with available evidence sources. In AQA, these sources provide quality estimates from different observations or modalities. This association is used in Section 3.1 to construct evidence bindings. Definition 2 further explains how the selected evidence is organized and combined to produce an overall quality score. Its three components, C\mathcal{C}, B\mathcal{B}, and R\mathcal{R}, represent what aspects are evaluated, which evidence sources are used, and how the evidence is combined, respectively. These components form the basis of relation induction and program compilation in Section 3.1.

W6.Program Compilation and Representation.

In CriScore, rule compilation converts the induced scoring criteria into executable scoring programs. The compiler determines which evidence sources are used, how their predictions are combined, and which scoring operations are applied. It also checks that the operations and their numerical parameters satisfy the predefined constraints. The program is a structured, executable representation of the scoring procedure, rather than a logical circuit or a Prolog program. During inference, the executor retrieves the required quality evidence and performs the specified operations to produce the final score. We have clarified the compilation process and the resulting program representation in Section 3.1.

Reviewer_uCfQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addressed the action quality assessment (AQA) problem by introducing CriScore, a neuro-symbolic framework for learning executable scoring criteria from domain evaluation knowledge and score supervision. The paper provide a detailed solution, validate it on four public datasets, compare it with existing methods, and analyze the experimental results.

Critical strengths

The paper provide a detailed solution, validate it on four public datasets, compare it with existing methods, and analyze the experimental results.

Critical weaknesses

1)Figure 1 is the Overview of CriScore. The flowchart is difficult to align with the algorithm description, and the excessive details in the figure hinder understanding of the overall algorithmic framework. The variables such as jj and cc shown in the figure are not defined in the main text, while the variables defined in the main text are not presented in the figure. It is not recommended to rely entirely on large language models to generate flowcharts. 2)The overall presentation of the paper and the description of the algorithm are unclear, making it difficult for readers to quickly grasp the intended meaning. For example, the following sentence defines sis_i, but provides insufficient contextual explanation of its purpose and how it is used after the definition. A base scorer f_θ is trained under the same supervision and produces s_i = f_θ(x_i).

3)The inconsistent definitions of equations and variables also make the paper difficult to understand. For example, in Equation (6), ll is defined as the prediction loss, whereas in Equation (7), it is defined as fold ℓ\ell. There are also many other instances of inconsistent and confusing notation throughout the paper. 4)The tables presenting experimental results in the experimental section should include citations to the original references of the compared methods. 4)Some of the methods used for comparison in this paper, such as CoRe (Yu et al., 2021), are relatively outdated. Many improved methods with competitive results have been proposed in recent years. I recommend that the authors conduct a more comprehensive literature review and reevaluate the performance of their proposed method against more recent state-of-the-art approaches.

Questions for authors

See weakness

1 comment

Thanks for your careful reading and constructive feedback. In the revised manuscript, we have revised Figure 1 to improve its clarity and organization, added citations and publication years for the compared methods, and further clarified the coverage of recent baseline approaches. In particular, our experimental comparisons already include several recent methods published in 2025 and 2026. The corresponding changes are highlighted in red in the revised manuscript. The specific responses are detailed below:

W1.Clarity of Figure 1 and its correspondence with the method.

Thanks for the comments. We have refined Figure 1 to simplify the overall presentation and highlight the key components and workflow of CriScore. Regarding the notations jj and cc, their definitions are already provided in Section 2. Specifically, cc denotes an evaluation concept, while jj denotes an evidence source. We have also reviewed the figure to ensure consistency between the notations used in the figure and those defined in the main text.

W2. Clarity of the algorithm description and the role of the base scorer.

Thanks for the comments. Here, sis_i denotes the overall quality score predicted by the base scorer for observation xix_i. We have updated the definition and the explanation in the revised manuscript.

W3. Consistency and clarity of mathematical notation.

Thanks for sharp catch. The prediction loss in Eq.(6) should be represented by Lpred,j\mathcal{L}_{pred,j}, while ℓ\ell denotes the fold index in Eq.(7). We have updated the notation in the revised manuscript.

W4. References for baselines.

Thanks for the suggestions. We have revised Table 1 to include the corresponding citations and publication years for all compared methods. This makes the origins of the baseline methods explicit and improves the clarity and traceability of the experimental comparisons.

W5. Coverage of recent methods.

Thanks for the comments. We agree that comparisons with recent state-of-the-art methods are important for evaluating the effectiveness of our proposed approach. We would like to clarify that our experimental evaluation already includes a comprehensive set of all baseline methods, rather than relying primarily on earlier approaches such as CoRe (Yu et al., 2021). These methods cover four categories: feature-based regression, general temporal models, graph-based motion models, and AQA-specific methods. In particular, our comparison includes a substantial number of recently proposed methods from 2025 and 2026, such as MambaTS (2026), Uni-FineParser (2025), PHI (2025), FineCausal (2025), UIL-AQA (2026), CaFlow (2026), MCMoE (2026), ASGTN (2025), and BriMA (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.