acceptodds
Under review as a conference paper at ICLR 2027

Learning from What It Learned: Decoding Post-Convergence States for Unified Action Quality Assessment

Abstract

Unified action quality assessment (AQA) trains a single model across heterogeneous sports, enabling cross-sport representation sharing while converging to a shared solution shaped by heterogeneous sport-specific preferences. Post-convergence analysis reveals limited directional agreement among sport-specific gradients and substantial variation in dominant local curvature around the shared solution, while the raw-to-EMA trajectory contains an empirically favorable interior weight state. Motivated by these observations, we propose TRAILS, a post-convergence decoding framework that exploits the model states produced during unified training in weight and statistics spaces. In weight space, Trajectory Interpolation (TI) searches the raw-to-EMA path for a favorable shared weight configuration. In statistics space, Statistics Recalibration (SR) re-estimates sport-specific BatchNorm statistics from unlabeled test samples, with sport identities inferred automatically from frozen visual features. TRAILS retains a single shared AQA weight configuration without gradient-based retraining of the AQA model. Experiments on three heterogeneous AQA benchmarks show that TRAILS achieves an average SRCC of and an R- of , improving the overall performance of unified AQA.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Action Quality Assessment, Unified Learning, Post-Convergence Decoding, Trajectory Interpolation, Statistics Recalibration

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_o25COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRAILS, a post-convergence approach for unified action quality assessment (AQA). It has two steps: Trajectory Interpolation (TI), which selects an intermediate model between the final raw checkpoint and its EMA counterpart based on training loss, and Statistics Recalibration (SR), which updates sport-specific BatchNorm statistics using unlabeled test data. TRAILS is evaluated on RG, FisV, and LOGO with MoAKE as the base model. It improves the average SRCC from 0.760 to 0.777 and reduces R-ℓ2 from 2.074 to 2.008.

Critical strengths

The paper offers useful analyses beyond the main results. Specifically, the authors examine several aspects of the converged model, including gradient conflicts across sports, dominant Hessian curvature, layer-wise parameter displacement, and sensitivity to different weight constructions. These analyses make the paper more informative than a straightforward combination of two post-processing techniques.

Critical weaknesses
  1. The proposed approach combines raw-to-EMA weight interpolation with sport-specific BatchNorm recalibration, both of which build closely on existing techniques. However, the paper falls short of clearly articulating its distinct methodological novelty beyond a straightforward combination of these existing components. The authors should better highlight what fundamental algorithmic insights or theoretical advantages differentiate TRAILS from prior works.
  2. The evaluation lacks comparisons against several prominent Action Quality Assessment (AQA) architectures and state-of-the-art baselines. Specifically, to robustly demonstrate TRAILS's effectiveness, the evaluation should include representative methods such as MUSDL (CVPR '20), CoRe (ICCV '21), TPT (ECCV '22), and NAE (CVPR '24).
  3. While the paper evaluates various test-set-based recalibration strategies, it omits evaluating sport-specific BatchNorm recalibration using only training data. This limitation makes it difficult to decouple the performance gains specifically attributable to sport-specific normalization from those driven by transductive test-time adaptation. An ablation study isolating these two factors is strongly recommended to clarify the source of improvement.
Questions for authors
  1. What is the key methodological novelty of TRAILS compared to existing weight interpolation and BatchNorm adaptation techniques? Furthermore, could the authors clarify how the optimization-geometry analysis directly informs or guides the proposed method's architectural design?
  2. Have the proposed components (TI and SR) been evaluated on other representative AQA architectures beyond MoAKE? Providing results on alternative backbones would significantly strengthen the claim that the proposed improvements are generalizable rather than architecture-specific.
  3. How does sport-specific BN recalibration using training data compare with recalibration using unlabeled test data? Including this comparison would help isolate and clarify whether the observed performance gains stem primarily from domain- or sport-specific normalization or from transductive test-time adaptation.
2 comments
AuthorsOpenReview ↗

We thank the reviewer for the constructive feedback and positive assessment of our post-convergence analyses. We provide point-by-point responses to the concerns raised below.

Q1: Methodological Novelty and Optimization-Guided Design

A1: The central contribution of TRAILS lies in task-driven post-convergence model-state decoding for unified AQA. While existing AQA research has primarily focused on architectural design and training-time optimization, TRAILS explores a complementary post-convergence perspective to address heterogeneous optimization behavior and sport-dependent feature distributions within a shared model.

Our gradient alignment and Hessian curvature analyses reveal distinct optimization characteristics across sports, motivating the exploration of alternative model states beyond the converged checkpoint. Inspired by mode connectivity, which suggests that favorable model configurations may exist along paths connecting different parameter states, we explore the training-derived Raw-to-EMA trajectory. TI then selects a favorable shared weight configuration by minimizing the sample-weighted training score loss, without additional gradient-based optimization. As demonstrated in the supplementary Complete Trajectory Coefficient Search, the selected state reduces the training score loss by 4.57% relative to Raw, providing empirical support for this optimization-guided selection strategy.

SR complements TI by recalibrating sport-specific BatchNorm statistics while preserving the selected weights, thereby accommodating heterogeneous feature distributions across sports. Their complementary contributions are supported by the ablation results: TI and SR individually improve average SRCC from 0.7600 to 0.7651 and 0.7683, respectively, while their combination further increases it to 0.7768. Together, TI and SR enable task-driven post-convergence model-state decoding in both weight and statistics spaces, improving unified AQA performance without additional gradient-based training or architectural modifications.

Q2: Baseline Comparisons and Cross-Architecture Generalization

A2: Existing AQA methods, including MUSDL, CoRe, and TPT, are primarily designed to train and evaluate separate models for individual sports, whereas unified AQA aims to assess heterogeneous sports using a single model with shared parameters. NAE addresses narrative action evaluation rather than numerical quality-score regression. We have further clarified these differences in the revised Related Work section.

To further address this concern, we conduct cross-architecture experiments under the unified multi-sport setting. Specifically, we evaluate a MoAKE variant with a shared temporal encoder and adapt TPT (ECCV 2022), an independent AQA architecture, to the same unified evaluation protocol. The results are reported in the revised Generalization Across Backbones section.

Table: Cross-architecture evaluation in terms of average SRCC (↑).

ArchitectureMethodAvg. SRCC ↑
MoAKE VariantRaw0.7427
TI0.7512
SR0.7478
TI + SR0.7574
TPT (ECCV 2022)Raw0.5698
EMA-best0.5767
TI0.5778

On the MoAKE variant, TI and SR independently improve average SRCC from 0.7427 to 0.7512 and 0.7478, respectively, while their combination further increases it to 0.7574, demonstrating their complementary effectiveness under an alternative unified AQA configuration. On the independent TPT backbone, TI improves average SRCC from 0.5698 to 0.5778, also exceeding the EMA-best result of 0.5767. Since TPT does not contain BatchNorm layers, SR is not applicable without architectural modifications. These results provide further evidence that score-guided post-convergence model-state decoding is effective beyond the original MoAKE architecture.

Q3: Sport-Specific Recalibration versus Test-Time Adaptation

A3: The primary objective of SR is to accommodate sport-specific feature distributions within a unified AQA model, rather than to perform global test-distribution adaptation. Although different sports share the same model weights, their feature statistics can differ substantially. SR addresses this heterogeneity through sport-specific BatchNorm recalibration without modifying the shared model weights. Importantly, this mechanism does not inherently require test-set statistics.

To further clarify this distinction, we conduct additional experiments comparing global and sport-specific recalibration using training data with the original test-set recalibration protocol. All configurations use identical TI-selected weights and differ only in their BatchNorm statistics.

Table: Effects of different BatchNorm recalibration protocols (SRCC ↑).

MethodTest-set BNRGFisVLOGOAvg.
TI (Original BN)No0.76540.67290.83360.7651
TI + Global-SR (Train)No0.77210.66460.83700.7668
TI + Sport-SR (Train)No0.77490.67290.84820.7750
TI + Sport-SR (Test)Yes0.77710.67290.85010.7767

As shown in the table, Sport-SR (Train) improves average SRCC from 0.7651 to 0.7750, outperforming Global-SR (Train) (0.7668) and approaching Sport-SR (Test) (0.7767). These results demonstrate that SR's improvements primarily arise from modeling sport-specific feature heterogeneity, rather than requiring access to the complete test distribution. Moreover, sport-specific BatchNorm statistics can be precomputed from training data and fixed during deployment, enabling independent online inference without accessing other test samples or future observations.

Reviewer_YANuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a post-convergence framework for unified AQA. The method consists of two components. Trajectory Interpolation searches between the final raw checkpoint and its exponential moving average (EMA) using the training-side score objective. Statistics Recalibration then updates sport-specific BatchNorm statistics using unlabeled test samples, while keeping the learned model parameters fixed. The authors also analyze sport-specific gradients, Hessian curvature, and the raw-to-EMA trajectory to motivate the proposed approach. Experiments on three AQA benchmarks show that TRAILS improves the average SRCC from 0.760 to 0.777 compared with the reproduced MoAKE baseline.

Critical strengths
  1. The method is relatively simple and does not require additional gradient-based training of the AQA model. The improvements over the reproduced MoAKE baseline are consistent across the three evaluated sports.
  2. The paper includes several useful analyses beyond the main comparison, including component ablations, trajectory interpolation, optimization geometry, and alternative weight constructions. These experiments help characterize the behavior of the proposed method.
Critical weaknesses
  1. The motivation and technical novelty are not sufficiently clear. The paper analyzes gradient disagreement and Hessian curvature differences across sports, but I find the connection between these observations and the proposed raw-to-EMA interpolation strategy somewhat unclear. In particular, it is not explained why the observed optimization geometry should favor an interpolated checkpoint. The alignment measurements in Table 5 are also very small, making their role in motivating TI difficult to understand.
  2. Moreover, TI is essentially a linear interpolation between two model checkpoints, while SR recalibrates BatchNorm statistics for different sports. Although the paper includes comparisons with alternative weight constructions, but I don't know why their combination is particularly suitable for unified AQA.
  3. The evaluation setting and robustness are unclear. SR uses the entire unlabeled test set to estimate sport-specific BatchNorm statistics, as described in Appendix A.4. This is a transductive setting, and it is unclear whether the methods compared in Table 1 have access to the same information. It would be helpful to clarify this difference and discuss how TRAILS would perform when test samples arrive individually.
  4. As a minor issue, some equations and annotations in Figure 2 are difficult to read, particularly in panels (b) and (c).
Questions for authors
  1. The authors should explain how the gradient disagreement and curvature analysis motivate raw-to-EMA interpolation. Do these observations provide any guidance for selecting the interpolation direction or coefficient?
  2. Since TI and SR are based on relatively standard operations, what is the main technical insight that makes their combination particularly effective for unified AQA? Technological innovation is very simple.
  3. How would TRAILS perform in an online inference setting where the complete unlabeled test set is not available for BatchNorm recalibration? The authors should clarify whether the baselines use the same test-time information.
2 comments
AuthorsOpenReview ↗

We thank the reviewer for the positive assessment of TRAILS' simplicity, consistent performance improvements, and extensive experimental analyses. We address each concern point by point below.

Q1: Motivation and Rationale for Trajectory Interpolation

A1: Our motivation stems from the heterogeneous optimization behavior of different sports under unified AQA. The gradient alignment and Hessian curvature analyses in the manuscript and appendix reveal distinct optimization characteristics across sports. In particular, limited gradient agreement suggests that different sports favor different optimization directions, motivating the exploration of alternative post-convergence model states beyond the final checkpoint.

Inspired by the perspective of mode connectivity, which suggests that favorable model configurations may exist along paths connecting different parameter states, we explore the Raw-to-EMA trajectory. Raw represents the post-convergence model state, whereas EMA incorporates historical information through exponential weight averaging. These complementary states provide a natural, training-derived interpolation path for exploring alternative model configurations. TI then selects a favorable intermediate state by minimizing the sample-weighted training score loss, without additional gradient-based optimization.

As shown in the supplementary Complete Trajectory Coefficient Search, TI achieves the lowest training-side score loss among 21 interpolation candidates at α=0.50\alpha=0.50, reducing the objective by 4.57% relative to Raw. Comparisons with alternative weight constructions, including SWA, further support the effectiveness of score-guided Raw-to-EMA interpolation. Together, these results provide empirical support for exploring and selecting post-convergence model states in unified AQA.

Q2: Technical Contribution and Complementarity

A2: The technical contribution of TRAILS lies in task-driven post-convergence model-state decoding for unified AQA, where heterogeneous sports must be accommodated within a shared model. While existing AQA research has primarily focused on architectural design and training-time optimization, TRAILS approaches unified AQA from a complementary post-convergence perspective, improving the trained model through task-driven weight selection and sport-specific statistical adaptation.

Specifically, TI performs score-guided shared weight selection by exploring the Raw-to-EMA trajectory and identifying a favorable model state using the training-side scoring objective. However, the selected weights still rely on BatchNorm statistics accumulated from heterogeneous sports. SR therefore performs sport-specific statistical recalibration while preserving the TI-selected weights. This design decouples shared weight selection from sport-specific statistical adaptation, enabling TRAILS to exploit complementary information in model parameters and normalization statistics without additional gradient-based training.

We further validate the advantages of TRAILS through comprehensive ablation studies, comparisons with alternative weight constructions, score-loss analyses, multi-seed evaluations, controlled perturbation experiments, and cross-architecture generalization studies. These experiments collectively provide empirical evidence for the effectiveness, complementarity, stability, robustness, and architectural applicability of our proposed approach.

Q3: Evaluation Fairness and Online Inference

A3: The primary objective of SR is to accommodate sport-specific feature distributions within a unified AQA model, rather than to rely on global test-distribution adaptation. Although different sports share the same model weights, their feature statistics can differ substantially. SR addresses this heterogeneity through sport-specific BatchNorm recalibration without modifying learnable parameters. Importantly, this mechanism does not inherently require test-set statistics.

To examine the source of SR's improvements and its applicability to online inference, we conduct additional experiments comparing global and sport-specific recalibration using training data with the original test-set recalibration protocol. All configurations use identical TI-selected weights and differ only in their BatchNorm statistics.

Table 1: Effects of different BatchNorm recalibration protocols (SRCC ↑).

MethodTest-set BNRGFisVLOGOAvg.
TI (Original BN)No0.76540.67290.83360.7651
TI + Global-SR (Train)No0.77210.66460.83700.7668
TI + Sport-SR (Train)No0.77490.67290.84820.7750
TI + Sport-SR (Test)Yes0.77710.67290.85010.7767

As shown in Table 1, sport-specific recalibration using training data alone improves average SRCC from 0.7651 to 0.7750, substantially outperforming global recalibration (0.7668) and approaching test-set recalibration (0.7767). These results indicate that SR's improvements primarily arise from modeling sport-specific feature heterogeneity, rather than requiring access to the complete test distribution.

Crucially, training-only SR enables online inference without test-set statistical estimation. Sport-specific BatchNorm statistics can be precomputed from training data and fixed during deployment. Each incoming sample can then be processed independently using the statistics associated with its sport category, without accessing other test samples or future observations. This provides an online-compatible alternative to the original test-set recalibration protocol.

Overall, these experiments demonstrate that TRAILS retains nearly all its performance gains without relying on test-set statistics, confirming the effectiveness of sport-specific statistical adaptation and its compatibility with online inference.

Reviewer_PwxPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRAILS, a post-training procedure for unified action quality assessment built on MoAKE. After unified training on RG, FisV and LOGO, Trajectory Interpolation (TI) searches the linear path between the final raw checkpoint and its EMA companion for the point with the lowest training score loss. Statistics Recalibration (SR) then re-estimates sport-specific BatchNorm statistics from unlabeled test samples, with sport identity predicted by a lightweight classifier on frozen features. A gradient and Hessian analysis around the converged checkpoint is presented as motivation. Experiments on the three benchmarks show modest improvements over the reproduced MoAKE without any retraining.

Critical strengths
  • The method is simple, cheap and entirely post-hoc. It requires no gradient updates, no architectural changes and no labels, so it can be applied to any trained unified model at negligible cost. If the gains hold up, this is a practically useful recipe.
  • The experimental section is comprehensive. Beyond the main results, the paper reports the complete interpolation sweep, random-direction and random-perturbation controls, sensitivity to the EMA decay, and a weight-perturbation stability test, and it openly notes where the training-selected coefficient differs from the test-side optimum.
Critical weaknesses
  • The reported gains are small and come from a single run with a fixed seed. The paper’s own reproduction of the baseline deviates from the published numbers by more than the improvement TRAILS brings, which suggests that run-to-run variation on these benchmarks is larger than the claimed effect. Without multiple seeds and variance estimates, the central claim is not established.
  • The contribution is thin and the comparison is not like-for-like. SR is essentially per-domain BatchNorm recalibration on test data, a known test-time adaptation technique that is not cited; it uses the test set transductively, while none of the baselines in the main table do. TI amounts to a one-parameter average between recent iterates, and the paper’s own SWA baseline, which should be closely related, performs worse than the raw checkpoint without explanation.
Questions for authors
  • Please report mean and standard deviation over several seeds for the raw checkpoint, EMA, TI, SR and the full method, per sport. Do the gains survive?
  • How was the late-checkpoint SWA baseline constructed, and why does it fall below the raw checkpoint? Was BatchNorm re-estimated for it?
  • Does SR improve the other unified baselines, and does TI help a second architecture?
  • Why is the training score loss lower at the interpolated point than at the final checkpoint after a full cosine schedule? Is dropout disabled when evaluating the objective, and do the routing and diversity terms behave the same way?
  • In the weight-perturbation test, are BatchNorm running statistics perturbed as well? Collapse at small relative noise may be a normalization effect rather than a weight-space property.

Thank you for the multi-seed results and the perturbation analysis. Over three seeds, the overall SRCC gain (0.7505 to 0.7620) is within one standard deviation (0.013), and SR adds no gain on FisV. The SWA baseline remains well below Raw without explanation, and the TPT result covers TI only. Given the limited novelty of TI and SR, I keep my score.

3 comments
AuthorsOpenReview ↗

We thank the reviewer for highlighting the practical value of TRAILS and the comprehensive experimental analysis. We address each concern point by point below.

Q1: Multi-Seed Robustness of TRAILS and Its Components

A1: We conduct additional experiments using three random seeds (42, 109, and 777), reporting the mean and standard deviation of SRCC and R-ℓ2\ell_2 for each sport in Table 3 (Table 8 of revised manuscript). We compare the raw MoAKE checkpoint (ff), its EMA counterpart (ee), TI-only (f+TIf+\mathrm{TI}), SR-only (f+SRf+\mathrm{SR}), and the full TRAILS method (f+TI+SRf+\mathrm{TI}+\mathrm{SR}).

Table 1: Multi-seed evaluation across three random seeds (mean ± std.).

SportMethodSRCC ↑R-ℓ2\ell_2 ↓
RGRaw (ff)0.7611 ± 0.00552.095 ± 0.076
EMA (ee)0.7593 ± 0.01852.007 ± 0.079
TI-only0.7672 ± 0.00312.010 ± 0.074
SR-only0.7662 ± 0.00442.055 ± 0.145
TRAILS0.7736 ± 0.00141.976 ± 0.135
FisVRaw (ff)0.6739 ± 0.02582.024 ± 0.122
EMA (ee)0.6627 ± 0.00962.116 ± 0.134
TI-only0.6774 ± 0.01812.027 ± 0.103
SR-only0.6739 ± 0.02582.025 ± 0.121
TRAILS0.6773 ± 0.01832.031 ± 0.106
LOGORaw (ff)0.8008 ± 0.03472.479 ± 0.381
EMA (ee)0.7853 ± 0.03362.574 ± 0.348
TI-only0.8083 ± 0.02462.407 ± 0.370
SR-only0.8041 ± 0.02652.469 ± 0.257
TRAILS0.8165 ± 0.02822.435 ± 0.322
ALLRaw (ff)0.7505 ± 0.01332.199 ± 0.130
EMA (ee)0.7406 ± 0.00992.232 ± 0.125
TI-only0.7562 ± 0.01132.148 ± 0.134
SR-only0.7534 ± 0.00992.183 ± 0.098
TRAILS0.7620 ± 0.01362.147 ± 0.133

As shown in the Table 1, TRAILS consistently improves mean SRCC over the raw checkpoint across all three sports, with gains of 0.0125, 0.0034, and 0.0157 on RG, FisV, and LOGO, respectively. The overall SRCC increases from 0.7505 to 0.7620, while R-ℓ2\ell_2 decreases from 2.199 to 2.147. TRAILS also reduces the SRCC standard deviation within each sport, from 0.0055 to 0.0014 on RG, 0.0258 to 0.0183 on FisV, and 0.0347 to 0.0282 on LOGO.

The component comparisons further demonstrate that both TI and SR contribute to the overall improvement. TRAILS achieves the highest overall mean SRCC (0.7620) and the lowest overall mean R-ℓ2\ell_2 (2.147) among all evaluated configurations. Together with the mean SRCC improvements observed across all three sports, these results demonstrate that the gains of TRAILS persist across different random seeds.

Q2: SWA Baseline Construction and BatchNorm Recalibration

A2: We construct the SWA baseline by uniformly averaging 8, 12, and 16 late-stage Raw checkpoints sampled every five epochs over the final 40, 60, and 80 training epochs, respectively. To examine the influence of BatchNorm statistics, we evaluate each configuration both with and without sport-specific Statistics Recalibration (SR).

Table 2: Comparison of post-convergence weight constructions (SRCC ↑).

MethodWithout SRWith SR
Raw0.76000.7683
SWA (40 epochs)0.70790.7241
SWA (60 epochs)0.70990.7255
SWA (80 epochs)0.71040.7284
TI (α=0.50\alpha=0.50)0.76510.7768

As shown in Table 2, BatchNorm recalibration improves all SWA variants by 0.0156–0.0180 SRCC, confirming that normalization statistics contribute to their performance. However, even the best recalibrated SWA achieves only 0.7284, remaining below both recalibrated Raw (0.7683) and TI (0.7768). This indicates that the performance gap cannot be attributed solely to BatchNorm statistics.

Unlike SWA, which uniformly averages late-stage checkpoints, TI explores the Raw-to-EMA trajectory and selects an intermediate state by minimizing the training-side score loss. These results demonstrate the advantage of score-guided interpolation over uniform late-checkpoint averaging under the evaluated setting.

Q3: Cross-Architecture Generalization

A3: We evaluate the generalizability of TRAILS on two additional backbone settings: a MoAKE variant and an existing one-by-one AQA architecture. For the MoAKE variant, we replace its sport-specific temporal encoders with a shared temporal encoder while preserving the unified multi-sport training protocol. For the latter, we adapt TPT (ECCV 2022), originally designed for one-by-one AQA, to the unified setting across RG, FisV, and LOGO.

Table 3: Cross-architecture generalization under unified multi-sport AQA (SRCC ↑).

ArchitectureMethodSRCC ↑
MoAKE VariantRaw0.7427
Raw+SR0.7478
TI0.7512
TI+SR0.7574
TPT (ECCV 2022)Raw0.5698
EMA0.5767
TI0.5778

On the MoAKE variant, TI and SR individually improve SRCC from 0.7427 to 0.7512 and 0.7478, respectively, while their combination further increases SRCC to 0.7574. On TPT, TI improves SRCC from 0.5698 to 0.5778, outperforming EMA (0.5767). Since TPT does not contain BatchNorm layers, SR is not applicable.

These results demonstrate that TI provides consistent improvements across different AQA architectures, while SR offers additional gains when BatchNorm is available. We have incorporated these experiments into the Generalization Across Backbones (Table 10) subsection of the revised manuscript.

Q4: Score-Guided Model-State Selection and Auxiliary Losses

A4: MoAKE jointly optimizes score, routing, and triplet losses during training, whereas TI selects the interpolation coefficient by minimizing the training-side score loss. Therefore, completing the cosine learning-rate schedule does not guarantee that the final Raw checkpoint minimizes the score loss along the Raw-to-EMA trajectory. Moreover, EMA incorporates historical model states through exponential weight averaging, providing an alternative parameter configuration for exploring favorable scoring solutions.

As shown in the Complete Trajectory Coefficient Search section of the Appendix, the selected TI state achieves a lower training score loss than both Raw and EMA, demonstrating that a more favorable scoring configuration exists along this trajectory. Importantly, TI identifies this state without additional training or gradient updates. During loss evaluation, dropout is disabled, and BatchNorm running statistics are held fixed across all interpolation states, ensuring consistent comparisons.

We additionally examine the routing and triplet losses along the same trajectory, averaged across sports according to their training sample counts. From Raw to the selected TI state (α=0.50\alpha=0.50), the routing loss changes from 0.751956 to 0.751976, while the triplet loss changes from 0 to 0.000041. Neither auxiliary loss decreases at the selected interpolation point, indicating that the auxiliary objectives do not follow the same trend as the score loss.

These observations clarify why score-guided interpolation can identify a favorable model state even after convergence of the original joint training procedure. Together with the observed SRCC improvements, they provide empirical support for selecting the interpolation coefficient using the task-relevant score objective.

Q5: Weight-Space Robustness and BatchNorm Perturbation Analysis

A5: The original perturbation protocol includes both trainable parameters and BatchNorm (BN) running statistics. To distinguish weight-space robustness from normalization-induced numerical instability, we conduct three controlled perturbation experiments: weight-only, BN-only, and full-state perturbations.

Weight-only robustness. With BN running statistics held fixed, both Raw and TI produce valid predictions at all evaluated noise levels. As shown in Table 4, TI consistently achieves higher SRCC than Raw under weight-only perturbations. At σ=0.50\sigma=0.50, TI retains an SRCC of 0.7604, compared with 0.7524 for Raw. Relative to their respective unperturbed baselines, TI exhibits a 38.2% smaller SRCC degradation (0.0047 vs. 0.0076). These results demonstrate that TI's advantage under weight perturbations persists independently of BN-statistics perturbation.

Table 4: Weight-only Gaussian perturbations with fixed BN running statistics (mean ± std.).

σ\sigmaRaw SRCC ↑TI SRCC ↑
0.000.76000.7651
0.010.7598 ± 0.00040.7656 ± 0.0005
0.020.7597 ± 0.00080.7656 ± 0.0007
0.050.7599 ± 0.00130.7655 ± 0.0011
0.100.7602 ± 0.00240.7661 ± 0.0021
0.200.7594 ± 0.00280.7653 ± 0.0019
0.300.7586 ± 0.00210.7645 ± 0.0032
0.500.7524 ± 0.00360.7604 ± 0.0036

BN-induced numerical instability. Table 5 reports the number of valid runs under different perturbation settings. BN-only and full-state perturbations exhibit identical validity patterns. At σ=0.05\sigma=0.05, all 20 Raw runs produce invalid predictions, whereas all 20 TI runs remain valid. At σ=0.10\sigma=0.10, TI retains 15 valid runs, while Raw has none. In contrast, all weight-only runs remain valid at these noise levels. These findings identify BN-statistics perturbation as the primary source of the observed numerical failures under the evaluated settings, rather than weight perturbation alone.

Table 5: Valid runs under controlled Gaussian perturbations (20 paired seeds per setting). W: weight-only; BN: BN-only; Full: full-state.

σ\sigmaRaw WRaw BNRaw FullTI WTI BNTI Full
0.01201919202020
0.022099202020
0.052000202020
0.102000201515

Together, these controlled experiments demonstrate that TI maintains higher SRCC under weight perturbations and greater numerical stability under BN-statistics perturbations. The weight-only results further establish that TI's robustness advantage is not solely attributable to normalization statistics.

We have incorporated these findings into the Stability Analysis subsection of the revised manuscript, with detailed experiments provided in the supplementary Model-State Stability Analysis.

Q6: Contribution and Generality of TRAILS

A6: TRAILS goes beyond a straightforward combination of weight interpolation and BatchNorm recalibration. Its central contribution is a task-driven post-convergence model-state decoding framework for unified AQA. Motivated by our analysis of heterogeneous optimization across sports, TI explores the Raw-to-EMA trajectory and selects a favorable weight configuration by minimizing the training-side score loss. SR further recalibrates sport-specific normalization statistics to accommodate heterogeneous feature distributions. Together, they refine the shared model in both weight and statistics spaces without additional gradient-based training.

Our experiments provide complementary evidence for this design. Comparisons with SWA and the score-loss analysis support the effectiveness of score-guided interpolation, while multi-seed and controlled perturbation experiments demonstrate its performance consistency and robustness under the evaluated settings. Furthermore, experiments on a MoAKE variant and the independent TPT architecture demonstrate the applicability of TI across different AQA backbones, with SR providing additional improvements when BatchNorm is available.

Reviewer_T7gbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRAILS, a post-training approach for unified action quality assessment (AQA). It first linearly interpolates between the final raw weights and the exponential moving average (EMA) weights from the same training run, selecting a shared weight configuration based on the training-set score prediction loss. It then re-estimates sport-specific BatchNorm statistics using unlabeled test videos grouped by automatically predicted sport identities. The procedure requires no further updates to the AQA model’s learnable parameters or architecture. Across RG, FisV, and LOGO, TRAILS improves the reproduced MoAKE baseline’s overall SRCC from 0.7600 to 0.7768 and reduces relative error from 2.074 to 2.008. The authors investigate the method’s effects and limitations through component ablations, interpolation coefficient searches, sport identification error tests, local optimization geometry analyses, and parameter perturbation experiments.

Critical strengths
  1. The method is simple and provides direct empirical gains. TRAILS requires neither architectural modifications nor additional gradient-based optimization of the AQA model, ultimately retaining a single shared set of scoring weights. Compared with the raw model from the same training run, it improves SRCC on all three datasets and reduces overall relative error. These results demonstrate the practical value of further exploiting model states retained during training and constitute the strongest argument for acceptance.
  2. The ablations distinguish the effects of the two components, and the reporting is reasonably transparent. Table 2 compares the raw model, EMA, TI alone, SR alone, and the complete method, showing that the gains do not simply come from using EMA. The interpolation coefficient is selected using the training loss. The authors also report performance degradation under sport identity corruption and unsuitable EMA decay settings. These results help clarify the conditions under which the method is useful.
  3. The central observation is concrete and reproducible, with an exceptionally low adoption cost. The claim that an interior state outperforms both endpoints on the training-side objective is supported by the complete 21-point sweep in Table 10, which exhibits a clear interior minimum. Each grid point requires only a forward evaluation, and the interpolation itself requires no retraining, additional parameters, or architectural changes. For practitioners who already maintain EMA weights in multi-task or unified training pipelines, this is an immediately actionable engineering finding.
Critical weaknesses
  1. The distinct value of the contribution has not been sufficiently established. TI builds on existing ideas in weight averaging and interpolation, while SR is closely related to domain adaptation through target-domain BatchNorm statistics. Applying these techniques to unified AQA has practical value, but the paper does not sufficiently establish what new insight emerges beyond what existing techniques would lead one to expect. Gradient disagreement and curvature differences provide motivation, but they do not explain why moving along the raw-to-EMA direction should improve generalization. The random-direction controls in the appendix also show that some of the very small cosine similarities fall within the normal range for high-dimensional random directions. With the main evaluation limited to a single unified training setup based on MoAKE, the contribution currently appears closer to a task-specific empirical improvement.
  2. The claim that “both components contribute” holds only for SRCC. For R-ℓ₂, TI alone achieves 1.998, the best reported value, whereas adding SR worsens it to 2.008. Similarly, raw weights with SR yield 2.096, compared with 2.074 for the raw weights alone. Thus, in both configurations where SR is added, it improves rank correlation while worsening score prediction error. The abstract and conclusion should explicitly acknowledge this trade-off, and the paper should explain why the two metrics move in opposite directions. One plausible interpretation is that sport-specific BatchNorm statistics improve within-sport ranking while shifting the predicted score scale. If supported, this would itself be a substantive finding rather than a footnote.
  3. The geometric motivation does not explain the observed effect. Table 9 shows that gradient alignment with the displacement falls within the 95% interval of isotropic random directions for RG and LOGO. The manuscript also explicitly acknowledges that the value of the interior state is established through direct evaluation rather than inferred from local alignment measurements. The first- and second-order analyses therefore serve as descriptive context around an empirical observation rather than an explanation of it. The appeal to mode connectivity also seems misplaced: (f) and (e) are neighboring states derived from the same training trajectory, so their linear connectivity is essentially trivial and does not require support from results of the kind established by Garipov or Frankle.
Questions for authors
  1. Distinction from existing methods and clarification of the controls. What does SR add beyond existing BatchNorm-based domain adaptation methods? Please also specify the SWA averaging window, checkpoint selection procedure, and normalization statistics handling used in Table 7. Does TI retain its advantage over these alternative weight constructions when all methods use the same statistics recalibration procedure?
  2. Why does SR improve SRCC while worsening R-ℓ₂, from 2.074 to 2.096 when added to the raw model, and from 1.998 to 2.008 when added to TI? Does this reflect a shift in the predicted score scale caused by sport-specific statistics? If so, could a simple sport-specific linear recalibration improve both metrics simultaneously?
  3. Table 11 reports a retrospective peak SRCC of 0.7794 for raw-to-EMA interpolation with SR, exceeding the reported operating-point result of 0.7768. Have the authors considered selecting (\alpha) using a held-out validation set rather than the training-side loss? If validation-based selection can consistently approach the retrospective peak without accessing test labels, this would substantially strengthen the case for the method.
2 comments
AuthorsOpenReview ↗

We thank the reviewer for the positive assessment of TRAILS' practical effectiveness, clear ablation studies, and low adoption cost. We address the concerns point by point below.

Q1: Novelty, Backbone Generalization, and Comparison with SWA and BN Recalibration

A1: Novelty. The contribution of TRAILS is a post-convergence model-state decoding framework for unified AQA, rather than a new interpolation or normalization algorithm. TI selects a favorable intermediate state along the Raw-to-EMA trajectory using training-side score loss, while SR recalibrates sport-specific BN statistics without updating model parameters. Together, they address suboptimal scoring configurations and sport-dependent feature distributions.

Backbone Generalization. We evaluate two additional settings: (1) a shared-temporal-encoder variant of MoAKE, and (2) TPT (ECCV 2022), an independently developed AQA architecture adapted to unified multi-sport evaluation. As shown in Table 1 (Table 10 of the revised manuscript), TI improves SRCC from 0.7427 to 0.7512 on the MoAKE Variant, while TI+SR further reaches 0.7574. On TPT, TI improves SRCC from 0.5698 to 0.5778. Since TPT contains no BN layers, SR is not applicable without architectural changes. These results support the transferability of TI beyond the original MoAKE backbone.

Table 1: Backbone generalization under unified multi-sport AQA (SRCC ↑).

BackboneMethodSRCC ↑
MoAKE VariantRaw0.7427
Raw+SR0.7478
TI0.7512
TI+SR0.7574
TPT (ECCV 2022)Raw0.5698
EMA0.5767
TI0.5778

SWA and BN Recalibration. We construct the standard SWA baseline by uniformly averaging 8, 12, and 16 late-stage Raw checkpoints sampled every five epochs over the final 40, 60, and 80 training epochs, respectively. We evaluate each configuration both with and without BatchNorm recalibration (SR) in Table 2.

Table 2: Comparison of model-state constructions with and without BN recalibration (SRCC ↑).

MethodWithout SRWith SR
Raw0.76000.7683
EMA0.75120.7588
SWA (40 epochs)0.70790.7241
SWA (60 epochs)0.70990.7255
SWA (80 epochs)0.71040.7284
TI (α = 0.50)0.76510.7768

As shown in the table, TI outperforms Raw, EMA, and all evaluated SWA configurations. With SR, TI achieves 0.7768 SRCC, compared with 0.7683 for Raw and 0.7241–0.7284 for SWA. SR also improves Raw and every evaluated SWA configuration, confirming its complementary effectiveness across different model-state constructions.

Q2: SRCC–R-ℓ2 Discrepancy and Affine Calibration

A2: SRCC measures ranking consistency, whereas R-ℓ2 evaluates prediction errors. These metrics capture different aspects of prediction quality, and improvements in ranking consistency do not necessarily imply lower numerical errors, even after affine calibration.

Both our MoAKE reproduction and TRAILS already employ sport-specific affine calibration, y^cal=asy^+bs\hat{y}_{\mathrm{cal}}=a_s\hat{y}+b_s, with coefficients fitted exclusively using training predictions and labels. No test quality labels are used. Therefore, the reported R-ℓ2 values are already calibrated, and affine calibration does not eliminate the observed discrepancies. Nevertheless, combining TI and SR improves both metrics over the reproduced MoAKE baseline, increasing SRCC from 0.760 to 0.777 and reducing R-ℓ2 from 2.074 to 2.008. We have clarified the calibration protocol in the revised Benchmarks and Metrics section.

Q3: Geometric Motivation and Mode Connectivity

A3: Our geometric analysis reveals heterogeneous optimization behavior across sports, including limited gradient agreement and differences in local curvature. These observations motivate exploring alternative post-convergence model states beyond the final checkpoint for unified AQA. Mode connectivity offers a useful perspective on the loss landscape, where intermediate parameter configurations may exhibit favorable properties. Motivated by this perspective, we explore the Raw-to-EMA trajectory, which connects the raw model state with its exponentially averaged counterpart.

Specifically, Raw and EMA represent two distinct model states derived from the training process, with EMA incorporating historical information through exponential weight averaging. TI explores their interpolation path, θ(α)=θf+α(θe−θf)\theta(\alpha)=\theta_f+\alpha(\theta_e-\theta_f), and selects a favorable intermediate state using the training-side score objective. The observed improvements over both endpoints provide empirical support for this strategy. We have also revised the Related Work section to clarify the relationship between our approach and mode connectivity.

Q4: Validation-Based Interpolation Coefficient Selection

A4: The original evaluation protocols adopted by existing AQA baselines do not include a separate validation split. To maintain consistency with these baselines, our original experiments followed the same data partition. Following your suggestion, we additionally hold out 5% of the training samples within each sport for validation, using the remaining 95% for training. We compare validation SRCC, validation score MSE, and training score MSE for selecting α\alpha from the same 21-point Raw-to-EMA grid. Given the limited amount of labeled training data in these AQA benchmarks, holding out 5% of the samples changes the training configuration and may affect model performance.

Table 3: Interpolation coefficient selection under a 95%/5% training–validation split. Each criterion selects α\alpha without SR.

Selection CriterionSelected α\alphaTI SRCC ↑TI R-ℓ2\ell_2 ↓TI+SR SRCC ↑TI+SR R-ℓ2\ell_2 ↓
Validation SRCC0.000.64623.32090.66322.8973
Validation score MSE0.850.65503.04050.66402.9388
Training score MSE0.400.66042.99570.66952.8769
Test SRCC oracle0.250.66173.01330.67162.8911

Validation SRCC selects Raw (α=0\alpha=0), whereas validation score MSE and training score MSE select α=0.85\alpha=0.85 and α=0.40\alpha=0.40, respectively. The training-MSE-selected state achieves 0.6604 test SRCC without SR, outperforming Raw (0.6462) and EMA (0.6557). With SR, it reaches 0.6695, approaching the test-sweep oracle of 0.6716.

These results indicate that coefficient selection depends on the evaluation criterion. In this split, training score MSE identifies a favorable intermediate state along the Raw-to-EMA trajectory. Since TI performs post-convergence model-state decoding, training score MSE provides a task-relevant criterion for selecting the interpolation coefficient without requiring a separate validation set.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Kind Reminder for Comments

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and reviewers for their time, constructive feedback, and careful evaluation of our work. We have provided detailed point-by-point responses and substantial additional experiments addressing all major concerns, including reproducibility and robustness, methodological motivation and novelty, cross-architecture generalization, transductive evaluation, and experimental fairness. These additional results and clarifications further strengthen the empirical evidence supporting TRAILS.

As we have not yet received further feedback from the reviewers, we would greatly appreciate the Area Chair's assistance in facilitating further discussion so that our responses and additional evidence can be fully considered. We welcome any remaining questions and would be happy to provide further clarification.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.